deadlock on console mutex in gdb

[Takashi Yano]

On Thu, 23 Dec 2021 15:28:58 -0400
David McFarland wrote:
> I've also been hitting another console related deadlock, even when
> there's no debugger involved:
[...]
> I can use this program:
> 
> =====test-pt.c
> #include <pthread.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <stdlib.h>
> 
> void *thread(void* p) {
>         for (int i = 0;; ++i) {
>                 int fd = posix_openpt(O_RDWR | O_NOCTTY);
>                 close(fd);
>                 fprintf(stderr, "test %i\n", i);
>         }
>         return 0;
> }
> 
> int main() {
>         pthread_t ids[3];
>         for (int i = 0; i < 3; ++i) {
>                 pthread_create(&ids[i], 0, &thread, &ids[i]);
>         }
>         for (int i = 0; i < 3; ++i) {
>                 pthread_join(ids[i], 0);
>         }
>         return 0;
> }
> =====
> 
> compiled with:
> 
> $ gcc -D_GNU_SOURCE test-pt.c -o test-pt
> 
> To hit those asserts.

Thanks for the report. I tried your test case, however I
encountered another problem of pty. The test case causes
memory leak and exhausts the system memory in a short time.

I will submit a patch for the memory leak issue.

> Perhaps the mutex should never be closed? It's also worrying that
> acquiring the mutex just silently fails before a PT is opened.
> There could be a thread that is acting as if it has a lock a the time
> that the mutex is created, and it will even try to release the lock.
> 
> It might just be safest to create it at process start, but it could have
> a significant performance cost.

With the patch above, I tried your test case again and
confirmed the assertion fails indeed. I also confirmed
the deadlock.

Thanks for pointing out this. Let me consider.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>