Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daily.CallClient thread leak #33

Open
LiangHeFRA opened this issue Dec 23, 2024 · 1 comment
Open

daily.CallClient thread leak #33

LiangHeFRA opened this issue Dec 23, 2024 · 1 comment

Comments

@LiangHeFRA
Copy link

LiangHeFRA commented Dec 23, 2024

import psutil
import threading
from typing import Any
import daily
daily.Daily.init()

def test_daily():
    print(f"test_daily.start pid={os.getpid()}, py threads={threading.active_count()}, os threads={psutil.Process(os.getpid()).num_threads()}")
    call_client = daily.CallClient(event_handler=daily.EventHandler())
    call_client.leave()
    call_client.release()
    print(f"test_daily.end pid={os.getpid()}, py threads={threading.active_count()}, os threads={psutil.Process(os.getpid()).num_threads()}")

for i in range(10):
    test_daily()

code above, print is

test_daily.start pid=1887293, py threads=1, os threads=8
test_daily.end pid=1887293, py threads=1, os threads=9
test_daily.start pid=1887293, py threads=1, os threads=9
test_daily.end pid=1887293, py threads=1, os threads=10
test_daily.start pid=1887293, py threads=1, os threads=10
test_daily.end pid=1887293, py threads=1, os threads=11
test_daily.start pid=1887293, py threads=1, os threads=11
test_daily.end pid=1887293, py threads=1, os threads=12
test_daily.start pid=1887293, py threads=1, os threads=12
test_daily.end pid=1887293, py threads=1, os threads=13
test_daily.start pid=1887293, py threads=1, os threads=13
test_daily.end pid=1887293, py threads=1, os threads=14
test_daily.start pid=1887293, py threads=1, os threads=14
test_daily.end pid=1887293, py threads=1, os threads=15
test_daily.start pid=1887293, py threads=1, os threads=15
test_daily.end pid=1887293, py threads=1, os threads=16
test_daily.start pid=1887293, py threads=1, os threads=16
test_daily.end pid=1887293, py threads=1, os threads=17
test_daily.start pid=1887293, py threads=1, os threads=17
test_daily.end pid=1887293, py threads=1, os threads=18

threre are leaking with call daily.CallClient
by gdb attach python process, and show the leak thread's function stack is:

(gdb) info thread
  Id   Target Id                                             Frame 
* 1    Thread 0x76a75f18e080 (LWP 1887440) "python"          0x000076a75eeecadf in __GI___clock_nanosleep (
    clock_id=1, flags=1, req=0x7fffc259b080, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
  2    Thread 0x76a7594006c0 (LWP 1887500) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x76a753e006c0 (LWP 1887495) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x76a7532006c0 (LWP 1887490) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  5    Thread 0x76a7536006c0 (LWP 1887484) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  6    Thread 0x76a758c006c0 (LWP 1887479) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x76a7598006c0 (LWP 1887473) "SoupSignalling"  syscall ()
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  8    Thread 0x76a7584006c0 (LWP 1887468) "SoupSignalling"  syscall ()
--Type <RET> for more, q to quit, c to continue without paging--
(gdb) thread 2
[Switching to thread 2 (Thread 0x76a7594006c0 (LWP 1887500))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
warning: 38     ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000076a75e421803 in std::thread::park ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#2  0x000076a75d125292 in std::sync::mpmc::array::Channel<T>::recv::{{closure}} ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#3  0x000076a75d124c8f in std::sync::mpmc::array::Channel<T>::recv ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#4  0x000076a75cf70aff in std::sys_common::backtrace::__rust_begin_short_backtrace ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#5  0x000076a75d1088a2 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#6  0x000076a75e42ba5b in std::sys::pal::unix::thread::Thread::new::thread_start ()
   from .venv/lib/python3.11/site-packages/daily/daily.abi3.so
#7  0x000076a75ee9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#8  0x000076a75ef29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
@LiangHeFRA
Copy link
Author

LiangHeFRA commented Dec 23, 2024

in the prod env, python process memory increase all the time, using gdb to find threading's num is more than 600, and threading's num increase by the time too.
the data below, threads which more than 80 are thread with daily/daily.abi3.so

/app# ps -T -p 22 | awk -F " " '{print $5}' | sort | uniq -c
1 AudioDeviceBuff
1 CMD
5 Daily
10 DecodingQueue
10 IncomingVideoSt
83 SoupSignalling
88 TaskQueuePacedS
2 daily-core
5 daily-emitter
1 daily-speaker-p
1 daily_network_t
1 daily_signaling
1 daily_worker_th
1 default-executo
4 event_engine
2 grpc_global_tim
1 lifeguard
83 mediasoup-manag
220 python
1 resolver-execut
1 rtc-low-prio
88 rtp_send_contro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant