Home Original page

What's the cost of using hundreds of threads?

Shitiz Bansal shitizb at yahoo.com
Tue Mar 1 15:42:18 EST 2005 
It depends on the OS.
I recently ran some 8000 threads in my program.

On Windows XP, the limit is around 2000 threads and i
dont quite know how to increase that.

On, Linux:
'ulimit -aH' yields this output on my fedora core2. It
cud be different for different flavors.

[01:55:18 wallace new]$ ulimit -aH
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 3967
virtual memory        (kbytes, -v) unlimited

Here the stack size and the max user processes are of
interest.
The default stack size is set as unlimited hence
python will allocate its own default stack size, which
is arnd 8Mb(for each thread).
On my machine this means the max no. of threads=400.
To increase this u can issue the command(valid only
for that terminal)- 'ulimit -s <stacksize in KB>'.
I limited the size to 64KB in my case.
Hence I could run 3967 threads which is the max user
processes,
Only root can increase this parameter by issuing
'ulimit -u <no. of processes>'.
This is not relevant in your case as 100 threads can
easily be handled i suppose, but i wrote this for the
benefit of other users.
As for the clumsy programming question- It depends on
your program.While using so many threads, you are
relying on your system's thread scheduler, which might
not be very efficient.

Cheers,
Shitiz
--- Jarek Zgoda <jzgoda at gazeta.usun.pl> wrote:

> Przemys³aw Ró¿ycki napisa³(a):
> 
> > - Is there any 'upper boundary' that limits the
> number of threads? (is 
> > it python / OS related)
> > - Is that the sign of 'clumsy programming' - i.e.
> Is creating so many 
> > threads a bad habit? (I must say that it
> simplified the solution of my 
> > problem very much).
> 
> I've read somewhere (I cann't recall where, though,
> was it MSDN?) that 
> Windows is not well suited to run more than 32
> threads per process. Most 
> of the code I saw doesn't spawn more threads than a
> half of this.
> 
> -- 
> Jarek Zgoda
> http://jpa.berlios.de/ | http://www.zgodowie.org/
> -- 
> http://mail.python.org/mailman/listinfo/python-list
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com
More information about the Python-list mailing list