Messages (10) msg56226 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:18

There used to be a bug open concerning this problem in the old bug
tracker but I can't find it in the new bug tracker:

http://mail.python.org/pipermail/python-bugs-list/2005-June/029290.html

msg56227 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:18

Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: The Written Word (Albert Chin) (tww-china)
Assigned to: Nobody/Anonymous (nobody)
Summary: semaphore errors from Python 2.3.x on AIX 5.2

Initial Comment:
Newer versions of the AIX 5.2 libc implement POSIX
semaphores and thus define _POSIX_SEMAPHORES in
<unistd.h>. However, when building Python 2.3.x (and
I'm sure any other version with sem_init()), a few
semaphore errors are encountered:
  building 'pcre' extension
  ./Modules/ld_so_aix gcc -bI:Modules/python.exp  
-L/opt/TWWfsw/python232/lib/support -Wl,-brtl  
-Wl,-blibpath:/opt/TWWfsw/python232/lib/support:/usr/lib  
build/temp.aix-5.2-2.3/pcremodule.o   build/temp.aix-5.2-2.3/pypcre.o
-L/opt/TWWfsw/python232/lib   -o build/lib.aix-5.2-2.3/pcre.so
  sem_trywait: Permission denied
  sem_wait: Permission denied
  sem_post: Permission denied

I ran the Python process under truss and saw that the
sem_init() from Python/thread_pthread.h worked
successfully. fork() was then called and a
sem_trywait() operation (probably from
PyThread_acquire_lock()) was called. From the
sem_init() man page
(http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/libs/basetrf2/sem_init.htm):
  If the pshared parameter has a nonzero value, the
semaphore
  is shared between processes. In this case, any
process that can
  access the sem parameter can use it for performing
sem_wait,
  sem_trywait, sem_post, and sem_destroy operations.

  Only the sem parameter itself may be used for performing
  synchronization.

  If the pshared parameter is zero, the semaphore is
shared between
  threads of the process. Any thread in this process
can use the sem
  parameter for performing sem_wait, sem_trywait,
sem_post, and
  sem_destroy operations. The use of the semaphore by
threads
  other than those created in the same process returns
an error.

Setting the 2nd value of sem_init to 1 (was 0) solved
the problem. However, I don't know if this is wise for
security. Another solution is to set
HAVE_BROKEN_POSIX_SEMAPHORES for "AIX/5" like for
"SunOS/5.8".

----------------------------------------------------------------------

Comment By: Leanid (lim_1)
Date: 2005-06-20 13:12

Message:
Logged In: YES 
user_id=701470

I can confirm this problem. I am using scons to build my 
project on AIX. I could not run parallel builds (-j4) on AIX. 
Randomly, but very often xlC invocation would stuck and I 
have 4 xlCcode hanging and taking all CPUs.


After adding HAVE_BROKEN_POSIX_SEMAPHORES for 
AIX and rebuilding Python parallel builds on AIX are fine. 
Trying first fix with changing 0 to 1 in sem_init also resolved 
this problem, but I started to get some other unexplained 
error from scons, so had to deactivate semaphores all 
together.

Albert, thanks for this report, it helped a lot.

----------------------------------------------------------------------

msg56228 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:30

I can confirm that this bug is still present in Python 2.4.1 (and
probably later versions) on AIX 5.2 and 5.3.

When HAVE_BROKEN_POSIX_SEMAPHORES is not defined at compile time, Python
will spend a lot of time in sem_wait, usually when acessing files.
The problem does not always happen as it seems to depend on the type of
file system in use.

In my programm running for 3m30s, Python would call about 400000 time
sem_wait and spend about 2mins waiting. When Python is compiled with
HAVE_BROKEN_POSIX_SEMAPHORES, the same programm will only take 1m30s.

I attach a patch which solves the problem.

msg57194 - (view) Author: Sébastien Sablé (sable) Date: 2007-11-07 08:53

The bug is still present in Python 2.5.1. The same patch applies.
The patch is rather trivival, could someone please integrate it in trunk?
Thanks in advance

msg57333 - (view) Author: Christian Heimes (christian.heimes) * Date: 2007-11-09 21:20

I'm setting the priority to high because it sounds important for AIX
users and the patch is *really* simple, just two additional lines for
configure.in

msg57512 - (view) Author: Marc-Andre Lemburg (lemburg) * Date: 2007-11-14 23:38

The problem is also present in Python 2.4 and 2.3. Confirmed on AIX 5.3.

msg57715 - (view) Author: Guido van Rossum (gvanrossum) * Date: 2007-11-20 22:35

I have no way to test this.

msg59199 - (view) Author: Guido van Rossum (gvanrossum) * Date: 2008-01-04 01:10

OK, accepted for 2.5 and 2.6.

Can someone with a stake and submit access (e.g. lemburg) check it in?

msg61878 - (view) Author: Marc-Andre Lemburg (lemburg) * Date: 2008-01-30 21:09

Christian, could you apply this patch ? It needs a working autoconf 2.61
installation which I currently don't have.

The patch itself, does the trick, so should go in.

Thanks.

msg61879 - (view) Author: Christian Heimes (christian.heimes) * Date: 2008-01-30 22:54

Fixed in r60464 (trunk) and r60465 (25)

History Date User Action Args 2022-04-11 14:56:27adminsetgithub: 45575 2017-03-31 16:36:19dstufftsetpull_requests: + pull_request930 2017-03-27 19:49:38gvanrossumsetnosy: - gvanrossum
2017-03-27 19:38:27MSeifertsetpull_requests: + pull_request753 2008-01-30 22:55:02christian.heimessetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg61879 2008-01-30 21:09:15lemburgsetassignee: lemburg -> christian.heimes
messages: + msg61878 2008-01-04 01:10:56gvanrossumsetassignee: lemburg
messages: + msg59199
resolution: accepted 2008-01-04 00:44:12christian.heimessetversions: - Python 2.4, Python 2.3 2007-11-20 22:35:00gvanrossumsetpriority: high -> normal
assignee: gvanrossum -> (no value)
messages: + msg57715 2007-11-20 01:16:13christian.heimessetassignee: gvanrossum
nosy: + gvanrossum 2007-11-14 23:38:29lemburgsetnosy: + lemburg
messages: + msg57512
versions: + Python 2.4, Python 2.3 2007-11-09 21:20:05christian.heimessetpriority: high
keywords: + patch
messages: + msg57333
nosy: + christian.heimes
versions: + Python 2.6, Python 2.5, Python 3.0 2007-11-07 08:53:52sablesetmessages: + msg57194 2007-10-04 09:43:07sablesetfiles: + patch_aix_broken_semaphores.diff 2007-10-04 09:30:54sablesetfiles: + patch_aix_broken_semaphores.diff
messages: + msg56228 2007-10-04 09:18:56sablesetmessages: + msg56227 2007-10-04 09:18:14sablecreate