RSS
The Python extension module posix_ipc gives Python access
to POSIX inter-process semaphores, shared memory and message queues on
systems that support the POSIX Realtime Extensions a.k.a. POSIX 1003.1b-1993.
That includes most (all?) Linuxes with kernel ≥ 2.6, FreeBSD ≥ 7.2, and
OpenSolaris ≥ 2008.05.
macOS/OS X and other Unix-y platforms (including Windows + Cygwin 1.7) provide partial (or partially broken) support. See the platform notes below for more details.
This module works under Python 2.7 and 3.x. It is released under a BSD license.
You can download
posix_ipc version 1.0.5
[MD5 sum]
[SHA1 sum]
which contains the source code, setup.py, installation instructions, tests, and
sample code. The exact same
posix_ipc tarball is also
available on PyPI. You can also find
the
posix_ipc source code on GitHub.
You might want to read all of the changes in this version and about some known bugs.
Note that this module doesn't support unnamed (anonymous) POSIX semaphores.
You might be interested in the very similar module
sysv_ipc which
provides Python access to IPC using System V semaphores, shared memory and
message queues.
System V IPC has broader OS support but is not as easy to use.
posix_ipc DocumentationJump to semaphores, shared memory, or message queues.
O_CREX are bitwise unique and can be
ORed together. O_CREX is shorthand for
O_CREAT | O_EXCL.
O_TRUNC is only useful when
creating SharedMemory objects.
sem_timedwait(). If False, all
timeouts > 0 passed to a semaphore's acquire() method are
treated as infinity.
As far as I know, this is only False under macOS.
sem_getvalue(). If False,
accessing the value attribute on a Semaphore instance
will raise an AttributeError.
As far as I know, this is only False under macOS.
max_messages
attribute. This can be quite small under Linux (e.g. 10)
but is usually LONG_MAX everywhere else.
max_message_size
attribute. This is 8k (or possibly smaller under Linux).
'0.9.8'. This is also
available as __version__.
In addition to standard Python errors (e.g. ValueError),
this module raises custom errors. These errors cover
situations specific to IPC.
sem.acquire()) is
interrupted by a signal other than KeyboardInterrupt.
This is a handle to a semaphore.
name must be None or
a string. If it is None, the module chooses a random
unused name. If it is a string, it
should begin with a slash and be valid according
to pathname rules on your system, e.g.
/wuthering_heights_by_semaphore
The flags specify whether you want to create a new semaphore or open an existing one.
0, the module attempts
to open an existing semaphore and raises an error if that semaphore
doesn't exist.
O_CREAT,
the module opens the semaphore if it exists (in which case mode and
initial value are ignored) or creates it if it doesn't.
O_CREAT | O_EXCL
(or O_CREX),
the module creates a new semaphore identified by
name. If a
semaphore with that name already exists, the call raises
an ExistentialError.
The timeout (which can be a float) specifies how many seconds this call should wait, if at all.
BusyError
if asked to wait. Since it will return immediately if not
asked to wait, this can be thought of as "non-blocking" mode.
This behavior is unaffected by whether or not the
platform supports sem_timedwait()
(see below).
BusyError.
On platforms that don't support the sem_timedwait() API,
a timeout > 0 is treated as
infinite. The call will not return until its wait
condition is satisfied.
Most platforms provide sem_timedwait(). macOS is a
notable exception. The module's Boolean constant
SEMAPHORE_TIMEOUT_SUPPORTED
is True on platforms that support sem_timedwait().
unlink(), below) the semaphore can be re-opened.
You must call close() explicitly; it is
not called automatically
when a Semaphore object is garbage collected.
Note, however, that once a semaphore has been unlinked,
calls to open() with the same name should
refer to a new semaphore. Sound confusing? It is, and you'd
probably be wise structure your code so as to avoid
this situation.
These semaphores support the context manager protocol so they can be
used with Python's with statement, just like
Python's threading.Semaphore. For instance --
with posix_ipc.Semaphore(name) as sem:
# Do something...
Entering the context acquires the semaphore, exiting the context releases
the semaphore. See demo4/child.py for a complete example. The context
manager only manages acquisition and release. If you create a new semaphore
as part of executing the with statement, you must explicitly unlink it.
This is a handle to a shared memory segment. POSIX shared memory segments masquerade as files, and so the handle to a shared memory segment is just a file descriptor that can be mmapped.
name must be None or
a string. If it is None, the module chooses a random
unused name. If it is a string, it
should begin with a slash and be valid according
to pathname rules on your system, e.g.
/four_yorkshiremen_sharing_memories
On some systems you need to have write access to the path.
The flags specify whether you want to create a new shared memory segment or open an existing one.
0, the module attempts
to open an existing segment and raises an error if that segment
doesn't exist.
O_CREAT,
the module opens the segment if it exists or creates it if it doesn't.
O_CREAT | O_EXCL
(or O_CREX),
the module creates a new shared memory segment identified by
name. If a
segment with that name already exists, the call raises
an ExistentialError.
If you pass a non-zero size, the segment will be (re)sized accordingly, regardless of whether or not it's a new or existing segment. Prior to version 1.0.4, this documentation incorrectly stated that size was ignored if the segment already existed.
To (re)size the segment, posix_ipc calls ftruncate().
The same function is available to Python via
os.ftruncate().
If you prefer to handle segment (re)sizing yourself, leave the
SharedMemory parameter size at its default of 0 and
call os.ftruncate() when and how you like.
Note that under macOS (up to and including 10.12 at least), you can
only call ftruncate() once on a segment during its lifetime.
This is a limitation of macOS, not posix_ipc.
When opening an existing shared memory segment, one can also specify
the flag O_TRUNC
to truncate the shared memory to zero bytes. macOS does not
support O_TRUNC.
close_fd() is the same as calling
os.close()
on a SharedMemory object's fd attribute.
You must call close_fd() or os.close()
explicitly; the file descriptor is not closed
automatically when a SharedMemory object is garbage collected.
Closing the file descriptor has no effect on any mmap
objects that were created from it. See the demo for an
example.
The
POSIX specification for shm_unlink() says, "Even if the object
continues to exist after the last shm_unlink(), reuse of the name shall subsequently
cause shm_open() to behave as if no shared memory object of this name exists
(that is, shm_open() will fail if O_CREAT is not set, or will create a new shared
memory object if O_CREAT is set)."
I'll bet a virtual cup of coffee that this tricky part of the standard is not well or consistently implemented in every OS. Caveat emptor.
This is a handle to a message queue.
name must be None or
a string. If it is None, the module chooses a random
unused name. If it is a string, it
should begin with a slash and be valid according
to pathname rules on your system, e.g.
/my_message_queue
On some systems you need to have write access to the path.
The flags specify whether you want to create a new queue or open an existing one.
0, the module attempts
to open an existing queue and raises an error if that queue
doesn't exist.
O_CREAT,
the module opens the queue if it exists (in which case
size and mode
are ignored) or creates it if it doesn't.
O_CREAT | O_EXCL
(or O_CREX),
the module creates a new message queue identified by
name. If a
queue with that name already exists, the call raises
an ExistentialError.
Max_messages defines how many messages
can be in the queue at one time. When the queue is full,
calls to .send() will wait.
Max_message_size defines the maximum size (in bytes) of a message.
Read and
write
default to True. If read/write
is False, calling .receive()/.send() on this object
is not permitted.
This doesn't affect other handles to the same queue.
The message string can contain embedded
NULLs (ASCII 0x00). Under Python 3, the message can
also be a bytes object.
The timeout (which can be a float)
specifies how many seconds this call should wait if the
queue is full. Timeouts are irrelevant when the block
flag is False.
BusyError
if asked to wait.
BusyError.
The priority allows you to order messages in the queue. The highest priority message is received first. By default, messages are sent at the lowest priority (0).
(message, priority). Messages are received in the order of
highest priority to lowest, and in FIFO order for messages of
equal priority.
Under Python 3, the returned message is a bytes object.
If the queue is empty, the call will not return immediately.
The optional timeout parameter controls the
wait just as for the function send(). It defaults to None.
None, any existing notification request is
cancelled.
signal.signal().
(function, param), notification will be sent
by invoking function(param) in a new
thread.
Message queues accept only one notification request at a time.
If another process has already requested notifications from
this queue, this call will fail with a BusyError.
The operating system delivers (at most) one notification
per request. If you want subsequent notifications, you must
request them by calling request_notification() again.
You must call close() explicitly; it is
not called automatically
when a MessageQueue object is garbage collected.
.send() and
.receive() may wait (block) if they cannot immediately
satisfy the send/receive request. When block is False,
the module will raise BusyError
instead of waiting.
This module comes with fairly complete unit tests in the tests
directory. To run them, install posix_ipc and then run this command
from the same directory as setup.py:
python -m unittest discover
This module comes with four demonstrations. The first (in the
directory demo) shows how to use shared memory and semaphores.
The second (in the directory demo2) shows how to use
message queues. The third (demo3) shows how to use message queue
notifications. The fourth (demo4) shows how to use a semaphore in
a context manager.
IPC objects are a little different from most Python objects and therefore require a little more care on the part of the programmer. When a program creates a IPC object, it creates something that resides outside of its own process, just like a file on a hard drive. It won't go away when your process ends unless you explicitly remove it. And since many operating systems don't even give you a way to enumerate existing POSIX IPC entities, it might be hard to figure out what you're leaving behind.
In short, remember to clean up after yourself.
I know it's verboten to talk about pointers in Python, but I'm going to do it anyway.
Each Semaphore object created by this module contains a C pointer to
the IPC object created by the system. When you call sem.close(),
the object's internal pointer is set to NULL. This leaves the
object in a not-quite-useless state. You can still call sem.unlink()
or print sem.name, but calls to sem.aquire() or
sem.release() will raise an ExistentialError.
If you know you're not going to use a Semaphore object after calling
sem.close() or sem.unlink(), you could you set your
semaphore variable to the return from the function (which is always
None) like so:
my_sem = my_sem.close()
That will ensure you don't have any nearly useless objects laying around that you might use by accident.
This doesn't apply to shared memory and message queues because they're referenced at the C level by a file descriptor rather than a pointer.
It appears that the read and write mode bits on IPC objects are
ignored by the operating system. For instance, on macOS, OpenSolaris and
Linux one can write to semaphores and message queues with a mode of
0400.
When creating a new message queue, you specify a maximum message size
which defaults to QUEUE_MESSAGE_SIZE_MAX_DEFAULT (currently 8192
bytes). You can create a queue with a larger value, but be aware that
posix_ipc allocates a buffer the size of the maximum message size
every time receive() is called.
man PagesThe posix_ipc module is just a wrapper around your system's API. If your
system's implementation has quirks, the man pages for
sem_open, sem_post,
sem_wait, sem_close, sem_unlink, shm_open, shm_unlink, mq_open, mq_send
mq_receive, mq_getattr, mq_close, mq_unlink and mq_notify will
probably cover them.
For Pythonistas –
I don't know of any bugs in this code, but FreeBSD users should check the platform notes.
If you need to support Python < 2.7, try posix_ipc version 0.9.9 [MD5 sum] [SHA1 sum].
This module is just a wrapper around the operating system's functions, so if the operating system doesn't provide a function, this module can't either. The POSIX Realtime Extensions (POSIX 1003.1b-1993) are, as the name implies, an extension to POSIX and so a platform can claim "POSIX conformance" and still not support any or all of the IPC functions.
Under 7.2, posix_ipc's demos fail unless they're run as
root. It's a simple permissions problem. Prefix the IPC object
names with /tmp in params.txt and the problem
goes away. I didn't see this behavior under FreeBSD 8.0, so it
must have been fixed at some point.
If you don't have the sem and mqueuefs kernel
modules loaded, you'll get a message like this (or something
similarly discouraging) when you
try to create a semaphore or message queue:
Bad system call: 12 (core dumped)
Type kldstat to list loaded modules, and
kldload sem or kldload mqueuefs if you need
to load either of these. BTW,
mqueuefs has
some cool features.
Prior to 7.2, FreeBSD POSIX semaphore support was broken.
Under FreeBSD 11.1, I have seen segfaults during the message queue
threaded notification rearm test. I don't know if posix_ipc
or FreeBSD (or both!) are culprits.
macOS' implementation of POSIX IPC has some significant holes.
Message queues are not supported at all. Also,
sem_getvalue() and sem_timedwait() are not
supported.
Versions of Cygwin prior to 1.7 didn't support POSIX IPC.
Under Cygwin 1.7 beta 62 (released in early October 2009),
posix_ipc compiles and runs both demos.