Hi,

the old and slow python implementation of pickle didn't block background
thread.
But the newer C-implementation blocks other threads while dump/load is
running.
Wouldn't it be possible to allow other threads during this time?
Especially could load/loads release the GIL, because Python objects are not available to the Python code until these functions have finished?

Regards,
Martin

um, something doesn't make sense about this. the python implementation of pickle never released the GIL (it can't, by definition -- it's written in python). The C implementation releasing the GIL wouldn't make sense, as the pickle api involves calls into python everywhere (for example, `__reduce__`)

This is about releasing the GIL periodically to allow other threads to run, as Python already does in its main interpreter loop.

A workaround is writing Python wrappers for IO:

def Writer:
    def __init__(self, file):
        self.file = file
    def write(self, data):
        return self.file.write(data)

def Reader:
    def __init__(self, file):
        self.file = file
    def read(self, size=-1):
        return self.file.read(size)
    def readline(self, size=-1):
        return self.file.readline(size)
    def peek(self, size=-1):
        return self.file.peek(size)

def mydump(obj, file, *args, **kwargs):
    return pickle.dump(obj, Writer(file), *args, **kwargs)

def myload(file, *args, **kwargs):
    return pickle.load(Reader(file), *args, **kwargs)

Maybe an optional parameter with the desired interval would be good idea. So that the coder can decide if he wants/needs that feature and which interval he needs for his application.
Otherwise it is hard to define a specific interval which fits for everyone.

The right way to do this is not to pass a timeout parameter but to check for GIL interrupts as done in the main bytecode evaluation loop.

Attaching proof-of-concept patch.

Attaching demonstration script.

(as the demo script shows, there is no detectable slowdown)

The demo script shows around 8% slowdown to me for

    data = list(map(float, range(N)))

Interesting, which kind of computer / system / compiler are you on?

CPU = Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Ubuntu 18.04
Linux 4.15.0 x86_64
gcc 7.3.0

Performing the check in save() can have not insignificant overhead (especially after implementing the issue34141 optimization). It can be reduced if perform it when flush a frame (in protocol 4) or buffer to the file, or after writing significant amount of bytes into buffer.

messages: + msg321827

messages: + msg321805
versions: + Python 3.8, - Python 3.6