Issue 36499: unpickling of a datetime object in 3.5 fails when pickled with 2.7

Issue36499

Created on 2019-04-01 23:02 by vadimf, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
pickle_unpickle.tar.gz vadimf, 2019-04-01 23:02 tgz with the source files needed to reproduce the problem
Messages (3)
msg339308 - (view) Author: Vadim (vadimf) Date: 2019-04-01 23:02
Unpickling fails when pickling is performed with 2.7 (pickledatetime.py) and unpickling is done with 3.5 (Tested on Ubuntu 16.04)
Please see detailed error description and workaround in the comments to the attached files.
msg339325 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-04-02 09:46
This seems to have been fixed with 8452ca15f41061c8a6297d7956df22ab476d4df4. I checked out your example locally and renamed pickle.py to another name to avoid module collision. Ran the below commands to pickle using Python 2 and to unpickle from master before and after the relevant commit. The fix is present in 3.7 and 3.6.

➜  bpo36499 python2 pickledatetime.py

# with 8452ca15f41061c8a6297d7956df22ab476d4df4
➜  bpo36499 ../cpython/python.exe unpickledatetime.py
Pickle:  {'timenode': datetime.datetime(2019, 4, 2, 15, 13, 2, 675110)}

# with 8452ca15f41061c8a6297d7956df22ab476d4df4~1
➜  bpo36499 ../cpython/python.exe unpickledatetime.py
Traceback (most recent call last):
  File "unpickledatetime.py", line 22, in <module>
    main()
  File "unpickledatetime.py", line 18, in main
    pkl = unpickler.load()
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1081, in load
    dispatch[key[0]](self)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1429, in load_reduce
    stack[-1] = func(*args)
TypeError: an integer is required (got type str)

# pickledatetime.py

import io
import pickle
from datetime import datetime


def main():
    """
    Uses python 2.7 to create a pickle file with a datetime object inside a dictionary.
    """
    data = {}
    data['timenode'] = datetime.now()
    with io.open('written_by_py27.pickle', 'wb') as handle:
        pickle.dump(data, handle, protocol=pickle.HIGHEST_PROTOCOL)

if __name__ == "__main__":
    main()

# unpickledatetime.py

import io
import pickle


def main():
    """
    Attempts to unpickle a dictionary with a datatype object inside
    """
    with io.open('written_by_py27.pickle', 'rb') as handle:
        unpickler = pickle._Unpickler(handle)
        unpickler.encoding = 'latin1'
        pkl = unpickler.load()
        print("Pickle: ", pkl)

if __name__ == "__main__":
    main()
msg339328 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-04-02 10:48
As noted, this is fixed in 3.6+. 3.5 is in security fix only mode ( https://devguide.python.org/#status-of-python-branches ), which is why the fix for #22005 wasn't backported.
History
Date User Action Args
2022-04-11 14:59:13adminsetgithub: 80680
2019-04-02 10:48:52josh.rsetstatus: open -> closed

superseder: datetime.__setstate__ fails decoding python2 pickle

nosy: + josh.r
messages: + msg339328
resolution: duplicate
stage: resolved

2019-04-02 09:46:38xtreaksetnosy: + serhiy.storchaka, xtreak
messages: + msg339325
2019-04-01 23:14:00xtreaksetnosy: + belopolsky, p-ganssle
2019-04-01 23:02:43vadimfcreate