Issue 33652: Improve pickling of typing types

Created on 2018-05-26 07:10 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (8) msg317727 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer)

Date: 2018-05-26 07:10

The following PR makes pickles for typing types more portable. Type variables no longer use _find_name() and can be unpickled in 3.6. Subscripted generics no longer expose internals and can be unpickled in 3.6 and future versions with changed internal implementation.

Before this PR:

>>> import pickle, pickletools, typing                                            
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))              
    0: \x80 PROTO      4
    2: \x95 FRAME      30
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x94 MEMOIZE    (as 0)
   20: \x8c SHORT_BINUNICODE '_find_name'
   32: \x93 STACK_GLOBAL
   33: h    BINGET     0
   35: \x8c SHORT_BINUNICODE 'T'
   38: \x86 TUPLE2
   39: R    REDUCE
   40: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))
    0: \x80 PROTO      4
    2: \x95 FRAME      198
   11: \x8c SHORT_BINUNICODE 'copyreg'
   20: \x8c SHORT_BINUNICODE '_reconstructor'
   36: \x93 STACK_GLOBAL
   37: \x8c SHORT_BINUNICODE 'typing'
   45: \x94 MEMOIZE    (as 0)
   46: \x8c SHORT_BINUNICODE '_GenericAlias'
   61: \x93 STACK_GLOBAL
   62: \x8c SHORT_BINUNICODE 'builtins'
   72: \x94 MEMOIZE    (as 1)
   73: \x8c SHORT_BINUNICODE 'object'
   81: \x93 STACK_GLOBAL
   82: N    NONE
   83: \x87 TUPLE3
   84: R    REDUCE
   85: }    EMPTY_DICT
   86: (    MARK
   87: \x8c     SHORT_BINUNICODE '_inst'
   94: \x88     NEWTRUE
   95: \x8c     SHORT_BINUNICODE '_special'
  105: \x89     NEWFALSE
  106: \x8c     SHORT_BINUNICODE '_name'
  113: N        NONE
  114: \x8c     SHORT_BINUNICODE '__origin__'
  126: h        BINGET     0
  128: \x8c     SHORT_BINUNICODE 'Union'
  135: \x93     STACK_GLOBAL
  136: \x8c     SHORT_BINUNICODE '__args__'
  146: h        BINGET     1
  148: \x8c     SHORT_BINUNICODE 'int'
  153: \x93     STACK_GLOBAL
  154: h        BINGET     1
  156: \x8c     SHORT_BINUNICODE 'str'
  161: \x93     STACK_GLOBAL
  162: \x86     TUPLE2
  163: \x8c     SHORT_BINUNICODE '__parameters__'
  179: )        EMPTY_TUPLE
  180: \x8c     SHORT_BINUNICODE '__slots__'
  191: N        NONE
  192: \x8c     SHORT_BINUNICODE '__module__'
  204: h        BINGET     0
  206: u        SETITEMS   (MARK at 86)
  207: b    BUILD
  208: .    STOP
highest protocol among opcodes = 4

With this PR:

>>> import pickle, pickletools, typing                                                              
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))                       
    0: \x80 PROTO      4
    2: \x95 FRAME      13
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x8c SHORT_BINUNICODE 'T'
   22: \x93 STACK_GLOBAL
   23: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))       
    0: \x80 PROTO      4
    2: \x95 FRAME      66
   11: \x8c SHORT_BINUNICODE '_operator'
   22: \x8c SHORT_BINUNICODE 'getitem'
   31: \x93 STACK_GLOBAL
   32: \x8c SHORT_BINUNICODE 'typing'
   40: \x8c SHORT_BINUNICODE 'Union'
   47: \x93 STACK_GLOBAL
   48: \x8c SHORT_BINUNICODE 'builtins'
   58: \x94 MEMOIZE    (as 0)
   59: \x8c SHORT_BINUNICODE 'int'
   64: \x93 STACK_GLOBAL
   65: h    BINGET     0
   67: \x8c SHORT_BINUNICODE 'str'
   72: \x93 STACK_GLOBAL
   73: \x86 TUPLE2
   74: \x86 TUPLE2
   75: R    REDUCE
   76: .    STOP
highest protocol among opcodes = 4

If there is a chance it would be nice to merge these changes into 3.7. Otherwise either pickles created in 3.7.0 will be incompatible not only with older Python versions, but with future Python versions too, or we will need to add complex code for supporting specific 3.7.0 pickles in future versions.

msg317752 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer)

Date: 2018-05-26 18:19

New changeset 09f3221fbbf72692308149054e4f7668b08b22eb by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
bpo-33652: Improve pickle support in the typing module. (GH-7123)
https://github.com/python/cpython/commit/09f3221fbbf72692308149054e4f7668b08b22eb

msg317753 - (view) Author: miss-islington (miss-islington) Date: 2018-05-26 18:38

New changeset d49862582ed3513debed6e919fd4f92e9d4eebbd by Miss Islington (bot) in branch '3.7':
bpo-33652: Improve pickle support in the typing module. (GH-7123)
https://github.com/python/cpython/commit/d49862582ed3513debed6e919fd4f92e9d4eebbd

msg317766 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer)

Date: 2018-05-26 21:19

There is a question -- what to do with all these __getstate__ and __setstate__ methods? They are part of the pickle protocol, but they are not used when define __reduce__. And they are not needed for supporting compatibility with older versions, because in 3.6 pickleable generic types were pickled by name. I think it is better to remove these methods.

msg317768 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer)

Date: 2018-05-26 21:42

Yes, these are just legacy from times when TypeVars were serialized by value, not by identity like now. I think it should be safe to remove them. Would you like to make a PR?

msg317837 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer)

Date: 2018-05-28 10:54

New changeset 97b523db7c79c18c48516fba9410014d9896abc4 by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
https://github.com/python/cpython/commit/97b523db7c79c18c48516fba9410014d9896abc4

msg317839 - (view) Author: miss-islington (miss-islington) Date: 2018-05-28 11:21

New changeset 98b089e2a178a309f20ac9ff498b230407da1f47 by Miss Islington (bot) in branch '3.7':
bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
https://github.com/python/cpython/commit/98b089e2a178a309f20ac9ff498b230407da1f47

msg317846 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer)

Date: 2018-05-28 12:57

I think this can be closed now.

History Date User Action Args 2022-04-11 14:59:00adminsetgithub: 77833 2018-05-28 12:57:46levkivskyisetstatus: open -> closed
resolution: fixed
messages: + msg317846

stage: patch review -> resolved

2018-05-28 11:21:56miss-islingtonsetmessages: + msg317839 2018-05-28 10:55:12miss-islingtonsetpull_requests: + pull_request6782 2018-05-28 10:54:59levkivskyisetmessages: + msg317837 2018-05-28 08:22:35serhiy.storchakasetpull_requests: + pull_request6778 2018-05-26 21:42:26levkivskyisetmessages: + msg317768 2018-05-26 21:19:09serhiy.storchakasetmessages: + msg317766 2018-05-26 18:38:02miss-islingtonsetnosy: + miss-islington
messages: + msg317753
2018-05-26 18:21:00miss-islingtonsetpull_requests: + pull_request6766 2018-05-26 18:19:28levkivskyisetmessages: + msg317752 2018-05-26 07:17:13serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request6758 2018-05-26 07:10:15serhiy.storchakacreate