Issue46195
Created on 2021-12-29 18:38 by med2277, last changed 2022-04-11 14:59 by admin. This issue is now closed.
| Pull Requests | |||
|---|---|---|---|
| URL | Status | Linked | Edit |
| PR 30304 | merged | sobolevn, 2021-12-30 18:40 | |
| Messages (8) | |||
|---|---|---|---|
| msg409319 - (view) | Author: Mehdi2277 (med2277) | Date: 2021-12-29 18:38 | |
This is two closely related issues with get_type_hints on an optional annotated member. I'm testing with Annotated/get_type_hints from typing extensions on 3.8 and assuming they're backport equivalent to current behavior.
The first issue is get_type_hints has inconsistent behavior depending on whether annotation comes from a function with a None default or an attribute with a None default.
class Foo:
def __init__(
self,
x:
Annotated[Optional[str], "doc string"] = None,
):
...
class Foo2:
x: Annotated[Optional[str], "doc string"] = None
get_type_hints(Foo.__init__)
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}
get_type_hints(Foo2)
# {'x': typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string']}
Attributes with a None default are not wrapped by get_type_hints, but function parameters. Which of the two behaviors is correct I don't know, but I'd expect the two to be equivalent annotations.
The second issue is for function arguments with a None default the optional wrapper happens even if the type inside annotated already has optional. Example,
from typing_extensions import Annotated, get_type_hints
class Foo:
def __init__(
self,
x:
Annotated[Optional[str], "doc string"] = None,
):
...
get_type_hints(Foo.__init__, include_extras=True)
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}
For Annotated types I would expect any type rules like wrapping to apply only to the first argument and not the entire annotation. I mainly ran into this for a runtime type introspection library (similar in spirit to pydantic).
As a note include_extras being True or False while it changes type is an issue in either case. With include_extras as False the Annotated goes away, but the type still gets double wrapped as an Optional.
|
|||
| msg409325 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2021-12-29 19:54 | |
Could you try with 3.10 and the stdlib typing.Annotated please? There might be changes (in the past a default of None automatically caused an Optional to be added, but we changed our minds.-- --Guido (mobile) |
|||
| msg409340 - (view) | Author: Nikita Sobolev (sobolevn) * ![]() |
Date: 2021-12-30 00:38 | |
I can verify that this happens on `3.10` and `main` branches:
```
from typing import Annotated, Optional, get_type_hints
class Foo:
def __init__(self, x: Annotated[Optional[str], "d"] = None): ...
class Foo2:
x: Annotated[Optional[str], "d"] = None
print(get_type_hints(Foo.__init__, include_extras=False)) # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo2, include_extras=False)) # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo.__init__, include_extras=True)) # not ok?
# {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}
print(get_type_hints(Foo2, include_extras=True)) # ok
# {'x': typing.Annotated[typing.Optional[str], 'd']}
```
Notice that 3rd case does not look correct: `{'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}`
In my opinion it should be `{'x': typing.Annotated[typing.Optional[str], 'd']}`
I will look into it! :)
|
|||
| msg409341 - (view) | Author: Nikita Sobolev (sobolevn) * ![]() |
Date: 2021-12-30 00:39 | |
And on 3.9 as well. |
|||
| msg409357 - (view) | Author: Nikita Sobolev (sobolevn) * ![]() |
Date: 2021-12-30 10:38 | |
As Guido said, the root cause of this problem is because `None` default automatically adds `Optional` to the resulting type. Source: https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/typing.py#L1854-L1856 So, what happens there: - correct `value` is passed to `_eval_type`, correct result `typing.Annotated[typing.Optional[str], 'd']` is returned at this point - then `if name in defaults and defaults[name] is None:` adds extra `Optional` annotation on top of `Annotated` > in the past a default of None automatically caused an Optional to be added, but we changed our minds Guido, are you talking about https://github.com/python/typing/issues/275 ? Now all type-checkers (AFAIK) support something similar to `--no-implicit-optional` mode. Having this in mind, I see different solutions to the current problem: 1. Remove `Optional` inference with `None` default. This is going to be a some-what breaking change. The only positive side of this is that we can really simplify our code (mainly because the other solution is to complicate our code even more). 2. Or we can change this place to explicitly check for `Annotated` type and its internal type. This should be the easiest to write and backport. But, it still has some complexity to it. I think that this is a better solution: we don't break existing behavior, change is local and pretty trivial. Also caused by this: - https://bugs.python.org/issue42921 - https://bugs.python.org/issue42288 |
|||
| msg409375 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2021-12-30 17:21 | |
Yes, we changed PEP 484 in https://github.com/python/peps/pull/689. So get_type_hints() should follow suit. |
|||
| msg414328 - (view) | Author: Jelle Zijlstra (JelleZijlstra) * ![]() |
Date: 2022-03-02 05:30 | |
New changeset 20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9 by Nikita Sobolev in branch 'main': bpo-46195: Do not add `Optional` in `get_type_hints` (GH-30304) https://github.com/python/cpython/commit/20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9 |
|||
| msg414329 - (view) | Author: Jelle Zijlstra (JelleZijlstra) * ![]() |
Date: 2022-03-02 05:31 | |
This is now fixed in 3.11, but we'll leave 3.10 and 3.9 alone. Thanks for your bug report! |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:59:54 | admin | set | github: 90353 |
| 2022-03-02 05:31:58 | JelleZijlstra | set | status: open -> closed resolution: fixed messages: + msg414329 stage: patch review -> resolved |
| 2022-03-02 05:30:27 | JelleZijlstra | set | nosy:
+ JelleZijlstra messages: + msg414328 |
| 2021-12-30 18:40:26 | sobolevn | set | keywords:
+ patch stage: patch review pull_requests: + pull_request28517 |
| 2021-12-30 17:21:30 | gvanrossum | set | messages: + msg409375 |
| 2021-12-30 10:38:57 | sobolevn | set | messages: + msg409357 |
| 2021-12-30 00:39:09 | sobolevn | set | messages:
+ msg409341 versions: + Python 3.9 |
| 2021-12-30 00:38:21 | sobolevn | set | versions: + Python 3.10, Python 3.11, - Python 3.8 |
| 2021-12-30 00:38:12 | sobolevn | set | nosy:
+ sobolevn messages: + msg409340 |
| 2021-12-29 19:54:58 | gvanrossum | set | messages: + msg409325 |
| 2021-12-29 19:10:03 | med2277 | set | type: behavior |
| 2021-12-29 18:38:47 | med2277 | create | |

