Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 764: Inlined typed dictionaries #4082

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

Viicos
Copy link
Contributor

@Viicos Viicos commented Oct 24, 2024

Basic requirements (all PEP Types)

  • Read and followed PEP 1 & PEP 12
  • File created from the latest PEP template
  • PEP has next available number, & set in filename (pep-NNNN.rst), PR title (PEP 123: <Title of PEP>) and PEP header
  • Title clearly, accurately and concisely describes the content in 79 characters or less
  • Core dev/PEP editor listed as Author or Sponsor, and formally confirmed their approval
  • Author, Status (Draft), Type and Created headers filled out correctly
  • PEP-Delegate, Topic, Requires and Replaces headers completed if appropriate
  • Required sections included
    • Abstract (first section)
    • Copyright (last section; exact wording from template required)
  • Code is well-formatted (PEP 7/PEP 8) and is in code blocks, with the right lexer names if non-Python
  • PEP builds with no warnings, pre-commit checks pass and content displays as intended in the rendered HTML
  • Authors/sponsor added to .github/CODEOWNERS for the PEP

Standards Track requirements

  • PEP topic discussed in a suitable venue with general agreement that a PEP is appropriate
  • Suggested sections included (unless not applicable)
    • Motivation
    • Rationale
    • Specification
    • Backwards Compatibility
    • Security Implications
    • How to Teach This
    • Reference Implementation
    • Rejected Ideas
    • Open Issues
  • Python-Version set to valid (pre-beta) future Python version, if relevant
  • Any project stated in the PEP as supporting/endorsing/benefiting from the PEP formally confirmed such
  • Right before or after initial merging, PEP discussion thread created and linked to in Discussions-To and Post-History

📚 Documentation preview 📚: https://pep-previews--4082.org.readthedocs.build/pep-0764/

@AA-Turner
Copy link
Member

Thank you for opening this PR. PEPs require a core developer or PEP editor who is willing to sponsor it, which your PR currently lacks. I would advise you to start a thread on the https://discuss.python.org/ forum to see if there is support for this proposal and if someone is willing to sponsor it.

A

@AA-Turner AA-Turner closed this Oct 24, 2024
Copy link
Contributor

@erictraut erictraut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing this. From my perspective, it's looking pretty good.

I added some comments and questions for your consideration.

don't have a name. For this reason, their :attr:`~type.__name__` attribute
will be set to an empty string.

It is not possible to specify any class arguments such as ``total``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Required, NotRequired and ReadOnly should be permitted here. Absent any mention of this, the typing spec could be interpreted as disallowing this. It's better to make it clear.

I presume that all inlined TypedDicts are implicitly "total" (i.e. all items are assumed to be Required unless explicitly NotRequired). This makes sense because it's consistent with precedent, but it would be good to spell it out in the spec.

I think it's also worth considering making all inlined TypedDicts implicitly "closed" (as defined in draft PEP 728). This would deviate from precedent, but I think one could make a good argument for why inlined TypedDicts should be closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Required, NotRequired and ReadOnly should be permitted here. Absent any mention of this, the typing spec could be interpreted as disallowing this. It's better to make it clear.

I presume that all inlined TypedDicts are implicitly "total" (i.e. all items are assumed to be Required unless explicitly NotRequired). This makes sense because it's consistent with precedent, but it would be good to spell it out in the spec.

Updated.

I think it's also worth considering making all inlined TypedDicts implicitly "closed" (as defined in draft PEP 728). This would deviate from precedent, but I think one could make a good argument for why inlined TypedDicts should be closed.

Sounds reasonable. I have to say I'm still a bit confused with PEP 728 and closed=True. If we disallow subclassing inlined typed dictionaries, it means that every inlined typed dictionary is implicitly @final. What's the purpose of having closed=True then? I guess I should ask my question in the PEP 728 discussion thread instead.

peps/pep-9995.rst Outdated Show resolved Hide resolved
don't have a name. For this reason, their :attr:`~type.__name__` attribute
will be set to an empty string.

It is not possible to specify any class arguments such as ``total``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I correct in assuming that there is no way for an inlined TypedDict to "extend" another (named) TypedDict? One could imagine supporting this using dictionary expansion syntax, but that adds complexity that may not be justifiable. It would also impose additional runtime requirements on TypedDict classes (they'd need to be iterable).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good way to support inheritance could be to allow more than one argument to the subscript: TypedDict[Base, {"a": int}].

peps/pep-9995.rst Outdated Show resolved Hide resolved
peps/pep-9995.rst Outdated Show resolved Hide resolved
parametrization overloads. On ther other hand, :class:`~typing.TypedDict` is
already a :term:`special form <typing:special form>`.

* If fufure work extends what inlined typed dictionaries can do, we don't have
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider reusing typing.Dict here? I can see some potential advantages. It's less verbose, doesn't have the baggage of builtins.dict, and the fact that this is a "typed" dict is already implicit given that it is imported from typing and involves type annotations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Showing my TypeScript bias here 😅 but I'm wondering how sacrilegious it would be to avoid the subscription all together and simply use the dictionary itself?

def get_movie() -> {'name': str, 'year': int}:
    return {
        'name': 'Blade Runner',
        'year': 1982,
    }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes problems with the | operator at least. I've been thinking of writing up something like "Why can't we ...?" discussing why certain "obvious" ways to make typing more ergonomic may not work well in practice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that such writeup would be very useful!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an extra section in the rejected ideas regarding the plain dict syntax.

On using typing.Dict, I kind of like the idea for the reasons you mentioned @erictraut. Dict[{...}] implicitly spells out as something like:

Dict[{...}]
|      |------> ...with some typed fields
|--> a dict... 

However, I still think this is debatable for a couple reasons:

  • typing.Dict is currently marked as deprecated in the Python documentation (although not scheduled for removal). It might be confusing to undeprecate it.
  • The same subscripting syntax — with the only difference being the number of type arguments — means two different things. Can be confusing as well.

I'll add an entry to the open issues so that we can further discuss this option.

peps/pep-9995.rst Outdated Show resolved Hide resolved
Subclassing an inlined typed dictionary
---------------------------------------

Should we allow the following?::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this shouldn't be allowed by a static type checker. In this example, InlinedTD is a type alias.

If you change this to InlinedTD = TypedDict('InlinedTD', {'a': int}), then it should be allowed (as it is today) because InlinedTD is now a class, which can be used as a base class.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should also discuss whether an inlined typed dictonary is a proper class or an instance of some new internal typing._InlinedTypedDict class. As specified currently in this PEP, an inlined typed dictionary is a also a class, with an empty string for __name__ (which isn't ideal to be fair). Initially, I wanted inlined typed dictionaries to be an instance of some _InlinedTypedDict class, but this complicates the runtime behavior. For example, should we allow accessing the introspection attributes?

InlinedTD = TypedDict[{'a': int}]
InlinedTD.__required_keys__
InlinedTD.__total__
# etc

@erictraut
Copy link
Contributor

@Viicos, as a member of the typing council, I'm willing to sponsor this PEP.

@JelleZijlstra JelleZijlstra reopened this Oct 24, 2024
@JelleZijlstra
Copy link
Member

Reopened as Eric offered to sponsor (thanks Eric!).

@Viicos
Copy link
Contributor Author

Viicos commented Oct 25, 2024

I just realized I opened this PR on the original repository, but meant to open it on my fork as part of this discussion.

But I'm happy to keep going with this PEP. Regardless of the chosen implementation for partial, pick, etc, I can see how inlined typed dictionaries can be useful (Partial[SomeTD] needs to return a new typed dictionary, and this raises the question of which __name__ to choose — inlined typed dictionaries fulfill this requirement).

Thanks for the feedback. Should I open a separate discussion on the forum? Should it be in the PEPs/Typing category?

@hugovk hugovk added the new-pep A new draft PEP submitted for initial review label Oct 25, 2024
Copy link
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized I opened this PR on the original repository, but meant to open it on my fork as part of this discussion.

Whichever you prefer :) Probably depends on how much more work you think it needs before you're ready to submit this to be merged as a draft.

Should I open a separate discussion on the forum? Should it be in the PEPs/Typing category?

Wait until this draft is merged, then open a new discussion in the Typing category so people can discuss the merged and proposed draft (see https://peps.python.org/pep-0001/#discussing-a-pep).

Also please review PEP 1 and PEP 12 for the PEP process.

I've updated the OP to include the new PEP checklist, please work through it and check off things as appropriate. Thanks!

peps/pep-9995.rst Outdated Show resolved Hide resolved
peps/pep-9995.rst Outdated Show resolved Hide resolved
@hugovk hugovk changed the title Inlined typed dictionaries PEP PEP 764: Inlined typed dictionaries Oct 25, 2024
peps/pep-9995.rst Outdated Show resolved Hide resolved

The new inlined syntax can be used to resolve these problems::

def get_movie() -> TypedDict[{'name': str, year: int, 'production': {'name': str, 'location': str}}]:
Copy link
Contributor

@erictraut erictraut Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should support nested inlined TypedDict definitions as shown above. We should require that the type annotation for each item be a valid type expression, and a dictionary expression not enclosed by a TypedDict is an invalid type expression. The above example should therefore be changed to:

def get_movie() -> TypedDict[{"name": str, "year": int, "production": TypedDict[{"name": str, "location": str}]}]: ...

The grammar definition proposed above by Jelle already implies this, but the spec should make it clear that using nested dictionary expressions (like in the current code sample) is not allowed and should result in a type checker error.

I understand this might seem needlessly verbose, but if we allow dictionary expressions in type expressions outside of a TyepdDict type argument, it will create a cascade of exceptions and future composability issues. For example, we'd need to support raw dictionary expressions within Required, NotRequired, and ReadOnly.

Copy link
Contributor Author

@Viicos Viicos Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed using a nested inlined typed dictionary is more consistent! Changed.

(Using typing.Dict instead of TypedDict might help reducing verbosity here).

peps/pep-9995.rst Outdated Show resolved Hide resolved
def test_values() -> {"int": int, "str": str}:
return {"int": 42, "str": "test"}

Pyright has added support in version `1.1.297`_ (using :class:`dict`), but was later
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pyright 1.1.387, which will be published in the next day, includes experimental support for the latest draft of this PEP.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, updated to reference the future release.

Some GH discussions are left opened for now, waiting for an answer.
@Viicos Viicos force-pushed the inlined-typed-dictionaries branch 2 times, most recently from f87336c to 2386695 Compare October 29, 2024 21:46
@Viicos Viicos force-pushed the inlined-typed-dictionaries branch from 2386695 to 2e0c379 Compare October 30, 2024 08:29
Copy link

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Viicos, awesome job here! So exciting to see this awesome contribution!!

peps/pep-0764.rst Outdated Show resolved Hide resolved
peps/pep-0764.rst Show resolved Hide resolved
peps/pep-0764.rst Show resolved Hide resolved
peps/pep-0764.rst Outdated Show resolved Hide resolved
peps/pep-0764.rst Show resolved Hide resolved
Although it is not possible to specify any class arguments such as ``total``,
any :external+typing:term:`type qualifier` can be used for individual fields::

Movie = TypedDict[{'name': NotRequired[str], 'year': ReadOnly[int]}]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder, is there a world in which eventually you could do something like:

Movie = Annotated[TypedDict[{'name': str, 'year': int}], Total(False)]

Or something like that for class args?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would require:

  • introducing a typing.Total class/function
  • setting a precedent regarding the meaning of Annotated metadata, which is currently ignored by type checkers.

peps/pep-0764.rst Outdated Show resolved Hide resolved
peps/pep-0764.rst Outdated Show resolved Hide resolved
While using :class:`dict` isn't ideal, we could make use of
:class:`typing.Dict` with a single argument::

def get_movie() -> Dict[{'title': str}]: ...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I like the strict parametrization of TypedDict better, I feel like this would set an odd precedent and confuse some users.

@ZeroIntensity
Copy link
Member

Apologies for the small (now undone) edit of one of the checkmarks, I was reading the PR and pushed the wrong button 😅

Copy link

From the perspective of this user of type annotations, I have only one reservation: isn't this encouraging an antipattern? In my mind, if you need to type-annotate a tree-like dictionary structure, then isn't it time to come up with a tree of classes or named tuples or any other data structure to make it explicit? I would think that a dict[A | B | C, D | E | F], ie. flat with a few possible types on each side is about as far as I'd encourage a dict use with type annotations.

With a complex tree-like dict, wouldn't it make more sense to add assert isinstance at the point of use to specify the structure in a scalable way? An annotation can be factored out with type statements, but I'm not convinced that the flexibility afforded by dictionaries warrants such rigidity. IMHO that's not what dicts-of-dicts are for...

@DetachHead
Copy link
Contributor

From the perspective of this user of type annotations, I have only one reservation: isn't this encouraging an antipattern? In my mind, if you need to type-annotate a tree-like dictionary structure, then isn't it time to come up with a tree of classes or named tuples or any other data structure to make it explicit?

regardless of what your opinion is on whether or not TypedDicts should be defined in-line or not, you can define them with a type alias anyway and this syntax is more flexible than the class syntax because it allows for keys that aren't valid python names, which can be useful when dealing with data structures you don't control, eg:

type Foo = TypedDict[{"foo bar": int}]

(yes i know you can also do type Foo = TypedDict("Foo", {"foo bar": int}) but this is gross because you have to duplicate its name)

another point i'd like to make is that the current class "syntax" for defining TypedDicts is quite confusing imo. it makes the type look like a regular class when that's not what it represents at all. i think allowing a proper syntax to define them in-line makes the most sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-pep A new draft PEP submitted for initial review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants