Brought up in #560:

It's unfortunate that none of these checks caught the SQLAlchemy issue in advance here. I think a good action item for us would be to add SQLAlchemy to the list of projects we run the test suites for on a nightly basis; that will help ensure that this doesn't happen again.

SQLAlchemy relies heavily on runtime introspection, is actively maintained, and has a test suite that runs pretty fast (~5m). Seems like a good candidate for the daily tests.

These tests are based on SQLAlchemy's run-test.yaml & run-on-pr.yaml workflows, with some setup borrowed from the pydantic tests.

A few notes about the setup:

• The tests pin the ubuntu-22.04 runner instead of using ubuntu-latest. This is done in the SQLAlchemy tests (sqlalchemy/sqlalchemy@8d73205). For whatever reason the tests fail on the Ubuntu 24.04 runner.
• PyPy tests are omitted. SQLAlchemy's CI includes PyPy tests, but they currently fail (example) and are ignored.
• The current failures are related to the issue from #560:

  AssertionError: <class 'typing.TypeAliasType'> is not <class 'typing_extensions.TypeAliasType'>
  

  The tests run green when using typing-extensions v4.12.2 (example run on my fork).

What just came to my mind, possibly we should also test against their rel_2_0 branch. Maybe one of them (cc: @CaselIT ) can comment on a good strategy here.

sqlalchemy/sqlalchemy#12472 has been merged - sqlachemy tests are now all passing 🎉 (thanks @Daraan!)

Thoughts from someone over at SQLAlchemy about which branches we should test against would be very much appreciated!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For whatever reason the tests fail on the Ubuntu 24.04 runner.

haven't really checked it, but it's a mistery to me. seemed like python behaviour is different for whatever reason in that version of ubuntu.

What just came to my mind, possibly we should also test against their rel_2_0 branch. Maybe one of them (cc: @CaselIT ) can comment on a good strategy here.

that may make sense, since that's what's released on pypi, at least until 2.1 is out later this year (hepefully). Not sure if you want to run them on main + last version on pypi?

main and rel_2_0 right now. by the end of this year we'll hopefully have 2.1 releases going and it will eventually move to "main + rel_2_1"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other than the branch consideration the change looks fine to me.

thanks for adding this test to your ci.
What will happen in case of failure? will an issue be filed?

What will happen in case of failure? will an issue be filed?

The workflow runs every day. If it fails, it opens an issue on this repo, and then somebody involved here will hopefully take a look and triage it; as a result we'll either change something on our side or file an issue on your side to get something changed.

main and rel_2_0 right now

Sounds good, let's do that.

What will happen in case of failure? will an issue be filed?

The workflow runs every day. If it fails, it opens an issue on this repo, and then somebody involved here will hopefully take a look and triage it; as a result we'll either change something on our side or file an issue on your side to get something changed.

main and rel_2_0 right now

Sounds good, let's do that.

ok, great. Thanks for adding this

Alright, I added rel_2_0 to the test matrix. The test setup needed a little tweaking since the pyproject.toml file in rel_2_0 doesn't have a project table, which made uv sync & uv run unhappy. I also noticed that tox might override typing-extensions in its test environment, so I made some changes that should ensure that typing-extensions-latest is always used

since the pyproject.toml file in rel_2_0 doesn't have a project table

yes, 2.0 still uses setup.cfg. It was migrated for 2.1 to pyproject, but 2.0 was kept as it was for consistency