We can have a "locked" CI and an unlocked one. That may help? However, I like to test against unlocked CIs to be sure we are hitting these errors and not the users.

By a locked CI you mean something like npm's lock files? I agree with you that we need to run the CI without version fixes as much as possible.

That was not what I had in mind, though.

My idea is to (after a successful release and as part of the CI), store the output of pip freeze somewhere for documentation purposes. Then, when a CI fails we can diff the current versions against the versions that produced the last successful CI run.

Another idea is to periodically schedule a CI run. That way we notice it earlier when the CI is broken and not on the next PR.