Fix order-dependent flaky tests related to UTF-8 support by dg98 · Pull Request #1093

Fix order-dependent flaky tests related to UTF-8 support by dg98 · Pull Request #1093 · prometheus/client_python

Conversation

For a university course, I tested this project for flaky tests, which are tests that can both pass or fail without changes to their code. I noticed that the tests changed in #1070 fail when their execution order is changed because the legacy validation is enabled in some tests but disabled in the test that follows. Since the execution order is not ensured, it's safer to disable them in the same test.
With these changes applied, the whole test suit passes every time even when executed in random order.

Signed-off-by: Dennis Gaebler <dennis.gaebler@uni-ulm.de>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

The downside to this approach is that if one of the assertions raise then the next test will also fail since disable_legacy_validation() will no longer be called. A couple options would be to:

Wrap the validation enablement/disabling with a With Statement Context Manager
Create a test fixture to be used when we need to use legacy validation.

Signed-off-by: Dennis Gaebler <dennis.gaebler@uni-ulm.de>

Thanks for your feedback, I didn't think of that. I added a context manager that enables and disables legacy validation and changed all uses of these functions to with statements

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2 participants