I have already discussed this with @janbuchar. This PR should not introduce any breaking changes. If additional_http_error_status_codes and ignore_http_error_status_codes are not specified, the previous behavior is preserved.

Status codes were previously discussed in PR: #498

However, we have differences with the JS version of crawlee:

Python:

• 400-499 codes cause errors without retries

JS:

• 400-499 codes are treated as successful
• Exceptions:
  • 401, 403, 429 - cause errors with retries
  • 406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, we have differences with the JS version of crawlee:

Python:

400-499 codes cause errors without retries
JS:

400-499 codes are treated as successful
Exceptions:
401, 403, 429 - cause errors with retries
406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

Any ideas how we want to align it? I believe we do not have issue for that, correct? cc @janbuchar

However, we have differences with the JS version of crawlee:
Python:
400-499 codes cause errors without retries
JS:
400-499 codes are treated as successful
Exceptions:
401, 403, 429 - cause errors with retries
406 - causes error without retries and ignores ignoreHttpErrorStatusCodes parameter

Any ideas how we want to align it? I believe we do not have issue for that, correct? cc @janbuchar

I believe apify/crawlee#812 tracks this. I also think that what we do in this PR is the correct way to handle things. The "success on 4xx" in JS is a crutch that prevented retries on such errors, and we can do that more cleanly now (in both implementations).

How does the Python version handle 406 again? Can it be ignored via ignoreHttpErrorStatusCodes?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.