Collaborator

Description

• additional_http_error_status_codes and ignore_http_error_status_codes have been removed from the constructor parameters for HTTP clients
• The method _raise_for_error_status_code has been removed from HttpClient and its logic has been moved to BasicCrawler
• Prioritized checking of status codes that indicate Session blocking. Codes (401, 403, 429) trigger retire for the Session and retry, while other 4XX codes are handled as client errors (errors without retries).
• additional_http_error_status_codes is no longer used when checking status codes that indicate Session blocking. According to current documentation, these codes should trigger a retry, not retire for the Session and retry. Blocking status codes can be modified in SessionPool through create_session_settings.
• Standardized error handling for status codes in both PlaywrightCrawler and HttpCrawler

Issues

• Closes: Refactor additional status codes arguments #998
• Closes: Implementing correct handling of 403 and the other codes that should trigger a SessionError #830

Testing

• Tests for client error status codes now use code 402.
• A separate test has been added for 403 since in this case the number of retries is affected by the 'max_session_rotations' parameter.

Pijukatel approved these changes Feb 27, 2025

View reviewed changes

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

janbuchar approved these changes Feb 28, 2025

View reviewed changes

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.