feat: Handle unprocessed requests in `add_requests_batched` by Pijukatel · Pull Request #1159

feat: Handle unprocessed requests in `add_requests_batched` by Pijukatel · Pull Request #1159 · apify/crawlee-python

added 2 commits

April 17, 2025 10:23

TODO:  finish tests and format and mypy

github-actions bot added t-tooling

Issues with this label are in the ownership of the tooling team.

tested

Temporary label used only programatically for some analytics.

labels

Apr 17, 2025

Pijukatel marked this pull request as ready for review

April 22, 2025 10:55

Pijukatel deleted the handle-unprocessed-requests branch

April 23, 2025 12:18

Mantisus pushed a commit to Mantisus/crawlee-python that referenced this pull request

Apr 24, 2025

### Description
Adds retry to unprocessed requests in call `add_requests_batched`.
Retry calls recursively `_process_batch`, which initially works on full
request batch and then on batches of unprocessed requests until retry
limit is reached or all requests are processed. Each retry is done after
linearly increasing delay with each attempt.

Unprocessed requests are not counted in `request_queue.get_total_count`
Add test.

### Issues

- Closes: [Handle unprocessed requests in
batch_add_requests](apify/apify-sdk-python#456)

Mantisus pushed a commit to Mantisus/crawlee-python that referenced this pull request

Apr 24, 2025

### Description
Adds retry to unprocessed requests in call `add_requests_batched`.
Retry calls recursively `_process_batch`, which initially works on full
request batch and then on batches of unprocessed requests until retry
limit is reached or all requests are processed. Each retry is done after
linearly increasing delay with each attempt.

Unprocessed requests are not counted in `request_queue.get_total_count`
Add test.

### Issues

- Closes: [Handle unprocessed requests in
batch_add_requests](apify/apify-sdk-python#456)