RequestManagerTandem | API | Crawlee for Python · Fast, reliable Python web crawlers.
- RequestManager
- RequestManagerTandem
Index
Methods
- __init__(request_loader, request_manager): None
Parameters
request_loader: RequestLoader
request_manager: RequestManager
Returns None
- async add_request(request, *, forefront): ProcessedRequest | None
Parameters
request: str | Request
optionalkeyword-onlyforefront: bool = False
Returns ProcessedRequest | None
- async add_requests(requests, *, forefront, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None
Parameters
requests: Sequence[str | Request]
optionalkeyword-onlyforefront: bool = False
optionalkeyword-onlybatch_size: int = 1000
optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(seconds=1)
optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None
Returns None
- async drop(): None
Returns None
- async fetch_next_request(): Request | None
Returns Request | None
- async get_handled_count(): int
Returns int
- async get_total_count(): int
Returns int
- async is_empty(): bool
Returns bool
- async is_finished(): bool
Returns bool
- async mark_request_as_handled(request): ProcessedRequest | None
Parameters
request: Request
Returns ProcessedRequest | None
- async reclaim_request(request, *, forefront): ProcessedRequest | None
Parameters
request: Request
optionalkeyword-onlyforefront: bool = False
Returns ProcessedRequest | None
- async to_tandem(request_manager): RequestManagerTandem
Parameters
optionalrequest_manager: RequestManager | None = None
Returns RequestManagerTandem