RequestQueue | API | Crawlee for Python · Fast, reliable Python web crawlers.
Index
Methods
- __init__(client, id, name): None
Parameters
client: RequestQueueClient
id: str
name: str | None
Returns None
- async add_request(request, *, forefront): ProcessedRequest | None
Parameters
request: str | Request
optionalkeyword-onlyforefront: bool = False
Returns ProcessedRequest | None
- async add_requests(requests, *, forefront, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None
Parameters
requests: Sequence[str | Request]
optionalkeyword-onlyforefront: bool = False
optionalkeyword-onlybatch_size: int = 1000
optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(seconds=1)
optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None
Returns None
- async drop(): None
Returns None
- async fetch_next_request(): Request | None
Returns Request | None
- async get_handled_count(): int
Returns int
- async get_request(unique_key): Request | None
Parameters
unique_key: str
Returns Request | None
- async get_total_count(): int
Returns int
- async is_empty(): bool
Returns bool
- async is_finished(): bool
Returns bool
- async mark_request_as_handled(request): ProcessedRequest | None
Parameters
request: Request
Returns ProcessedRequest | None
- async open(*, id, name, alias, configuration, storage_client): Storage
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
optionalkeyword-onlyalias: str | None = None
optionalkeyword-onlyconfiguration: Configuration | None = None
optionalkeyword-onlystorage_client: StorageClient | None = None
Returns Storage
- async purge(): None
Returns None
- async reclaim_request(request, *, forefront): ProcessedRequest | None
Parameters
request: Request
optionalkeyword-onlyforefront: bool = False
Returns ProcessedRequest | None
- async to_tandem(request_manager): RequestManagerTandem
Parameters
optionalrequest_manager: RequestManager | None = None
Returns RequestManagerTandem