feat: Add AdaptivePlaywrightCrawler by Pijukatel · Pull Request #872 · apify/crawlee-python

added 17 commits

December 24, 2024 09:38
Add method to BasicCrawler to handle just one request.
Add statistics

TODO:
Make mypy happy about statistics.
Wrap existing statistics from init in adaptive statistics.
Silent subcrawler statistics and loggers in general. (Set level to error?)
Ignore and create TODO follow up issue for refactoring Statistics class after technical discussion.
Pre-navigation hooks delegation to sub crawler hooks.
Statistics were marked as generics, but in reality were not.
Hardcoding state_model to make it explicit and clear.
WIP KVS handling. Currently it does not go through Result handler.

@github-actions github-actions bot added t-tooling

Issues with this label are in the ownership of the tooling team.

tested

Temporary label used only programatically for some analytics.

labels

Jan 6, 2025
Fix wrong id for predictor_state persistence.
Add test for pre nav hook
Add test for statistics in crawler init
…nternals of sub crawlers)

Cleanup commit results.

@Pijukatel

janbuchar

Add it in adaptive crawler instead at the cost of accessing many private members.

@Pijukatel

janbuchar

@Pijukatel

@Pijukatel

janbuchar

janbuchar

janbuchar

Co-authored-by: Jan Buchar <jan.buchar@apify.com>

@Pijukatel

janbuchar

vdusek