feat: Add adaptive context helpers by Pijukatel · Pull Request #964 · apify/crawlee-python

added 30 commits

December 24, 2024 09:38
Add method to BasicCrawler to handle just one request.
Add statistics

TODO:
Make mypy happy about statistics.
Wrap existing statistics from init in adaptive statistics.
Silent subcrawler statistics and loggers in general. (Set level to error?)
Ignore and create TODO follow up issue for refactoring Statistics class after technical discussion.
Pre-navigation hooks delegation to sub crawler hooks.
Statistics were marked as generics, but in reality were not.
Hardcoding state_model to make it explicit and clear.
WIP KVS handling. Currently it does not go through Result handler.
Fix wrong id for predictor_state persistence.
Add test for pre nav hook
Add test for statistics in crawler init
…nternals of sub crawlers)

Cleanup commit results.
Add it in adaptive crawler instead at the cost of accessing many private members.
Use different url for unit tests.
(By temporal wrapper context.)
This adds some complexity, but adds more flexibility.
Add one sanity check test for parsel variant.
Update some doc strings.

@github-actions github-actions bot added t-tooling

Issues with this label are in the ownership of the tooling team.

tested

Temporary label used only programatically for some analytics.

labels

Feb 7, 2025

@Pijukatel

vdusek

janbuchar

@Pijukatel

janbuchar

@Pijukatel

@Pijukatel

janbuchar

Mantisus pushed a commit to Mantisus/crawlee-python that referenced this pull request

Feb 19, 2025
### Description

Add adaptive context helpers and documentation for
AdaptivePlaywrightCrawler.

### Issues

- Closes: apify#249

---------

Co-authored-by: Jan Buchar <Teyras@gmail.com>
Co-authored-by: Jan Buchar <jan.buchar@apify.com>