fix: Improve crawler statistics logging by vdusek · Pull Request #1364 · apify/crawlee-python
$ python run_crawler.py
[ParselCrawler] INFO Crawled 0/8292 pages, 0 failed requests, desired concurrency 1.
[ParselCrawler] INFO Current request statistics:
┌───────────────────────────────┬────────┐
│ requests_finished │ 0 │
│ requests_failed │ 0 │
│ retry_histogram │ [0] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ None │
│ requests_finished_per_minute │ 0 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 0s │
│ requests_total │ 0 │
│ crawler_runtime │ 14.1ms │
└───────────────────────────────┴────────┘
[crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] ERROR Request to https://crawlee.dev failed and reached maximum retries
Traceback (most recent call last):
File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_basic_crawler.py", line 1312, in __run_task_function
await self._run_request_handler(context=context)
File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_basic_crawler.py", line 1407, in _run_request_handler
await wait_for(
...<5 lines>...
)
File "/home/vdusek/Projects/crawlee-python/src/crawlee/_utils/wait.py", line 37, in wait_for
return await asyncio.wait_for(operation(), timeout.total_seconds())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vdusek/.local/share/uv/python/cpython-3.13.0-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
return await fut
^^^^^^^^^
File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_context_pipeline.py", line 114, in __call__
await final_context_consumer(cast('TCrawlingContext', crawling_context))
File "/home/vdusek/Projects/crawlee-python/src/crawlee/router.py", line 98, in __call__
return await self._default_handler(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vdusek/Projects/crawlee-python/run_crawler.py", line 12, in request_handler
raise SessionError('Blah blah blah')
crawlee.errors.SessionError: Blah blah blah
[crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
[ParselCrawler] INFO Error analysis: total_errors=1 unique_errors=1
[ParselCrawler] INFO Final request statistics:
┌───────────────────────────────┬────────────────────────────────┐
│ requests_finished │ 0 │
│ requests_failed │ 1 │
│ retry_histogram │ [0, 0, 0, 0, 0, 0, 0, 0, 0, 1] │
│ request_avg_failed_duration │ 607.8ms │
│ request_avg_finished_duration │ None │
│ requests_finished_per_minute │ 0 │
│ requests_failed_per_minute │ 8 │
│ request_total_duration │ 607.8ms │
│ requests_total │ 1 │
│ crawler_runtime │ 7.31s │
└───────────────────────────────┴────────────────────────────────┘