Logs: incorrect number of pages to crawl
Logs in crawlee 1.0.0
[BeautifulSoupCrawler] INFO Crawled 0/1 pages, 0 failed requests, desired concurrency 10. [BeautifulSoupCrawler] INFO Current request statistics: ┌───────────────────────────────┬────────┐ │ requests_finished │ 0 │ │ requests_failed │ 0 │ │ retry_histogram │ [0] │ │ request_avg_failed_duration │ None │ │ request_avg_finished_duration │ None │ │ requests_finished_per_minute │ 0 │ │ requests_failed_per_minute │ 0 │ │ request_total_duration │ 0s │ │ requests_total │ 0 │ │ crawler_runtime │ 42.7ms │ └───────────────────────────────┴────────┘ [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 10; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0 [BeautifulSoupCrawler] INFO Crawled 22/70 pages, 0 failed requests, desired concurrency 10. [BeautifulSoupCrawler] INFO Crawled 61/70 pages, 0 failed requests, desired concurrency 11. [crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish [BeautifulSoupCrawler] INFO Final request statistics: ┌───────────────────────────────┬────────────┐ │ requests_finished │ 70 │ │ requests_failed │ 0 │ │ retry_histogram │ [70] │ │ request_avg_failed_duration │ None │ │ request_avg_finished_duration │ 2.82s │ │ requests_finished_per_minute │ 196 │ │ requests_failed_per_minute │ 0 │ │ request_total_duration │ 3min 17.4s │ │ requests_total │ 70 │ │ crawler_runtime │ 21.38s │ └───────────────────────────────┴────────────┘
Same code in crawlee 1.2.0 and 1.3.0
[BeautifulSoupCrawler] INFO Current request statistics: ┌───────────────────────────────┬──────┐ │ requests_finished │ 0 │ │ requests_failed │ 0 │ │ retry_histogram │ [0] │ │ request_avg_failed_duration │ None │ │ request_avg_finished_duration │ None │ │ requests_finished_per_minute │ 0 │ │ requests_failed_per_minute │ 0 │ │ request_total_duration │ 0s │ │ requests_total │ 0 │ │ crawler_runtime │ 0s │ └───────────────────────────────┴──────┘ [BeautifulSoupCrawler] INFO Crawled 0/318 pages, 0 failed requests, desired concurrency 10. [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 10; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0 [BeautifulSoupCrawler] INFO Crawled 25/387 pages, 0 failed requests, desired concurrency 10. [BeautifulSoupCrawler] INFO Crawled 55/387 pages, 0 failed requests, desired concurrency 11. [crawlee._autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish [BeautifulSoupCrawler] INFO Final request statistics: ┌───────────────────────────────┬────────────┐ │ requests_finished │ 70 │ │ requests_failed │ 0 │ │ retry_histogram │ [70] │ │ request_avg_failed_duration │ None │ │ request_avg_finished_duration │ 3.37s │ │ requests_finished_per_minute │ 178 │ │ requests_failed_per_minute │ 0 │ │ request_total_duration │ 3min 55.7s │ │ requests_total │ 70 │ │ crawler_runtime │ 23.55s │ └───────────────────────────────┴────────────┘