fix: Fix memory leak in `PlaywrightCrawler` on browser context creation by Pijukatel · Pull Request #1446 · apify/crawlee-python
Description
- Add
LocktoPlaywrightBrowserController._browser_contextcreation to ensure no orphaned duplicate contexts are created.
Issues
Testing
- Unit test added.
- Benchmark actor with PlaywrightCrawler used to confirm normal memory consumption with this fix.
Pijukatel
changed the title
fix: Fix memory leak in PlaywrightCrawler on context creation
fix: Fix memory leak in PlaywrightCrawler on browser context creation
Memory consumption of PlaywrightCrawler on 2000 pages:
- With fix: Average:378.4 MB Maximum:638.8 MB
- Without fix: Average:2.8 GBMaximum:3.8 GB -> Gets memory starved and stops.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job, thanks
Ignoring the failing test due to #1441
We should get this out soon, doesn't have to be right ahead, but I'd ship it this week.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters