Home Original page

fix: Resolve compatibility issue between `SqlStorageClient` and `AdaptivePlaywrightCrawler` by Mantisus · Pull Request #1496 · apify/crawlee-python

@Mantisus

@Mantisus

@vdusek vdusek changed the title fix: Add special deepcopy logic for SqlStorageClient for compatibility with AdaptivePlaywrightCrawler. fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler

Oct 17, 2025

vdusek

@Mantisus

vdusek

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm

@vdusek vdusek added the t-tooling

Issues with this label are in the ownership of the tooling team.

label

Oct 19, 2025

@janbuchar

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

janbuchar

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment above

@Mantisus

@Mantisus

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

Updated.

@Pijukatel could you take a look at this as well? You are the most familiar with the AdaptivePlaywrightCrawler. I just want to make sure I'm not missing any important cases where deepcopy was used.

vdusek

janbuchar

@Mantisus

janbuchar

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@janbuchar

Perhaps we could add a test that shows that adaptive crawler can be instantiated with sql storage client, that wouldn't be too hard 🙂

@Mantisus

Pijukatel