I've encountered an issue with the redirection handling logic in CurlImpersonateHttpClient. Currently, there's no straightforward way to disable automatic redirect following when initializing CurlImpersonateHttpClient. Ideally, it should be possible to configure it like this:

crawler = HttpCrawler(http_client=CurlImpersonateHttpClient(allow_redirects=False))

While the allow_redirects argument is correctly passed to the underlying AsyncSession, CurlImpersonateHttpClient overrides this behavior in the crawl method by explicitly setting allow_redirects=True:

Actaully, this is a default value for AsyncSession, so there is no need to force it. I guess this is somehow connected to HttpxHttpClient and it's default behaviour. Related issue: #1013