Default crawlee[cli] template project broken on Windows
This could just be a misunderstanding on my part, first time using crawlee. I followed the guide on the homepage and used pipx to create a project
pipx run 'crawlee[cli]' create my-crawler
This produces a project that fails to run after I activate the venv in powershell and attempt to run the project
cd my-crawler .\.venv\Scripts\activate python -m my-crawler
[asyncio] ERROR Task exception was never retrieved future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()> Traceback (most recent call last): File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run await self._transport.connect() File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect raise exc File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect self._proc = await asyncio.create_subprocess_exec( File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec transport = await self._make_subprocess_transport( File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport raise NotImplementedError NotImplementedError
Annoyingly this took me ages to figure out. It seems to be related to this line in the _main_.py that gets generated, as commenting it out resolves the problem:
if platform.system() == 'Windows': # This mitigates a warning raised by curl-cffi. If you do not need to use curl-impersonate, you may remove this. asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) # __comment out ^ to fix__
Is this just an issue with my configuration? I've updated pip and all packages to the latest versions, I guess my problems resolved with the above but I couldn't find a relevant issue anywhere and wasted a lot of my workday trying to figure this out so I figured I'd create this issue to hopefully prevent others from getting stuck on the same problem. (I'm not a Windows person - entirely possible this is something I'm just not understanding)