feat: AutoscaledPool implementation by janbuchar ยท Pull Request #55 ยท apify/crawlee-python

@janbuchar

The current concurrency model is "launch desired_concurrency worker tasks that execute the task function in a loop". Maybe we could achieve simpler and slightly more efficient code if each execution of the task function had its own asyncio task, similarly to the original implementation. Follow-up tasks would be started in the done_callback, or in a recurring task (both of these would be needed).

@janbuchar

@janbuchar

vdusek

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just comments regarding the measure time util.

vdusek

@janbuchar

@janbuchar

vdusek

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

@janbuchar

@janbuchar

vdusek

vdusek

vdusek

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we will have a different opinion on how much we should document the code ๐Ÿ™‚. However, I'd be glad if you could add at least some comments explaining what's going on in main functions like run(), autoscale, and worker. So that it's easier to understand it when we come back to this module. In a similar way as it is in the original TS implementation. Also, I'd like the class docstring to be extended to a similar length as it is in the TS implementation. Again, I believe you can copy the doc from there and just adjust it.

@vdusek

At first, good job ๐Ÿ‘. I went through all of it, some parts were harder to understand, so I'd be glad if we could have more comments, as I wrote earlier, otherwise, it's great.

To your question, let's discuss it on Slack...

vdusek

@janbuchar

Maybe we will have a different opinion on how much we should document the code ๐Ÿ™‚. However, I'd be glad if you could add at least some comments explaining what's going on in main functions like run(), autoscale, and worker.

I filled in what seemed non-obvious. Please do point out if something is still unclear.

Also, I'd like the class docstring to be extended to a similar length as it is in the TS implementation. Again, I believe you can copy the doc from there and just adjust it.

Honestly, the TS implementation seems overdocumented to me (can't believe that I said that ๐Ÿ˜„). It seems like a very internal thing and as such it should not need things like usage examples. Also, copying documentation between projects just feels wrong.

vdusek

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! basically just formatting things, otherwise it's great

@janbuchar @vdusek

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

vdusek

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ๐Ÿš€