feat: AutoscaledPool implementation by janbuchar ยท Pull Request #55 ยท apify/crawlee-python
- closes Implement
AutoscaledPool#19
The current concurrency model is "launch desired_concurrency worker tasks that execute the task function in a loop". Maybe we could achieve simpler and slightly more efficient code if each execution of the task function had its own asyncio task, similarly to the original implementation. Follow-up tasks would be started in the done_callback, or in a recurring task (both of these would be needed).
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just comments regarding the measure time util.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we will have a different opinion on how much we should document the code ๐. However, I'd be glad if you could add at least some comments explaining what's going on in main functions like run(), autoscale, and worker. So that it's easier to understand it when we come back to this module. In a similar way as it is in the original TS implementation. Also, I'd like the class docstring to be extended to a similar length as it is in the TS implementation. Again, I believe you can copy the doc from there and just adjust it.
At first, good job ๐. I went through all of it, some parts were harder to understand, so I'd be glad if we could have more comments, as I wrote earlier, otherwise, it's great.
To your question, let's discuss it on Slack...
Maybe we will have a different opinion on how much we should document the code ๐. However, I'd be glad if you could add at least some comments explaining what's going on in main functions like
run(),autoscale, andworker.
I filled in what seemed non-obvious. Please do point out if something is still unclear.
Also, I'd like the class docstring to be extended to a similar length as it is in the TS implementation. Again, I believe you can copy the doc from there and just adjust it.
Honestly, the TS implementation seems overdocumented to me (can't believe that I said that ๐). It seems like a very internal thing and as such it should not need things like usage examples. Also, copying documentation between projects just feels wrong.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! basically just formatting things, otherwise it's great
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM ๐
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters