fix: Handle invalid URLs in `RequestList` by Mantisus · Pull Request #1803 · apify/crawlee-python
This could be a very long list, and if we iterate fully in the constructor just for validation, it feels like an overhead. However, if we wrap it in a generator that performs the validation, that would essentially be the current solution. This is because, with persistence disabled, validation occurs the moment a new URL is received from the iterator. And when persistence is enabled, we iterate through all URLs before saving in the storage.