Add specialized/parametrized RequestQueueClients

Context
Default RequestQueue offers many features apart from request managment, like the possibility of multiple producers/consumers working with it, real-time persistence, and integration with console. To support all of it, it comes with some costs - actual financial costs to use request queue and increased network traffic as the request queue is sending many API calls related to request handling and locking. Some use cases do not need those additional features of the request queue and thus are only affected by the costs with no benefit.

Proposed solution

  • Add single consumer variant of the RequestQueue - this will drop the locking mechanism of the RequestQueue and reduce costs and network traffic, while preserving all other benefits of RequestQueue

  • Add new specialized RequestQueue client that will not be based on platform-backed RequestQueue to even further reduce the costs. Possibly just extend MemoryRequestQueueClient with persistence capabilities to KVS. Use case is single consumer/producer request queue with on-demand persistence that is massively cheaper compared to platform-backed RequestQueue. - For example this POC

  • Update documentation and describe all the variants of RequestQueue, their benefits, drawbacks, and recommended use cases. Also mention crawlee-based RequestQueue clients that still have some niche use case even on Apify platform.