Add Support for Specifying Headers in `context.enqueue_links` Method
When using the context.enqueue_links method, there is currently no way to specify custom headers (e.g., cookies or other required headers) for the links being enqueued. Some websites require specific headers or cookies to properly retrieve data from detail pages. It would be helpful to add support for passing headers or request options to context.enqueue_links to handle such cases.
Example use case:
headers = {
"Cookie": "session_id=abc123",
"User-Agent": "Custom User Agent"
}
context.enqueue_links(selector="a.detail-page", headers=headers)
This feature would greatly enhance flexibility when crawling websites that require authentication or specific headers for access.
Thank you!