fix: add `payload` to `SendRequestFunction` to support `POST` request by Mantisus · Pull Request #1202

fix: add `payload` to `SendRequestFunction` to support `POST` request by Mantisus · Pull Request #1202 · apify/crawlee-python

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for POST requests by introducing an optional payload parameter to the send request functionality and updating related tests.

Updated test cases to handle both GET and POST requests using parameterized inputs.
Modified the send_request function signatures in both _basic_crawler.py and _types.py to accept an optional payload.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tests/unit/crawlers/_basic/test_basic_crawler.py	Updated test to include POST requests with payload and JSON response validation
src/crawlee/crawlers/_basic/_basic_crawler.py	Added the payload parameter to the send_request function and passed it along to the HTTP client
src/crawlee/_types.py	Modified the SendRequestFunction interface to include the payload parameter

		pytest.param('POST', 'post', 'Hello, world!', id='post send_request'),
		],
		)
		async def test_send_request_works(server_url: URL, method: HttpMethod, path: str, payload: None \| bytes) -> None:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POST test case is providing a payload as a string ('Hello, world!'), while the function signature indicates a payload of type 'None | bytes'. Ensure type consistency by either converting the string to bytes or updating the type annotation if string payloads are supported.

Copilot uses AI. Check for mistakes.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

vdusek changed the title ~~fix: add payload in SendRequestFunction for support POST request~~ fix: add payload to SendRequestFunction to support POST request

May 20, 2025