feat: Add `MySQL` and `MariaDB` support for `SqlStorageClient` by Mantisus · Pull Request #1749 · apify/crawlee-python
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds MySQL and MariaDB support to SqlStorageClient, expanding the database backend options beyond SQLite and PostgreSQL. The implementation includes database-specific configurations, SQL dialect handling, and special error handling for MariaDB's unique locking behavior.
Changes:
- Added
sql_mysqloptional dependency group with aiomysql, cryptography, and SQLAlchemy - Implemented MySQL/MariaDB-specific SQL operations (INSERT IGNORE, ON DUPLICATE KEY UPDATE)
- Added MariaDB-specific error handling for locked rows (error 1020)
- Updated documentation to include installation and configuration instructions for MySQL/MariaDB
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Added aiomysql 0.3.2, cryptography 46.0.5, and pymysql 1.1.2 dependencies; updated package metadata to include sql-mysql extra |
| pyproject.toml | Added sql_mysql optional dependency group with required packages |
| src/crawlee/storage_clients/_sql/_storage_client.py | Extended dialect validation to include mysql/mariadb; added MySQL/MariaDB-specific engine configuration (connect_timeout, READ COMMITTED isolation level) |
| src/crawlee/storage_clients/_sql/_request_queue_client.py | Reorganized transaction handling; enabled skip_locked for MySQL/MariaDB in fetch operations |
| src/crawlee/storage_clients/_sql/_client_mixin.py | Implemented MySQL/MariaDB insert-with-ignore and upsert statements; added MariaDB error 1020 handling for locked rows |
| docs/guides/storage_clients.mdx | Updated documentation to include MySQL/MariaDB in supported databases; added installation instructions and configuration warnings |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.