feat: Add `MySQL` and `MariaDB` support for `SqlStorageClient` by Mantisus · Pull Request #1749 · apify/crawlee-python

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds MySQL and MariaDB support to SqlStorageClient, expanding the database backend options beyond SQLite and PostgreSQL. The implementation includes database-specific configurations, SQL dialect handling, and special error handling for MariaDB's unique locking behavior.

Changes:

  • Added sql_mysql optional dependency group with aiomysql, cryptography, and SQLAlchemy
  • Implemented MySQL/MariaDB-specific SQL operations (INSERT IGNORE, ON DUPLICATE KEY UPDATE)
  • Added MariaDB-specific error handling for locked rows (error 1020)
  • Updated documentation to include installation and configuration instructions for MySQL/MariaDB

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
uv.lock Added aiomysql 0.3.2, cryptography 46.0.5, and pymysql 1.1.2 dependencies; updated package metadata to include sql-mysql extra
pyproject.toml Added sql_mysql optional dependency group with required packages
src/crawlee/storage_clients/_sql/_storage_client.py Extended dialect validation to include mysql/mariadb; added MySQL/MariaDB-specific engine configuration (connect_timeout, READ COMMITTED isolation level)
src/crawlee/storage_clients/_sql/_request_queue_client.py Reorganized transaction handling; enabled skip_locked for MySQL/MariaDB in fetch operations
src/crawlee/storage_clients/_sql/_client_mixin.py Implemented MySQL/MariaDB insert-with-ignore and upsert statements; added MariaDB error 1020 handling for locked rows
docs/guides/storage_clients.mdx Updated documentation to include MySQL/MariaDB in supported databases; added installation instructions and configuration warnings

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.