Updating Macro to be lazy loaded by Tom-Fynes · Pull Request #5708 · SQLMesh/sqlmesh

Fixes #5692

This pull request refactors the macro evaluation logic in sqlmesh/core/macros.py to introduce lazy loading for Python environments, improve error handling for missing macro dependencies, and enhance code readability. The changes enable projects to share state databases without requiring all external dependencies to be present, and provide clearer error messages when macros fail to load. Additionally, the code has been reformatted for better readability, and error messages have been made more consistent throughout the file.

Macro Environment Loading and Error Handling:

  • Introduced lazy loading for Python environment executables, deferring imports and macro definitions until they are actually needed. This allows projects to share state databases without requiring all dependencies to be present at load time. Added _unloaded_executables and _failed_imports tracking, and refactored macro loading logic into a new _load_python_env method. (sqlmesh/core/macros.py) [1] [2]
  • Enhanced error handling in macro invocation: if a macro fails to load due to a missing dependency, a clear error message is raised, explaining the likely cause and how to proceed. (sqlmesh/core/macros.py)

Code Readability and Consistency:

Error Message Improvements:

  • Standardized error messages for missing variables, schema issues, and invalid arguments, making them more descriptive and user-friendly. (sqlmesh/core/macros.py) [1] [2] [3] [4] [5] [6] [7] [8]

General Cleanup:

  • Removed unused import prepare_env from the file. (sqlmesh/core/macros.py)

These changes make the macro evaluation system more robust, maintainable, and easier for users to debug when issues arise.