Add local Airflow 3 support by jeremybeard · Pull Request #1811

Add local Airflow 3 support by jeremybeard · Pull Request #1811 · astronomer/astro-cli

Description

This change adds support to the CLI for locally running Airflow 3 using the existing astro dev commands.

After the first Astro runtime image for Airflow 3 is released by Astronomer astro dev init will automatically begin to initialize new projects using Airflow 3
Airflow 3 replaces the webserver process with the api-server process, which is reflected in the Airflow 3 container names after running astro dev start
The DAG processor component will run as a separate dag-processor container, instead of inline in the scheduler
The astro dev upgrade-check command is removed as it was only applicable for upgrading to Airflow 2.0. Note this is a different command to the still-supported astro dev upgrade-test
Locally the Airflow 3 UI will not require "admin/admin" credentials as is the case for the Airflow 2 UI

🎟 Issue(s)

Resolves #1807

🧪 Functional Testing

Unit tests updated
Integration tests passing
Manually ran through each astro dev command

📋 Checklist

Rebased from the main (or release if patching) branch (before testing)
Ran make test before taking out of draft
Ran make lint before taking out of draft
Added/updated applicable tests
Tested against Astro-API (if necessary).
Tested against Houston-API and Astronomer (if necessary).
Communicated to/tagged owners of respective clients potentially impacted by these changes.
Updated any related documentation

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great improvement 🚀 , I have left some minor comments to squash but otherwise LGTM

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we would have to update this dag to be compatible with Airflow 3 or replace wit ith other working example short-term

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm it was working but not in the most recent builds. I'll find out whether the DAG is incompatible or if it's a pre-release bug.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated it to use assets instead of datasets (thanks @TJaniF!) but there are OSS fixes to come before it will succeed.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I just realized that our integration tests could be improved by adding tests for AF 3 deployments. I am ok with doing it later once there is a final 3.0-0 release so that we don't have to add temporary custom Dockerfile edit logic and also given that we have tested it manually to cover for more cases than the ITs.

LGTM, I just realized that our integration tests could be improved by adding tests for AF 3 deployments. I am ok with doing it later once there is a final 3.0-0 release so that we don't have to add temporary custom Dockerfile edit logic and also given that we have tested it manually to cover for more cases than the ITs.

@neel-astro @jeremybeard can you create an issue to track that post 3.0 release work?

BasPH mentioned this pull request

Mar 24, 2025

8 tasks