Apache Airflow 3.4.0: Task Loops and Dags in Any Language

• Task Loops are added with .loop() on a decorated task group, with a required max_iterations bound and an optional until predicate.
• The post describes the TypeScript, Java and Go SDKs as generally available, while its upgrade section marks the language SDKs and the Dag Importer as experimental.
• YAML Dags use an opt-in importer on top of AIP-85.
• Dags defined outside Python run through a language runtime on the Dag processor and workers, so they need a coordinator configured for their queue.

Task Loops Repeat a Task Group Up to a Hard Cap #

Until now, a Dag whose number of passes was not known in advance needed a fixed depth or a Dag that triggered itself. In 3.4, .loop() repeats a decorated task group a number of times decided at runtime, and each pass can read the previous result through loop.previous. The body runs before the check, so it always runs at least once. The post states that a max_iterations value is required, so a loop cannot run without bound.

A reader may wonder how a loop fits a directed acyclic graph. The post's answer is that no dependency loop is introduced: the tasks still form a cycle-free graph, and the repetition is handled at runtime only. The loop context offers loop.index, loop.previous and loop.result, and the optional until predicate can return a LoopDecision(stop=..., reason=...) to record why it stopped.

A loop with no until runs exactly max_iterations times and succeeds. If a loop reaches its cap while the predicate is still false, the final iteration is marked failed, so a loop that never converges shows up as a failure. Airflow creates each iteration only when the loop decides to continue, so no tail of skipped iterations appears. Loops can be combined with .partial(), .override() and .expand(), but they cannot be nested, cannot sit inside a mapped task group, and cannot themselves be mapped. The UI adds an Iterations view that shows each pass and the reason its gate let the loop continue.

Whole Dags in TypeScript, Java and Go Through AIP-85 #

Airflow 3.3 allowed individual tasks in Java or Go while Python still owned the Dag. In 3.4, the entire Dag, including its schedule, tasks and dependencies, can be written in TypeScript, Java or Go with no Python file. The post attributes this to AIP-85, the Dag Importer, a pluggable framework that lets the Dag processor read Dag sources beyond .py. For a language SDK, the Dag processor runs that language's runtime to read the Dags an artifact declares, and then treats them like Python Dags in the scheduler and UI.

Each SDK builds one self-contained artifact, and the runtime each artifact needs differs. The post lists these requirements:

Supported Dag features include nested task groups, if and switch branching, labeled edges, order-only dependencies, and triggering another Dag. The post also notes that the TypeScript SDK adds human-in-the-loop tasks. Routing is configured in the [sdk] section: each language is registered with a coordinator, such as airflow.sdk.coordinators.node.NodeCoordinator for ts, and queue_to_coordinator maps each Dag's queue to that coordinator. The same configuration is needed on the Dag processor and on the workers.

YAML Dags Need a $schema Marker and Explicit Templating #

The YAML importer is opt-in and needs no code and no build step. A .yaml file carrying a $schema marker is discovered, validated against a schema and built into a Dag with real operators. Each task either names a built-in operator with uses: or a code task with run:, binds arguments with with:, and declares dependencies with needs:. The post's example is marked as illustrative, and it asks readers to confirm exact keys against the released schema.

Templating is explicit. A field renders only when the author marks it with $t, reads an XCom with $x, reads a file with $f, or sets a literal with $const. A marker on a field the operator does not template is a build error rather than being ignored, so a reader should expect invalid markers to stop the build.

Pin the Exact Versions of the Language SDKs and Importer #

The release post presents the three language SDKs as generally available, but its upgrade section says the language SDKs, the Dag Importer framework and several common.ai previews are explicitly marked experimental. Both statements appear in the same post. The upgrade guidance is to review the 3.4.0 release notes for Significant Changes, and to pin the exact versions you test with, since the experimental label is the more cautious reading for production planning.

Who Should Check Their Dags Before Upgrading to 3.4.0 #

Teams with iterative workflows, such as a refinement or review cycle that today runs as a fixed number of tasks or as a Dag that triggers itself, can replace that pattern with .loop(). Teams that write Java or Go tasks inside a Python Dag can evaluate whether the whole Dag should move to the language SDK. Teams with declarative pipelines can evaluate the YAML importer. Any team that adopts one of these should register the matching coordinator and route each Dag's queue to it on both the Dag processor and the workers.

Also in This Release #

• A Dag Folders panel in the Dags list, which groups Dags by their directory layout in each bundle.
• Dags not written in Python show their source language in the grid and graph.
• An AI Model tab on the task instance page, showing the model, token usage and estimated cost for common.ai tasks.
• Stability, performance and quality-of-life fixes across the scheduler, API server and UI.