Visitar URL original
Fix scheduler crash loop from large Edge deadline callbacks by KidAmnesiac1 · Pull Request #74445 · apache/airflow · GitHub
Skip to content

Fix scheduler crash loop from large Edge deadline callbacks - #74445

Open
KidAmnesiac1 wants to merge 1 commit into
apache:mainfrom
KidAmnesiac1:fix-edge-job-command-length
Open

KidAmnesiac1 wants to merge 1 commit into
apache:mainfrom
KidAmnesiac1:fix-edge-job-command-length

Conversation

@KidAmnesiac1

@KidAmnesiac1 KidAmnesiac1 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

Changes the edge_job.command column from varchar(2048) to Text, so the EdgeExecutor can store workloads of any size.

Reason for the change

The EdgeExecutor stores each workload as JSON in edge_job.command. A SyncCallback deadline workload carries a snapshot of the Dag run, including its conf, so it can easily exceed 2048 characters. Ours was about 2.3 KB (Airflow 3.3.2, edge3 4.3.2, PostgreSQL), and the insert failed:

psycopg2.errors.StringDataRightTruncation: value too long for type character varying(2048)

The error escapes the scheduler loop and the transaction rolls back, so the deadline stays pending and every restarted scheduler crashes on it again. Scheduling stops for the whole deployment until the Dag run is deleted.

Fix

  • New edge3 migration 0007 changes edge_job.command to Text. On PostgreSQL, converting varchar to text only changes metadata, so the upgrade doesn't rewrite the table.
  • EdgeJobModel.command is now Text, and the 5.0.0 revision head points at the new migration.
  • A new test queues a callback workload over 2048 characters and checks it's stored unchanged. On PostgreSQL it fails with the error above without this change. SQLite doesn't enforce varchar lengths, so the test only catches the bug on PostgreSQL and MySQL.

The migration is labelled 5.0.0 to match 0006, which landed after 5.0.0rc2 was cut. Both may need relabelling to the next edge3 version.

Related: #74446 makes the scheduler survive a callback that fails to queue for any reason. The Dag run snapshot itself comes from get_simple_context() in Deadline.handle_miss, whose TODO covers fetching it at runtime instead (attempted in #66608, reverted in #68909).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

A missed deadline with a SyncCallback is queued to the EdgeExecutor as
JSON in edge_job.command, which is limited to 2048 characters. The
callback carries a snapshot of the Dag run (including its conf), so it
can easily go over the limit; ours was about 2.3 KB. The database
rejects the insert, the scheduler loop fails and rolls back, and every
restart picks up the same deadline and crashes again, stopping
scheduling for the whole deployment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:edge Edge Executor / Worker (AIP-69) / edge3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant