Repository navigation
Fix scheduler crash loop from large Edge deadline callbacks - #74445
Open
KidAmnesiac1 wants to merge 1 commit into
Open
KidAmnesiac1 wants to merge 1 commit into
KidAmnesiac1 wants to merge 1 commit into
Conversation
A missed deadline with a SyncCallback is queued to the EdgeExecutor as JSON in edge_job.command, which is limited to 2048 characters. The callback carries a snapshot of the Dag run (including its conf), so it can easily go over the limit; ours was about 2.3 KB. The database rejects the insert, the scheduler loop fails and rolls back, and every restart picks up the same deadline and crashes again, stopping scheduling for the whole deployment.
KidAmnesiac1
requested review from
dheerajturaga,
jscheffl and
shubhamraj-git
as code owners
October 8, 2026 06:38
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Changes the
edge_job.commandcolumn fromvarchar(2048)toText, so theEdgeExecutorcan store workloads of any size.Reason for the change
The
EdgeExecutorstores each workload as JSON inedge_job.command. ASyncCallbackdeadline workload carries a snapshot of the Dag run, including itsconf, so it can easily exceed 2048 characters. Ours was about 2.3 KB (Airflow 3.3.2, edge3 4.3.2, PostgreSQL), and the insert failed:The error escapes the scheduler loop and the transaction rolls back, so the deadline stays pending and every restarted scheduler crashes on it again. Scheduling stops for the whole deployment until the Dag run is deleted.
Fix
0007changesedge_job.commandtoText. On PostgreSQL, convertingvarchartotextonly changes metadata, so the upgrade doesn't rewrite the table.EdgeJobModel.commandis nowText, and the5.0.0revision head points at the new migration.varcharlengths, so the test only catches the bug on PostgreSQL and MySQL.The migration is labelled
5.0.0to match0006, which landed after5.0.0rc2was cut. Both may need relabelling to the next edge3 version.Related: #74446 makes the scheduler survive a callback that fails to queue for any reason. The Dag run snapshot itself comes from
get_simple_context()inDeadline.handle_miss, whoseTODOcovers fetching it at runtime instead (attempted in #66608, reverted in #68909).Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines