Astronomer

Carter Page joined Inside the CTO Circle to talk about how we actually build with agents at Astronomer. The short version: every team runs its own agents. They work overnight, and engineers review the output in the morning. When an incident fires, the context is already there by the time a human sits down. What makes that safe enough to run every day: 🔁 Airflow is the outer loop: scheduling, retries, event triggers, all the logs in one place 🤖 Agents are the inner loop and keep iterating until the result holds up 👤 Agents never push to production. Whoever ships the code owns it, no matter who wrote it Agents can take the night shift. Orchestration and ownership are why we trust them with it. Thanks for having Carter on. Worth the full listen 👇

Generic AI assistants can read your log files, but that doesn’t make them good at diagnosing broken pipelines. When a DAG fails, the real evidence is scattered across task logs, lineage graphs, code diffs, and upstream data. Apache Airflow is the only place where all of this context actually comes together. Featured live at #AirflowSummit Austin 2026, the team from Astronomer (Stephanie Niu & Christine S.) brings one of the most talked-about sessions to Airflow Summit Online Reconnect: "How We Taught an Agent Airflow"! Whether you missed it during the live tracks in Austin or couldn't attend in person, here is what you'll learn: ✅ Orchestration-Level Diagnosis: Why grounding AI investigations in live Airflow context (lineage, diffs, and upstream state) outperforms surface-level LLM log readers. ✅ Encoding Operational Knowledge: How Otto, Astronomer's data engineering agent, encodes failure patterns from eight years of running Airflow at enterprise scale. ✅ 95% MTTR Reduction Case Study: A real-world production story of a team that cut Mean Time To Resolution by 95% in eight weeks—shifting morning routines from firefighting to reviewing proposed fixes. ✅ The Self-Healing Blueprint: A practical mental model for trustworthy AI diagnosis, including which layers of self-healing are worth building yourself and which ones aren't. 🗓️ November 4 & 5, 2026 · Free Online Event 👉 Save your spot & explore the agenda: https://lnkd.in/eQ7cjmKm #DataEngineering #AIOps #Orchestration #DataOps #AirflowSummit #MLOps

What you should know about ETL/ELT pipelines in 2026. Join Airflow expert Marc Lamberti on October 15 to learn the latest Airflow features that make ETL/ELT orchestration easier today. We'll cover core best practices, plus how to handle new challenges like prepping data for both humans and agents, so you're fully up to speed on modern data engineering.

How it started: Ramp's applied science team started with just one scheduled notebook. How it's going.... Today it runs more than 100 production ML models, with Airflow orchestrating every step. At Airflow Summit 2026, Ryan Stevens, Director of Applied Science at Ramp, shared how the team got there: by crawling, walking, and then running on an internal ML platform built on Airflow with Astro. One standard way to build, schedule and monitor every model means the team can keep adding models without adding engineers at the same rate. Read the full rundown and catch the recording of Ryan's talk below. #mlops #apacheairflow #appliedml

If you're in Bangalore 🇮🇳 and work with data pipelines join us in 3 days. 👇 We're hosting the next Bangalore Airflow Meetup this 𝗪𝗲𝗱𝗻𝗲𝘀𝗱𝗮𝘆, 𝗢𝗰𝘁𝗼𝗯𝗲𝗿 𝟳𝘁𝗵 at the 𝗨𝗯𝗲𝗿 𝗕𝗮𝗻𝗴𝗮𝗹𝗼𝗿𝗲 𝗢𝗳𝗳𝗶𝗰𝗲 — and we'd love 💚 to see you there. On the agenda: • 𝗦𝘁𝗮𝗻𝗱𝗮𝗿𝗱𝗶𝘇𝗶𝗻𝗴 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 w/ Dinesh Jagannathan & Sambit Acharya from Uber • 𝗠𝗶𝗰𝗿𝗼𝘀𝗲𝗿𝘃𝗶𝗰𝗲-𝗚𝗿𝗮𝗱𝗲 𝗗𝗲𝗹𝗶𝘃𝗲𝗿𝘆 𝗮𝗻𝗱 𝗥𝗲𝗹𝗲𝗮𝘀𝗲 𝗳𝗼𝗿 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗗𝗔𝗚𝘀 𝗮𝘁 𝗨𝗯𝗲𝗿 w/ Piyush Maheshwari & @Sameer Raj from Uber • 𝗪𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘆𝗼𝘂 𝘀𝗮𝘃𝗶𝗻𝗴 𝘀𝘁𝗮𝘁𝗲 𝗶𝗻 𝗔𝗽𝗮𝗰𝗵𝗲 𝗔𝗶𝗿𝗳𝗹𝗼𝘄? w/ Amogh Desai, Senior Software Engineer at Astronomer  • 𝗪𝗵𝗲𝗻 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗧𝗮𝘀𝗸𝘀 𝗛𝗮𝘃𝗲 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗖𝗼𝗻𝘀𝗲𝗾𝘂𝗲𝗻𝗰𝗲𝘀 w/ Shubham Raj from Qualcomm • 𝗦𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗶𝗻𝗴 𝗬𝗼𝘂𝗿 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗨𝗽𝗴𝗿𝗮𝗱𝗲: 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹 𝗧𝗼𝗼𝗹𝘀 𝗳𝗼𝗿 𝗠𝗶𝗴𝗿𝗮𝘁𝗶𝗻𝗴 𝗳𝗿𝗼𝗺 𝟮.𝘅 𝘁𝗼 𝟯 w/ Ankit Chaurasia from Apple If you're scaling Airflow pipeline authoring, deployment, or delivery across a large engineering org, this one's packed with real production lessons from Uber's own team. 🗓 𝗪𝗲𝗱𝗻𝗲𝘀𝗱𝗮𝘆, 𝗢𝗰𝘁𝗼𝗯𝗲𝗿 𝟳𝘁𝗵 | 𝟮:𝟯𝟬–𝟲:𝟬𝟬 𝗣𝗠 (𝗜𝗦𝗧) 📍 𝗨𝗯𝗲𝗿 𝗕𝗮𝗻𝗴𝗮𝗹𝗼𝗿𝗲 𝗢𝗳𝗳𝗶𝗰𝗲 Snacks included. RSVP here → https://lnkd.in/epyx5qzC #ApacheAirflow #DataEngineering #Bangalore #Airflow3 #Meetup

We're proud to announce that Astronomer is now a Premier Partner in the Snowflake Partner Network. Thousands of data teams rely on Snowflake and Astro together to run the pipelines behind their analytics and AI. Premier status reflects the depth of that partnership, and our commitment to making Astro the most reliable way to orchestrate data and AI workloads on Snowflake. Thanks to the Snowflake team for the partnership. There's a lot more to come. #Snowflake #ApacheAirflow #DataEngineering

3 exciting updates for anyone writing Airflow Dags... 1️⃣ Astro IDE is officially GA! After almost 7,000 projects across 2,300 teams using it, all Astro users can now write, test, and deploy Dags right in your browser (no local setup needed). 2️⃣ The Astro extension for VS Code is live today. Health checks, local Airflow runs, Otto built in, and docs search, all without leaving your editor. Grab it in the VS Code marketplace (link below). Cursor users: Your dedicated Astro extension arrives next week. 3️⃣ Big improvements to the Astro CLI are coming soon. We also believe the future must be friendly to agents. Our new experience will have improvements that make the Astro CLI easier for agents to work with, while still providing an excellent user experience for data engineers. More on this soon. Yetunde Dada breaks down all of these updates in the post below.

If you’re a ChatGPT or Codex user and you manage Airflow pipelines, you’re in luck. We partnered with OpenAI to release a ChatGPT plugin that lets you code, debug, upgrade, trace lineage, and query data from Airflow all from within ChatGPT. Best of all, it's totally free for all Airflow users (whether you’re an Astronomer customer or not). Check out Julian’s post below for more.

Some personal news: I'm joining Astronomer as an Advisor, and today we're sharing the first thing we've built together. Most data engineering benchmarks test SQL or dbt models in isolation. Real production platforms are orchestrated, usually by Apache Airflow, and shared across teams with their own conventions and legacy systems. We didn't have a good way to measure whether an agent, and in particular Astronomer's data engineering agent, Otto, can handle that. So we built the Astronomer Data Engineering Benchmark, an open-source benchmark of tasks on a platform modeled on real customer environments at Astronomer. The results were interesting. Frontier models are already strong at this work, but the harness still makes a measurable difference. Holding the model and thinking level fixed, Otto had a higher pass rate than the native harness (Claude Code or Codex) in most configurations. On Airflow, its pass rate jumped to 20 percentage points. This was a real collaboration, and I want to give a special shoutout to the team who did the heavy lifting: Julian LaNeve, Anubhav Sharma, and Kaxil Naik. I've enjoyed working with this team, and I'm looking forward to what comes next. Blog and repo are in the comments.