When the Automation Platform Is What Failed
Who detects a failed automation platform, what it does when it cannot establish authority, which identities it runs under, and what happens to the work it missed.
revisualized.com, Joseph Tracy, Infrastructure Engineer, Storage and Automation
About
Full background →I'm Joseph Tracy, an infrastructure engineer in Seattle with over two decades running enterprise systems end to end: scale-out NAS and SAN storage, Linux and Windows Server, virtualization, campus networking, and the identity and collaboration platforms sitting on top of all of it. Most of that time I have worked at the failure end, as the escalation point a hard problem routes to when the frontline is out of moves, which is a different education from building something once and handing it over.
Automation came out of that work rather than alongside it. Bringing hundreds of wireless access points onto a network, or large numbers of IP video cameras into one system, is the kind of job that stops being possible by hand somewhere around the third rack, and the scripts that made those deployments repeatable are the same instinct behind what I build now: the check that should have run before the job started, the snapshot that should have been taken before the config changed, the timing history that turns "it feels slow" into a measurement. Each one ships tested, documented, and with its source.
Everything here is built and tested on my own time, against public documentation and my own lab. This site is where that work happens in public.
Featured
Shell Reliability Library: The Plumbing Every Script Rewrote
The shared reliability plumbing extracted from my monitoring scripts: dependency preflight, locking with guaranteed cleanup, temporary directories that cannot leak, structured logging, and retry with backoff. Not a framework. A library with exactly the functions its two consumers earned.
Projects
All projects →Mount Audit Reporter
A Bash auditor that reads, without changing, the mounts visible in its current Linux mount namespace and compares them with a reviewed manifest: source, filesystem type, options that must be present, and options that must never appear. It reports every mismatch in one run, refuses a manifest it cannot trust, and prints a candidate manifest from a known-good host.
Systemd Evidence Collector
A pair of service-state read-only tools for a failed systemd service. One captures a private evidence bundle with a manifest and a per-command outcome record. The other reads that bundle and writes a review report that states what was observed and leaves the conclusion to a person.
Alert Mail Dispatcher: Delivered, or Still on Disk
A small, dependency-free alert deliverer for hosts with no mail agent. Callers hand it a subject and body; it delivers to the relay via STARTTLS, and when the relay is unreachable it spools the message to disk atomically and retries on the next flush, because the moment your network is degraded is exactly the moment your alerts must not silently vanish.
Articles
All articles →Say Only What the Evidence Supports in the Postmortem
Part two of two. What each postmortem claim costs in evidence, how to state a finding plainly once the review earns it, what to write instead of a finding about a person, and how to record prevention, action items, and residual risk so the document stays accurate after it leaves the room.
Say Only What the Evidence Supports: Incident Language for the People Who Decide
Part one of two, for IT staff who explain running incidents to directors, managers, and VPs: how to label what you know, carry the same evidence into three audience-specific updates, ask for a decision, and report recovery honestly, with rewrites of the sentences that most often go wrong.
The Directive That Was Ignored
systemd-analyze verify can warn and continue with a unit file whose meaning has changed. By default it still exits zero, so a gate reading exit status lets the change through unless it asks for warnings to count.
What a Bounded Blast Radius Still Lets Through
A target-count cap can pass while a job is about to touch the wrong things. This piece works through four operational questions a cap can't answer on its own: what gets touched, how fast it happens, whether it can be undone, and how soon you'd know, each backed by an executed, reproducible demonstration. Every experimental or calculated number here is generated by the reproduction script or an explicit calculation shown in the piece.