systemd-tty-ask-password-agent paints the passphrase prompt before any keyboard exists, silently discarding the first keystrokes #44332

systemd version the issue has been seen with

260.5

Used distribution

NixOS unstable (26.05.20261008.7c8764b)

Linux kernel version used

6.12.112

CPU architectures issue was seen on

x86_64

Component

systemd-cryptsetup, systemd-ask-password

Expected behaviour you didn't see

An interactive passphrase prompt should not be painted before it is possible for a human to answer it. Either the prompt waits until an input device capable of producing key events exists, or keystrokes are accepted from the moment the prompt becomes visible.

Unexpected behaviour you saw

In a systemd initrd, systemd-ask-password-console.service paints the LUKS passphrase prompt on /dev/tty1 roughly 830 ms before the internal keyboard is usable. Keys pressed in that window produce no input event at all and are silently lost. In practice this costs the first character of the passphrase on essentially every boot, because the window is almost exactly human reaction time to a prompt appearing.

The resulting failure is indistinguishable from a typo: the passphrase is rejected and the user retypes it. The subjective experience is that the keyboard "needs waking up" before it starts registering.

This is deterministic, not intermittent. The root device is PCIe NVMe and is present by ~2.7 s, while the keyboard requires xhci_pci to load, then sequential hub port enumeration, then hid-apple to bind. The disk wins that race on every boot.

Steps to reproduce the problem

1. A machine whose root device enumerates faster than its keyboard — in practice, any NVMe root plus a USB-attached (including internal USB) keyboard.
2. LUKS2 root with a passphrase keyslot, no TPM/FIDO2 enrolment.
3. boot.initrd.systemd.enable = true (systemd initrd), no plymouth.
4. Reboot and begin typing the moment the prompt appears.
5. The first character is missing. journalctl -b -o short-monotonic shows the prompt predating the keyboard's HID bind.

Additional program output to the terminal or log subsystem illustrating the issue

Two consecutive boots, monotonic seconds, unmodified configuration:

boot A                                   boot B
[2.254] fbcon: Taking over console       [2.342]   <- prompt is visible from here on
[2.768] systemd-tty-ask-password-agent:  [2.826]   <- prompt PAINTED
        Starting password query on /dev/tty1
[2.902] usbcore: registered usbhid       [2.958]
[3.353] usb 1-5: new full-speed USB      [3.412]   <- keyboard enumerated
        device number 4 using xhci_hcd
[3.501] input: Apple Inc. Apple Internal [3.569]
        Keyboard / Trackpad as .../input1
[3.613] apple 0003:05AC:0291.0004:       [3.649]   <- keyboard USABLE
        input,hiddev98,hidraw3: USB HID
        v1.10 Keyboard

Dead window: 845 ms and 823 ms. The console framebuffer is already up at 2.25/2.34 s, so the prompt really is on screen for the whole window — this is not a case of the text being buffered and displayed later.

This is not a termios flush

Worth stating explicitly, because every community report of this symptom I could find assumes the keystroke is read and thrown away. It is not.

In src/shared/ask-password-api.c at v260, ask_password_tty() contains exactly three tc*attr calls:

819:  if (tcgetattr(ttyfd, &old_termios) < 0)
829:  r = RET_NERRNO(tcsetattr(ttyfd, TCSANOW, &new_termios));
965:  (void) tcsetattr(ttyfd, TCSANOW, &old_termios);

There is no tcflush, no TCSAFLUSH, and no TCSADRAIN anywhere in the file; the read loop begins at line 903. Anything already buffered on the tty would be read normally. The keypress is lost because no input device exists yet to generate an event from it — nothing in userspace is discarding anything.

So this cannot be fixed by changing how the agent handles the terminal. It can only be fixed by not asking until asking is meaningful.

Why this is not a hardware bug

#43399 was closed with "there's some issue between the kernel, the usb stack and the hw. Please contact the driver people." I want to address that directly, because I think the class of report has been consistently misread — including by the reporters themselves.

• The keyboard here is completely healthy. It works perfectly 830 ms later, through the entire rest of the boot, and in userspace. There is no -71, no failed enumeration, no driver error. The kernel log above is a clean, successful, unremarkable USB probe.
• Slow enumeration is the premise, not the defect. USB HID enumeration takes the time it takes; no kernel change will make it instant, and it shouldn't have to. The defect is that systemd asks a human a question before the human can possibly answer.
• This is precisely the class of race systemd exists to eliminate. Declared ordering is the project's central abstraction, and systemd-ask-password-console.service already carries ordering against other units. It has no ordering whatsoever against input availability.
• systemd is the only component with the necessary knowledge. The kernel does not know a prompt is pending; the agent does not currently ask whether input is possible. Only systemd sits on both facts at once — and udev, in the same project, already computes ID_INPUT_KEYBOARD.

The same mechanism has been reported repeatedly across distributions and consistently misattributed to hardware:

• Arch BBS 307773 — "Cryptsetup prompt appears before the keyboard gets detected", ~5 s window behind a USB hub, unresolved. The reporter explicitly asks for the prompt to wait.
• Arch BBS 255520 — "the first keystroke isn't recorded, the decryption fails and I have to type my passphrase again", misattributed to the "press TAB" hint, unresolved.
• Manjaro 122483 — "LUKS passphrase drops keypresses", worked around by switching initramfs hooks.
• NixOS/nixpkgs#297538 — open, same family.
• Red Hat bz#470740, bz#1445353, Debian #1080203 — older instances.

That several independent reporters concluded their hardware was failing, when it was working correctly, is itself part of the cost of this bug.

I've also filed NixOS/nixpkgs#576661 asking that distribution to expose a stopgap option, since users there currently have no supported way to avoid this. I don't think that's the right long-term home for the fix, which is why I'm reporting it here too.

Impact

Silent input loss on a credential prompt. The user cannot tell a dropped character from a mistyped one, so the natural conclusions are "I fat-fingered it", "my keyboard is dying", or "my disk is failing" — all wrong. On a machine where the window is longer (a keyboard behind a hub, per Arch 307773), several characters are lost and the prompt looks broken.

Proposed fix

In rough order of how well I think each addresses the actual defect:

1. Order the console password agent after input availability. Have udev-tagged keyboard arrival gate systemd-ask-password-console.service, with a bounded timeout so a machine with no keyboard at all still reaches the prompt. This is the minimal change and matches how systemd already handles device readiness elsewhere.

2. Make the agent wait internally before painting. Before the first paint, if no ID_INPUT_KEYBOARD device is present, wait (bounded, a few seconds) for one. Keeps the logic in the one place that knows a prompt is pending.

3. Re-paint when a keyboard arrives. Strictly better than either of the above if it's cheap: paint immediately, and when an input device appears, clear and re-paint. The user then sees a prompt that visibly resets at the moment typing becomes possible, which removes the silent-loss property even if the window can't be eliminated.

A bounded timeout matters in all three: headless and serial-console systems must not regress. Ordering against ID_INPUT_KEYBOARD only when such a device could plausibly appear (e.g. skip when the ask is being answered over a non-tty agent) would keep the change inert for those cases.

Happy to carry any of these as a PR if a maintainer indicates which shape is acceptable.

Workaround, for anyone who finds this before it's fixed

A oneshot unit that polls sysfs for the keyboard, ordered before the agent via a drop-in:

# systemd-ask-password-console.service.d/overrides.conf
[Unit]
After=wait-for-keyboard.service
Wants=wait-for-keyboard.service

Measured effect on this machine — the gap flips sign, so the prompt now always follows the keyboard:

Total boot time is unchanged, because the boot was already blocked waiting for a human to type.

One trap for anyone implementing this generically: "wait for any keyboard" is not sufficient. A plugged-in YubiKey registers as an HID keyboard with a full letter-key bitmap and enumerated ~700 ms before the real keyboard on this machine, so a naive check is satisfied early and fixes nothing. The trackpad on this machine also shares the keyboard's USB vendor:product. A correct generic implementation needs more than "an input device with keys exists".