Add history JSONL file format proposal - #12982

typo: delimiters

if we play some tricks to avoid depending on the number of history entries, we could probably avoid parsing offsets eagerly too. But that's unrelated.

Agreed, that's a very valid possible future optimization

I guess history append in concurrent sessions could trigger a collision.
Dropping some parts of those history items (at vacuum time) seems okay.
If we use the session ID in addition to the item ID, then collisions are even less likely.

This is a good point. I think you've hit on the right idea and we should take it a step farther by folding the session ID into the item ID. This pushes it up to 22 chars but makes collisions between sessions impossible unless the sessions themselves collide:

A fish history item is identified by its "id:" a 128 bit unsigned integer. This contains the item's 48 bit timestamp, a 16 bit nonce, and the 64 bit session id.
The session ID is itself composed of a 48 bit session timestamp and a 16 bit session nonce.

  Bits 127            80 79       64 63                  16 15         0
  +---------------------+-----------+----------------------+------------+
  | timestamp (48)      | nonce (16)| session timestamp (48)| nonce (16)|
  +---------------------+-----------+----------------------+------------+

The net effect is tiny since we remove the session ID as a separate field. It slightly complicates decoding but not in a significant way.

I wonder why we'd fold timestamp, session ID and nonce into a single opaque JSON field?
We could also make it

{"started": <timestamp>, "session": <timestamp>, "nonce": <nonce>, ...}

and use all three as composite primary key.

This gives more information to readers and enables some jq workflows though I don't have a concrete one.

It's hard to estimate how deduplication affects item counts in practice.
I feel like there are some commands I re-run very often.
It wouldn't be great if this change would in practice cause history from a few years ago to roll off, for people who run hundreds(?) of commands a day.
Possibly we should deduplicate only items older than a year or so. Or maybe compress them and/or move them to different files.

Looking at what others do, I'm surprised to read that Chrome rolls off history after 90 days.

zsh has HIST_EXPIRE_DUPS_FIRST, to roll off duplicate entries before any unique entry. I think that sounds promising, unless we need an accurate history for something.
IIRC, the motivation for non-deduplicated history is purely simplicity of implementation. For example, I imagine that up-arrow history search will still present a deduplicated view to the user. So that should be fine

we may want two limits (high/low watermarks) so we don't vacuum at every command once we hit the limit

Ok so if people have multiple history files, they can do for h in foo bar; fish_history=$h fish; end to migrate them all. This only works when stdin is a TTY, or possibly also for fish_history=$h fish -c 'history search ...'

1 million sounds okay. Given the example

{"id":"AZ9TBVz27qQ","cmd":"cat file.txt","paths":["file.txt"],"cwd":"~","exit":0,"dur":4
,"sid":"AAA44pHL-_o"}

is 109 bytes, that comes out to 109 MB for a million entries. Tho the example command is a bit short. My current history has ~41k entries (in a fish_history file of 3.5M), with an average length of ~46 codepoints for the command (which I'm gonna say is like 99% ascii so bytes), and the cwd is minimal here.

So this might end up as ~160MB in practice, which shouldn't be an issue for a fully-loaded complete history of everything. Might not fit on floppy disks, but I hear you can get those new-fangled zip drives with that much memory.

Of course my current history as-is would be 4.1MB, but had I run those same commands with the new format they'd all be stored, so I wouldn't be at 41k, I'd be at.... gluteal extraction 100k (for 10M).

As a point of comparison, here's what Atuin stores: https://github.com/atuinsh/atuin/blob/372b092bde6e36cab4fab0bc8696a18e1e4a5d57/crates/atuin/src/command/client/history.rs#L667

It's pretty much the same fields.

They have an explicit timestamp meanwhile in Fish, it would have to be extracted from the ULID-like id prefix.
The host / hostname may be useful when an history is shared between multiple machines.
The user / author may be useful to share history between team members.
Not sure about the purpose of intent.

struct TailJsonHistory<'a> {
    id: HistoryId,
    timestamp: String,
    timestamp_unix_ns: u64,
    command: &'a str,
    cwd: &'a str,
    session: &'a str,
    hostname: &'a str,
    host: &'a str,
    user: &'a str,
    author: &'a str,
    #[serde(skip_serializing_if = "Option::is_none")]
    intent: Option<&'a str>,
    #[serde(skip_serializing_if = "Option::is_none")]
    exit: Option<i64>,
    #[serde(skip_serializing_if = "Option::is_none")]
    duration_ns: Option<i64>,
    #[serde(skip_serializing_if = "Option::is_none")]
    duration: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    success: Option<bool>,
    #[serde(skip_serializing_if = "Option::is_none")]
    finished_at: Option<String>,
}