Repository navigation
Conversation
e611efb to
ca0214d
Compare
| A new pump-side primitive, `evict_local_segments_up_to(up_to)`, is staged by the cleaner through | ||
| `LifecycleFrame::EvictLocalSegments` and applied on the owner shard under `write_lock`, exactly like | ||
| `CleanPartition`. It walks sealed local segments from the front and stops at the first of: the | ||
| active segment, `end_offset > tiered_up_to_offset`, `end_offset >= persistence checkpoint`, or the | ||
| local retention budget being satisfied. It does not consult the consumer barrier. Per victim it | ||
| writes the marker, fsyncs the directory, unlinks the `.log`, flips `location` to `Remote`, drops the | ||
| cached fd via `reset_read_state`, and leaves `PartitionStats.size_bytes` untouched. A per-pass budget | ||
| of 16 mirrors `SEGMENT_REMOVAL_BUDGET_PER_PASS`. |
There was a problem hiding this comment.
I like the overall design. A first-read question on local eviction: it is staged by the cleaner, so a replica waits up to one cleaner interval (1 m by default) after applying AdvanceTieredFloor. Could applying the floor stage EvictLocalSegments directly, at least for partitions under local storage pressure?
| On success the primary commits `AdvanceTieredFloor { up_to_offset }` to the partition's control | ||
| log. Only the primary submits it, and a stale primary's command is rejected by the view of whichever | ||
| group realizes that log, so no further fencing is needed. Every replica applies it through the | ||
| partition state machine below. |
There was a problem hiding this comment.
Why is upload prompt rather than at the point a segment leaves the local retention envelope? I assume it's the backup property and keeping eviction free of network I/O during a store outage. Could Rationale and alternatives say so?
|
|
||
| Open design decisions: | ||
|
|
||
| - **Piece size and caps.** `upload_piece_size`, `upload_concurrency` and `fetch_concurrency` bound |
There was a problem hiding this comment.
Phase 3 builds the pump tick latency gauge and tests a single 1 GiB upload , but an upload backlog only exists once the upload loop lands in Phase 4 . Could the Phase 4 PR include the measurement from, with eviction passes and primary object deletion on the same shard, and could the Phase 3 defaults stay provisional until then?
|
we're dropping purge, removal pr goes up today. so |
05d74d2 to
82a6d1d
Compare
82a6d1d to
2aac350
Compare
|
@hubcio @numinnex thanks, read through this against master. Agree with the direction: keep the replicated local log as the write path, copy sealed segments out after, one replica uploads and tells the others an offset, stubs keep offset lookups off the network. A couple of questions/observations that I had
|
This branch has not been deployed
4 participants
No description provided.