Visitar URL original
vmm: memory_manager: Skip snapshot file holes in on-demand restore by umangPokhriyall · Pull Request #9021 · cloud-hypervisor/cloud-hypervisor · GitHub
Skip to content

vmm: memory_manager: Skip snapshot file holes in on-demand restore - #9021

Merged
rbradford merged 2 commits into
cloud-hypervisor:mainfrom
umangPokhriyall:ondemand-prefault-skip-holes
Oct 9, 2026
Merged

vmm: memory_manager: Skip snapshot file holes in on-demand restore#9021
rbradford merged 2 commits into
cloud-hypervisor:mainfrom
umangPokhriyall:ondemand-prefault-skip-holes

Conversation

@umangPokhriyall

@umangPokhriyall umangPokhriyall commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

With memory_restore_mode=ondemand, the uffd handler prefaults every page of the snapshot, holes of a sparse snapshot file included, so the guest RAM ends up fully populated. Copy restore already skips the holes.

  • vmm: memory_manager: Pass the served bitmap to the uffd handler
  • vmm: memory_manager: Skip snapshot file holes in on-demand restore

1 GiB shared=on Ubuntu 22.04 guest, 209 MiB of data in the snapshot:

  • before: 1024 MiB allocated after the prefault, which takes 521 ms
  • after: 209 MiB allocated, the prefault takes 108 ms

Pages that hold data are still prefaulted, so a snapshot taken after the restore matches the original (the #8525 case). A snapshot without holes restores as before.

Fixes: #9019

Assisted-by: Claude:Opus-5

@umangPokhriyall
umangPokhriyall requested a review from a team as a code owner October 6, 2026 16:09
@lucido-simon

lucido-simon commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

I don't like that we're adding a method to the memory source trait for a behavior that's specific to prefaulting + file memory source, but I don't see any other, cleaner way. Let see what others think.

@umangPokhriyall

umangPokhriyall commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

I don't like that we're adding a method to the memory source trait for a behavior that's specific to prefaulting + file memory source, but I don't see any other, cleaner way. Let see what others think.

I had the same hesitation before sending and couldn't find a cleaner way either, so I left it as is. I'll go with what they decide.

@umangPokhriyall
umangPokhriyall force-pushed the ondemand-prefault-skip-holes branch from a936035 to 6f005e2 Compare October 7, 2026 16:29
@umangPokhriyall umangPokhriyall mentioned this pull request Oct 7, 2026
Closed
@umangPokhriyall umangPokhriyall changed the title vmm: Skip snapshot holes in the on-demand restore prefault vmm: memory_manager: Skip snapshot file holes in on-demand restore Oct 7, 2026
@umangPokhriyall

umangPokhriyall commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

I looked at how QEMU and CRIU handle this in their userfaultfd restores. Both register all of guest memory and only prefault the pages that hold data.

I first tried registering only the data, but a fragmented snapshot then needs more mappings than the kernel allows by default. So I went with the same approach as QEMU and CRIU. The holes are now marked in the handler's served bitmap before it starts, and the trait is unchanged.

lucido-simon
lucido-simon suggested changes Oct 8, 2026
View reviewed changes

@lucido-simon lucido-simon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a much cleaner approach, thanks! I left one minor comment.

FYI, I let claude gather some data: VMs seems to have few, big holes, even after guest activity (~7 holes avg on a 32GB ubuntu guest). On my btrfs mount, on a fast system, the latency this introduces before guest startup is < 1ms even on the worst case (restoring from a guest that had a lot of activity before snapshotting).

Even if startup latency is king in for on demand restores, this seems to be fine.

Comment thread vmm/src/memory_manager.rs Outdated
Comment on lines +1075 to +1078
// Holes in the snapshot file need no prefault: mark the pages they
// fully cover as served, so the prefault skips them. A fault on one is
// served like a discarded page, and the guest mapping's natural
// zero-fill takes over once the handler is done.

@lucido-simon lucido-simon Oct 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very LLMish and hard to parse. Can you reword this? ie.

For sparse files, we mark the holes as served so we don't prefault them. If the guest do actually access it, we map it normally (as we do for pages that were discarded). Once prefault andUFFD unregistered, the kernel takes over and installs zero pages on access, which is the correct behavior

@umangPokhriyall umangPokhriyall Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks :) updated with your wording.

@umangPokhriyall
umangPokhriyall force-pushed the ondemand-prefault-skip-holes branch from 6f005e2 to 88e7acf Compare October 8, 2026 10:15
@lucido-simon

lucido-simon commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

LGTM! cc @sboeuf

@umangPokhriyall

umangPokhriyall commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

That's a much cleaner approach, thanks! I left one minor comment.

FYI, I let claude gather some data: VMs seems to have few, big holes, even after guest activity (~7 holes avg on a 32GB ubuntu guest). On my btrfs mount, on a fast system, the latency this introduces before guest startup is < 1ms even on the worst case (restoring from a guest that had a lot of activity before snapshotting).

Even if startup latency is king in for on demand restores, this seems to be fine.

Thanks for testing it on a bigger guest. Good to see the holes stay few and big there too.
Would it be worth doing the same for postcopy and the offload daemon? If it fits CH, I can look into it.

sboeuf
sboeuf requested changes Oct 8, 2026
View reviewed changes
Comment thread vmm/src/memory_manager.rs Outdated
Comment thread vmm/src/memory_manager.rs Outdated
Comment thread vmm/src/memory_manager.rs
@umangPokhriyall
umangPokhriyall force-pushed the ondemand-prefault-skip-holes branch from 88e7acf to b58d31f Compare October 9, 2026 07:21
@umangPokhriyall
vmm: memory_manager: Pass the served bitmap to the uffd handler
fb3d078 
Create the bitmap of served pages before spawning the uffd handler, so
the on-demand restore can mark pages the prefault does not need to
serve.

No functional change intended.

Signed-off-by: Umang Pokhriyal <umangpokhriyall@gmail.com>
@umangPokhriyall
vmm: memory_manager: Skip snapshot file holes in on-demand restore
462dab5 
On-demand restore prefaults every page of the saved regions, holes of a
sparse snapshot file included, so the guest RAM ends up fully populated.

Mark the pages fully covered by a hole of the snapshot file as served
before the uffd handler starts. The prefault skips them, and once the
handler is done the guest mapping's natural zero-fill provides them.
That matches the source content, as on the copy restore path.

Signed-off-by: Umang Pokhriyal <umangpokhriyall@gmail.com>
@umangPokhriyall
umangPokhriyall force-pushed the ondemand-prefault-skip-holes branch from b58d31f to 462dab5 Compare October 9, 2026 07:34
sboeuf
sboeuf approved these changes Oct 9, 2026
View reviewed changes

@sboeuf sboeuf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rbradford
rbradford requested a review from lucido-simon October 9, 2026 15:22
@rbradford
rbradford added this pull request to the merge queue Oct 9, 2026
Merged via the queue into cloud-hypervisor:main with commit c91a654 Oct 9, 2026
44 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sboeuf sboeuf sboeuf approved these changes
@rbradford rbradford rbradford approved these changes
+1 more reviewer
@lucido-simon lucido-simon lucido-simon approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

On-demand restore: skip holes in the snapshot memory file

4 participants

@umangPokhriyall @lucido-simon @rbradford @sboeuf