Repository navigation
Avoiding MmProbeAndLockPages overhead in ReadFile #622
Description
Describe the bug
Technically a performance bug, in the combination of npcap with libpcap, when trying to tune npcap for 10GBit+ operation.
This assumes 8MB userspace and kernel side buffers for npcap, in order to get the rate of syscalls to a manageable level in the first place.
For context have a look at the opposing side:
https://github.com/the-tcpdump-group/libpcap/blob/fbcc461fbc2bd3b98de401cc04e6a4a10614e99f/pcap-npf.c#L542
https://github.com/the-tcpdump-group/libpcap/blob/fbcc461fbc2bd3b98de401cc04e6a4a10614e99f/pcap-npf.c#L432
When using PacketReceivePacket with above buffer sizes, a significant share of time (70%+) is spent in MmProbeAndLockPages on the level of NtReadFile prior to NPF_read respectively MmUnlockPages at IoCompleteRequest.
As a user of libpcap, the memory for the user space buffer is allocated privately within libpcap, and is never exposed in raw form to the user of the API. Only PacketInitPacket or PacketReceivePacket on the npcap end has the guarantee to see the raw buffer.
Given the design of these 2 APIs, as a user I can't do anything to speed up MmProbeAndLockPages from the outside. I can't choose large pages, I can't VirtualLock.
What's worse, is that the overhead scales linearly with the size of the user space buffer - not with the actual amount of data transferred.
Expected behavior
Specifying a large user space buffer on libpcap side doesn't result in excessive overhead on npcap side.
Either npcap optimizes the buffer for repeated use (e.g. by explicitly applying VirtualLock in PacketInitPacket), or libpcap turns smarter when allocating the memory.
Activity
Ext3h commentedon Aug 8, 2022
Issue appears to have been exacerbated by not activating the blocking mode - and thereby calling
PacketReceivePacketwith a large buffer excessively often. The overhead is significantly lower in blocking mode when the majority of calls does actual work.dmiller-nmap commentedon Aug 19, 2022
This is an interesting idea, and I'm reopening it for further investigation. I would be interested in what your bottlenecks turn out to be once you've optimized transfers using the existing tools: pcap_setmintocopy() and pcap_getevent() or pcap_set_timeout(), as well as managing buffer sizes with pcap_setuserbuffer() and pcap_set_buffer_size().
I do think we could avoid some of this overhead, but VirtualLock may not be the right tool. Here are some links I found so far on the topic of locking a buffer to be used for DMA between user-mode and kernel-mode:
dmiller-nmap commentedon Feb 17, 2025
We are giving more thought to this issue and hope to have a plan for some improvement soon. Here are some current thoughts:
pcap_set_timeout()and avoidingpcap_set_nonblock()) is used by Packet.dll to throttle calls toReadFile()to allow each one to retrieve more packets at a time. TheMinToCopyvalue (pcap_setmintocopy()) sets a desired minimum amount of packet data for eachReadFile()call. The Npcap driver signals Packet.dll when this amount is available, interrupting the timeout wait. The result is that a call topcap_dispatch()will return when the timeout expires or the minimum data is available, whichever happens first. See Npcap buffer sizes are not well-documented #30 for discussion about improving Npcap's documentation on buffer sizes and tuning.VirtualLock()is an appropriate solution for this issue, libpcap currently does not usePacketAllocatePacket()/PacketFreePacket(), so that is not a suitable place to insert the call.PacketInitPacket()does not have a corresponding de-initialization function in which to callVirtualUnlock(), so that is also not suitable; also it is used for packet injection as well, which may use many different buffers. It seems more appropriate, then, to add it to libpcap.ReadFile()and instead using a shared buffer the way that memory-mapped capture on Linux does. This would be a big departure from the current Packet API, so it makes sense to discuss that at Potential enhancement: alternative to Packet.dll #569 instead.Experimental feature: adaptive buffer sizing. See #622
dmiller-nmap commentedon Nov 13, 2025
VirtualLock is not an appropriate solution, and the existing overhead-reducing mechanisms are sufficient for now. As we do further documentation for buffer sizing (#30), it is important to note that readfile calls incur overhead proportional to the user buffer size even if only a small amount of data is transferred. Closing this issue in favor of better documentation and existing performance suggestions.