Process channels in banks in preprocessors to reduce RAM usage #4837

Activity

1. alejoe91

   added this to the 0.106.0 milestone on Oct 7, 2026
2. alejoe91

   assigned

   h-mayorquin

   on Oct 7, 2026
3. alejoe91

   added

   preprocessingRelated to preprocessing moduleRelated to preprocessing module

   on Oct 7, 2026
4. h-mayorquin

   changed the title [-]Implement filtering by banks of channels to reduce RAM[/-] [+]Process channels in banks in preprocessors to reduce RAM usage[/+] on Oct 8, 2026
5. h-mayorquin commented on Oct 8, 2026

   

   h-mayorquin

   on Oct 8, 2026

   Last edited by h-mayorquin

   Contributor

   More actions

   @alejoe91, @samuelgarcia, @JoeZiminski. I got excited after #4840 (Filter one channel at a time to reduce peak memory) and I did an overview of which preprocessors could be improved in a similar way. Here are the results:

   preprocessor	peak memory, full chunk	peak memory, one channel at a time	time, full chunk vs one channel at a time
   gaussian_filter	2904% of the trace	108%	12.5 s vs 12.5 s
   resample	500%	68%	2.3 s vs 2.5 s
   phase_shift	2223%	106%	4.0 s vs 7.1 s
   highpass_spatial_filter	2075%	517% (see below)	8.1 s vs 7.3 s

   384 channels, 30 kHz, 10 s of int16, calling the real get_traces. The peak is from tracemalloc as a percentage of the int16 trace and the time is the best of 3. The output is identical in every row. The highpass_spatial_filter row is a different split: the gain control step runs one channel at a time, but the filter itself runs across channels and is independent for each time sample, so I ran it on blocks of time samples (and the floor is about 500% because the gains have to be kept).

   decimate with antialiasing goes through the same helper as resample (get_polyphase_resampled_traces), so it would get the same fix (I only read that one). phase_shift is the exception on time: a strict loop over channels throws away the rotation reuse from #4802 (Speed up FFT phase shifts for Neuropixels recordings). A prototype with one block per distinct delay keeps it and still cuts the peak by about 80% at 1 s chunks (384 to 79 MiB) with the same speed, but it is 20 to 30% slower at 5 to 10 s chunks.

   There are other two groups: some are elementwise (scale, astype, rectify, clip and similar) and their only temporary is the output, so there is nothing to gain with this approach. Others use a combination of all the channels like whiten, common_reference and interpolate_bad_channels, so a per-channel loop does not apply to them.

   I think we should tackle those improvements one by one, with #4840 (Filter one channel at a time to reduce peak memory) first, and once we have them all we can think about a public interface.

   @alejoe91 I edited the title so this works as a tracker.

   ❤️React with ❤️2Reacted by Alessio Buccino and Joe Ziminski
6. h-mayorquin

   added

   performancePerformance issues/improvementsPerformance issues/improvements

   on Oct 8, 2026
7. alejoe91 commented on Oct 8, 2026

   

   alejoe91

   on Oct 8, 2026

   MemberAuthor

   More actions

   Thanks @h-mayorquin

   Maybe for phase_shift we could use banks with the same rotation, which is a natural follow up after #4802 ?
8. h-mayorquin commented on Nov 4, 2026

   

   h-mayorquin

   on Nov 4, 2026

   Contributor

   More actions

   Here #4904