Repository navigation
extremely slow scan for "copyright" files #278
Description
First of all, thank you for DISABLE_COPYRIGHT_FILES_DEPLOYMENT, it really saves the day!
Right now each library / executable is scanned running a separate dpkg -S. Then linuxdeploy checks for /usr/share/doc/PKG/copyright. This is extremely slow. dpkg -S actually supports multiple query paths in a single call, in case you want to batch the calls.
Another solution would be to add an --output-deployed-files FNAME to output the list of files for which the copyright files need to be found. After all, linuxdeploy has valuable logic to include / exclude deployed files from lookup.
I currently extract the AppImage created by linuxdeploy, list all .so libraries, look them up all at once in a single dpkg -S operation, and copy the results as appropriate. But mine is a simple case.
I've noticed that there already is a framework for deferred operations, so maybe that can be co-opted.
Activity
[-]extremely slow scan for "copyright" file[/-][+]extremely slow scan for "copyright" files[/+]TheAssassin commentedon Apr 10, 2024
This can easily happen on Debian when the dpkg cache is on a slow drive and is very large. For instance, in (temporary) CI environments, Docker containers etc., the scan is really fast. Ever since I replaced my SSHD (HDD + SSD cache) with a real SSD, I haven't had any issues with this any more; before, linuxdeploy took 2-3 seconds per item.
mralusw commentedon Apr 11, 2024
I've had the cache in
/zramfor... ages. I've tried with/tmpand it makes no difference.Doesn't matter if the patterns are found or not found. Debian 12, Ubuntu 22.04, Ubuntu 20.04 — some slower than others.
I have no idea what it's doing, since
strace dpkg -S '*/zzz' '*/xxx' '*/yyy' '*/uuu'shows delays just before it prints each message, not in-between syscalls. Probably some very inefficient grepping.Anyway, the first improvement was to use xargs to run multiple
dpkg-query's in parallel, and batch 10 of queries for each:But the real improvement (blazing fast) came by ditching
dpkgand searching directly in/var/lib/dpkg/info/*.list:(
grep -f -reads regex's from stdin).I might turn this into a plugin. Problem is, I don't really know what LD deploys. I'm just searching for identical library names, and the setup is specific to my app. I'd still like to obtain a map of which files where automagically deployed to which location. Maybe I (shudder) could parse the
ldlog.TheAssassin commentedon May 15, 2024
Nobody's ever suggested machine readable output from this application. I'm open to such features, but please open another issue.
I'm not sure I want to implement my own parser for those files.
linuxdeploy runs strictly sequentially. It primarily tries to speed things up by trying to avoid running the same operation twice on a file. Adding any kind of parallel operations would be a lot of work given the lack of proper C++ libraries for such a purpose (Python would make it quite easy, though).
Well, the code is free/open-source, so you can have a look for yourself. While it may be C++, you should be able to get a basic idea of what it does from there. You can grep for
dpkgto find the exact commands linuxdeploy uses. Maybe there's a better way?mralusw commentedon May 16, 2024
Cool, I think that makes sense given linuxdeploy's strenghts.
Yeah, not a good idea and I'm not proposing that. I'm thinking more along the lines of making more operations pluggable by external scripts.
Yeah, me neither, and it's ultimately a
dpkgissue (unlikely to be solved). With pluggable scripts though it can be addressed.In the case at hand, FWIW, I think
dpkgmight be searching for files created by package install scripts (i.e. not in the list file), and probably also bydpkg-divert. These make sense for admin usage ofdpkgon a given machine, but not for package specification.I did, obviously :) (and I've had a long stint with C++, though years pass and C++ mutates). "I don't know" meant there is no machine-readable output, which goes back to point 0 — thanks for clarifying that there is no current design