Repository navigation
Plan caches hold a copy of the package descriptions for every component #12463
Description
since #11703, every unit's ElaboratedConfiguredPackage carries the package's whole GenericPackageDescription, beside the PackageDescription and configured programs it already had. in memory these are shared: each component is a record update of its package's one elaborated value. but Binary writes a full copy per unit, and improved-plan holds two plans, so the plan caches grow with the square of a package's component count, and every decode makes the copies separate again.
repro: one package with N public sublibraries, one module each, a 16-flag common stanza. cabal build all once, then a no-op cabal build all under +RTS -s; master at 538d465, ghc 9.14.1, macOS arm64, medians of three runs:
| sublibs | improved-plan |
allocated | max live | no-op |
|---|---|---|---|---|
| 10 | 0.79 MB | 0.20 GB | 8.7 MB | 0.07 s |
| 40 | 7.5 MB | 1.33 GB | 56 MB | 0.34 s |
| 80 | 27.3 MB | 4.21 GB | 190 MB | 1.06 s |
on our real project (538 components in 44 packages, the largest has 60) improved-plan is 56 MB under 3.16.1.0 and 130 MB under 3.18.1.0, and a no-op cabal build of one library takes 3.1 s, allocates 10.9 GB and peaks at 590 MB live.
the descriptions can't just be dropped from the plan: the in-library configure reads the whole GPD (computePackageInfoFromIndex for the internal libraries, checkPackageProblems in finalCheckPackage). the duplication only exists in how the caches are written, so that's where i'd fix it. first mentioned on #11703; PR to follow.
Activity
Write each package's descriptions once in the plan caches
Write each package's descriptions once in the plan caches
Drop elabGPkgDescription from the elaborated plan
andreabedini commentedon Oct 26, 2026
Thank you for the report @PaleRoses, I am giving this a look.
andreabedini commentedon Oct 26, 2026
I can reproduce this. I compared cabal-install 3.16.1.0 with 3.18.1.0 (GHC 9.12.2, Linux) on a package with N public sublibraries, where the common stanza has 16
if flag(..)blocks settingghc-options. The GPD keeps every one of those branches, so it grows a lot faster than the PD. At N=80improved-plangoes from 10 MB to 128 MB.There are a few things the issue doesn't mention:
The per-component
cache/configfile is also affected. Its key is the whole elaborated package (PackageFileMonitor.hs), so every component decodes a GPD and compares it with==on every build. At N=40 each file goes from 34 KB to 432 KB. With the GPD moved into a per-package map inElaboratedSharedConfig, as suggested on Drop elabGPkgDescription from the elaborated plan #12464, this goes away too.The extra PATH copies don't come from Directly call in-library functions to build packages #11703. They come from Configure the compiler ProgramDb once per project #11768 (aa71936).
configureAllKnownProgramsmakeselabConfiguredProgramshold about 11 programs instead of 2. Each one stores the whole PATH inprogramOverrideEnvand every location it tried inprogramMonitorFiles. As far as I can see nothing reads that field; it only takes part in serialisation andEq. This grows linearly with the number of units, but also with the length of PATH: at N=10 it's 157 KB → 863 KB of PATH strings alone. Stripping the field the wayprogramDbSignaturedoes should be enough, but that is a separate fix.The value stored in
improved-planincludes the elaborated plan as well as the improved one. That is where the factor of 2 comes from, and it was already there in 3.16.michaelpj commentedon Nov 7, 2026
I believe #10817 also contributes to copies of
PATH. If we want to detect whenPATHhas changed perhaps we could store a hash instead of the whole thing?michaelpj commentedon Nov 7, 2026
Somewhat hilariously, I believe the plan caches being so big means that on my project it is actually faster for cabal to re-plan than to read the cache!
PaleRoses commentedon Nov 9, 2026
@michaelpj yep! Since #11768 every component keeps 11 copies of
PATH, and #12500 swaps them for a hash.You're right about the flip! With a long nix
PATHmain's HLS plan hits 152 MB and redoing the plan after aPATHchange wins. With both PRs it stays at 23 MB and the flip is gone. PLEASE do tell whether this fixes it on your project, or give both PRs a try!!@andreabedini with a full store the store-only descriptions are about 1 MB of the 23 MB plan, and it's still faster than 3.16, so I'd leave the map alone.
Also, half the plan file is a second copy of the elaborated plan. I have a small fix that nearly halves it, happy to open it as a follow-up!
michaelpj commentedon Nov 9, 2026
Yep, I've tested your changes on my project and they help a lot, great work!
Drop elabGPkgDescription from the elaborated plan (haskell#12464)
Drop elabGPkgDescription from the elaborated plan (haskell#12464)
Drop elabGPkgDescription from the elaborated plan (haskell#12464)