Repository navigation
Improve devcontainer build speed #8
Description
Activity
The below is extracted from the logs, the main
dnfstep takes 2,038 seconds (34 minutes).Notably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second
builddepcommand, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".cc @hroncok for any advice (as author of python/cpython#103283)
dnf install ...:[linux/arm64 2/4] RUN dnf -y --nodocs --setopt=install_weak_deps=False install /usr/bin/{blurb,clang,curl,git,ln,tar,xz} 'dnf-command(builddep)' && dnf -y --nodocs --setopt=install_weak_deps=False builddep python3 && dnf -y clean all 729.3 Fedora 40 openh264 (From Cisco) - aarch64 476 B/s | 2.1 kB 00:04 785.2 Fedora 40 - aarch64 - Updates 1.1 MB/s | 38 MB 00:33 1156.0 Last metadata expiration check: 0:00:23 ago on Wed Sep 25 18:39:54 2024. 1241.3 Package [...] is already installed. [x4] 1250.7 Dependencies resolved. 1251.6 ================================================================================ 1251.6 Package Arch Version Repo Size 1251.6 ================================================================================ 1251.6 Installing: ... 1251.6 Transaction Summary 1251.6 ================================================================================ 1251.6 Install 47 Packages 1251.6 1251.7 Total download size: 140 M 1251.7 Installed size: 562 M 1251.7 Downloading Packages: ... 1267.9 -------------------------------------------------------------------------------- 1267.9 Total 8.7 MB/s | 140 MB 00:16 1293.7 Running transaction check 1295.7 Transaction check succeeded. 1295.7 Running transaction test 1305.3 Transaction test succeeded. 1305.3 Running transaction 1313.6 Preparing : 1/1 1315.1 Installing : systemd-libs-255.12-1.fc40.aarch64 1/47 ... 1344.5 Running scriptlet: dnf-plugins-core-4.9.0-1.fc40.noarch 47/47 ... 1348.7 Complete!dnf builddep python3:1358.6 enabling fedora-source repository 1358.6 enabling fedora-cisco-openh264-source repository 1358.6 enabling updates-source repository 1370.6 Fedora 40 - Source 1.2 MB/s | 6.6 MB 00:05 1433.9 Fedora 40 openh264 (From Cisco) - aarch64 - Sou 30 B/s | 134 B 00:04 1452.2 Fedora 40 - Updates Source 359 kB/s | 1.7 MB 00:04 1544.5 Last metadata expiration check: 0:00:08 ago on Wed Sep 25 18:51:10 2024. 1708.0 Package [...] is already installed. [x12] 1722.1 Dependencies resolved. 1724.3 ================================================================================ 1724.3 Package Arch Version Repo Size 1724.3 ================================================================================ 1724.3 Installing: ... 1724.3 Transaction Summary 1724.3 ================================================================================ 1724.3 Install 233 Packages 1724.3 Upgrade 11 Packages 1724.3 1724.7 Total download size: 123 M 1724.8 Downloading Packages: ... 1746.9 -------------------------------------------------------------------------------- 1746.9 Total 5.6 MB/s | 123 MB 00:21 1861.1 Running transaction check 1867.8 Transaction check succeeded. 1867.8 Running transaction test 1893.0 Transaction test succeeded. 1893.0 Running transaction 1914.6 Preparing : 1/1 1917.6 Upgrading : zlib-ng-compat-2.1.7-2.fc40.aarch64 1/255 ... 1989.6 Running scriptlet: expat-2.6.0-1.fc40.aarch64 255/255 ... 2025.9 Complete!dnf clean:2036.9 37 files removed DONE 2037.9sNotably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second
builddepcommand, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".That's pretty bad, but I have no idea why would it take so long. Perhaps it's the emulation/virtualization that is there that is so slow?
I was having a quick nosey here 👋🏽 but looking at the Actions runs, the build is taking
< 3minsand the 35 mins + was an outlier.Reacted by Miro Hrončok- added 2 commits that reference this issue
on Oct 7, 2024 - Reacted by Tania Allard and Erlend E. Aasland
Ah that makes sense. Will have a look over there and maybe do some digging on why this is taking so long then.
Last release by Brett took 28 mins.
(36mins -> 28mins!!)
https://github.com/python/cpython-devcontainers/actions/runs/11245912749Reacted by Tania AllardWe have a container job at github actions that runs dnf on 4 architectures. The arm64 one (third in the screenshot) is regularly extremely slow:
What do you think about splitting actual container creation into two steps? One with big
dnfsteps, which rarely changes. And another container with all other steps. Unfortunately, we should use some sort of links to actual containers. But regular updates can be much faster with this scheme.Reacted by Erlend E. AaslandI'm personally fine with that if @corona10 and @erlend-aasland are. I have #12 open to split things up in regards to WASI, but I wasn't sure about following through with it due to having to change the CPython repo when new containers go out. We might need to settle on some policy in #11 to know how much work would be expected.
I'm personally fine with that if
@corona10and@erlend-aaslandare.+1
+1
Based on the last release run for the dev container, this is now down to under 11 minutes, which seems reasonable to me, so I'm going to close this issue.
@brettcannon
The 35-minute release period is pretty painful. We may need to find out whether we can reduce the build time.
https://github.com/python/cpython-devcontainers/actions/runs/11038928730