Repository navigation
ue4-docker build 4.22.2 fails with 'out of space' with new-enough Docker EE #44
Description
Output of the ue4-docker info command: See below.
I reproduced the same failure as #37 (and the troubleshooting guide) on Windows Server 2019 with Docker 18.09.5, and upgrading to Docker 18.09.6 did not fix it.
So I don't think this is #37, unless somehow the 8GB file copy fix (moby/moby#37771) was completely excluded from the Docker EE 18.09 series, despite the release notes.
Before:
(venv) PS C:\Users\localadmin> python -m ue4docker info
ue4-docker version: 0.0.34 (latest available version is 0.0.35)
Operating system: Windows Server 2019 Standard Version 1809 (OS Build 17763.557)
Docker daemon version: 18.09.5
NVIDIA Docker supported: No
Maximum image size: 200GB
Available disk space: 677.09 GiB
Total system memory: 63.89 GiB physical, 73.39 GiB virtual
Number of processors: 20 physical, 40 logical
(venv) PS C:\Users\localadmin> docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 49
Server Version: 18.09.5
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Standard Version 1809 (OS Build 17763.557)
Architecture: x86_64
CPUs: 40
Total Memory: 63.89GiB
Name: au1-bldwin-22
ID: IN3O:ZULX:EYEV:VGNW:EE33:WOFD:FUF3:NRTO:I2NK:6WEP:CPAL:HSPL
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Upgrade per Microsoft's documentation
(venv) PS C:\Users\localadmin> Install-Package -Name Docker -ProviderName DockerMsftProvider -Update -Force
Name Version Source Summary
---- ------- ------ -------
Docker 18.09.6 DockerDefault Contains Docker EE for use with Windows Server.
(venv) PS C:\Users\localadmin> Start-Service Docker
After:
(venv) PS C:\Users\localadmin> python -m ue4docker info
ue4-docker version: 0.0.34 (latest available version is 0.0.35)
Operating system: Windows Server 2019 Standard Version 1809 (OS Build 17763.557)
Docker daemon version: 18.09.6
NVIDIA Docker supported: No
Maximum image size: 200GB
Available disk space: 676.86 GiB
Total system memory: 63.89 GiB physical, 73.39 GiB virtual
Number of processors: 20 physical, 40 logical
(venv) PS C:\Users\localadmin> docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 49
Server Version: 18.09.6
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Standard Version 1809 (OS Build 17763.557)
OSType: windows
Architecture: x86_64
CPUs: 40
Total Memory: 63.89GiB
Name: au1-bldwin-22
ID: IN3O:ZULX:EYEV:VGNW:EE33:WOFD:FUF3:NRTO:I2NK:6WEP:CPAL:HSPL
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Reacted by Samuel Lorétan
Activity
TBBle commentedon Jun 19, 2019
On examination, 200gB is probably simply not enough:
d6d0454b25e0is the final state of the 'builder' image:So it's possible that
COPY --fromrequires the "image size limit" to be large enough to hold both input and output.The command I'm using is
and it's happily resuming at the problematic
COPYstep.I've tried changing the daemon.json to have a 400gB limit, and executing
Stop-Service Docker+Start-Service Docker, but it still failed. I'm going to reboot the host, and try again. But a quick check suggests that the 400gB limit has taken effect:docker run --rm -it mcr.microsoft.com/dotnet/framework/sdk:4.7.2-windowsservercore-ltsc2019 powershell:One thing I did notice, is that
ue4docker infosaw the 400gB limit change before I restarted Docker. I didn't check, but it's not documented as a dynamic configuration parameter, so without looking I suspect ue4-docker could be fooled by a daemon.json change that hasn't had the requisite service restart.adamrehn commentedon Jun 20, 2019
Hmmm, that's a bit puzzling. The 8GB limit fix should definitely be included in those versions of Docker.
Regarding the
daemon.jsonchange: yes, ue4-docker will be fooled by this. To my great annoyance, there's no way to query the Docker daemon about those configuration settings, so my only option is to read the JSON file and assume that the Docker daemon was restarted after the last modification.TBBle commentedon Jun 20, 2019
To get the real image-size limit, perhaps the script could fire up a small, known image, and query the free disk space, as I did above.
After a reboot, the problem persists. I'll try and debug it some more later, but in the meantime, I'm going to try and put the Linux containers (which built fine) into CI first, as that's a larger pain-point in our CI system.
TBBle commentedon Jun 20, 2019
I tried to create a simple reproduction of the "8gB problem", but didn't succeed. A null-byte file (
fsutil file createNew) happily copied, and the random powershell scripts I grabbed ran out of RAM trying to create the file.I tried switching from MS's 18.09.6 (from
DockerMSFTProvider, installed above) to Docker's 18.09.6 (fromDockerProvider), as the packages have different SHA256, but that didn't change anything.I've installed 19.03.2-rc2 from
DockerProviderjust in case there's a problem that persists in the 18.09 series:and am now rebuilding to see if that passes. Interestingly, step 6 (the first RUN in ue4-minimal) didn't pull from the cache after the Docker update, I wonder if something has changed upstream that would cause that to not match.
I actually had to bypass the SHA256 verification of the downloaded .zip of 19.03.0-rc2. I downloaded it from a couple of places, and always got the same SHA256, but it never matched the manifest SHA256.
TBBle commentedon Jun 20, 2019
Issue reproduced with 19.03.0-rc2 as well.
adamrehn commentedon Jun 20, 2019
Damn, that's a bit worrying, the 8GB fix should well and truly be integrated in versions of Docker that new. Out of curiosity, does the build succeed if you use the
--exclude debugflag?TBBle commentedon Jun 21, 2019
I'll try
--exclude debugnext.I've just been playing with debug logging for Docker (still 19.03.0-rc2), but it doesn't really tell me much:
I'm not sure if the 8gB problem gives those logs. I was working with an abbreviated Dockerfile which started from the final state of the build container before the second
FROM:I also confirmed that
docker cpworks, which I think also relies on the tar system:TBBle commentedon Jun 21, 2019
I can confirm that with
--exclude debug, it passes this step fine.If I remember correctly, the last set of images I built were under a version of ue4-docker where the PDBs were automatically excluded, which was under an AWS-hosted by otherwise similar build environment (ltsc2019), for 4.21.1. So I'm wondering if somehow this problem is exclusive to Docker EE, and the people who've fixed #37 by upgrading to 18.09.6 were actually unintentionally switching to Docker CE or something.
If I find some time, I might try and finish my effort to put together a small repro case, as all the repro-cases I found for this issue were for Linux containers. Then I could also check it against Docker CE easily.
adamrehn commentedon Jun 21, 2019
Okay, so it's definitely a size issue then if the version with truncated debug symbols works.
To the best of my knowledge, it's not possible to install Docker CE under Windows Server (unless using Docker Desktop under a version of Windows Server with the Desktop Experience), but I'd be genuinely interested to see if Docker CE 19.03.0-beta3 suffers from the same issue when running the edge release of Docker Desktop under Windows 10. (The latest stable release of Docker Desktop currently includes Docker CE 18.09.2, which definitely does suffer from this issue.)
A reliable repro Dockerfile for Windows containers would be very useful, since we could then systematically test various versions of both Docker CE and Docker EE to determine exactly which release(s) do and do not include the 8GB limit fix.
adamrehn commentedon Jul 14, 2019
Although I know you were encountering this problem under Windows Server 1809, it's interesting to note that a related bug has just been positively identified in Windows Server 1903: docker/for-win#3884 (comment)
TBBle commentedon Aug 6, 2019
Docker Enterprise for Windows Server (DockerMsftProvider) now has a release of 19.03.01, so sometime in the next couple of weeks I expect to be trying this out again with UE 4.22, same as the previous attempt, but slightly patched this time for Server and Client builds.
adamrehn commentedon Aug 7, 2019
Yeah, I noticed yesterday that the latest version of Docker Desktop now includes 19.03.1, so it'll be interesting to see how it goes. I might actually spin up a Windows Server 1809 VM and test it out too.
adamrehn commentedon Jan 31, 2020
@TBBle I've just added functionality to ue4-docker that allows users to test for the presence of the 8GiB filesystem layer bug: #63 (comment)
TBBle commentedon Feb 1, 2020
Awesome. I gave the new version a go at home (as a baseline) on Docker Desktop 2.2.0.0 (44247)
I'll have a run on the system at work about which I originally logged this ticket. It's probably due to have its Docker installation upgraded anyway.
Linux containers (VM) mode
I think it's trying to
_getdiskusageon /var/lib/docker. Not a supported build-mode, as I recall.Annoyingly, without
--linuxit doesn't work as it tries to use a Windows base image. It'd be nice if it could detect and tell me which platform the Docker daemon was configured for.Windows Containers mode:
Same results so far as described in #63 (comment).
I turned on 'experimental mode' to get access to LCOW
So yeah, LCOW mode passed, and (if I understand correctly) that's the same daemon as WCOW, so it's not a missing daemon patch in the Windows build, as surmised.
Without experimental mode enabled in the daemon, --linux failed similarly to how lack of --linux failed in Linux Containers mode, with a complaint that the requested base image did not have a matching platform. I'm not sure if this is usefully detectable... Perhaps there's some reliable combination of server OS/arch and available storage drivers that can report the right failure earlier.
Thought I'd give process isolation mode a chance while I was in Windows Containers mode. I had to do it by hand, because ue4docker sees the kernel version (via
platform.win_ver()=>sys.getwindowsversion().platform_versionrather than the OS versionsys.getwindowsversion().build, and the kernel is shared between 1903 and 1909, but MS provides distinct container bases. I'm not sure if it matters.Anyway, looks like docker/for-win#3884 affects Windows 1909 as well.
Also, that issue got hijacked by discussions of a different error, caused by filesystem filters. Probably worth changing the link in the docs to docker/for-win#4100, where it's clearly visible that there'll be a workaround for this in Docker Engine 19.03.6. A
ue4-docker diagnosticfor this might help too.Edit: Opened UnrealContainers/unrealcontainers.github.io#2 for the doc updates in that last paragraph.
TBBle commentedon Feb 1, 2020
I captured the relevant log snippet in case I or someone goes looking into this later, with "log-level debug" and "debug true" configured, and the process-isolation command (it's faster) from the end of my last comment.
The parent layer's own logs:
I disabled re-exec (by setting the
DOCKER_WINDOWSFILTER_NOREEXECsystem env-var to "true", and restarting Docker), and now we can see the logs fromImportlayer:I confirmed that the file in
importFolderPath(C:\ProgramData\Docker\tmp\hcs557293350 here) exists during the run, and is 8GB, with the file named file present.TBBle commentedon Feb 1, 2020
Confirmed a slightly simpler test-case, which I'll post upstream.
Given an named 8gfile that is 8589934592 bytes long (i.e. I ran the PowerShell script from the diagnostics locally), and the following Dockerfile:
the problem reproduces.
adamrehn commentedon Feb 2, 2020
Thanks for the thorough testing and investigation! I've now added code to detect when Docker Desktop is in Linux container mode and provided more helpful error messages (in both
ue4-docker buildand the diagnostics), blacklisted Windows 1909, and added a diagnostic for testing for the storage-opt issue:ue4-docker diagnostics maxsize. The changes are live in ue4-docker version 0.0.48.adamrehn commentedon Feb 2, 2020
It's a bit worrying that the diagnostic passes when running in LCOW mode, since as you've noted that suggests something a bit subtler than Windows builds of the daemon not including the relevant bugfix. Thanks for opening the issue on the Moby issue tracker, hopefully someone can identify what's actually going on there.
TBBle commentedon Feb 2, 2020
No worries. I'm pretty sure (having spent some time with hcsshim and containerd recently while trying to get buildkit working for WCOW) that the problem is either an issue in system call hcsshim is making (
ImportLayer), or something Docker is doing is wrong or out-of-order such that the prerequisites forImportLayerare not correct.For example, I haven't ruled out that Docker is somehow getting out of sync and cleaning up a directory that
ImportLayeris still using.I'm leaving it alone for now, hoping someone upstream can track this down, but if I get more time, I might try and work out if I can put together a repro using just hcsshim calls, and/or add a ton of debugging to Docker to try and track down all the things going on. I'm still having trouble working out exactly how Docker ends up calling
ImportLayeranyway, the Go debugger won't let me breakpoint it for some reason, and I can't find an actual call-site by text but the logs clearly show it being called. >_<I also need to finish some games so I can clear the disk space to keep experimenting... Or fix
FROM scratchsupport for Windows in Docker.adamrehn commentedon Feb 2, 2020
BuildKit support for Windows containers would be excellent, I'll be keen to see how that progresses. Combined with a fix for this problem, it'd finally provide enough parity between Linux and Windows containers to make some serious improvements to how ue4-docker builds images.
TBBle commentedon Mar 11, 2020
So it's visible, the BuildKit tracking bug for Windows Container support is moby/buildkit#616
tynril commentedon Apr 29, 2020
Hi! While I understand that this is caused by an upstream issue, is there any known configuration of Windows/Docker versions that work around this issue?
I am currently hitting it with the following configuration:
The diagnostic run looks like that:
If there is any configuration known to work at this point, I'd love to hear about it!
adamrehn commentedon Apr 29, 2020
@tynril I'm not currently aware of any combination of Windows/Docker version that does not exhibit this behaviour, and judging by the comments on the upstream issue that @TBBle opened, it looks like it's still an unsolved problem. In the meantime, you can successfully build Windows container images by excluding debug symbols with the
--exclude debugflag.TBBle commentedon Sep 25, 2020
The fix has gone into https://github.com/moby/moby/ (upstream for Docker Engine) and will be part of the 20.03.0 release, whenever that is.
TBBle commentedon Oct 27, 2020
Thanks to @Agendum for the prompt, it appears that the Docker Engine 20.10.0 beta1 release included in Docker Desktop for Windows Community 2.4.2.0 (the Edge channel).
However, I haven't tested the 8gig diagnostic yet to confirm that.
TBBle commentedon Oct 27, 2020
Per a discussion on #99, it's possible there's a different 8gig issue in Docker Engine still (relating to COPY a different target in a multi-target Dockerfile) which is going to hit in the same circumstances as this ticket, but might be a problem in Docker Engine itself, rather than in the underlying container support.
TBBle commentedon Oct 28, 2020
I've confirmed that with Docker Desktop for Windows Community Edge (2.4.2.0), the
ue4-docker diagnostics 8gigtest passes. For some unrelated reason, process isolation is failing on both my test machines, so I used hyperv isolation.This was using the updated version of the test from #97.
Test machines:
For
ue4-docker diagnostics 8gig --isolation hyperv -basetag 2004:For
ue4-docker diagnostics 8gig --isolation hyperv -basetag 1909:adamrehn commentedon Dec 11, 2020
I can confirm that the 8gig diagnostic passes when using process isolation mode with Docker CE 20.10.0 under Windows 10 version 20H2. I'm going to close this issue now and we can continue to track discussions around the potential new bug in #99.