worker: add Hyper-V isolation and image caching to the Windows containerd runtime - #9786

Windows workers only have the houdini runtime, which runs every step
directly on the host. Add containerd as an opt-in runtime for Windows
workers with `--runtime containerd`. houdini stays the default, and the
flag description marks the Windows containerd runtime as experimental.

The worker starts containerd.exe on a named pipe and runs the existing
Garden backend against it. The shared runtime files now build for
linux || windows, with Windows versions of the Linux-only pieces:

- spec_windows.go turns a Garden container spec into a Windows OCI
  spec. It mounts init.exe (cmd/init/windows) into the container as the
  init process, converts the POSIX paths the web node sends
  (/tmp/build/...) to C:\ paths, and maps CPU shares and memory limits
  to Windows resources.
- prepareRootfs converts the streamed rootfs volume into a Windows base
  layer with hcsshim and creates the scratch layer that the runhcs shim
  needs. Flattened multi-layer images carry registry deltas and machine
  state from the containers that built them. It merges the last layer's
  delta hives into the base hives and removes that state before it
  processes the layer.
- HCNNetwork gives each container a network namespace and an endpoint
  on a NAT HNS network, and creates the network when it is missing.
  `--containerd-network-name`, `--containerd-network-pool` and
  `--containerd-network-gateway` configure it.
- The killer reads the runhcs process details, and stdin stays open
  after start because the runhcs shim closes it as soon as CloseIO is
  called.
- rootfsDir accepts raw:// URIs whose Windows path the web node
  percent-encoded.
- CmdRunner kills containerd when the shutdown signal cannot be
  delivered, which is always the case on Windows.

Windows workers also register the resource types in `--resource-types`,
as Linux workers do.

concourse#4365

Signed-off-by: Kurt McAlpine <kurt@linux.com>

A Windows image built on a case-sensitive filesystem can hold two paths
that differ only in case. Extracted onto a case-insensitive filesystem,
the second entry overwrites the first, and that fails when the first
entry is read-only.

When creating a regular file fails with a permission error, make the
existing file writable and create it again.

Signed-off-by: Kurt McAlpine <kurt@linux.com>

A streamed rootfs volume loses NTFS security descriptors, reparse point
metadata and the per-layer registry state. Servercore images need these
to boot. A natively pulled servercore image boots, and the same image
streamed as a volume fails with driver initialization errors.

Tasks on Windows workers can now set `rootfs_uri: oci://<image-ref>`.
The Windows spec records the reference in an annotation, and the
backend creates the container with Client.NewContainerWithImage. That
pulls the image through containerd and lets the Windows snapshotter
manage the layers, so pulled images are cached across containers.
Container.Delete removes the container's snapshot.

concourse#4365

Signed-off-by: Kurt McAlpine <kurt@linux.com>

The Windows containerd runtime had two ways to get a container's rootfs.
A streamed rootfs volume was flattened into a single base layer. That
drops the NTFS security descriptors and file attributes that Windows
layers keep in their tar headers, and all but the last layer's registry
hive deltas, so servercore images failed to boot. An oci:// rootfs made
containerd pull the image itself, which bypasses resources and their
credentials, versions and caching.

The registry-image resource now writes the rootfs of a Windows image as
an OCI image layout. The Windows runtime copies the layout's blobs into
containerd's content store under a lease, unpacks them with the windows
snapshotter, and creates the container on a snapshot of the image. A
Windows image reaches a task the way any image does: through a get step
or a worker's resource types, streamed between workers as a volume.

Unpacking first checks that the snapshotter supports the image's
platform, so an image for another Windows build fails before any layer
is extracted. Layers stay unpacked while a container uses them, so
containers that share an image share its layers. Deleting a container
removes its snapshot. When creating the container fails, the runtime
deletes the network namespace it attached.

This removes the oci:// rootfs scheme, the rootfs flattening and the
registry hive merging that flattening needed.

concourse#4365

Signed-off-by: Kurt McAlpine <kurt@linux.com>

containerd names a process's stdio pipes on Windows after the process
ID alone, as \\.\pipe\ctr-<id>-stdin and so on. Every task step runs
its process with the ID "task", and pipe names are global to the host.
A task that started on a worker while another task was running failed
with:

    task exec: failed to create stdin pipe \\.\pipe\ctr-task-stdin:
    open \\.\pipe\ctr-task-stdin: Access is denied.

The Windows runtime now names the pipes after the container handle and
the process ID. The ID that containerd knows the process by is
unchanged.

concourse#4365

…ntime

A Windows worker ran every container with process isolation, which
shares the host's kernel, so it could only run images built for the
host's Windows version.

`--containerd-isolation=hyperv` boots each container in its own utility
VM, which runs the image's kernel. Unpacking skips the snapshotter
platform check for these containers, because that check requires the
image to match the host's Windows build. A Server 2022 worker ran a
Server 2019 nanoserver image this way. With process isolation the same
image failed the platform check.

`--containerd-hyperv-cpus` and `--containerd-hyperv-memory-mb` size the
utility VM. Unset, the runtime's defaults of 2 processors and 1024 MB
apply. A container's memory limit sizes its utility VM in place of the
memory flag.

The host commits a utility VM's whole memory when the VM starts, so the
VM size times the number of containers running at once has to fit the
host's commit limit. Past the limit a container fails to create.

The worker needs the Hyper-V role. Without it the worker starts, and
each container fails to create.

concourse#4365

The Windows runtime unpacked an image under a lease and never stored
it. Once the last container on the image was deleted, nothing referred
to the image's content or its unpacked layers, and containerd's garbage
collector removed them. The next build copied and unpacked the whole
image again, inside the worker's create lock.

The runtime now stores the image in containerd's image store, named by
its digest. Importing an image layout labels each blob with the blobs
it refers to, which garbage collection needs in order to keep the
content of a stored image. A later container on the same image skips
the copy and the unpack.

Nothing removes a stored image. The worker's disk has to hold every
image the worker has run, and replacing the worker reclaims it.

concourse#4365