Repository navigation
Pushes to gitlab fail with SSL error #150
Description
from bazelbuild/rules_docker#820:
When attempting to run a docker push through a container_push rule to a registry.gitlab.com target, I am recieving the following error:
registry.gitlab.com/gauntletwizard/bazel-go:{BUILD_EMBED_LABEL} was resolved to registry.gitlab.com/gauntletwizard/bazel-go:foo
F0430 20:27:05.873595 4586 fast_pusher_.py:194] Error publishing registry.gitlab.com/gauntletwizard/bazel-go:foo: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
Users report they have made no change to their dependencies. Error is coming from containerregistry libraries so posting here. My suspicion is something changed in registry.gitlab.com that makes it so that containerregistry is no longer being able to validate certificates.
Activity
s-garg commentedon May 2, 2019
@nlopezgi Is there a way to set PYTHONHTTPSVERIFY property in bazel, such that it trickles down to containerregistry? (found this thread to turn off cert verification in python: http://blog.pengyifan.com/how-to-fix-python-ssl-certificate_verify_failed/)
Also is it possible to figure out for which URL cert verification is failing? The SSL cert for https://registry.gitlab.com seems to be fine - current cert is valid from March 23rd, 2019 - May 22nd, 2020.
s-garg commentedon May 6, 2019
The issue is coming from httplib2. You can reproduce the issue using the following python code:
import httplib2h = httplib2.Http()h.request('https://registry.gitlab.com')And the issue can be resolved by adding the root certificate to the Http definition.
import httplib2h = httplib2.Http('./USERTrustRSACertificationAuthority.crt')h.request('https://registry.gitlab.com')The root certificates used by httplib2 are coming from the cacerts.txt file.
(https://github.com/httplib2/httplib2/blob/master/python2/httplib2/cacerts.txt)
registry.gitlab.com probably switched the root CA last week and that has triggered the problem.
I don't see any option in the containerregistry registry code to specify a CACerts file.
I see three open PRs related to configuring CA Certs.
However, they have been open for a while. Is there any workaround for this problem that others have used?
KaylaNguyen commentedon May 7, 2019
I'll try updating httplib2 for containerregistry.
arcticwaters commentedon May 11, 2019
Until support is available in httplib2, use an http_archive patch: https://gist.github.com/arcticwaters/d3310d2b6e362a277501615c6ff07878. This patch works for gitlab.
Containerregistry won't register that repository if one is already defined so it has to go at the top of your WORKSPACE. Downside: you'll have to remember to update when containerregistry updates httplib2.
s-garg commentedon May 11, 2019
I came up with my own hack to get it to work. But the patch approach seems better. Thanks for sharing!
palvarez89 commentedon May 28, 2019
This is still a problem. Is anybody looking into it?
patch bb-remote-execution httplib2