Harbor, Cert-manager, self-signed CA and Containerd/Docker Troubleshooting
Recently, I switch the container registry from docker hub to harbor and encountered “x509: certificate signed by unknown issuer error” using Docker Desktop and Harbor private registry. This article is about how I resolved this issue in my Docker desktop on Mac and my home lab k8s containerd.
In my home lab, I deployed a harbor registry to replace the default one (docker hub) and exposed it as a load balancer service. This way I can build docker image on my mac, push the image to harbor, then deploy the app to my k8s environment. As a devop beginner, my simple pipeline looks like this:
- Write the app codes
- Wrap the codes in Dockerfile
- Use Docker Desktop to build the image
- Tag the image and point the tag to my container registry
- Push the image to remote container registry
- Write k8s deployment yaml and deploy the app as a K8S service
- Test/Write another app to connect to the service
These steps worked fine until I try to “secure” my K8S home lab and moved the container registry to Harbor. I started with installing cert-manager with a self-signed CA, then deployed Harbor with cert-manager issued certificate.
After deploying cert-manager in my home lab, I created a self-signed x509 key pair like the following.
In order to get cert-manager to pickup this key pair as cluster issuer, I put this key pair as secret in the cert-manager namespace.
Next step, I created a certificate for Harbor and deployed harbor to K8s.
With this certificate, I can direct helm install with my custom_values.yaml to use the certificate turn on TLS. My yaml file looked like this:
I used helm chart to deploy harbor. The steps are here:
After a while, I logged in to Harbor webui and checked the certificate is exactly what I created in cert-manager.
After I added a user and a project to harbor, I thought I can resume my devop project by switch my docker desktop registry to harbor. I hit an error “x509: certificate signed by unknown authority” when “docker login harbor.home.lab”. Aha, I thought I need to import the ca.crt to my mac, but docker desktop didn’t pickup the root ca. I googled docker x509 and found the solution. I copied the root ca.crt to .docker/certs.d in my home dir and restart docker desktop. Now docker can recognize and trust my self-signed ca.
Now I can tag and push my image to harbor!
From the harbor web ui, I can see the image!
Not bad. Now it is time to deploy this image as k8s deployment.
Hmm….. “kubectl describe pod” got an image pull error and showed the same “x509: certificate signed by unknown authority”. I guess I need to do the same thing (copying ca.crt to somewhere) on the worker nodes, but where? I switch docker to containerd in my k8s homel ab. I googled again and found ca should be in ‘/usr/local/share/ca-certificates’. On each worker node, copy the ca.crt to ‘/usr/local/share/ca-certificates’ and restart containerd.
After this change, I finally can pull the image from the worker.
“kubectl logs” show that my image is not good as there is no FLASK_APP env to start the web app. What an devop rookie mistake.