Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker Registry certificate does not autorenew #21

Open
petermikitsh opened this issue Jun 15, 2019 · 0 comments
Open

Docker Registry certificate does not autorenew #21

petermikitsh opened this issue Jun 15, 2019 · 0 comments

Comments

@petermikitsh
Copy link
Member

The resource Certificate with name docker-reg-cert expires without auto-renewing. This breaks deployments, because once the certificate expires and deployments are issued, which leads to pods attempt to spin up, you'll get errors like:

  Warning  Failed     5s (x2 over 21s)   kubelet, epic-borg-ud9d  Failed to pull image "10.245.8.222:5000/digitalocean:1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.245.8.222:5000/v2/: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "docker-registry.default.svc.cluster.local")

or, more commonly:

Warning  Failed     17s                kubelet, epic-borg-ud9v  Failed to pull image "10.245.8.222:5000/digitalocean:1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.245.8.222:5000/v2/: x509: certificate has expired or is not yet valid

Configuring the certificate resource with the renewBefore option [0] should fix the problem, e.g.,

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example
spec:
  secretName: example-tls
  duration: 24h
  renewBefore: 12h <--- this is the important part we need to add
  dnsNames:
  - foo.example.com
  - bar.example.com
  issuerRef:
    name: my-internal-ca
    kind: Issuer

Additionally, we create a DaemonSet resource [1] that copies over the latest certificate daily to all nodes in all node pools. This permits the node pools to trust pulling images from the docker registry.

Here's a couple things to review to ensure high availability of the docker registry, and ensure the capability to deploy whenever within the cluster:

  1. Ensure the renewBefore spec setting actually works as excepted
  2. Ensure that the docker registry "hot reloads" the new certificate (if not, we need to force it to restart after the certificate is renewed)
  3. Ensure that the DaemonSet is performing as we except it to (described above)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@petermikitsh