Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bad --insecure-registry values cause: Job for docker.service failed because the control process exited with error code. #8790

Closed
anencore94 opened this issue Jul 21, 2020 · 8 comments
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@anencore94
Copy link
Contributor

anencore94 commented Jul 21, 2020

Steps to reproduce the issue:

  1. minikube start --insecure-registry " :5000" --driver=virtualbox
  2. Just wait until failed with printing stdout, stderr.

Full output of failed command:

  • same as next log

Full output of minikube start command used, if not already included:

😄 Ubuntu 18.04 위의 minikube v1.12.1 ✨ 유저 환경 설정 정보에 기반하여 virtualbox 드라이버를 사용하는 중 👍 Starting control plane node minikube in cluster minikube 🔥 virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) 를 생성하는 중 ... 🔥 virtualbox 의 "minikube" 를 삭제하는 중 ... 🤦 StartHost failed, but will try again: creating host: create: provisioning: ssh command error: command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; } err : Process exited with status 1 output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service. Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

🔥 virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) 를 생성하는 중 ...
😿 Failed to start virtualbox VM. "minikube start" may fix it: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

❌ [DOCKER_RESTART_FAILED] error provisioning host Failed to start host: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err : Process exited with status 1
output : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

💡 Suggestion: Remove the incompatible --docker-opt flag if one was provided
⁉️ Related issue: #7070

What I expect

  • I found this error(?) when I tried to start minikube with minikube start --insecure-registry=$REGISTRY_IP:$REGISTRY_PORT, but those env was not enrolled in bash.
  • It would be nice to check insecure-registry flag value with regular expression checking.
  • It does not failed when minikube start with non-existing registry but url is ipv4 type. I mean starting minikube with --insecure-registry "localhost:5000" before creating local registry does not failed in current version

version info

minikube version v1.12.1 docker version 19.03.6 ubuntu 18.04.3 LTS
@sharifelgamal sharifelgamal added needs-solution-message Issues where where offering a solution for an error would be helpful kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. priority/backlog Higher priority than priority/awaiting-more-evidence. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Jul 27, 2020
@sharifelgamal
Copy link
Collaborator

Yeah, we should probably do a sanity check for URL parameters. Help wanted!

@priyawadhwa priyawadhwa added the kind/bug Categorizes issue or PR as related to a bug. label Aug 12, 2020
@tstromberg tstromberg removed the needs-solution-message Issues where where offering a solution for an error would be helpful label Sep 1, 2020
@kadern0
Copy link
Contributor

kadern0 commented Sep 8, 2020

@sharifelgamal I've been looking at a validation function. Are these all the supported formats for the registry or am I missing something?

addresses := []string{"10.0.0.1/24", "example.com", "localhost:5000", "localhost", "127.0.0.1", "www.example.com"}

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 8, 2020
@tstromberg tstromberg changed the title minikube start with insecure-registry flag failed when registry url is invalid. bad --insecure-registry values cause: Job for docker.service failed because the control process exited with error code. Dec 14, 2020
@tstromberg tstromberg added good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 14, 2020
@tstromberg
Copy link
Contributor

tstromberg commented Dec 14, 2020

@kadern0 - I did some digging and discovered that we pass the value directly to dockerd's --insecure-registry flag. Here's the documentation: https://docs.docker.com/engine/reference/commandline/dockerd/#insecure-registries

In order to communicate with an insecure registry, the Docker daemon requires --insecure-registry in one of the following
two forms:

--insecure-registry myregistry:5000 tells the Docker daemon that myregistry:5000 should be considered insecure.
--insecure-registry 10.1.0.0/16 tells the Docker daemon that all registries whose domain resolve to an IP address is part of > the subnet described by the CIDR syntax, should be considered insecure.

That suggests to me that the only valid forms should be:

  • <ip>:<port>
  • <hostname>:<port>
  • <network>/<netmask> (CIDR)

I can confirm that this bug still occurs at master. Honestly, you could probably solve 95% of issues by providing a regular expression that asserts that the value contains only alphanumerics, slashes, colons, dashes, square brackets, or dots (basically: DNS, IPv4, IPv6, or CIDR)

@kadern0
Copy link
Contributor

kadern0 commented Dec 15, 2020

Thanks for your answer, @tstromberg. This should be easy to achieve with regular expressions although it might be interesting looking at using some validator within the ClusterConfig struct although this change will be bigger.

I mention this because it seems to me this function is not 100% correct since it won't reject some malformed URLs (like this "http://;;..,,----__???**"):

func validateRegistryMirror() {

With the validator in place, many validation functions could be either removed or simplified.

@kadern0
Copy link
Contributor

kadern0 commented Jan 6, 2021

@tstromberg, this one is fixed by #9977

@cvila84
Copy link

cvila84 commented Jan 28, 2021

Hello, would it be possible to also consider IP and hostname in the regexp without having to mention a port ?

In our situation, until 1.17, we were using --insecure-registry dockerhub.example.com in our CI/CD with all K8S manifests pointing to docker images built with tag of kind dockerhub.example.com/xxx/yyy

With 1.17+, to pass the validation, I have to set --insecure-registry dockerhub.example.com:80 (which means the same thing) but the result is that no K8S pod can start as images with tag of kind dockerhub.example.com/xxx/yyy are tried to be pulled in a secure way because it does not precisely match the insecure registry setting which contain the port...

Changing all the jobs that build our docker images in order to add port 80 to the tag is not really an option as we would like to keep consistency in our tags (as they are parsed by some other internal tools we have)

Thanks !

@kadern0
Copy link
Contributor

kadern0 commented Feb 3, 2021

@tstromberg it seems we might need to update the validation fuction according to docker's insecure registry validation (port is not mandatory):
https://github.com/docker/engine/blob/master/registry/config_test.go

Do you agree?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests

9 participants