Add a machine readable reason to all error paths #9126

tstromberg · 2020-08-31T00:31:47Z

This gargantuan PR updates all error paths to provide a machine readable reason. This PR also significantly expands the dynamic range of exit codes, providing a framework to use most of the ~125 exit codes available in UNIX.

This replaces the previous "problem" error matching package with a new general purpose "reason" package that can be used directly from anywhere. As such, this PR gets rid of many attempts to provide output that looked similar to the problem package. Like the old problem package, the reason ID is propagated into both the JSON output and display output.

Unlike the previous problem implementation, reasons no longer require an error regexp, and now supports custom exit codes and styles.

Fixes #9080

Example 1: Not enough Docker storage

Old (exit code 251, UNDEFINED)

💣  docker daemon out of memory. No space left on device

In JSON form:

{"data":{"exitcode":"507","message":"docker daemon out of memory. No space left on device\n"},"datacontenttype":"application/json","id":"e2edb0f5-f8df-47d3-af2f-a15f22145223","source":"https://minikube.sigs.k8s.io/","specversion":"1.0","type":"io.k8s.sigs.minikube.error"}

New (exit code 26, ExInsufficientStorage)

❌  Exiting due to RSRC_DOCKER_STORAGE: Docker is out of disk space! (/var is at 99% of capacity)
💡  Suggestion: 

    Try at least one of the following to free up space on the device:
    
    1. Run "docker system prune" to remove unused docker data
    2. Increase the amount of memory allocated to Docker for Desktop via
    Docker icon > Preferences > Resources > Disk Image Size
    3. Run "minikube ssh -- docker system prune" if using the docker container runtime
🍿  Related issue: https://github.com/kubernetes/minikube/issues/9024

In JSON form:

{"data":{"advice":"Try at least one of the following to free up space on the device:\n\t\n\t\t\t1. Run \"docker system prune\" to remove unused docker data\n\t\t\t2. Increase the amount of memory allocated to Docker for Desktop via \n\t\t\t\tDocker icon \u003e Preferences \u003e Resources \u003e Disk Image Size\n\t\t\t3. Run \"minikube ssh -- docker system prune\" if using the docker container runtime","exitcode":"26","issues":"https://github.com/kubernetes/minikube/issues/9024","message":"Exiting due to RSRC_DOCKER_STORAGE: Docker is out of disk space! (/var is at 100%% of capacity)","name":"RSRC_DOCKER_STORAGE","url":""},"datacontenttype":"application/json","id":"e6d5030a-b2d4-4e03-84d1-b2db7a8020f9","source":"https://minikube.sigs.k8s.io/","specversion":"1.0","type":"io.k8s.sigs.minikube.error"}

Example 2: Not enough Docker cores

Old (exit status 64, EX_USAGE)

💥  Your Docker Desktop has less than 2 CPUs. Increase CPUs for Docker Desktop. 
	
	Docker icon > Settings > Resources > CPUs
				
				
📘  https://docs.docker.com/config/containers/resource_constraints/
💣  Ensure your Docker for Desktop system has enough CPUs. The minimum allowed is 2 CPUs.

New (exit status 29, ExInsufficientCores)

⛔  Exiting due to RSRC_DOCKER_CORES: Docker Desktop has less than 2 CPUs configured, but Kubernetes requires at least 2 to be available
💡  Suggestion: 

    1. Click on "Docker for Desktop" menu icon
    2. Click "Preferences"
    3. Click "Resources"
    4. Increase "CPUs" slider bar to 2 or higher
    5. Click "Apply & Restart"
📘  Documentation: https://docs.docker.com/docker-for-mac/#resources

📘 https://docs.docker.com/config/containers/resource_constraints/
💣 Ensure your Docker for Desktop system has enough CPUs. The minimum allowed is 2 CPUs.

Example 3: Docker is not running (exit status 69: EX_UNAVAILABLE)

Old

❗  'docker' driver reported an issue: "docker version --format {{.Server.Os}}-{{.Server.Version}}" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
💡  Suggestion: Start the Docker service
📘  Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/

💣  Failed to validate 'docker' driver

New (exit status 63: ExProviderNotRunning)


💣  Exiting due to PROVIDER_DOCKER_NOT_RUNNING: "docker version --format -" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
💡  Suggestion: Start the Docker service
📘  Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/

Example 4: Permissions to $HOME/.kube/config

Old (exit 78, EX_CONFIG)

❌  [KUBECONFIG_DENIED] failed to start node startup failed: Failed to update kubeconfig file.: writing kubeconfig: Error writing file /Users/tstromberg/.kube/config: writefile failed for /Users/tstromberg/.kube/config: open /Users/tstromberg/.kube/config: permission denied
💡  Suggestion: Run: 'chmod 600 $HOME/.kube/config'
⁉️   Related issue: https://github.com/kubernetes/minikube/issues/5714

New (exit 37, ExHostPermission)

🚫  Exiting due to HOST_KUBECONFIG_PERMISSION: Failed kubeconfig update: writing kubeconfig: Error writing file /Users/tstromberg/.kube/config: open /Users/tstromberg/.kube/config: permission denied
💡  Suggestion: Run: 'sudo chown $USER $HOME/.kube/config && chmod 600 $HOME/.kube/config'
🍿  Related issue: https://github.com/kubernetes/minikube/issues/5714

Example 5: Docker memory allocation warning

Old

❗  Your system has 32768MB memory but Docker has only 2240MB. For a better performance increase to at least 3GB.

	Docker for Desktop  > Settings > Resources > Memory

New

💨  Docker Desktop has access to only 2240MiB of the 32768MiB in available system memory. Consider increasing this for improved performance.
💡  Suggestion: 

    1. Click on "Docker for Desktop" menu icon
    2. Click "Preferences"
    3. Click "Resources"
    4. Increase "Memory" slider bar to 3 GB or higher
    5. Click "Apply & Restart"
📘  Documentation: https://docs.docker.com/docker-for-mac/#resources

k8s-ci-robot · 2020-08-31T00:32:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tstromberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tstromberg · 2020-08-31T05:12:10Z

/ok-to-test

minikube-pr-bot · 2020-08-31T05:26:27Z

kvm2 Driver
error collecting results for kvm2 driver: timing run 0 with Minikube (PR 9126): timing cmd: [/home/performance-monitor/.minikube/minikube-binaries/9126/minikube start --driver=kvm2]: starting cmd: fork/exec /home/performance-monitor/.minikube/minikube-binaries/9126/minikube: exec format error
docker Driver
error collecting results for docker driver: timing run 0 with Minikube (PR 9126): timing cmd: [/home/performance-monitor/.minikube/minikube-binaries/9126/minikube start --driver=docker]: starting cmd: fork/exec /home/performance-monitor/.minikube/minikube-binaries/9126/minikube: exec format error

…h new behavior

minikube-pr-bot · 2020-08-31T18:31:51Z

kvm2 Driver
Times for minikube: 69.9s 65.6s 69.0s
Average time for minikube: 68.2s

Times for Minikube (PR 9126): 63.5s 64.0s 62.7s
Average time for Minikube (PR 9126): 63.4s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9126) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.0s    | 39.6s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.0 | 25.0s    |                    |
| on Docker 19.03.12 ...         |          |                    |
| * Verifying Kubernetes         | 1.6s     | 1.4s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.4s     | 0.0s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         | 0.1s     | 0.1s               |
| configured to use "minikube"   |          |                    |
|                                | 0.0s     | 0.0s               |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 30.4s 27.8s 29.5s
Average time for minikube: 29.2s

Times for Minikube (PR 9126): 27.0s 28.2s 26.4s
Average time for Minikube (PR 9126): 27.2s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9126) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.0s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 9.0s     | 8.8s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.0 | 19.0s    |                    |
| on Docker 19.03.8 ...          |          |                    |
| * Verifying Kubernetes         | 0.9s     | 1.0s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.0s     | 0.0s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         | 0.1s     | 0.1s               |
| configured to use "minikube"   |          |                    |
|                                | 0.0s     | 17.3s              |
+--------------------------------+----------+--------------------+

priyawadhwa

This is awesome! Just a few comments but the code is looking way cleaner after this refactor :)

cmd/minikube/cmd/mount.go

pkg/minikube/reason/match.go

pkg/minikube/registry/registry.go

cmd/minikube/cmd/start.go

minikube-pr-bot · 2020-08-31T20:12:17Z

kvm2 Driver
Times for minikube: 61.8s 63.1s 65.0s
Average time for minikube: 63.3s

Times for Minikube (PR 9126): 63.1s 63.6s 65.7s
Average time for Minikube (PR 9126): 64.1s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9126) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the kvm2 driver based  | 0.0s     | 0.0s               |
| on user configuration          |          |                    |
| * Starting control plane node  | 0.0s     | 0.0s               |
| minikube in cluster minikube   |          |                    |
| * Creating kvm2 VM (CPUs=2,    | 38.5s    | 39.1s              |
| Memory=3700MB, Disk=20000MB)   |          |                    |
| ...                            |          |                    |
| * Preparing Kubernetes v1.19.0 | 22.8s    | 22.8s              |
| on Docker 19.03.12 ...         |          |                    |
| * Verifying Kubernetes         | 1.5s     | 1.6s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.3s     | 0.4s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         | 0.1s     | 0.1s               |
| configured to use "minikube"   |          |                    |
|                                | 0.0s     | 0.0s               |
+--------------------------------+----------+--------------------+

docker Driver
Times for minikube: 30.6s 28.6s 30.3s
Average time for minikube: 29.8s

Times for Minikube (PR 9126): 30.0s 28.2s 28.2s
Average time for Minikube (PR 9126): 28.8s

Averages Time Per Log

+--------------------------------+----------+--------------------+
|              LOG               | MINIKUBE | MINIKUBE (PR 9126) |
+--------------------------------+----------+--------------------+
| * minikube v1.12.3 on Debian   | 0.1s     | 0.1s               |
|                           9.11 |          |                    |
| * Using the docker driver      | 0.1s     | 0.0s               |
| based on user configuration    |          |                    |
| * Starting control plane node  | 0.1s     | 0.1s               |
| minikube in cluster minikube   |          |                    |
| * Creating docker container    | 9.0s     | 8.7s               |
| (CPUs=2, Memory=3700MB) ...    |          |                    |
| * Preparing Kubernetes v1.19.0 | 19.5s    | 19.0s              |
| on Docker 19.03.8 ...          |          |                    |
| * Verifying Kubernetes         | 0.9s     | 0.9s               |
| components...                  |          |                    |
| * Enabled addons:              | 0.0s     | 0.0s               |
| default-storageclass,          |          |                    |
| storage-provisioner            |          |                    |
| * Done! kubectl is now         | 0.1s     | 0.1s               |
| configured to use "minikube"   |          |                    |
|                                | 0.0s     | 0.0s               |
+--------------------------------+----------+--------------------+

medyagh

great refactor

Add detailed JSON & status codes for all error paths

2c5e8fd

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 31, 2020

k8s-ci-robot requested review from prasadkatti and RA489 August 31, 2020 00:32

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 31, 2020

tstromberg changed the title ~~Add a machine readable reason to all erorr paths~~ Add a machine readable reason to all error paths Aug 31, 2020

tstromberg added 4 commits August 30, 2020 20:48

Unwrap unnecessary level of error annotation for improved messages

cd91fb8

Unwrap unnecessary level of error annotation for improved messages

7f5a06b

Improve error disambiguation, especially for non-running drivers

f3cce50

Shorten Docker memory warning

8ef7410

tstromberg requested review from priyawadhwa and sharifelgamal and removed request for prasadkatti and RA489 August 31, 2020 04:05

kubernetes deleted a comment from TravisBuddy Aug 31, 2020

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 31, 2020

Update TestFunctional/parallel/DryRun and TestJSONOutputError to matc…

09cdf60

…h new behavior

priyawadhwa reviewed Aug 31, 2020

View reviewed changes

cmd/minikube/cmd/mount.go Show resolved Hide resolved

pkg/minikube/reason/match.go Outdated Show resolved Hide resolved

pkg/minikube/registry/registry.go Outdated Show resolved Hide resolved

cmd/minikube/cmd/start.go Show resolved Hide resolved

Code review comments

750cf39

medyagh approved these changes Aug 31, 2020

View reviewed changes

tstromberg merged commit 51a3155 into kubernetes:master Sep 1, 2020

tstromberg mentioned this pull request Sep 1, 2020

Warn if Docker is likely to encounter swap issues (performance impact) #8892

Closed

zhijianli88 mentioned this pull request Sep 2, 2020

fix make extract #9152

Merged

briandealwis mentioned this pull request Oct 9, 2020

vm-driver-none fallback detection GoogleContainerTools/skaffold#4868

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a machine readable reason to all error paths #9126

Add a machine readable reason to all error paths #9126

tstromberg commented Aug 31, 2020 •

edited

Loading

k8s-ci-robot commented Aug 31, 2020

tstromberg commented Aug 31, 2020

minikube-pr-bot commented Aug 31, 2020

minikube-pr-bot commented Aug 31, 2020

priyawadhwa left a comment

minikube-pr-bot commented Aug 31, 2020

medyagh left a comment

Add a machine readable reason to all error paths #9126

Add a machine readable reason to all error paths #9126

Conversation

tstromberg commented Aug 31, 2020 • edited Loading

Example 1: Not enough Docker storage

Old (exit code 251, UNDEFINED)

New (exit code 26, ExInsufficientStorage)

Example 2: Not enough Docker cores

Old (exit status 64, EX_USAGE)

New (exit status 29, ExInsufficientCores)

Example 3: Docker is not running (exit status 69: EX_UNAVAILABLE)

Old

New (exit status 63: ExProviderNotRunning)

Example 4: Permissions to $HOME/.kube/config

Old (exit 78, EX_CONFIG)

New (exit 37, ExHostPermission)

Example 5: Docker memory allocation warning

Old

New

k8s-ci-robot commented Aug 31, 2020

tstromberg commented Aug 31, 2020

minikube-pr-bot commented Aug 31, 2020

minikube-pr-bot commented Aug 31, 2020

priyawadhwa left a comment

Choose a reason for hiding this comment

minikube-pr-bot commented Aug 31, 2020

medyagh left a comment

Choose a reason for hiding this comment

tstromberg commented Aug 31, 2020 •

edited

Loading