kvm: recover from minikube-net network left over failures #9641
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @prezha. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Travis tests have failedHey @prezha, 1st Buildmake testTravisBuddy Request Identifier: 2599aa80-2239-11eb-b1e0-17a3b2b69400 |
|
Can one of the admins verify this patch? |
medyagh
requested changes
Nov 9, 2020
|
/ok-to-test |
|
kvm2 Driver |
|
/ok-to-test |
this should never happen (during |
|
but even with that, it should still work: i've intentionally avoided calling deleteNetwork() from createNetwork() in Create(), as it looked awkward and also we have ensureNetwork() call later on in Start() that would check and fix any unexpected deviations (ie, delete/recreate network if needed), and it's working as expected - from logs: so, we've correctly detected that there is the previous also, there are out of curiosity: are there any chances that minikube is run also under another account (ie, root)? that could explain why we cannot reactivate network as an ordinary user... minikube/.github/workflows/pr.yml Line 610 in 07ff90c and minikube/.github/workflows/master.yml Line 612 in 07ff90c note: in the example above in the description section, i've demonstrated that we now can reactivate inactive minikube-net network that was manipulated (ie, inactivated) under root account (using virsh), but that network was originally created under a regular user account, and here we might deal with the minikube-net that was created under root account and cannot be reactivated under regular account |
|
@prezha that makes sense. I think we should do two things:
Would you be comfortable adding one or both of those to this PR? |
52f78cc to
9e11413
Compare
agree, i'm on it (both of them) |
Codecov Report
@@ Coverage Diff @@
## master #9641 +/- ##
==========================================
- Coverage 29.64% 29.49% -0.16%
==========================================
Files 173 173
Lines 10791 10846 +55
==========================================
Hits 3199 3199
- Misses 7143 7198 +55
Partials 449 449
|
medyagh
approved these changes
Nov 10, 2020
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: medyagh, prezha The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
we can add the jenkins script in a follow up PR |
|
thank you verty much for this PR @prezha this will improve user's experience on kvm |
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes: #9610
fixes: #9049
fixes: #8952
as explained in original issues, cluster with minikube-net network for kvm driver might fail because eg, the network is inactive or already in use, and this pr tries to handle those situations by either trying to avoid race conditions between ops in the first place, then fix the issue, or recreate the network as the last resort
example for safely recreating minikube-net network (if all else fails)
step 1 - create cluster with kvm2 driver:
step 2 - stop the cluster:
step 3 - check networks:
step 4 - 'break' minikube-net network:
step 5 - check networks:
step 6 - disable 'easy fix' (reactivate inactive network) in kvm.setupNetwork()
step 7 - start the cluster (and see the network being safely recreated, and cluster brought up successfully):
relevant output sections:
step 8 - finally, check networks again: