Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cephfs: Log clone progress during a clone operation #4918

Merged
merged 1 commit into from
Nov 22, 2024

Conversation

Nikhil-Ladha
Copy link
Contributor

Describe what this PR does

Add clone progress report in the clone status operation and report the same in the logs for user's reference.

Fixes: #4813

Checklist:

  • Commit Message Formatting: Commit titles and messages follow
    guidelines in the developer
    guide
    .
  • Reviewed the developer guide on Submitting a Pull
    Request
  • Pending release
    notes

    updated with breaking and/or notable changes for the next major release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@mergify mergify bot added the component/cephfs Issues related to CephFS label Oct 28, 2024
@Nikhil-Ladha
Copy link
Contributor Author

cc @Madhu-1 @nixpanic

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch 2 times, most recently from f923e0a to 3f24712 Compare October 28, 2024 06:25
Copy link
Collaborator

@Madhu-1 Madhu-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if we are using older version of the ceph that doesn't support the new fields? i assume it will be empty just want to confirm on it.

internal/cephfs/store/fsjournal.go Show resolved Hide resolved
internal/cephfs/core/clone.go Show resolved Hide resolved
@Nikhil-Ladha
Copy link
Contributor Author

Nikhil-Ladha commented Oct 28, 2024

what happens if we are using older version of the ceph that doesn't support the new fields? i assume it will be empty just want to confirm on it.

Yes, it will be empty.

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from 3f24712 to 992ff84 Compare October 28, 2024 07:03
@Nikhil-Ladha Nikhil-Ladha requested a review from Madhu-1 October 28, 2024 07:06
@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from 992ff84 to 500fde7 Compare October 28, 2024 07:46
Copy link
Collaborator

@Madhu-1 Madhu-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please test it out and paste the PVC describe output, just to make sure we are surfacing it to user, feel free to add E2E if you think it can be done as well

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from 500fde7 to 946609f Compare October 28, 2024 07:54
Madhu-1
Madhu-1 previously approved these changes Oct 28, 2024
Copy link
Collaborator

@Madhu-1 Madhu-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Lets get the testing results

@Madhu-1 Madhu-1 requested a review from nixpanic October 28, 2024 07:59
@Nikhil-Ladha
Copy link
Contributor Author

Testing results:

PVC Events:

Events:
  Type     Reason                 Age                From                                                                                                             Message
  ----     ------                 ----               ----                                                                                                             -------
  Normal   ExternalProvisioning   68s (x2 over 68s)  persistentvolume-controller                                                                                      Waiting for a volume to be created either by the external provisioner 'rook-ceph.cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed     68s                rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f5df68b95-9stgl_d702fa82-d9f0-4f54-a5b9-3bd7b2cdd7d5  failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = clone from snapshot is pending
  Warning  ProvisioningFailed     67s                rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f5df68b95-9stgl_d702fa82-d9f0-4f54-a5b9-3bd7b2cdd7d5  failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = clone from snapshot is already in progress. progress report: percentage cloned=0%, amount cloned=306/0, files cloned=3/1
  Normal   Provisioning           66s (x3 over 68s)  rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f5df68b95-9stgl_d702fa82-d9f0-4f54-a5b9-3bd7b2cdd7d5  External provisioner is provisioning volume for claim "default/cephfs-pvc-clone"
  Normal   ProvisioningSucceeded  58s                rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f5df68b95-9stgl_d702fa82-d9f0-4f54-a5b9-3bd7b2cdd7d5  Successfully provisioned volume pvc-70f1e45e-7d31-4e52-babe-8ff083860505

CSI Logs:

I1028 09:34:09.854647       1 fsjournal.go:137] ID: 38 Req-ID: pvc-70f1e45e-7d31-4e52-babe-8ff083860505 clone from snapshot is already in progress. progress report: percentage cloned=0%!,(MISSING) amount cloned=306/0, files cloned=3/1
E1028 09:34:09.854817       1 utils.go:271] ID: 38 Req-ID: pvc-70f1e45e-7d31-4e52-babe-8ff083860505 GRPC error: rpc error: code = Aborted desc = clone from snapshot is already in progress. progress report: percentage cloned=0%, amount cloned=306/0, files cloned=3/1

I am unable to understand what does this "MISSING" indicates in the log, can someone please help?

@iPraveenParihar
Copy link
Contributor

I am unable to understand what does this "MISSING" indicates in the log, can someone please help?

MISSING in log messages is occurred when there is mismatch between format specifiers and the arguments provided.
something, like fmt.Printf("key1: %s, key2: %s", "value1") should log MISSING.

In your PR, I don't see that to be the case 🤔. You could verify that once with you latest PR changes?

@Nikhil-Ladha
Copy link
Contributor Author

I am unable to understand what does this "MISSING" indicates in the log, can someone please help?

MISSING in log messages is occurred when there is mismatch between format specifiers and the arguments provided. something, like fmt.Printf("key1: %s, key2: %s", "value1") should log MISSING.

In your PR, I don't see that to be the case 🤔. You could verify that once with you latest PR changes?

The above output is using the latest changes only.

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Oct 28, 2024

/test ci/centos/mini-e2e-helm/k8s-1.31

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from 946609f to f7ff8c3 Compare October 29, 2024 08:38
@mergify mergify bot dismissed Madhu-1’s stale review October 29, 2024 08:39

Pull request has been modified.

Copy link
Contributor

mergify bot commented Nov 6, 2024

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from f7ff8c3 to a91c8aa Compare November 7, 2024 14:36
@Nikhil-Ladha Nikhil-Ladha requested a review from Madhu-1 November 11, 2024 05:24
Madhu-1
Madhu-1 previously approved these changes Nov 11, 2024
Copy link
Contributor

@Rakshith-R Rakshith-R left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just few small nits.
rest looks good to me.

internal/cephfs/core/clone.go Show resolved Hide resolved
Copy link
Contributor

@Rakshith-R Rakshith-R left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦 Sorry,
didn't click add review on this requested change.

Please take a look, small nit.

@Nikhil-Ladha
Copy link
Contributor Author

🤦 Sorry, didn't click add review on this requested change.

Please take a look, small nit.

I don't see the code for which you are suggesting the change, can you please re-highlight?

@mergify mergify bot dismissed Madhu-1’s stale review November 12, 2024 15:04

Pull request has been modified.

Madhu-1
Madhu-1 previously approved these changes Nov 13, 2024
Copy link
Contributor

@iPraveenParihar iPraveenParihar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thanks!

Comment on lines 139 to 140
log.UsefulLog(ctx, err.Error())

return nil, err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
log.UsefulLog(ctx, err.Error())
return nil, err
}
}
log.ErrorLog(ctx, err.Error())
return nil, err

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be logically incorrect to term it as error here.
Since, there is no error as such with the clone, rather it is just in progress and should get complete in sometime. So, I don't think we should log it as an error log.

Copy link
Contributor

@Rakshith-R Rakshith-R Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is returned as a error. I would be fine with using debuglog, we don't use useful log anywhere else.
And your recent change no longer returns err if progress is missing. This needs to be changed as well. Observe the change in curly bracket

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right.
Updated the code now, please take a look.

@Nikhil-Ladha Nikhil-Ladha force-pushed the issue4813/log-clone-progress branch from b9023dc to c5da289 Compare November 21, 2024 13:39
@mergify mergify bot dismissed stale reviews from iPraveenParihar and Madhu-1 November 21, 2024 13:39

Pull request has been modified.

@iPraveenParihar
Copy link
Contributor

@Mergifyio rebase

log cephfs clone progress report during cephfs clone
operation

Signed-off-by: Nikhil-Ladha <[email protected]>
Copy link
Contributor

mergify bot commented Nov 22, 2024

rebase

✅ Branch has been successfully rebased

@iPraveenParihar iPraveenParihar force-pushed the issue4813/log-clone-progress branch from c5da289 to 68b4f9c Compare November 22, 2024 05:55
@iPraveenParihar
Copy link
Contributor

@Mergifyio queue

Copy link
Contributor

mergify bot commented Nov 22, 2024

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 98cf078

@mergify mergify bot added the ok-to-test Label to trigger E2E tests label Nov 22, 2024
@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.30

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.29

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.30

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.29

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.30

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.29

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.31

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-cephfs

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.31

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/upgrade-tests-rbd

@ceph-csi-bot
Copy link
Collaborator

/test ci/centos/mini-e2e/k8s-1.31

@ceph-csi-bot ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Nov 22, 2024
@mergify mergify bot merged commit 98cf078 into ceph:devel Nov 22, 2024
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cephfs Issues related to CephFS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Cephfs] Log clone progress
5 participants