492 questions
0
votes
1
answer
18
views
While uploading a file to Ceph bucket , getting a S3 exception with code 403, after upgrading the s3 java dependency
While uploading a file to Ceph bucket , getting a S3 exception with code 403, after upgrading the s3 java dependency.
Error -
software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status ...
0
votes
0
answers
27
views
libvirt migration failed : unknown protocol 'rbd'
I've been using KVM/Qemu/libvirt/DRBD/Ceph for years but I've just had a strange issue when migrating a VM that has a Ceph RBD volume attached.
I've run this command to migrate the VM :
# virsh ...
- 3,015
0
votes
0
answers
22
views
Ceph erasure coding issue
See I have total host 5, each host holding 24 HDD and each HDD is of size 9.1TiB.
So, a total of 1.2PiB out of which i am getting 700TiB.
I did erasure coding 3+2 and placement group 128.
But, the ...
- 1
0
votes
0
answers
17
views
rook-ceph (rgw) s3 website doesn't redirect to the index.html
The config override:
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-config-override
namespace: rook-ceph
data:
config: |
[global]
osd_pool_default_size = 1
...
- 5,798
0
votes
1
answer
37
views
Resolve certificate issues with Ceph
I've just deployed a three-node Ceph cluster using cephadm. I'm seeing SSL-related errors due to the fact we have an SSL-inspecting firewall/proxy.
The CA is added to all the hosts so they're able to ...
- 148
0
votes
0
answers
13
views
ceph says host is added, but it is not
I have a cephadm-managed ceph-cluster consisting of 2 nodes.
When trying to add a third node, cephadm says it was successful:
main@node01:~$ sudo ceph orch host add node03 10.0.0.155
Added host '...
- 643
0
votes
0
answers
14
views
CRUSH rule resulted in duplicated OSD for PG
My goal is to have primary on a specific host (due to read-replicas not an option for non-RBD), and replicas on any host (including the host already chosen), but not the primary OSD.
My current CRUSH ...
- 4,856
0
votes
0
answers
47
views
Moving Ceph OSD daemons to a new service specification
I had a service specification that assigned all free SSDs to OSDs:
service\_type: osd
service\_id: 34852880
service\_name: 34852880
placement:
host\_pattern: '*'
spec:
data\_devices:
...
- 567
0
votes
1
answer
223
views
Unable to Use Backup Immutability in Veeam with S3 Bucket: Default Retention Not Supported
I’m encountering an issue with configuring backup immutability in Veeam when adding an S3 bucket as a repository. Here’s a summary of the problem:
Issue: After enabling retention lock in compliance ...
0
votes
0
answers
44
views
Ceph storage: erasure code used space calculation
I have an EC pool used for object storage in Ceph. The EC is 3+2 scheme, and the pool has stored 289 TiB.
As I understand, the total storage will be used after doing EC will be:
289 TiB * 5 / 3 = 481 ...
- 54
0
votes
0
answers
84
views
Why doesn't the "norebalance" flag prevent a healthy Ceph cluster from moving data to a newly added osd?
With a clean install of Ceph version 18.2.4, I tried to add a new OSD in a HEALTH_OK cluster without triggering data movement.
Setting the "norebalance" flag didn't prevent ceph to ...
- 175
0
votes
0
answers
84
views
Ceph S3 and Signature Version 4 (SigV4) Support with Mountpoint for S3
I am currently working with a Ceph S3 Storage and using the S3-compatible API to interact with an object storage bucket. I attempted to mount the bucket using Mountpoint for S3 and I followed the same ...
- 21
0
votes
0
answers
103
views
ceph orchestrator unavailbale after ceph upgrade
I was trying to issue ceph upgrade from 17 to 18.2.4, as outlined here
ceph orch upgrade start --ceph-version 18.2.4
Initiating upgrade to quay.io/ceph/ceph:v18.2.4
After this, however, the ...
- 243
3
votes
1
answer
251
views
How to use fake s3 bucket
I tried to use aiobotocore for async work with Ceph, but i have no access to real bucket, then i need to mock s3 bucket to create a fake one
I'm trying to use moto but it anyway want to connect a real ...
- 31
0
votes
3
answers
877
views
Ceph faild to add osd node to a new ceph cluster (Error EINVAL: Traceback (most recent call last))
I'm deploying a Ceph cluster with 3 nodes on Ubuntu 24.04. Everything works fine until I add OSD nodes to the cluster.
Here are my steps:
First of all, unfortunately, I encountered a bug while ...
0
votes
0
answers
23
views
list role working but create role not working with aws cli for ceph
I am using the below command to list roles and its working
$ aws iam list-roles --endpoint-url https://xxxs.com
but when I try to create a role, it gives an error.
$ aws iam create-role --role-name ...
- 1,355
0
votes
1
answer
350
views
Is it normal for a ceph orch osd rm drain to take so long? [closed]
I tried to drain the host by running
sudo ceph orch host drain node-three
But it stuck at removing osd with the below status
node-one@node-one:~$ sudo ceph orch osd rm status
OSD HOST STATE ...
- 1,614
0
votes
0
answers
55
views
how to create a ceph notification and receive object's changes?
I tried to get Ceph's message notification API, but found that it couldn't be turned on, so I ignored it, and then created a topic, created a rabbitmq message notification, but still couldn't receive ...
-1
votes
1
answer
92
views
CEPH storage can be used via RGW service or directly by librados. Which one is better to use from microservice? [closed]
I am planning to use CEPH as storage of video files. RHEL CEPH provides options to store directly using librados or using RGW. I am curious to know which implementation is used more in the industry. ...
- 81
0
votes
1
answer
398
views
Ceph Maximum number of replication
I am studying about Replication in Ceph.
I can't find about max number of copies in replication.
I guess maximum value depends on size of object, but I want to know about max value approximately.
I ...
- 1
0
votes
0
answers
65
views
Error with DomainAttachDevice method from go-libvirt library: "No secret with id 'libvirt-11-format-encryption-secret0'"
I'm integrating the digitalocean's go-libvirt library into my Ubuntu environment alongside Microceph. However, when attempting to use DomainAttachDevice, I encounter the error
unable to execute QEMU ...
- 83
0
votes
0
answers
204
views
Unable to delete multiple files via s3cmd (Ceph)
When attempting to delete multiple files in a Ceph bucket, via the s3cmd cli the following error is presented:
s3cmd rm --recursive --force s3://search-backup/
ERROR: S3 error: 403 (AccessDenied)
...
- 980
-1
votes
1
answer
167
views
How to use SCP and AWS to share data from remote Ceph (S3) and a remote standard filesystem?
I need to transfer data between machine A (linux filesystem) and machine B (Ceph/S3 object filesystem) from machine C (the only one I can control). On machine C, I installed aws cli for my transfers.
...
- 41
3
votes
1
answer
1k
views
How to format Ceph S3 bucket-policy Principal?
I'm trying to set the bucket policy for a Ceph S3 bucket.
This policy works but enables public access
{
"Version": "2012-10-17",
"Statement": [
{
...
- 31.9k
1
vote
1
answer
90
views
I'm trying to extract a string from json using jq but got an error "cannot index string with string "num_objects"
My json snippet is as follows
{
"bucket": "sample-bucket",
"tenant": "",
"num_objects": ...
- 13
1
vote
0
answers
439
views
How to allow access to Ceph/S3 objects between multiple users?
We have a Ceph filesystem we're accessing via Ceph/S3 interface. We have a bucket owner user user-00 and multiple individual user accounts user-01, user-02, etc.
When user-02 uploads a file we're ...
- 31.9k
0
votes
1
answer
125
views
Ceph And Subuser
Can Bucket Quota and User Quota be specified for Ceph SubUser?
Bucket Quota settings did not work like normal user methods.
I used the Bucket Quota and User Quota setting method of Ceph official ...
0
votes
2
answers
149
views
Is it possible for anonymous user to upload on S3 storage to object which has public-read-write ACL without presigned URL?
I am setting object with public-read-write ACL with following code in python:
o = s3.Object(bucket, key)
o.Acl().put(ACL='public-read-write')
I can now access object on https://hostname/tenant:bucket/...
- 825
0
votes
2
answers
231
views
Is it possible for anonymous user to get object metadata (etag) without "full request"?
I am setting object with public-read-write ACL with following code:
o = s3.Object(bucket, key)
o.Acl().put(ACL='public-read-write')
I can now access object on https://hostname/tenant:bucket/key ...
- 825
0
votes
1
answer
769
views
What mistake did I make in setting up the iSCSI Target Attach?
I built an iscsi gateway using ceph-iscsi and set up an iSCSI target and an initiator that can attach to the target.
And as shown in the figure below, I entered the iSCSI Initiator and Target ...
0
votes
1
answer
84
views
s3 object storage bucket list operation implementation mechanism, how to ensure performance
How can the file list of s3 buckets be stored to ensure the performance of the list and fast data recovery.How does the usual Internet solve this problem, and what are the general solutions.Is ...
- 23
0
votes
1
answer
285
views
Is it dangerous for a cloned image to be flattened while being used as a volume?
I am currently learning about Ceph.
I found an interesting feature called flatten in the official documentation, and I have a question about this feature, so I'm asking.
Assuming that Openstack's ...
- 3
1
vote
1
answer
582
views
Spark simple query to Ceph cluster - Unable to execute HTTP request: Unsupported or unrecognized SSL message
I feel I'm missing something trivial here, please help.
Solution is based on these articles:
Theory: https://www.redhat.com/en/blog/why-spark-ceph-part-1-3
Implementation: https://radanalytics.io/...
0
votes
1
answer
315
views
Custom endpoint for `ros3` driver in `h5py`
h5py supports the native S3 driver for HDF5 (the ros3 driver). We've enabled this with a local build of HDF5.
>>> import h5py
>>>
>>> print(f'Registered drivers: {h5py....
- 31.9k
0
votes
1
answer
3k
views
Ceph: fix active+undersized+degraded pgs after removing an osd?
I can't find clear information anywhere. How to make Ceph cluster healthy again after osd removing?
I just removed one of the 4 osd. Deleted as in manual.
kubectl -n rook-ceph scale deployment rook-...
- 2,511
0
votes
1
answer
670
views
ceph cluster break down after power failure, how can I save my data?
I have a 4 node ceph cluster deployed with cephadm on rocky9.0, ceph version 17.2.5. podman version 4.1.1
The entire cluster was broke down after an unexpected power failure.
cluster conf
# minimal ...
- 1
1
vote
1
answer
218
views
Ceph Rados : Facing errors during dependencies installation on Ubuntu
I am getting few errors during dependencies installation step in the install-deps.sh script.
Building dependency tree
Reading state information... Done
E: Unable to locate package ceph-libboost-...
0
votes
1
answer
145
views
wget -c --content-disposition results in 403 forbidden
Downloading a file from ceph object store using presigned url results in 403 error when options -c and --content-disposiiton are used.
Debug output shows it requests a HEAD which is why it fails, but ...
0
votes
1
answer
675
views
Forcing the removal of a ceph OSD
Our current situation is as follow: We do have a server that was used as a ceph OSD. The disks of this server crashed and were replace by brand new disks. Then the machine got reinstalled without the ...
- 2,143
1
vote
1
answer
1k
views
Need help to setup a crush rule in ceph for ssd and hdd osd
we are having a proxmox cluster with 3 nodes. Each node have 4 ssd and 12 hdd.
My plan is to create 2 crush rules (one for ssd devices and another one for hdd devices).
With these 2 rules I will ...
- 99
1
vote
1
answer
154
views
Is there solution for viewing chunks distribution in ceph's pools with erasure code?
According to the picture from this page ceph splits objects on chunks and writes them on osds.
For viewing objects distribution with replication algorithm I can use commands like ceph pg dump or ceph ...
- 13
0
votes
1
answer
618
views
Trying to mount Kubernetes volume to ceph rbd storage but getting FailedMount error
Showing FailedMount error while mounting Kubernetes volume to ceph storage rbd
Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-mxlkm]: timed out ...
3
votes
0
answers
1k
views
AWS S3 using Ceph how to provide a url for an object
I'm starting to work with AWS SDK S3 with Ceph storage platform,
and my server needs to provide a URL to some large objects without getting the objects themselves, but only a URL to access the object.
...
- 992
0
votes
0
answers
299
views
Flink S3 Unknown Host Exception
I have a flink Job that's run on flink environment we deployed with helm on kubernetes cluster.
The job uses rocksdb s3(ceph) state backend.
But every once in a while the pod can't reach the host name ...
1
vote
1
answer
218
views
How to configure artifactory CEPH S3 with Direct Upload Mechanism?
Artifactory: 7.38.8
It works fine with the Eventual Upload Mechanism. I try to change it to the Direct Upload Mechanism, but I cannot upload any file.
My binarystore.xml configuration:
<config ...
- 23
0
votes
1
answer
277
views
Create Bucket Ceph with Spring
I am not sure how to create a bucket Ceph with Spring.
Are there any documents I can prefer?
How to get this accessKey and secretKey?
String accessKey = "insert your access key here!";
...
user19709202
0
votes
1
answer
462
views
how to seperate traffic in Rook
AFAIK ceph have 2 specific trafic path:
Traffic between client and ceph nodes,
Traffic between ceph nodes (Inter Ceph-node).
So, let say my network is like this
Note :
Kube-node-4 is a kubernet ...
- 605
0
votes
2
answers
4k
views
How to get better performace in ProxmoxVE + CEPH cluster
We have been running ProxmoxVE since 5.0 (now in 6.4-15) and we noticed a decay in performance whenever there is some heavy reading/writing.
We have 9 nodes, 7 with CEPH and 56 OSDs (8 on each node). ...
1
vote
1
answer
640
views
PySpark write data to Ceph returns 400 Bad Request
I have a problem with pySpark configuration when writing data inside a ceph bucket.
With the following Python code snippet I can read data from the Ceph bucket but when I try to write inside the ...
- 372
0
votes
1
answer
766
views
Ceph gateway for Azure blob support
Does ceph gateway support Azure blob storage?
In below link docs, they mentioned only S3 and Swift API supported. Can we use for azure blob?
https://docs.ceph.com/en/quincy/radosgw/index.html
Or any ...
- 647