Questions tagged [ceph]
Ceph is a free software storage platform designed to present object, block, and file storage from a single distributed computer cluster.
216 questions
0
votes
0
answers
72
views
Ceph High Latency
I am seeking assistance with a challenging issue related to Ceph that has significantly impacted the company I work for.
Our company has been operating a cluster with three nodes hosted in a data ...
0
votes
1
answer
101
views
cephadm does not add mon-daemon after add host
I have a cephadm-managed ceph-cluster consisting of 2 nodes.
When trying to add a third node, cephadm says it was successful:
main@node01:~$ sudo ceph orch host add node03 10.0.0.155
Added host '...
- 101
0
votes
0
answers
32
views
Auto-Create multiple RBD-pools using rook-ceph possible?
Maybe somebody here can help me out - the microk8s / rook-ceph docs don't really mention this properly: does microk8s connect-external-ceph support multiple pools?
I have created 2 pools on my ...
- 101
1
vote
0
answers
41
views
How to setup Ceph with S3 Bucket Encryption using Hashicorp Vault with Ansible?
I stood up a 3-node HA Vault cluster via Ansible playbook I designed. I have a 3-node Ceph cluster environment (Reef) up and running using cephadm (https://computingforgeeks.com/install-ceph-storage-...
- 13
0
votes
1
answer
70
views
Moving Ceph OSD daemons to a new service specification
I had a service specification that assigned all free SSDs to OSDs:
service\_type: osd
service\_id: dashboard-tintin-7634852880
service\_name: osd.dashboard-tintin-7634852880
placement:
host\...
- 195
0
votes
1
answer
54
views
Ceph encountered problems after running "ceph dashboard iscsi-gateway-add -i 1.conf images"
I have correctly configured 3 storage nodes and 1 c
ontrol node (the control node has storage node services), as well as correctly configured iSCI storage. However, I have noticed that in order to ...
- 23
0
votes
0
answers
87
views
Writes to ceph rbd devices mapped on multiple nodes do not propagate until unmap
I am running an OpenShift environment backed by a Ceph service providing RBD block devices. When the same volume is mapped to multiple nodes (e.g., when using block volume mode), I'm seeing unexpected ...
- 46.8k
0
votes
1
answer
233
views
ceph orchestrator unavailbale after ceph upgrade
I was trying to issue ceph upgrade from 17 to 18.2.4, as outlined here
ceph orch upgrade start --ceph-version 18.2.4
Initiating upgrade to quay.io/ceph/ceph:v18.2.4
After this, however, the ...
- 151
0
votes
1
answer
51
views
Ceph dashboard show image usage only for static provisioned volumes
I have a simple question. Why Ceph Dashboard => Block => Images in column "Usage" does not show the size for dynamically created Persistent Volume?
If I create a static volume then, ...
- 123
1
vote
1
answer
175
views
Ceph Manager Daemon: SMB Module is not available
I currently have a Ceph Cluster in version 18.2.2 (reef). It runs an NFS-Ganesha cluster with some NFS exports. I wanted to deploy an SMB cluster (as documented at https://docs.ceph.com/en/latest/mgr/...
- 11
0
votes
1
answer
98
views
Rook-ceph ceph:v18.2.2 mon pods failing in loop
rook-ceph mons failling rook v1.13
debug 2024-07-01T15:25:42.499+0000 7f43a7779700 1 mon.d@1(peon).paxos(paxos active c 2794..3317) lease_timeout -- calling new election
debug 2024-07-01T15:25:42.500+...
- 111
1
vote
2
answers
250
views
Rook Ceph storage and MySQL space very big difference
Can someone please advise me what could be the cause of this condition?
I have a Rook Ceph cluster on which MySQL database with 3x replication is stored. This database is used by me for development ...
- 123
1
vote
1
answer
130
views
CephFS snapshot mirroring: how does the snapshot sizes inside .snap work exactly?
we're currently experimenting with CephFS snapshot mirroring to evaluate whether it will serve our purposes of establishing a passive storage backup to a secondary site. We setup two CEPH clusters ...
- 41
0
votes
1
answer
148
views
Extent ceph storage volume size
have installed ceph storage clustering on three nodes that first node is bootstrap. also i created a pool and then a created volume with 20GB size and then a linux client added to this network that ...
- 1
0
votes
1
answer
134
views
Rook-ceph. Blok Storage. FailedAttachVolume. Pod of sample is not started
I installed ceph cluster in kuber according to the instructions https://rook.io/docs/rook/v1.12/Getting-Started/intro/
on 3 nodes
wn-1 - /dev/sdb 200 GB HDD
wn-2 - /dev/sdb 200 GB SSD
wn-3 - /dev/...
0
votes
1
answer
312
views
Ceph - Too many objects () are misplaced;
I'm currently running a 3-Node Hyperconverged Proxmox/Ceph cluster. I'm in the process of transferring a large amount of data (100TB+) from an old unRAID instance to the new cluster infrastructure. ...
0
votes
0
answers
335
views
Ceph OSD rebalancing/redistribution of data
I'm new to ceph and I'm experimenting to find out, how it works. I started with a 3 node cluster where each node got one OSD (2x 500GB, 1x 1TB). I created the pool and the filesystem, mounted the ...
- 83
0
votes
0
answers
320
views
Proxmox Network Interfaces Setup for Cluster
I was curious if the experts here could give some feedback on preferred /etc/network/interfaces config for Proxmox VE 8 with 4 available NICs. I plan to use Clustering with Corosync and Ceph.
I have ...
0
votes
0
answers
47
views
There is a Base Clock difference by service in ubuntu 20.04
Currently using Intel Xeon 4216, 6226R CPU in bare metal.
Base clocks of 2.1GHz and 2.9GHz are normal, but only 800MHz and 1200MHz are confirmed as the lowest clocks.
For servers used by other ...
- 1
3
votes
0
answers
156
views
CEPH Quincy installation with multipathd enabled
we have a setup comprising of 6 servers hosting CentOS 8 Minimal Installation with CEPH Quincy version 18.2.2 supported by 20Gbps fiber optics NICs and a dual Xeon Intel processors, bootstrapped the ...
- 41
2
votes
1
answer
1k
views
Ceph cluster mixing nvme disks with different sizes
I am deploying a ceph cluster, the cluster will have three controller servers and 27 osd nodes.
Each osd node has 3x3.8Tb nvme + 1x1.9Tb nvme disks, for a total of 4 nvme disks per node.
The failure ...
- 81
0
votes
0
answers
98
views
Ceph s3 LDAP not working despite following instructions
I have a problem with LDAP configuration in Ceph. I have a fresh cluster (for testing purposes). No operation has been performed on it except adding certificates to Dashborad and Grafana and issuing . ...
- 11
0
votes
1
answer
330
views
How to enable dm-crypt on Ceph OSD with cephadm?
I had originally used the following website (https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-servers/) to stand up a Ceph cluster on Ubuntu 22.04 with Cephadm and Ansible. ...
- 13
0
votes
0
answers
609
views
Setting Read-Only Access On Ceph Rgw Bucket For Another User On Same Tenant Not Working
Two users can't see to see each other's buckets on Ceph Reef.
First, I create the users on a ceph admin vm:
radosgw-admin user create --tenant=homelab --uid=user1 --display-name="First User" ...
- 101
2
votes
1
answer
397
views
Where does CEPH store device metrics for each device? [ceph reef]
I just deployed my first ceph cluster and have been trying multiple things so far.
Though, I couldn't understand how ceph store metric for the devices.
I can easily access those through:
# ceph device ...
- 51
0
votes
1
answer
217
views
Completely Reprovisioning Ceph Rgw Without Reprovisioning Ceph Cluster
I recently began learning about Ceph and setup my own ansible scripts to setup a Ceph cluster (yes, I'm aware that cephadm-ansible exists, but I want to get comfortable managing ceph).
Initially, I ...
- 101
0
votes
0
answers
128
views
why rook cephfs storage class store pv data in 2 places?
I am a newbie to ceph and run ceph using rook in bare metal environment.my question is :
when you use rook cephfs provisioner as storage class and create a pvc to assign to a pod , when the pv is ...
- 101
1
vote
0
answers
885
views
Deleting file in ceph does not free up disk space
Ceph version: 16.2.13 (pacific) (I know that pacific is deprecated but the whole environment is legacy (like centos 7.3). I am not authorized to upgrade as well.) The cluster has 6 servers (22 osds, ...
0
votes
0
answers
125
views
Ceph rbd commands are hanging forever when executed on remote server
I have an openstack compute node connected to ceph cluster with 10G. The compute can ping and reach the all ceph nodes and port 3300 but somehow rbd commands are hanging which is stopping the compute ...
0
votes
0
answers
217
views
openstack fails to create an instance after integrating with rbd ceph
I have configured openstack kolla to enable ceph for glance, cinder and nova. I must say that cinder and glance are working perfectly fine with ceph. Nova on the other hand refuses to work properly. ...
0
votes
0
answers
110
views
openstack nova goes down when ceph pool is set to have more than 1 pgs
I have openstack connected to ceph cluster.
When i increase the pgs of the vms pool to be more than 1 and launch instance in openstack the instance is stuck on spawning state and nova compute ...
- 115
0
votes
1
answer
1k
views
Can I create buckets in ceph object storage using a command-line interface?
Is it possible to create buckets in the ceph object store via the rados gateway from a CLI? I am aware of the radosgw-admin tool, however, it doesn't appear to have this ability. It appears mainly to ...
- 29
0
votes
0
answers
22
views
cephfs mount: can't open files with LibreOffice
I'm able to open/write different relevant filetypes with other applications (vim,code,gedit,...) from remote cephfs storage.
libreoffice has been configured to prevent file-locks.
All dirs and files ...
- 21
0
votes
1
answer
170
views
ceph snapshots are not running
Our ceph cluster (quincy 17.2.7) runs in a docker container. The snapshot schedule was created as:
ceph fs snap-schedule add / 1h 2023-12-13T09:00:00 nerfcomputing2024
and retention:
ceph fs snap-...
- 1
0
votes
0
answers
645
views
RADOS permission denied
I'm trying to install a ceph cluster using 3 ubuntu 20.04 machine each with 4 disk.
I've followed the manual installation from here https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-...
0
votes
1
answer
319
views
how number of replicas is aligned with crush rule in a ceph cluster
I have some difficulties to understand whether the crush rule is aligned with the number of replicas. So, imagine I create a crush rule named replicate_in_2_racks as follows:
ceph osd crush rule ...
- 1
0
votes
2
answers
928
views
how to get the used space (size in bytes) of a directory in a ceph file system
I have a ceph cluster quincy 17.2.7
I wonder if ceph has some tools to quickly get the the used space (size in bytes) of a directory in a ceph file system. I know one can get that using du -hs /fs/dir-...
- 1
0
votes
0
answers
985
views
Unable to mount CephFS on client - Rocky8.8
When attempting to mount a Ceph file system the mount command times out:
mount error 110 = Connection timed out
dmesg shows the following:
ceph: device name is missing path (no : separator in splunk@$...
- 101
0
votes
1
answer
83
views
ceph remove spurious data
I’d like to remove some “spurious" data:
root@XX03:/# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.0 PiB 1.0 PiB 47 GiB 47 GiB 0
TOTAL ...
- 1
0
votes
0
answers
387
views
Ceph mounted rbd poor performance
all tests are done from an external server connected with 10G to all 3 ceph nodes and the cluster is all nvme flash.
when i ran fio with this config:
[global]
ioengine=rbd
clientname=admin
pool=...
0
votes
1
answer
845
views
Now that the in-tree Kubernetes CephFS persistent volume plugin is deprecated how can I use Rook to get the same mounting behavior?
As of Kubernetes v1.28 the CephFS in-tree persistent volume plugin is deprecated. The official suggestion is to now instead use the third-party CSI driver. However, I am not sure how I can use for ...
- 29
1
vote
1
answer
623
views
Kubernetes Rook-Ceph: different pools for different use-cases?
I'm experimenting with rook-ceph on kubernetes for storage again. I have 3 nodes, which each have
a local ssd-partition mounted at /dev/sda2, and
a NAS-volume at /dev/sdb.
In my rook-config, I use ...
- 458
0
votes
2
answers
238
views
how to use the images that have been create by "Offload rbd’s copy_volume_to_image" method in openstack?
I have an openstack integrated with ceph.
I'm working on the fastest way to create a copy of an instance's volume without using cinder_backup module.
in the Ceph it is possible to access the rbd files ...
- 3
0
votes
1
answer
354
views
extract openstack instances' volume from ceph
I have an OpenStack-kolla environment with the Ceph backend.
I've configured my cinder backend with NFS. I can create a Backup with Cinder from all instances and store them on NFS. In this case, I don'...
- 3
1
vote
1
answer
463
views
Ceph: Rados Gateway web endpoint not responding
I'm creating a simple Ceph cluster and trying to connect to Ceph Gateway.
This is ceph status output of my Ceph cluster:
cluster:
id: a7f64266-0894-4f1e-a635-d0aeaca0e993
health: ...
- 11
0
votes
1
answer
913
views
kolla-ansible openstack bootstrap-servers failed on enabling ceph apt repository (ubuntu 22.04)
This is to deploy the recent kolla-ansible openstack. The problem happens since last week. I retested it on a good environment successfully deployed on 7/14/2023 and got bootstrap-servers failed ...
- 11
0
votes
0
answers
448
views
Ceph's failure to automount after network failure
I'm having some problems with the mounting of a ceph-cluster on debian machines, don't know if I'm doing something wrong, if it's a version problem or anything else.
I'm using the ceph cluster from ...
- 155
0
votes
0
answers
235
views
ceph - is a drive mapping change recoverable /dev/sdi /dev/sdj
I was testing drive hot plugging on my server, and unplugged one of the drives, then plugged it back in. Though, when plugged back in the drive letter changed from /dev/sdj to /dev/sdl. Now ceph will ...
- 111
0
votes
1
answer
3k
views
ceph "permission denied" when adding storage OSD
I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the ...
3
votes
1
answer
184
views
Pooled storage with varying redundancy per file system
I have some files that I want stored mirrored. I have some files that I only need single copies (i.e. scratch data, easily regenerated data, etc). And I have some files that are so critical, I want it ...
- 214