Test Ceph RBD during an OSD outage and recovery

Verify access during an OSD outage

You’ll test whether the replicated RADOS Block Device (RBD) volume stays readable and writable when one object storage daemon (OSD) stops. Run all commands on ceph1 using the mounted volume and checksum baseline from the RBD exercise. You’ll restore the OSD and verify both the original file and a file written during the outage.

Record the healthy state

On ceph1, confirm that the cluster is healthy before introducing a failure:

    

        
        
sudo /usr/local/sbin/cephadm shell -- ceph -s
sudo /usr/local/sbin/cephadm shell -- ceph osd tree
sudo sha256sum -c "$HOME/agi-rbd-baseline.sha256"

    

The checksum check should report OK.

Make sure to reload the helper with source "$HOME/agi-rbd-tools.sh" if this is a new SSH session.

Use rbd_kernel device list to identify the mapping for agi-rbd/demo-volume and set RBD_DEVICE to that path. The cluster should report three OSDs up and in, and all placement groups should be active+clean.

Stop one OSD

Stop the OSD on ceph3 to test loss of one storage daemon while keeping the rest of the cluster running.

First, identify its daemon name:

    

        
        
sudo /usr/local/sbin/cephadm shell -- ceph orch ps --hostname ceph3 --daemon-type osd

    

Copy the daemon name from the NAME column, such as osd.2, and assign it to OSD_NAME:

    

        
        
OSD_NAME=osd.2
sudo /usr/local/sbin/cephadm shell -- ceph orch daemon stop "$OSD_NAME"

    

Inspect the cluster:

    

        
        
sudo /usr/local/sbin/cephadm shell -- ceph -s
sudo /usr/local/sbin/cephadm shell -- ceph osd tree

    

Wait for the OSD tree to report the selected OSD down and the other two up. The health status reports degraded redundancy. Keep the outage brief. Because the pool has three copies and min_size is 2, the mapped RBD volume remains available.

Verify the file and its checksum:

    

        
        
sudo cat /mnt/ceph-rbd/hello.txt
sudo sha256sum -c "$HOME/agi-rbd-baseline.sha256"

    

The baseline check should report OK. Write new data and flush it to storage while the OSD is down:

    

        
        
sudo timeout 30s dd if=/dev/urandom of=/mnt/ceph-rbd/during-outage.bin \
  bs=1M count=8 conv=fsync status=progress

    

Save the checksum of the new file outside the RBD volume, then unmount and remount to check persistence:

    

        
        
sudo sha256sum /mnt/ceph-rbd/during-outage.bin | tee "$HOME/agi-rbd-outage.sha256"
cd "$HOME"
sudo umount /mnt/ceph-rbd
sudo mount "$RBD_DEVICE" /mnt/ceph-rbd
sudo sha256sum -c "$HOME/agi-rbd-baseline.sha256"
sudo sha256sum -c "$HOME/agi-rbd-outage.sha256"

    

Both checks should report OK.

Restore the OSD

Restart the stopped OSD to restore the third copy of the pool’s data:

    

        
        
sudo /usr/local/sbin/cephadm shell -- ceph orch daemon start "$OSD_NAME"
watch sudo /usr/local/sbin/cephadm shell -- ceph -s

    

Ceph detects the returning OSD and repairs any objects that changed while it was unavailable. Stop watch with Ctrl+C after the cluster returns to HEALTH_OK and all placement groups are active+clean.

Verify both files again after recovery:

    

        
        
sudo sha256sum -c "$HOME/agi-rbd-baseline.sha256"
sudo sha256sum -c "$HOME/agi-rbd-outage.sha256"

    

Both checks should report OK.

Clean up the mapped volume

When you’re done, unmount and unmap the RBD image.

Reload the helper first if this is a new SSH session, then identify the current mapping:

    

        
        
source "$HOME/agi-rbd-tools.sh"
rbd_kernel device list

    

Set RBD_DEVICE to the path shown for agi-rbd/demo-volume, then unmount every mount point for that device and unmap it:

    

        
        
RBD_DEVICE=/dev/rbd0
sudo umount -A "$RBD_DEVICE" 2>/dev/null || true
rbd_kernel device unmap "$RBD_DEVICE"

    

Using umount -A with the device path unmounts the device wherever it’s mounted, even if the mount point differs from /mnt/ceph-rbd.

If the unmap reports Device or resource busy, first confirm that the device isn’t mounted and no process holds it open:

    

        
        
findmnt --source "$RBD_DEVICE"
sudo fuser -vm "$RBD_DEVICE"

    

If both commands report nothing, the kernel can still hold a watch on the image, which keeps the mapping busy even with no filesystem mounted. After confirming that the device is unmounted, force the unmap:

    

        
        
rbd_kernel device unmap -o force "$RBD_DEVICE"

    

Confirm that the image is no longer mapped:

    

        
        
rbd_kernel device list

    

The list should no longer show agi-rbd/demo-volume. This leaves the Ceph cluster and RBD image in place for further exploration.

What you’ve accomplished

You’ve built a three-node Ceph cluster on Arm virtual machines, used RBD as Linux block storage, and verified data availability and integrity during an OSD outage and recovery. You’ve learned how replication settings and Controlled Replication Under Scalable Hashing (CRUSH) placement work together to protect data, and how to monitor cluster health.

You’ve also learned about OSD recovery. Protecting against physical-host failure requires spreading the cluster across independent hosts. You can apply the same Ceph concepts to independent Arm servers and define failure domains that match the physical infrastructure.