Ceph on Bare-Metal Kubernetes: How It Actually Works

1. The Baseline: What Ceph Is, and Why It Belongs on Bare Metal

Ceph is a distributed storage engine that takes independent disks spread across independent machines and presents them as one storage pool. There is no storage “server” in the classic sense. Three properties define it:

No central lookup. Clients compute where data lives using a deterministic algorithm (CRUSH), instead of asking a metadata server per request.
Replication by default. Every object is written to multiple OSDs on multiple hosts. The failure domain is a host, a rack, or whatever your CRUSH rules say.
One engine, many interfaces. Block (RBD), shared filesystem (CephFS), and object (S3 via RGW) are all thin layers over the same replicated object store (RADOS).

Why run it on bare-metal Kubernetes?

On-prem has no managed block storage. You own chassis with real disks in them; Ceph turns those disks into an EBS-equivalent.
Rook makes those disks a managed Kubernetes resource: disk inventory, daemon deployment, and upgrades become reconciliation loops instead of SSH sessions.
The volume client (kernel RBD driver) runs in the same host kernel as your pods. On bare metal, a pod’s write reaches physical platters in as few hops as physics allows: pod → kernel → network → OSD.

The capacity math you must internalize: with 3× replication, usable capacity is raw / 3. The reference cluster for this article runs 7 HDDs, one per storage host, ~0.47 TiB each → 3.3 TiB raw → ~1.1 TiB usable. Every “we’re out of space” conversation starts with that division.

The trade-off you accept: storage traffic shares the node’s NIC and kernel with your workloads, and every write waits for the slowest of three disks. This is fine for most workloads; it is a design constraint you must respect, not a defect.

2. The Control Plane: Rook

Rook is a Kubernetes operator — a control loop with admin credentials. It translates Custom Resources into running software:

CephCluster → MON, MGR, and OSD daemons
CephBlockPool → a RADOS pool with size/replication settings
CephFilesystem → MDS daemons + metadata/data pools
CephObjectStore → RGW daemons + object pools

What Rook does:

Bootstraps the cluster: generates the FSID, monmap, and admin keyring on first reconcile.
Runs disk discovery (rook-discover DaemonSet) and one-shot osd-prepare jobs that zap candidate devices, format them with BlueStore, and enroll each OSD into the CRUSH map.
Deploys every Ceph daemon as a Deployment or DaemonSet, and keeps them running: restarts, upgrades, node cleanup.
Deploys and configures the CSI drivers (provisioner + node plugins) with cephx credentials.
Reports cluster health into CR status conditions and Prometheus.

What Rook does NOT do:

It is not in the data path. Once a volume is mapped and mounted, pods do I/O through the kernel client directly to OSDs. You could delete the operator pod and mounted volumes would keep serving reads and writes indefinitely.
It does not place objects, replicate data, or serve I/O — Ceph daemons do that.
It does not create PersistentVolumes — the CSI provisioner sidecars do that.

Correct mental model: Rook is day-0/day-2 lifecycle management. CSI is volume lifecycle. Ceph daemons are the storage itself. All three are separable at runtime.

A detail from the reference cluster worth copying: the operator, MONs, and RGWs run on hosts that contribute no OSDs. Control-plane daemons don’t compete with OSDs for disk I/O, and a storage host failure doesn’t threaten quorum or the S3 endpoint.

3. The Storage Engine: OSD, MON, MGR

OSD — one daemon, one disk

An OSD (Object Storage Daemon) owns exactly one raw block device. That’s the ratio: 1 OSD = 1 disk. The reference cluster’s seven HDDs are seven OSDs, each on a different host — which is precisely why its pools use failureDomain: host with size: 3: any object survives the loss of any two hosts.

Inside an OSD:

BlueStore is the storage engine. Object data is written directly to the raw device — no intervening filesystem. BlueStore manages its own space, checksums, and compression.
RocksDB (via BlueFS) holds the metadata: object keys, extent maps, stats. RocksDB runs on a WAL/DB device that can be a separate, faster disk — this matters, see section 5.
Each OSD acts as primary for some placement groups and replica for others. The primary receives client writes, persists locally, forwards to peer replicas, and only then acknowledges.
OSDs heartbeat each other constantly and report failures to the MONs. The MONs mark an OSD down only after enough independent reports arrive — one flaky peer can’t evict a healthy OSD.

MON — the map, not the I/O

The MON maintains the cluster map: the authoritative state of what exists (monmap, osdmap, CRUSH map, PG map, MDS map) and who may talk to it (auth). Three MONs run a Paxos quorum — 3 tolerates 1 failure; there is no such thing as 2 MONs.

Key facts:

Clients fetch the map once, cache it, and subscribe to incremental updates. The MON is not consulted per I/O. Its failure slows map distribution and blocks management operations; it does not sit between your pod and its data.
MON quorum loss doesn’t immediately stop in-flight I/O — clients keep serving with the last known map — but the control plane is frozen and nothing new can be placed.
MON placement deserves intent: on the reference cluster, the three MONs sit on non-storage hosts, so a disk/node failure in the storage tier can’t shrink quorum.

MGR — telemetry and automation

The MGR is an active/standby pair that holds no data and serves no client I/O. The active one runs the modules that make Ceph operable:

prometheus exporter — per-OSD/per-pool metrics (consumed by a per-node exporter DaemonSet)
dashboard — the web UI
pg_autoscaler — keeps PG counts sane across pools
crash — aggregates daemon crash dumps from the per-node crash collectors

Standby takes over in seconds if the active dies. MGR crashes are annoying — a health warning — but invisible to workloads.

Honorable mentions: the MDS pair (active + hot standby) serves CephFS metadata only — file data never traverses the MDS, clients go straight to OSDs for reads/writes. The RGW pair fronts the object store; unlike everything else in this section, RGW is in its data path (it proxies S3 requests to RADOS via librados).

4. The Data Path: Tracing One Write

This is the section that matters. A volume has two distinct lives: a setup phase that happens once, and a data path that runs on every write().

Phase A — Provisioning and mount (once, at pod start)

Pod spec → PVC → K8s API
  → CSI provisioner deployment (sidecar watching PVC events)
      → creates RBD image in the pool (via cephx creds from Secret)
      → returns PV with a VolumeHandle
  → kubelet on the target node
      → gRPC "NodeStageVolume" to the CSI node plugin (privileged DaemonSet pod)
          → rbd map → kernel RBD driver creates /dev/rbdX on the HOST
          → mkfs.ext4/xfs (filesystem mode) or raw device (block mode)
      → "NodePublishVolume" → bind-mount into the pod's filesystem namespace

Participants: the external-provisioner sidecar in the CSI provisioner deployment, kubelet, and the CSI node plugin DaemonSet. Two different pieces of software — the provisioner deployment creates volumes against the Ceph API; the node plugin performs attach/format/bind-mount on each bare-metal node. The provisioner never touches node I/O; the node plugin never creates volumes.

Note what RBD “attach” means here: nothing is attached in the SCSI sense. The volume is network-attached — mapping it just tells the host kernel’s rbd module to open the object set over the network.

After the mount completes, the node plugin is done. For RBD it exits the data path entirely.

Phase B — The write itself (every single time)

1. App calls write(fd, buf)
2. ext4/xfs (filesystem mode) → page cache / direct I/O
3. Kernel RBD driver (krbd) — host kernel, NOT a container
4. krbd computes placement LOCALLY:
     hash(object) mod pg_num  →  Placement Group (PG)
     CRUSH(PG, osdmap)        →  ordered OSD list [primary, replica, replica]
5. One message to the primary OSD over the network
6. Primary writes to its BlueStore disk AND forwards to 2 peer OSDs on other hosts
7. Both replicas persist + ack
8. Primary acks the client → write() returns

Who is NOT in this path:

The MON — map was cached at mount time
Rook — reconciles CRs, doesn’t move bytes
The K8s API — kubelet did its job at mount time
The CSI provisioner — volume already exists
The CSI node plugin pod — the kernel driver owns the device

That is the entire point of the architecture: every component that “manages” the volume is out of the steady-state path. A pod’s write is: local kernel → primary OSD → two replicas. Three network hops, three disk flushes.

The latency budget: your write latency ≈ network RTT (×3) + the slowest of three disk flushes. On HDD-backed OSDs, rotational latency dominates: single-digit-to-tens of milliseconds per sync write is normal, and that’s before BlueStore metadata overhead. Throughput scales with PG spread; latency does not improve with more HDDs — it’s bounded by the platter.

The CephFS variant: same RADOS destination, different client. The node plugin mounts the filesystem with ceph-fuse (a userspace daemon inside the plugin pod) and bind-mounts it into the app pod. So for CephFS, unlike RBD, the plugin pod is in the data path — data flows app → FUSE → OSDs, metadata ops go to the active MDS, file data still goes directly to OSDs. The trade: shared RWX semantics and multi-pod access, at the cost of a userspace hop on every operation.

5. Real-World Grounding: What This Looks Like on a Real Cluster

The reference topology: 13 bare-metal nodes. Seven are storage hosts contributing exactly one HDD OSD each (~0.47 TiB, all BlueStore). Three are storage-control hosts running MONs (quorum of 3), the RGW pair, and the operator — no OSDs. One is the Kubernetes control-plane node running no Ceph daemons at all. One is an edge node running only the CSI RBD plugin, so pods scheduled there can still mount RBD volumes. Twelve pools, ~170 placement groups, all active+clean, serving roughly 95 bound PVCs.

Storage class design tells you the workload tiering:

A general RBD class — Delete reclaim, immediate binding, expansion enabled. App state (message queues, databases, KV stores, log stores, container registries) lives here. Delete the PVC, the RBD image is garbage-collected.
A database RBD class — Retain reclaim, WaitForFirstConsumer binding, expansion enabled. Retain means an admin must delete the underlying image manually — a dropped PVC on a database does not destroy the data. WaitForFirstConsumer defers volume creation until the scheduler has placed the pod.
A CephFS class — for the couple of workloads needing ReadWriteMany shared mounts.
A bucket class — provisions S3 buckets as ObjectBucketClaims for object-native consumers.

Two RBD classes against the same pool, differentiated only by policy — that’s the pattern to copy, not a quirk.

Reading the health warnings — a masterclass in HDD-only Ceph:

“4 OSDs experiencing stalled reads in db device of BlueFS” and “1 OSD with slow operations in BlueStore”: BlueStore’s RocksDB WAL/DB lives on the same HDD as the data. Metadata compactions and client I/O fight over one spindle. The fix is the standard one: give OSDs a small NVMe/SSD as a dedicated block.db/block.wal device. One fast device shared across that host’s OSDs is enough — RocksDB metadata traffic is small but latency-sensitive.
“mon is low on available space”: MONs write small amounts of persistent state constantly. A control host with a tiny root disk will warn; it’s cosmetic until it isn’t.
“2 mgr modules recently crashed”: restart the loop and check the crash collector’s dumps. The MGR’s absence never touched the data path — workloads didn’t notice, which is the design working.

Operational rules this cluster teaches:

One disk per host → failure domain host is not a choice, it’s arithmetic. With all replicas needing distinct hosts, capacity headroom is hosts - 2, not disks.
3× replication on 7 hosts: you lose ~2/3 of raw capacity. Erasure coding would change the math but raises latency on HDDs.
Expansion is real-time: allowVolumeExpansion: true on the RBD classes means the resizer sidecar grows the RBD image in-place; growing the pool itself means adding hosts (the actual hard part on bare metal).

The summary worth keeping: Rook, the API server, the provisioner, and the plugins are control plane — they create, mount, and heal, then get out of the way. Ceph clients and OSDs are data plane — they compute placement locally and talk peer-to-peer. A pod writing to a PVC on bare-metal Kubernetes is, after the mount, indistinguishable from a bare-metal process writing to a replicated disk. That’s the whole trick.

Ceph architecture for RADOS, daemon responsibilities, placement, and client interfaces.
Rook Ceph documentation for Kubernetes storage management and operator responsibilities.
Ceph CSI for RBD and CephFS volume provisioning and node plugins.
Kubernetes StorageClasses for reclaim policy, binding mode, and volume expansion.
Read a Ceph health warning before changing the cluster for health-identifier-driven diagnosis.
A snapshot is not a restore plan for recovery inputs and restore verification.