Read a Ceph health warning before changing the cluster
- Ceph
- storage
- Kubernetes
- observability
Start with the health identifier
HEALTH_WARN is an entry point, not a diagnosis. Ceph exposes specific health identifiers because a down monitor, an OSD near capacity, and a slow storage device call for different investigations. I keep this distinction in mind because storage warnings can quickly turn into broad application symptoms if they are treated too generically.
Run these read-only checks from an administrative environment with access to the intended Ceph cluster:
ceph -s
ceph health detail
ceph osd tree
ceph df
Record the health code, affected daemon or pool, and current client impact. This guide does not assume a particular Ceph deployment operator or toolbox pod.
Separate the failure domains
| Signal | First question |
|---|---|
MON_DOWN | Is a monitor unavailable, and does the cluster retain a majority? |
OSD_DOWN | Is the daemon, host, or network path unavailable? |
OSD_NEARFULL | Which device or pool is approaching its capacity threshold? |
OSD_BACKFILLFULL | Is recovery or rebalancing blocked by available space? |
OSD_FULL | Are writes blocked because a full threshold has been reached? |
MON_CLOCK_SKEW | Are monitor hosts synchronizing their clocks correctly? |
Look for a shared cause before treating several warnings as independent failures. A host or network problem can affect multiple daemons at once.
Relate storage health to the workload
In Kubernetes, storage problems may surface as mounting failures, unavailable volumes, or rising application latency. Connect the Ceph signal with the affected workload rather than assuming every application symptom originates in the storage layer.
Compare the timing of the health warning with workload events and the relevant observability data. Preserve enough context to distinguish capacity pressure from a failed device or an unavailable network path.
Prefer a cause-specific recovery
Restoring a daemon, correcting time synchronization, adding capacity, and changing a threshold are different operations. Select the procedure that matches the actual health identifier and the deployed Ceph version.
Muting a warning changes its visibility; it does not fix the condition. Raising a full threshold is not a substitute for understanding why capacity is exhausted. Validate both the health state and the affected client workload after recovery. The useful outcome is a storage explanation that matches what clients actually experienced.
Source
Ceph health-check reference, reviewed on 13 September 2026. The latest documentation tracks development, so use the release selector for version-specific recovery procedures.
Related articles
LearningsStorage
Ceph on Bare-Metal Kubernetes: How It Actually Works
Understand Ceph on bare-metal Kubernetes through Rook, OSDs, CRUSH placement, CSI provisioning, RBD and CephFS data paths, and storage policies.
LearningsStorage
A snapshot is not a restore plan
Treat volume snapshots as one recovery input and verify application restore, consistency, permissions, and target-cluster compatibility.
RunbooksKubernetes & GitOps
When a rollout stops progressing
Diagnose a stalled Kubernetes deployment by checking rollout state, pod events, readiness, and GitOps recovery paths.