Skip to content
EN DE
Back to Knowledge
RunbooksPublished

Read a Ceph health warning before changing the cluster

  • Ceph
  • storage
  • Kubernetes
  • observability

Start with the health identifier

HEALTH_WARN is an entry point, not a diagnosis. Ceph exposes specific health identifiers because a down monitor, an OSD near capacity, and a slow storage device call for different investigations. I keep this distinction in mind because storage warnings can quickly turn into broad application symptoms if they are treated too generically.

Run these read-only checks from an administrative environment with access to the intended Ceph cluster:

ceph -s
ceph health detail
ceph osd tree
ceph df

Record the health code, affected daemon or pool, and current client impact. This guide does not assume a particular Ceph deployment operator or toolbox pod.

Separate the failure domains

SignalFirst question
MON_DOWNIs a monitor unavailable, and does the cluster retain a majority?
OSD_DOWNIs the daemon, host, or network path unavailable?
OSD_NEARFULLWhich device or pool is approaching its capacity threshold?
OSD_BACKFILLFULLIs recovery or rebalancing blocked by available space?
OSD_FULLAre writes blocked because a full threshold has been reached?
MON_CLOCK_SKEWAre monitor hosts synchronizing their clocks correctly?

Look for a shared cause before treating several warnings as independent failures. A host or network problem can affect multiple daemons at once.

Relate storage health to the workload

In Kubernetes, storage problems may surface as mounting failures, unavailable volumes, or rising application latency. Connect the Ceph signal with the affected workload rather than assuming every application symptom originates in the storage layer.

Compare the timing of the health warning with workload events and the relevant observability data. Preserve enough context to distinguish capacity pressure from a failed device or an unavailable network path.

Prefer a cause-specific recovery

Restoring a daemon, correcting time synchronization, adding capacity, and changing a threshold are different operations. Select the procedure that matches the actual health identifier and the deployed Ceph version.

Muting a warning changes its visibility; it does not fix the condition. Raising a full threshold is not a substitute for understanding why capacity is exhausted. Validate both the health state and the affected client workload after recovery. The useful outcome is a storage explanation that matches what clients actually experienced.

Source

Ceph health-check reference, reviewed on 13 September 2026. The latest documentation tracks development, so use the release selector for version-specific recovery procedures.

LearningsStorage

A snapshot is not a restore plan

Treat volume snapshots as one recovery input and verify application restore, consistency, permissions, and target-cluster compatibility.

RunbooksKubernetes & GitOps

When a rollout stops progressing

Diagnose a stalled Kubernetes deployment by checking rollout state, pod events, readiness, and GitOps recovery paths.