Rookio Ceph cluster : mon c is low on available space message [closed]
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 12 days ago .
I have setup RookIO 1.4 cluster in Kubernetes 1.18. with 3 nodes allocated 1TB storage on each of them. after creating cluster. when I run the ceph status cluster status shows as HEALTH_WARN with mon c is low on available space . There is no data stored yet. why status how low on available space? How to clear this error?
[root@rook-ceph-tools-6bdcd78654-sfjvl /]# ceph status cluster: id: ad42764d-aa28-4da5-a828-2d87205aff08 health: HEALTH_WARN mon c is low on available space services: mon: 3 daemons, quorum a,b,c (age 37m) mgr: a(active, since 36m) osd: 3 osds: 3 up (since 37m), 3 in (since 37m) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 3.0 GiB used, 3.6 TiB / 3.6 TiB avail pgs: 1 active+clean
All three node has same size storage:
sdb 8:16 0 1.2T 0 disk └─ceph--a6cd601d--7584--4b1f--bf82--48c95437f351-osd--data--ae1bc856--8ded--4b1e--8c87--30ca0f0959a3 253:3 0 1.2T 0 lvm sdb 8:16 0 1.2T 0 disk └─ceph--ccaf7144--d6a0--441c--bcd5--6a09d056bd7a-osd--data--36a9b28c--7207--400a--936b--edfb3255ce0b 253:3 0 1.2T 0 lvm sdb 8:16 0 1.2T 0 disk └─ceph--53e9b8a9--8925--4b21--a6ea--f8e17a322d5c-osd--data--6b1e779c--a18a--4e4d--960e--73ca9473d02f 253:3 0 1.2T 0 lvm
Health checks
There is a finite set of health messages that a Ceph cluster can raise. These messages are known as health checks. Each health check has a unique identifier.
The identifier is a terse human-readable string — that is, the identifier is readable in much the same way as a typical variable name. It is intended to enable tools (for example, UIs) to make sense of health checks and present them in a way that reflects their meaning.
This page lists the health checks that are raised by the monitor and manager daemons. In addition to these, you might see health checks that originate from MDS daemons (see CephFS health messages ), and health checks that are defined by ceph-mgr python modules.
Definitions
Monitor
DAEMON_OLD_VERSION
Warn if one or more old versions of Ceph are running on any daemons. A health check is raised if multiple versions are detected. This condition must exist for a period of time greater than mon_warn_older_version_delay (set to one week by default) in order for the health check to be raised. This allows most upgrades to proceed without the occurrence of a false warning. If the upgrade is paused for an extended time period, health mute can be used by running ceph health mute DAEMON_OLD_VERSION —sticky . Be sure, however, to run ceph health unmute DAEMON_OLD_VERSION after the upgrade has finished.
MON_DOWN
One or more monitor daemons are currently down. The cluster requires a majority (more than one-half) of the monitors to be available. When one or more monitors are down, clients might have a harder time forming their initial connection to the cluster, as they might need to try more addresses before they reach an operating monitor.
The down monitor daemon should be restarted as soon as possible to reduce the risk of a subsequent monitor failure leading to a service outage.
MON_CLOCK_SKEW
The clocks on the hosts running the ceph-mon monitor daemons are not well-synchronized. This health check is raised if the cluster detects a clock skew greater than mon_clock_drift_allowed .
This issue is best resolved by synchronizing the clocks by using a tool like ntpd or chrony .
If it is impractical to keep the clocks closely synchronized, the mon_clock_drift_allowed threshold can also be increased. However, this value must stay significantly below the mon_lease interval in order for the monitor cluster to function properly.
MON_MSGR2_NOT_ENABLED
The ms_bind_msgr2 option is enabled but one or more monitors are not configured to bind to a v2 port in the cluster’s monmap. This means that features specific to the msgr2 protocol (for example, encryption) are unavailable on some or all connections.
In most cases this can be corrected by running the following command:
ceph mon enable-msgr2
After this command is run, any monitor configured to listen on the old default port (6789) will continue to listen for v1 connections on 6789 and begin to listen for v2 connections on the new default port 3300.
If a monitor is configured to listen for v1 connections on a non-standard port (that is, a port other than 6789), then the monmap will need to be modified manually.
MON_DISK_LOW
One or more monitors are low on disk space. This health check is raised if the percentage of available space on the file system used by the monitor database (normally /var/lib/ceph/mon ) drops below the percentage value mon_data_avail_warn (default: 30%).
This alert might indicate that some other process or user on the system is filling up the file system used by the monitor. It might also indicate that the monitor database is too large (see MON_DISK_BIG below).
If space cannot be freed, the monitor’s data directory might need to be moved to another storage device or file system (this relocation process must be carried out while the monitor daemon is not running).
MON_DISK_CRIT
One or more monitors are critically low on disk space. This health check is raised if the percentage of available space on the file system used by the monitor database (normally /var/lib/ceph/mon ) drops below the percentage value mon_data_avail_crit (default: 5%). See MON_DISK_LOW , above.
MON_DISK_BIG
The database size for one or more monitors is very large. This health check is raised if the size of the monitor database is larger than mon_data_size_warn (default: 15 GiB).
A large database is unusual, but does not necessarily indicate a problem. Monitor databases might grow in size when there are placement groups that have not reached an active+clean state in a long time.
This alert might also indicate that the monitor’s database is not properly compacting, an issue that has been observed with some older versions of RocksDB. Forcing a compaction with ceph daemon mon. compact might shrink the database’s on-disk size.
This alert might also indicate that the monitor has a bug that prevents it from pruning the cluster metadata that it stores. If the problem persists, please report a bug.
To adjust the warning threshold, run the following command:
ceph config set global mon_data_size_warn
AUTH_INSECURE_GLOBAL_ID_RECLAIM
One or more clients or daemons that are connected to the cluster are not securely reclaiming their global_id (a unique number that identifies each entity in the cluster) when reconnecting to a monitor. The client is being permitted to connect anyway because the auth_allow_insecure_global_id_reclaim option is set to true (which may be necessary until all Ceph clients have been upgraded) and because the auth_expose_insecure_global_id_reclaim option is set to true (which allows monitors to detect clients with “insecure reclaim” sooner by forcing those clients to reconnect immediately after their initial authentication).
To identify which client(s) are using unpatched Ceph client code, run the following command:
ceph health detail
If you collect a dump of the clients that are connected to an individual monitor and examine the global_id_status field in the output of the dump, you can see the global_id reclaim behavior of those clients. Here reclaim_insecure means that a client is unpatched and is contributing to this health check. To effect a client dump, run the following command:
ceph tell mon.\* sessions
We strongly recommend that all clients in the system be upgraded to a newer version of Ceph that correctly reclaims global_id values. After all clients have been updated, run the following command to stop allowing insecure reconnections:
ceph config set mon auth_allow_insecure_global_id_reclaim false
If it is impractical to upgrade all clients immediately, you can temporarily silence this alert by running the following command:
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM 1w # 1 week
Although we do NOT recommend doing so, you can also disable this alert indefinitely by running the following command:
ceph config set mon mon_warn_on_insecure_global_id_reclaim false
AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED
Ceph is currently configured to allow clients that reconnect to monitors using an insecure process to reclaim their previous global_id . Such reclaiming is allowed because, by default, auth_allow_insecure_global_id_reclaim is set to true . It might be necessary to leave this setting enabled while existing Ceph clients are upgraded to newer versions of Ceph that correctly and securely reclaim their global_id .
If the AUTH_INSECURE_GLOBAL_ID_RECLAIM health check has not also been raised and if the auth_expose_insecure_global_id_reclaim setting has not been disabled (it is enabled by default), then there are currently no clients connected that need to be upgraded. In that case, it is safe to disable insecure global_id reclaim by running the following command:
ceph config set mon auth_allow_insecure_global_id_reclaim false
On the other hand, if there are still clients that need to be upgraded, then this alert can be temporarily silenced by running the following command:
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 1w # 1 week
Although we do NOT recommend doing so, you can also disable this alert indefinitely by running the following command:
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
Manager
MGR_DOWN
All manager daemons are currently down. The cluster should normally have at least one running manager ( ceph-mgr ) daemon. If no manager daemon is running, the cluster’s ability to monitor itself will be compromised, and parts of the management API will become unavailable (for example, the dashboard will not work, and most CLI commands that report metrics or runtime state will block). However, the cluster will still be able to perform all I/O operations and to recover from failures.
The “down” manager daemon should be restarted as soon as possible to ensure that the cluster can be monitored (for example, so that the ceph -s information is up to date, or so that metrics can be scraped by Prometheus).
MGR_MODULE_DEPENDENCY
An enabled manager module is failing its dependency check. This health check typically comes with an explanatory message from the module about the problem.
For example, a module might report that a required package is not installed: in this case, you should install the required package and restart your manager daemons.
This health check is applied only to enabled modules. If a module is not enabled, you can see whether it is reporting dependency issues in the output of ceph module ls .
MGR_MODULE_ERROR
A manager module has experienced an unexpected error. Typically, this means that an unhandled exception was raised from the module’s serve function. The human-readable description of the error might be obscurely worded if the exception did not provide a useful description of itself.
This health check might indicate a bug: please open a Ceph bug report if you think you have encountered a bug.
However, if you believe the error is transient, you may restart your manager daemon(s) or use ceph mgr fail on the active daemon in order to force failover to another daemon.
OSDs
OSD_DOWN
One or more OSDs are marked “down”. The ceph-osd daemon might have been stopped, or peer OSDs might be unable to reach the OSD over the network. Common causes include a stopped or crashed daemon, a “down” host, or a network outage.
Verify that the host is healthy, the daemon is started, and the network is functioning. If the daemon has crashed, the daemon log file ( /var/log/ceph/ceph-osd.* ) might contain debugging information.
OSD__DOWN
(for example, OSD_HOST_DOWN, OSD_ROOT_DOWN)
All of the OSDs within a particular CRUSH subtree are marked “down” (for example, all OSDs on a host).
OSD_ORPHAN
An OSD is referenced in the CRUSH map hierarchy, but does not exist.
To remove the OSD from the CRUSH map hierarchy, run the following command:
ceph osd crush rm osd.
OSD_OUT_OF_ORDER_FULL
The utilization thresholds for nearfull , backfillfull , full , and/or failsafe_full are not ascending. In particular, the following pattern is expected: nearfull < backfillfull , backfillfull < full , and full < failsafe_full .
To adjust these utilization thresholds, run the following commands:
ceph osd set-nearfull-ratio ceph osd set-backfillfull-ratio ceph osd set-full-ratio
OSD_FULL
One or more OSDs have exceeded the full threshold and are preventing the cluster from servicing writes.
To check utilization by pool, run the following command:
ceph df
To see the currently defined full ratio, run the following command:
ceph osd dump | grep full_ratio
A short-term workaround to restore write availability is to raise the full threshold by a small amount. To do so, run the following command:
ceph osd set-full-ratio
Additional OSDs should be deployed in order to add new storage to the cluster, or existing data should be deleted in order to free up space in the cluster.
OSD_BACKFILLFULL
One or more OSDs have exceeded the backfillfull threshold or would exceed it if the currently-mapped backfills were to finish, which will prevent data from rebalancing to this OSD. This alert is an early warning that rebalancing might be unable to complete and that the cluster is approaching full.
To check utilization by pool, run the following command:
ceph df
OSD_NEARFULL
One or more OSDs have exceeded the nearfull threshold. This alert is an early warning that the cluster is approaching full.
To check utilization by pool, run the following command:
ceph df
OSDMAP_FLAGS
One or more cluster flags of interest have been set. These flags include:
- full — the cluster is flagged as full and cannot serve writes
- pauserd, pausewr — there are paused reads or writes
- noup — OSDs are not allowed to start
- nodown — OSD failure reports are being ignored, and that means that the monitors will not mark OSDs “down”
- noin — OSDs that were previously marked out are not being marked back in when they start
- noout — “down” OSDs are not automatically being marked out after the configured interval
- nobackfill, norecover, norebalance — recovery or data rebalancing is suspended
- noscrub, nodeep_scrub — scrubbing is disabled
- notieragent — cache-tiering activity is suspended
With the exception of full, these flags can be set or cleared by running the following commands:
ceph osd set ceph osd unset
OSD_FLAGS
One or more OSDs or CRUSH have a flag of interest set. These flags include:
- noup: these OSDs are not allowed to start
- nodown: failure reports for these OSDs will be ignored
- noin: if these OSDs were previously marked out automatically after a failure, they will not be marked in when they start
- noout: if these OSDs are “down” they will not automatically be marked out after the configured interval
To set and clear these flags in batch, run the following commands:
ceph osd set-group ceph osd unset-group
ceph osd set-group noup,noout osd.0 osd.1 ceph osd unset-group noup,noout osd.0 osd.1 ceph osd set-group noup,noout host-foo ceph osd unset-group noup,noout host-foo ceph osd set-group noup,noout class-hdd ceph osd unset-group noup,noout class-hdd
OLD_CRUSH_TUNABLES
The CRUSH map is using very old settings and should be updated. The oldest set of tunables that can be used (that is, the oldest client version that can connect to the cluster) without raising this health check is determined by the mon_crush_min_required_version config option. For more information, see Tunables .
OLD_CRUSH_STRAW_CALC_VERSION
The CRUSH map is using an older, non-optimal method of calculating intermediate weight values for straw buckets.
The CRUSH map should be updated to use the newer method (that is: straw_calc_version=1 ). For more information, see Tunables .
CACHE_POOL_NO_HIT_SET
One or more cache pools are not configured with a hit set to track utilization. This issue prevents the tiering agent from identifying cold objects that are to be flushed and evicted from the cache.
To configure hit sets on the cache pool, run the following commands:
ceph osd pool set hit_set_type ceph osd pool set hit_set_period ceph osd pool set hit_set_count ceph osd pool set hit_set_fpp
OSD_NO_SORTBITWISE
No pre-Luminous v12.y.z OSDs are running, but the sortbitwise flag has not been set.
The sortbitwise flag must be set in order for OSDs running Luminous v12.y.z or newer to start. To safely set the flag, run the following command:
ceph osd set sortbitwise
OSD_FILESTORE
Warn if OSDs are running Filestore. The Filestore OSD back end has been deprecated; the BlueStore back end has been the default object store since the Ceph Luminous release.
The ‘mclock_scheduler’ is not supported for Filestore OSDs. For this reason, the default ‘osd_op_queue’ is set to ‘wpq’ for Filestore OSDs and is enforced even if the user attempts to change it.
ceph report | jq -c '."osd_metadata" | .[] | select(.osd_objectstore | contains("filestore")) | '
In order to upgrade to Reef or a later release, you must first migrate any Filestore OSDs to BlueStore.
If you are upgrading a pre-Reef release to Reef or later, but it is not feasible to migrate Filestore OSDs to BlueStore immediately, you can temporarily silence this alert by running the following command:
ceph health mute OSD_FILESTORE
Since this migration can take a considerable amount of time to complete, we recommend that you begin the process well in advance of any update to Reef or to later releases.
OSD_UNREACHABLE
Registered v1/v2 public address of one or more OSD(s) is/are out of the defined public_network subnet, which will prevent these unreachable OSDs from communicating with ceph clients properly.
Even though these unreachable OSDs are in up state, rados clients will hang till TCP timeout before erroring out due to this inconsistency.
POOL_FULL
One or more pools have reached their quota and are no longer allowing writes.
To see pool quotas and utilization, run the following command:
ceph df detail
If you opt to raise the pool quota, run the following commands:
ceph osd pool set-quota max_objects ceph osd pool set-quota max_bytes
If not, delete some existing data to reduce utilization.
BLUEFS_SPILLOVER
One or more OSDs that use the BlueStore back end have been allocated db partitions (that is, storage space for metadata, normally on a faster device), but because that space has been filled, metadata has “spilled over” onto the slow device. This is not necessarily an error condition or even unexpected behavior, but may result in degraded performance. If the administrator had expected that all metadata would fit on the faster device, this alert indicates that not enough space was provided.
To disable this alert on all OSDs, run the following command:
ceph config set osd bluestore_warn_on_bluefs_spillover false
Alternatively, to disable the alert on a specific OSD, run the following command:
ceph config set osd.123 bluestore_warn_on_bluefs_spillover false
To secure more metadata space, you can destroy and reprovision the OSD in question. This process involves data migration and recovery.
It might also be possible to expand the LVM logical volume that backs the db storage. If the underlying LV has been expanded, you must stop the OSD daemon and inform BlueFS of the device-size change by running the following command:
ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$ID
BLUEFS_AVAILABLE_SPACE
To see how much space is free for BlueFS, run the following command:
ceph daemon osd.123 bluestore bluefs available
This will output up to three values: BDEV_DB free , BDEV_SLOW free , and available_from_bluestore . BDEV_DB and BDEV_SLOW report the amount of space that has been acquired by BlueFS and is now considered free. The value available_from_bluestore indicates the ability of BlueStore to relinquish more space to BlueFS. It is normal for this value to differ from the amount of BlueStore free space, because the BlueFS allocation unit is typically larger than the BlueStore allocation unit. This means that only part of the BlueStore free space will be available for BlueFS.
BLUEFS_LOW_SPACE
If BlueFS is running low on available free space and there is not much free space available from BlueStore (in other words, available_from_bluestore has a low value), consider reducing the BlueFS allocation unit size. To simulate available space when the allocation unit is different, run the following command:
ceph daemon osd.123 bluestore bluefs available
BLUESTORE_FRAGMENTATION
As BlueStore operates, the free space on the underlying storage will become fragmented. This is normal and unavoidable, but excessive fragmentation causes slowdown. To inspect BlueStore fragmentation, run the following command:
ceph daemon osd.123 bluestore allocator score block
The fragmentation score is given in a [0-1] range. [0.0 .. 0.4] tiny fragmentation [0.4 .. 0.7] small, acceptable fragmentation [0.7 .. 0.9] considerable, but safe fragmentation [0.9 .. 1.0] severe fragmentation, might impact BlueFS’s ability to get space from BlueStore
To see a detailed report of free fragments, run the following command:
ceph daemon osd.123 bluestore allocator dump block
For OSD processes that are not currently running, fragmentation can be inspected with ceph-bluestore-tool . To see the fragmentation score, run the following command:
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score
To dump detailed free chunks, run the following command:
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump
BLUESTORE_LEGACY_STATFS
One or more OSDs have BlueStore volumes that were created prior to the Nautilus release. (In Nautilus, BlueStore tracks its internal usage statistics on a granular, per-pool basis.)
If all OSDs are older than Nautilus, this means that the per-pool metrics are simply unavailable. But if there is a mixture of pre-Nautilus and post-Nautilus OSDs, the cluster usage statistics reported by ceph df will be inaccurate.
The old OSDs can be updated to use the new usage-tracking scheme by stopping each OSD, running a repair operation, and then restarting the OSD. For example, to update osd.123 , run the following commands:
systemctl stop ceph-osd@123 ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123 systemctl start ceph-osd@123
To disable this alert, run the following command:
ceph config set global bluestore_warn_on_legacy_statfs false
BLUESTORE_NO_PER_POOL_OMAP
One or more OSDs have volumes that were created prior to the Octopus release. (In Octopus and later releases, BlueStore tracks omap space utilization by pool.)
If there are any BlueStore OSDs that do not have the new tracking enabled, the cluster will report an approximate value for per-pool omap usage based on the most recent deep scrub.
The OSDs can be updated to track by pool by stopping each OSD, running a repair operation, and then restarting the OSD. For example, to update osd.123 , run the following commands:
systemctl stop ceph-osd@123 ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123 systemctl start ceph-osd@123
To disable this alert, run the following command:
ceph config set global bluestore_warn_on_no_per_pool_omap false
BLUESTORE_NO_PER_PG_OMAP
One or more OSDs have volumes that were created prior to Pacific. (In Pacific and later releases Bluestore tracks omap space utilitzation by Placement Group (PG).)
Per-PG omap allows faster PG removal when PGs migrate.
The older OSDs can be updated to track by PG by stopping each OSD, running a repair operation, and then restarting the OSD. For example, to update osd.123 , run the following commands:
systemctl stop ceph-osd@123 ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123 systemctl start ceph-osd@123
To disable this alert, run the following command:
ceph config set global bluestore_warn_on_no_per_pg_omap false
BLUESTORE_DISK_SIZE_MISMATCH
One or more BlueStore OSDs have an internal inconsistency between the size of the physical device and the metadata that tracks its size. This inconsistency can lead to the OSD(s) crashing in the future.
The OSDs that have this inconsistency should be destroyed and reprovisioned. Be very careful to execute this procedure on only one OSD at a time, so as to minimize the risk of losing any data. To execute this procedure, where $N is the OSD that has the inconsistency, run the following commands:
ceph osd out osd.$N while ! ceph osd safe-to-destroy osd.$N ; do sleep 1m ; done ceph osd destroy osd.$N ceph-volume lvm zap /path/to/device ceph-volume lvm create --osd-id $N --data /path/to/device
Wait for this recovery procedure to completely on one OSD before running it on the next.
BLUESTORE_NO_COMPRESSION
One or more OSDs is unable to load a BlueStore compression plugin. This issue might be caused by a broken installation, in which the ceph-osd binary does not match the compression plugins. Or it might be caused by a recent upgrade in which the ceph-osd daemon was not restarted.
To resolve this issue, verify that all of the packages on the host that is running the affected OSD(s) are correctly installed and that the OSD daemon(s) have been restarted. If the problem persists, check the OSD log for information about the source of the problem.
BLUESTORE_SPURIOUS_READ_ERRORS
One or more BlueStore OSDs detect spurious read errors on the main device. BlueStore has recovered from these errors by retrying disk reads. This alert might indicate issues with underlying hardware, issues with the I/O subsystem, or something similar. In theory, such issues can cause permanent data corruption. Some observations on the root cause of spurious read errors can be found here: https://tracker.ceph.com/issues/22464
This alert does not require an immediate response, but the affected host might need additional attention: for example, upgrading the host to the latest OS/kernel versions and implementing hardware-resource-utilization monitoring.
To disable this alert on all OSDs, run the following command:
ceph config set osd bluestore_warn_on_spurious_read_errors false
Or, to disable this alert on a specific OSD, run the following command:
ceph config set osd.123 bluestore_warn_on_spurious_read_errors false
Device health
DEVICE_HEALTH
One or more OSD devices are expected to fail soon, where the warning threshold is determined by the mgr/devicehealth/warn_threshold config option.
Because this alert applies only to OSDs that are currently marked in , the appropriate response to this expected failure is (1) to mark the OSD out so that data is migrated off of the OSD, and then (2) to remove the hardware from the system. Note that this marking out is normally done automatically if mgr/devicehealth/self_heal is enabled (as determined by mgr/devicehealth/mark_out_threshold ).
To check device health, run the following command:
ceph device info
Device life expectancy is set either by a prediction model that the mgr runs or by an external tool that is activated by running the following command:
ceph device set-life-expectancy
You can change the stored life expectancy manually, but such a change usually doesn’t accomplish anything. The reason for this is that whichever tool originally set the stored life expectancy will probably undo your change by setting it again, and a change to the stored value does not affect the actual health of the hardware device.
DEVICE_HEALTH_IN_USE
One or more devices (that is, OSDs) are expected to fail soon and have been marked out of the cluster (as controlled by mgr/devicehealth/mark_out_threshold ), but they are still participating in one or more Placement Groups. This might be because the OSD(s) were marked out only recently and data is still migrating, or because data cannot be migrated off of the OSD(s) for some reason (for example, the cluster is nearly full, or the CRUSH hierarchy is structured so that there isn’t another suitable OSD to migrate the data to).
This message can be silenced by disabling self-heal behavior (that is, setting mgr/devicehealth/self_heal to false ), by adjusting mgr/devicehealth/mark_out_threshold , or by addressing whichever condition is preventing data from being migrated off of the ailing OSD(s).
DEVICE_HEALTH_TOOMANY
Too many devices (that is, OSDs) are expected to fail soon, and because mgr/devicehealth/self_heal behavior is enabled, marking out all of the ailing OSDs would exceed the cluster’s mon_osd_min_in_ratio ratio. This ratio prevents a cascade of too many OSDs from being automatically marked out .
You should promptly add new OSDs to the cluster to prevent data loss, or incrementally replace the failing OSDs.
Alternatively, you can silence this health check by adjusting options including mon_osd_min_in_ratio or mgr/devicehealth/mark_out_threshold . Be warned, however, that this will increase the likelihood of unrecoverable data loss.
Data health (pools & placement groups)
PG_AVAILABILITY
Data availability is reduced. In other words, the cluster is unable to service potential read or write requests for at least some data in the cluster. More precisely, one or more Placement Groups (PGs) are in a state that does not allow I/O requests to be serviced. Any of the following PG states are problematic if they do not clear quickly: peering, stale, incomplete, and the lack of active.
For detailed information about which PGs are affected, run the following command:
ceph health detail
In most cases, the root cause of this issue is that one or more OSDs are currently down : see OSD_DOWN above.
To see the state of a specific problematic PG, run the following command:
ceph tell query
PG_DEGRADED
Data redundancy is reduced for some data: in other words, the cluster does not have the desired number of replicas for all data (in the case of replicated pools) or erasure code fragments (in the case of erasure-coded pools). More precisely, one or more Placement Groups (PGs):
- have the degraded or undersized flag set, which means that there are not enough instances of that PG in the cluster; or
- have not had the clean state set for a long time.
For detailed information about which PGs are affected, run the following command:
ceph health detail
In most cases, the root cause of this issue is that one or more OSDs are currently “down”: see OSD_DOWN above.
To see the state of a specific problematic PG, run the following command:
ceph tell query
PG_RECOVERY_FULL
Data redundancy might be reduced or even put at risk for some data due to a lack of free space in the cluster. More precisely, one or more Placement Groups have the recovery_toofull flag set, which means that the cluster is unable to migrate or recover data because one or more OSDs are above the full threshold.
For steps to resolve this condition, see OSD_FULL above.
PG_BACKFILL_FULL
Data redundancy might be reduced or even put at risk for some data due to a lack of free space in the cluster. More precisely, one or more Placement Groups have the backfill_toofull flag set, which means that the cluster is unable to migrate or recover data because one or more OSDs are above the backfillfull threshold.
For steps to resolve this condition, see OSD_BACKFILLFULL above.
PG_DAMAGED
Data scrubbing has discovered problems with data consistency in the cluster. More precisely, one or more Placement Groups either (1) have the inconsistent or snaptrim_error flag set, which indicates that an earlier data scrub operation found a problem, or (2) have the repair flag set, which means that a repair for such an inconsistency is currently in progress.
OSD_SCRUB_ERRORS
Recent OSD scrubs have discovered inconsistencies. This alert is generally paired with PG_DAMAGED (see above).
OSD_TOO_MANY_REPAIRS
The count of read repairs has exceeded the config value threshold mon_osd_warn_num_repaired (default: 10 ). Because scrub handles errors only for data at rest, and because any read error that occurs when another replica is available will be repaired immediately so that the client can get the object data, there might exist failing disks that are not registering any scrub errors. This repair count is maintained as a way of identifying any such failing disks.
LARGE_OMAP_OBJECTS
One or more pools contain large omap objects, as determined by osd_deep_scrub_large_omap_object_key_threshold (threshold for the number of keys to determine what is considered a large omap object) or osd_deep_scrub_large_omap_object_value_sum_threshold (the threshold for the summed size in bytes of all key values to determine what is considered a large omap object) or both. To find more information on object name, key count, and size in bytes, search the cluster log for ‘Large omap object found’. This issue can be caused by RGW-bucket index objects that do not have automatic resharding enabled. For more information on resharding, see RGW Dynamic Bucket Index Resharding .
To adjust the thresholds mentioned above, run the following commands:
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold ceph config set osd osd_deep_scrub_large_omap_object_value_sum_threshold
CACHE_POOL_NEAR_FULL
A cache-tier pool is nearly full, as determined by the target_max_bytes and target_max_objects properties of the cache pool. Once the pool reaches the target threshold, write requests to the pool might block while data is flushed and evicted from the cache. This state normally leads to very high latencies and poor performance.
To adjust the cache pool’s target size, run the following commands:
ceph osd pool set target_max_bytes ceph osd pool set target_max_objects
There might be other reasons that normal cache flush and evict activity are throttled: for example, reduced availability of the base tier, reduced performance of the base tier, or overall cluster load.
TOO_FEW_PGS
The number of Placement Groups (PGs) that are in use in the cluster is below the configurable threshold of mon_pg_warn_min_per_osd PGs per OSD. This can lead to suboptimal distribution and suboptimal balance of data across the OSDs in the cluster, and a reduction of overall performance.
If data pools have not yet been created, this condition is expected.
To address this issue, you can increase the PG count for existing pools or create new pools. For more information, see Choosing the Number of PGs .
POOL_PG_NUM_NOT_POWER_OF_TWO
One or more pools have a pg_num value that is not a power of two. Although this is not strictly incorrect, it does lead to a less balanced distribution of data because some Placement Groups will have roughly twice as much data as others have.
This is easily corrected by setting the pg_num value for the affected pool(s) to a nearby power of two. To do so, run the following command:
ceph osd pool set pg_num
To disable this health check, run the following command:
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
POOL_TOO_FEW_PGS
One or more pools should probably have more Placement Groups (PGs), given the amount of data that is currently stored in the pool. This issue can lead to suboptimal distribution and suboptimal balance of data across the OSDs in the cluster, and a reduction of overall performance. This alert is raised only if the pg_autoscale_mode property on the pool is set to warn .
To disable the alert, entirely disable auto-scaling of PGs for the pool by running the following command:
ceph osd pool set pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs for the pool, run the following command:
ceph osd pool set pg_autoscale_mode on
Alternatively, to manually set the number of PGs for the pool to the recommended amount, run the following command:
ceph osd pool set pg_num
TOO_MANY_PGS
The number of Placement Groups (PGs) in use in the cluster is above the configurable threshold of mon_max_pg_per_osd PGs per OSD. If this threshold is exceeded, the cluster will not allow new pools to be created, pool pg_num to be increased, or pool replication to be increased (any of which, if allowed, would lead to more PGs in the cluster). A large number of PGs can lead to higher memory utilization for OSD daemons, slower peering after cluster state changes (for example, OSD restarts, additions, or removals), and higher load on the Manager and Monitor daemons.
The simplest way to mitigate the problem is to increase the number of OSDs in the cluster by adding more hardware. Note that, because the OSD count that is used for the purposes of this health check is the number of in OSDs, marking out OSDs in (if there are any out OSDs available) can also help. To do so, run the following command:
ceph osd in id(s)>
POOL_TOO_MANY_PGS
One or more pools should probably have fewer Placement Groups (PGs), given the amount of data that is currently stored in the pool. This issue can lead to higher memory utilization for OSD daemons, slower peering after cluster state changes (for example, OSD restarts, additions, or removals), and higher load on the Manager and Monitor daemons. This alert is raised only if the pg_autoscale_mode property on the pool is set to warn .
To disable the alert, entirely disable auto-scaling of PGs for the pool by running the following command:
ceph osd pool set pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs for the pool, run the following command:
ceph osd pool set pg_autoscale_mode on
Alternatively, to manually set the number of PGs for the pool to the recommended amount, run the following command:
ceph osd pool set pg_num
POOL_TARGET_SIZE_BYTES_OVERCOMMITTED
One or more pools have a target_size_bytes property that is set in order to estimate the expected size of the pool, but the value(s) of this property are greater than the total available storage (either by themselves or in combination with other pools).
This alert is usually an indication that the target_size_bytes value for the pool is too large and should be reduced or set to zero. To reduce the target_size_bytes value or set it to zero, run the following command:
ceph osd pool set target_size_bytes 0
The above command sets the value of target_size_bytes to zero. To set the value of target_size_bytes to a non-zero value, replace the 0 with that non-zero value.
POOL_HAS_TARGET_SIZE_BYTES_AND_RATIO
One or more pools have both target_size_bytes and target_size_ratio set in order to estimate the expected size of the pool. Only one of these properties should be non-zero. If both are set to a non-zero value, then target_size_ratio takes precedence and target_size_bytes is ignored.
To reset target_size_bytes to zero, run the following command:
ceph osd pool set target_size_bytes 0
TOO_FEW_OSDS
The number of OSDs in the cluster is below the configurable threshold of osd_pool_default_size . This means that some or all data may not be able to satisfy the data protection policy specified in CRUSH rules and pool settings.
SMALLER_PGP_NUM
One or more pools have a pgp_num value less than pg_num . This alert is normally an indication that the Placement Group (PG) count was increased without any increase in the placement behavior.
This disparity is sometimes brought about deliberately, in order to separate out the split step when the PG count is adjusted from the data migration that is needed when pgp_num is changed.
This issue is normally resolved by setting pgp_num to match pg_num , so as to trigger the data migration, by running the following command:
ceph osd pool set pgp_num
MANY_OBJECTS_PER_PG
One or more pools have an average number of objects per Placement Group (PG) that is significantly higher than the overall cluster average. The specific threshold is determined by the mon_pg_warn_max_object_skew configuration value.
This alert is usually an indication that the pool(s) that contain most of the data in the cluster have too few PGs, or that other pools that contain less data have too many PGs. See TOO_MANY_PGS above.
To silence the health check, raise the threshold by adjusting the mon_pg_warn_max_object_skew config option on the managers.
The health check will be silenced for a specific pool only if pg_autoscale_mode is set to on .
POOL_APP_NOT_ENABLED
A pool exists but the pool has not been tagged for use by a particular application.
To resolve this issue, tag the pool for use by an application. For example, if the pool is used by RBD, run the following command:
rbd pool init
Alternatively, if the pool is being used by a custom application (here ‘foo’), you can label the pool by running the following low-level command:
ceph osd pool application enable foo
POOL_FULL
One or more pools have reached (or are very close to reaching) their quota. The threshold to raise this health check is determined by the mon_pool_quota_crit_threshold configuration option.
Pool quotas can be adjusted up or down (or removed) by running the following commands:
ceph osd pool set-quota max_bytes ceph osd pool set-quota max_objects
To disable a quota, set the quota value to 0.
POOL_NEAR_FULL
One or more pools are approaching a configured fullness threshold.
One of the several thresholds that can raise this health check is determined by the mon_pool_quota_warn_threshold configuration option.
Pool quotas can be adjusted up or down (or removed) by running the following commands:
ceph osd pool set-quota max_bytes ceph osd pool set-quota max_objects
To disable a quota, set the quota value to 0.
Other thresholds that can raise the two health checks above are mon_osd_nearfull_ratio and mon_osd_full_ratio . For details and resolution, see Storage Capacity and No Free Drive Space .
OBJECT_MISPLACED
One or more objects in the cluster are not stored on the node that CRUSH would prefer that they be stored on. This alert is an indication that data migration due to a recent cluster change has not yet completed.
Misplaced data is not a dangerous condition in and of itself; data consistency is never at risk, and old copies of objects will not be removed until the desired number of new copies (in the desired locations) has been created.
OBJECT_UNFOUND
One or more objects in the cluster cannot be found. More precisely, the OSDs know that a new or updated copy of an object should exist, but no such copy has been found on OSDs that are currently online.
Read or write requests to unfound objects will block.
Ideally, a “down” OSD that has a more recent copy of the unfound object can be brought back online. To identify candidate OSDs, check the peering state of the PG(s) responsible for the unfound object. To see the peering state, run the following command:
ceph tell query
On the other hand, if the latest copy of the object is not available, the cluster can be told to roll back to a previous version of the object. For more information, see Unfound Objects .
SLOW_OPS
One or more OSD requests or monitor requests are taking a long time to process. This alert might be an indication of extreme load, a slow storage device, or a software bug.
To query the request queue for the daemon that is causing the slowdown, run the following command from the daemon’s host:
ceph daemon osd. ops
To see a summary of the slowest recent requests, run the following command:
ceph daemon osd. dump_historic_ops
To see the location of a specific OSD, run the following command:
ceph osd find osd.
PG_NOT_SCRUBBED
One or more Placement Groups (PGs) have not been scrubbed recently. PGs are normally scrubbed within an interval determined by osd_scrub_max_interval globally. This interval can be overridden on per-pool basis by changing the value of the variable scrub_max_interval . This health check is raised if a certain percentage (determined by mon_warn_pg_not_scrubbed_ratio ) of the interval has elapsed after the time the scrub was scheduled and no scrub has been performed.
PGs will be scrubbed only if they are flagged as clean (which means that they are to be cleaned, and not that they have been examined and found to be clean). Misplaced or degraded PGs will not be flagged as clean (see PG_AVAILABILITY and PG_DEGRADED above).
To manually initiate a scrub of a clean PG, run the following command:
PG_NOT_DEEP_SCRUBBED
One or more Placement Groups (PGs) have not been deep scrubbed recently. PGs are normally scrubbed every osd_deep_scrub_interval seconds at most. This health check is raised if a certain percentage (determined by mon_warn_pg_not_deep_scrubbed_ratio ) of the interval has elapsed after the time the scrub was scheduled and no scrub has been performed.
PGs will receive a deep scrub only if they are flagged as clean (which means that they are to be cleaned, and not that they have been examined and found to be clean). Misplaced or degraded PGs might not be flagged as clean (see PG_AVAILABILITY and PG_DEGRADED above).
To manually initiate a deep scrub of a clean PG, run the following command:
ceph pg deep-scrub
PG_SLOW_SNAP_TRIMMING
The snapshot trim queue for one or more PGs has exceeded the configured warning threshold. This alert indicates either that an extremely large number of snapshots was recently deleted, or that OSDs are unable to trim snapshots quickly enough to keep up with the rate of new snapshot deletions.
The warning threshold is determined by the mon_osd_snap_trim_queue_warn_on option (default: 32768).
This alert might be raised if OSDs are under excessive load and unable to keep up with their background work, or if the OSDs’ internal metadata database is heavily fragmented and unable to perform. The alert might also indicate some other performance issue with the OSDs.
The exact size of the snapshot trim queue is reported by the snaptrimq_len field of ceph pg ls -f json-detail .
Stretch Mode
INCORRECT_NUM_BUCKETS_STRETCH_MODE
Stretch mode currently only support 2 dividing buckets with OSDs, this warning suggests that the number of dividing buckets is not equal to 2 after stretch mode is enabled. You can expect unpredictable failures and MON assertions until the condition is fixed.
We encourage you to fix this by removing additional dividing buckets or bump the number of dividing buckets to 2.
UNEVEN_WEIGHTS_STRETCH_MODE
The 2 dividing buckets must have equal weights when stretch mode is enabled. This warning suggests that the 2 dividing buckets have uneven weights after stretch mode is enabled. This is not immediately fatal, however, you can expect Ceph to be confused when trying to process transitions between dividing buckets.
We encourage you to fix this by making the weights even on both dividing buckets. This can be done by making sure the combined weight of the OSDs on each dividing bucket are the same.
Miscellaneous
RECENT_CRASH
One or more Ceph daemons have crashed recently, and the crash(es) have not yet been acknowledged and archived by the administrator. This alert might indicate a software bug, a hardware problem (for example, a failing disk), or some other problem.
To list recent crashes, run the following command:
ceph crash ls-new
To examine information about a specific crash, run the following command:
ceph crash info
To silence this alert, you can archive the crash (perhaps after the crash has been examined by an administrator) by running the following command:
ceph crash archive
Similarly, to archive all recent crashes, run the following command:
ceph crash archive-all
Archived crashes will still be visible by running the command ceph crash ls , but not by running the command ceph crash ls-new .
The time period that is considered recent is determined by the option mgr/crash/warn_recent_interval (default: two weeks).
To entirely disable this alert, run the following command:
ceph config set mgr/crash/warn_recent_interval 0
RECENT_MGR_MODULE_CRASH
One or more ceph-mgr modules have crashed recently, and the crash(es) have not yet been acknowledged and archived by the administrator. This alert usually indicates a software bug in one of the software modules that are running inside the ceph-mgr daemon. The module that experienced the problem might be disabled as a result, but other modules are unaffected and continue to function as expected.
As with the RECENT_CRASH health check, a specific crash can be inspected by running the following command:
ceph crash info
To silence this alert, you can archive the crash (perhaps after the crash has been examined by an administrator) by running the following command:
ceph crash archive
Similarly, to archive all recent crashes, run the following command:
ceph crash archive-all
Archived crashes will still be visible by running the command ceph crash ls but not by running the command ceph crash ls-new .
The time period that is considered recent is determined by the option mgr/crash/warn_recent_interval (default: two weeks).
To entirely disable this alert, run the following command:
ceph config set mgr/crash/warn_recent_interval 0
TELEMETRY_CHANGED
Telemetry has been enabled, but because the contents of the telemetry report have changed in the meantime, telemetry reports will not be sent.
Ceph developers occasionally revise the telemetry feature to include new and useful information, or to remove information found to be useless or sensitive. If any new information is included in the report, Ceph requires the administrator to re-enable telemetry. This requirement ensures that the administrator has an opportunity to (re)review the information that will be shared.
To review the contents of the telemetry report, run the following command:
ceph telemetry show
Note that the telemetry report consists of several channels that may be independently enabled or disabled. For more information, see Telemetry Module .
To re-enable telemetry (and silence the alert), run the following command:
ceph telemetry on
To disable telemetry (and silence the alert), run the following command:
ceph telemetry off
AUTH_BAD_CAPS
One or more auth users have capabilities that cannot be parsed by the monitors. As a general rule, this alert indicates that there are one or more daemon types that the user is not authorized to use to perform any action.
This alert is most likely to be raised after an upgrade if (1) the capabilities were set with an older version of Ceph that did not properly validate the syntax of those capabilities, or if (2) the syntax of the capabilities has changed.
To remove the user(s) in question, run the following command:
ceph auth rm
(This resolves the health check, but it prevents clients from being able to authenticate as the removed user.)
Alternatively, to update the capabilities for the user(s), run the following command:
ceph auth [ . ]
For more information about auth capabilities, see User Management .
OSD_NO_DOWN_OUT_INTERVAL
The mon_osd_down_out_interval option is set to zero, which means that the system does not automatically perform any repair or healing operations when an OSD fails. Instead, an administrator an external orchestrator must manually mark “down” OSDs as out (by running ceph osd out ) in order to trigger recovery.
This option is normally set to five or ten minutes, which should be enough time for a host to power-cycle or reboot.
To silence this alert, set mon_warn_on_osd_down_out_interval_zero to false by running the following command:
ceph config global mon mon_warn_on_osd_down_out_interval_zero false
DASHBOARD_DEBUG
The Dashboard debug mode is enabled. This means that if there is an error while processing a REST API request, the HTTP error response will contain a Python traceback. This mode should be disabled in production environments because such a traceback might contain and expose sensitive information.
To disable the debug mode, run the following command:
ceph dashboard debug disable
Brought to you by the Ceph Foundation
The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.
© Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0). Revision bf82a7bd .
[SOLVED] Ceph data not available
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
Ax2020
Member
Jan 8, 2021 3 0 6 36
we had a strange issue, we got a call asking for assistance on a proxmox cluster, apparently today the system restarted (apparently one node was already down for «problems» and the 2 other node seems to have gone down around the same time).
When they restarted the system they tried to start again the VM but after a few seconds the console timed out and even if the vm was started nothing seems to work, trying to navigate inside the ceph storage result in another timeout.
Here the result of the oh ceph -s:
root@SV3:~# ceph -s cluster: id: 89fd82e2-031d-4309-bbf9-454dcc2a4021 health: HEALTH_WARN Reduced data availability: 345 pgs inactive Degraded data redundancy: 5956939/13902540 objects degraded (42.848%), 1003 pgs degraded, 1003 pgs undersized 1023 pgs not deep-scrubbed in time 1023 pgs not scrubbed in time services: mon: 3 daemons, quorum SV1,SV2,SV3 (age 90m) mgr: SV2(active, since 90m), standbys: SV3 osd: 18 osds: 18 up (since 88m), 18 in (since 115m); 1003 remapped pgs data: pools: 1 pools, 1024 pgs objects: 4.63M objects, 18 TiB usage: 47 TiB used, 51 TiB / 98 TiB avail pgs: 33.691% pgs not active 5956939/13902540 objects degraded (42.848%) 656 active+undersized+degraded+remapped+backfill_wait 344 undersized+degraded+remapped+backfill_wait+peered 21 active+clean 2 active+undersized+degraded+remapped+backfilling 1 undersized+degraded+remapped+backfilling+peered io: recovery: 43 MiB/s, 10 objects/s
root@SV3:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 98.24387 root default -3 32.74796 host SV1 0 hdd 5.45799 osd.0 up 1.00000 1.00000 1 hdd 5.45799 osd.1 up 1.00000 1.00000 2 hdd 5.45799 osd.2 up 1.00000 1.00000 3 hdd 5.45799 osd.3 up 1.00000 1.00000 4 hdd 5.45799 osd.4 up 1.00000 1.00000 15 hdd 5.45799 osd.15 up 1.00000 1.00000 -5 32.74796 host SV2 5 hdd 5.45799 osd.5 up 1.00000 1.00000 6 hdd 5.45799 osd.6 up 1.00000 1.00000 7 hdd 5.45799 osd.7 up 1.00000 1.00000 8 hdd 5.45799 osd.8 up 1.00000 1.00000 9 hdd 5.45799 osd.9 up 1.00000 1.00000 16 hdd 5.45799 osd.16 up 1.00000 1.00000 -7 32.74796 host SV3 10 hdd 5.45799 osd.10 up 1.00000 1.00000 11 hdd 5.45799 osd.11 up 1.00000 1.00000 12 hdd 5.45799 osd.12 up 1.00000 1.00000 13 hdd 5.45799 osd.13 up 1.00000 1.00000 14 hdd 5.45799 osd.14 up 1.00000 1.00000 17 hdd 5.45799 osd.17 up 1.00000 1.00000
not the best situation, but they absolutely need to access the file inside of the storage:
root@SV3:~# rbd list Storage vm-101-disk-0 vm-101-disk-1 vm-101-disk-2 vm-102-disk-0 vm-102-disk-1 vm-103-disk-0 vm-103-disk-1 vm-103-disk-2
but it is not possible to do a backup or a move. Any ideas?
Ceph Common Issues
Many of these problem cases are hard to summarize down to a short phrase that adequately describes the problem. Each problem will start with a bulleted list of symptoms. Keep in mind that all symptoms may not apply depending on the configuration of Rook. If the majority of the symptoms are seen there is a fair chance you are experiencing that problem.
If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have registered for the Rook Slack, proceed to the #ceph channel to ask for assistance.
Troubleshooting Techniques¶
There are two main categories of information you will need to investigate issues in the cluster:
- Kubernetes status and logs documented here
- Ceph cluster status (see upcoming Ceph tools section)
Ceph Tools¶
After you verify the basic health of the running pods, next you will want to run Ceph tools for status of the storage components. There are two ways to run the Ceph tools, either in the Rook toolbox or inside other Rook pods that are already running.
- Logs on a specific node to find why a PVC is failing to mount
- See the log collection topic for a script that will help you gather the logs
- Other artifacts:
- The monitors that are expected to be in quorum: kubectl -n get configmap rook-ceph-mon-endpoints -o yaml | grep data
Tools in the Rook Toolbox¶
The rook-ceph-tools pod provides a simple environment to run Ceph tools. Once the pod is up and running, connect to the pod to execute Ceph commands to evaluate that current state of the cluster.
Ceph Commands¶
Here are some common commands to troubleshoot a Ceph cluster:
- ceph status
- ceph osd status
- ceph osd df
- ceph osd utilization
- ceph osd pool stats
- ceph osd tree
- ceph pg stat
The first two status commands provide the overall cluster health. The normal state for cluster operations is HEALTH_OK, but will still function when the state is in a HEALTH_WARN state. If you are in a WARN state, then the cluster is in a condition that it may enter the HEALTH_ERROR state at which point all disk I/O operations are halted. If a HEALTH_WARN state is observed, then one should take action to prevent the cluster from halting when it enters the HEALTH_ERROR state.
There are many Ceph sub-commands to look at and manipulate Ceph objects, well beyond the scope this document. See the Ceph documentation for more details of gathering information about the health of the cluster. In addition, there are other helpful hints and some best practices located in the Advanced Configuration section. Of particular note, there are scripts for collecting logs and gathering OSD information there.
Cluster failing to service requests¶
Symptoms¶
- Execution of the ceph command hangs
- PersistentVolumes are not being created
- Large amount of slow requests are blocking
- Large amount of stuck requests are blocking
- One or more MONs are restarting periodically
Investigation¶
Create a rook-ceph-tools pod to investigate the current state of Ceph. Here is an example of what one might see. In this case the ceph status command would just hang so a CTRL-C needed to be sent.
1 2 3 4
Another indication is when one or more of the MON pods restart frequently. Note the ‘mon107’ that has only been up for 16 minutes in the following output.
1 2 3 4 5 6 7 8
kubectl -n rook-ceph get all -o wide --show-all
Solution¶
What is happening here is that the MON pods are restarting and one or more of the Ceph daemons are not getting configured with the proper cluster information. This is commonly the result of not specifying a value for dataDirHostPath in your Cluster CRD.
The dataDirHostPath setting specifies a path on the local host for the Ceph daemons to store configuration and data. Setting this to a path like /var/lib/rook , reapplying your Cluster CRD and restarting all the Ceph daemons (MON, MGR, OSD, RGW) should solve this problem. After the Ceph daemons have been restarted, it is advisable to restart the rook-tools pod.
Monitors are the only pods running¶
Symptoms¶
- Rook operator is running
- Either a single mon starts or the mons start very slowly (at least several minutes apart)
- The crash-collector pods are crashing
- No mgr, osd, or other daemons are created except the CSI driver
Investigation¶
When the operator is starting a cluster, the operator will start one mon at a time and check that they are healthy before continuing to bring up all three mons. If the first mon is not detected healthy, the operator will continue to check until it is healthy. If the first mon fails to start, a second and then a third mon may attempt to start. However, they will never form quorum and the orchestration will be blocked from proceeding.
The crash-collector pods will be blocked from starting until the mons have formed quorum the first time.
There are several common causes for the mons failing to form quorum:
- The operator pod does not have network connectivity to the mon pod(s). The network may be configured incorrectly.
- One or more mon pods are in running state, but the operator log shows they are not able to form quorum
- A mon is using configuration from a previous installation. See the cleanup guide for cleaning the previous cluster.
- A firewall may be blocking the ports required for the Ceph mons to form quorum. Ensure ports 6789 and 3300 are enabled. See the Ceph networking guide for more details.
- There may be MTU mismatch between different networking components. Some networks may be more susceptible to mismatch than others. If Kubernetes CNI or hosts enable jumbo frames (MTU 9000), Ceph will use large packets to maximize network bandwidth. If other parts of the networking chain don’t support jumbo frames, this could result in lost or rejected packets unexpectedly.
Operator fails to connect to the mon¶
First look at the logs of the operator to confirm if it is able to connect to the mons.
Likely you will see an error similar to the following that the operator is timing out when connecting to the mon. The last command is ceph mon_status , followed by a timeout message five minutes later.
1 2 3 4 5 6
The error would appear to be an authentication error, but it is misleading. The real issue is a timeout.
Solution¶
If you see the timeout in the operator log, verify if the mon pod is running (see the next section). If the mon pod is running, check the network connectivity between the operator pod and the mon pod. A common issue is that the CNI is not configured correctly.
To verify the network connectivity:
- Get the endpoint for a mon
- Curl the mon from the operator pod
For example, this command will curl the first mon from the operator:
kubectl -n rook-ceph deploy/rook-ceph-operator -- curl -n rook-ceph get svc -l -o
If «ceph v2» is printed to the console, the connection was successful. If the command does not respond or otherwise fails, the network connection cannot be established.
Failing mon pod¶
Second we need to verify if the mon pod started successfully.
kubectl -n rook-ceph get pod -l
If the mon pod is failing as in this example, you will need to look at the mon pod status or logs to determine the cause. If the pod is in a crash loop backoff state, you should see the reason by describing the pod.
1 2 3 4 5 6 7 8
The pod shows a termination status that the keyring does not match the existing keyring kubectl -n rook-ceph describe pod -l
See the solution in the next section regarding cleaning up the dataDirHostPath on the nodes.
Solution¶
This is a common problem reinitializing the Rook cluster when the local directory used for persistence has not been purged. This directory is the dataDirHostPath setting in the cluster CRD and is typically set to /var/lib/rook . To fix the issue you will need to delete all components of Rook and then delete the contents of /var/lib/rook (or the directory specified by dataDirHostPath ) on each of the hosts in the cluster. Then when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected.
Deleting the dataDirHostPath folder is destructive to the storage. Only delete the folder if you are trying to permanently purge the Rook cluster.
See the Cleanup Guide for more details.
PVCs stay in pending state¶
Symptoms¶
- When you create a PVC based on a rook storage class, it stays pending indefinitely
For the WordPress example, you might see two PVCs in pending state.
1 2 3 4
kubectl get pvc
Investigation¶
There are two common causes for the PVCs staying in pending state:
- There are no OSDs in the cluster
- The CSI provisioner pod is not running or is not responding to the request to provision the storage
Confirm if there are OSDs¶
To confirm if you have OSDs in your cluster, connect to the Rook Toolbox and run the ceph status command. You should see that you have at least one OSD up and in . The minimum number of OSDs required depends on the replicated.size setting in the pool created for the storage class. In a «test» cluster, only one OSD is required (see storageclass-test.yaml ). In the production storage class example ( storageclass.yaml ), three OSDs would be required.
1 2 3 4 5 6 7 8 9
ceph status
OSD Prepare Logs¶
If you don’t see the expected number of OSDs, let’s investigate why they weren’t created. On each node where Rook looks for OSDs to configure, you will see an «osd prepare» pod.
kubectl -n rook-ceph get pod -l
See the section on why OSDs are not getting created to investigate the logs.
CSI Driver¶
The CSI driver may not be responding to the requests. Look in the logs of the CSI provisioner pod to see if there are any errors during the provisioning.
There are two provisioner pods:
Get the logs of each of the pods. One of them should be the «leader» and be responding to requests.
Operator unresponsiveness¶
Lastly, if you have OSDs up and in , the next step is to confirm the operator is responding to the requests. Look in the Operator pod logs around the time when the PVC was created to confirm if the request is being raised. If the operator does not show requests to provision the block image, the operator may be stuck on some other operation. In this case, restart the operator pod to get things going again.
Solution¶
If the «osd prepare» logs didn’t give you enough clues about why the OSDs were not being created, please review your cluster.yaml configuration. The common misconfigurations include:
- If useAllDevices: true , Rook expects to find local devices attached to the nodes. If no devices are found, no OSDs will be created.
- If useAllDevices: false , OSDs will only be created if deviceFilter is specified.
- Only local devices attached to the nodes will be configurable by Rook. In other words, the devices must show up under /dev .
- The devices must not have any partitions or filesystems on them. Rook will only configure raw devices. Partitions are not yet supported.
OSD pods are failing to start¶
Symptoms¶
- OSD pods are failing to start
- You have started a cluster after tearing down another cluster
Investigation¶
When an OSD starts, the device or directory will be configured for consumption. If there is an error with the configuration, the pod will crash and you will see the CrashLoopBackoff status for the pod. Look in the osd pod logs for an indication of the failure.
kubectl -n rook-ceph logs rook-ceph-osd-fl8fs
One common case for failure is that you have re-deployed a test cluster and some state may remain from a previous deployment. If your cluster is larger than a few nodes, you may get lucky enough that the monitors were able to start and form quorum. However, now the OSDs pods may fail to start due to the old state. Looking at the OSD pod logs you will see an error about the file already existing.
1 2 3 4 5 6 7
kubectl -n rook-ceph logs rook-ceph-osd-fl8fs
Solution¶
If the error is from the file that already exists, this is a common problem reinitializing the Rook cluster when the local directory used for persistence has not been purged. This directory is the dataDirHostPath setting in the cluster CRD and is typically set to /var/lib/rook . To fix the issue you will need to delete all components of Rook and then delete the contents of /var/lib/rook (or the directory specified by dataDirHostPath ) on each of the hosts in the cluster. Then when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected.
OSD pods are not created on my devices¶
Symptoms¶
- No OSD pods are started in the cluster
- Devices are not configured with OSDs even though specified in the Cluster CRD
- One OSD pod is started on each node instead of multiple pods for each device
Investigation¶
First, ensure that you have specified the devices correctly in the CRD. The Cluster CRD has several ways to specify the devices that are to be consumed by the Rook storage:
- useAllDevices: true : Rook will consume all devices it determines to be available
- deviceFilter : Consume all devices that match this regular expression
- devices : Explicit list of device names on each node to consume
Second, if Rook determines that a device is not available (has existing partitions or a formatted filesystem), Rook will skip consuming the devices. If Rook is not starting OSDs on the devices you expect, Rook may have skipped it for this reason. To see if a device was skipped, view the OSD preparation log on the node where the device was skipped. Note that it is completely normal and expected for OSD prepare pod to be in the completed state. After the job is complete, Rook leaves the pod around in case the logs need to be investigated.
1 2 3 4 5 6
Get the prepare pods the cluster kubectl -n rook-ceph get pod -l
view the logs the node of interest the container kubectl -n rook-ceph logs rook-ceph-osd-prepare-node1-fvmrp provision
Here are some key lines to look for in the log:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
A device will be skipped Rook sees it has partitions or a filesystem Other messages about a disk being unusable by ceph include: A device is going to be configured For each device configured you will see a report printed to the log
Solution¶
Either update the CR with the correct settings, or clean the partitions or filesystem from your devices. To clean devices from a previous install see the cleanup guide.
After the settings are updated or the devices are cleaned, trigger the operator to analyze the devices again by restarting the operator. Each time the operator starts, it will ensure all the desired devices are configured. The operator does automatically deploy OSDs in most scenarios, but an operator restart will cover any scenarios that the operator doesn’t detect automatically.
Restart the operator to ensure devices are configured. A new pod will automatically be started when the current operator pod is deleted. kubectl -n rook-ceph delete pod -l
Node hangs after reboot¶
This issue is fixed in Rook v1.3 or later.
Symptoms¶
- After issuing a reboot command, node never returned online
- Only a power cycle helps
Investigation¶
On a node running a pod with a Ceph persistent volume
1 2 3 4
_netdev mount option is absent, also occurs cephfs OS is not aware PV is mounted over network
When the reboot command is issued, network interfaces are terminated before disks are unmounted. This results in the node hanging as repeated attempts to unmount Ceph persistent volumes fail with the following error:
Solution¶
The node needs to be drained before reboot. After the successful drain, the node can be rebooted as usual.
Because kubectl drain command automatically marks the node as unschedulable ( kubectl cordon effect), the node needs to be uncordoned once it’s back online.
--ignore-daemonsets --delete-local-data
Uncordon the node:
Using multiple shared filesystem (CephFS) is attempted on a kernel version older than 4.7¶
Symptoms¶
- More than one shared filesystem (CephFS) has been created in the cluster
- A pod attempts to mount any other shared filesystem besides the first one that was created
- The pod incorrectly gets the first filesystem mounted instead of the intended filesystem
Solution¶
The only solution to this problem is to upgrade your kernel to 4.7 or higher. This is due to a mount flag added in the kernel version 4.7 which allows to chose the filesystem by name.
For additional info on the kernel version requirement for multiple shared filesystems (CephFS), see Filesystem — Kernel version requirement.
Set debug log level for all Ceph daemons¶
You can set a given log level and apply it to all the Ceph daemons at the same time. For this, make sure the toolbox pod is running, then determine the level you want (between 0 and 20). You can find the list of all subsystems and their default values in Ceph logging and debug official guide. Be careful when increasing the level as it will produce very verbose logs.
Assuming you want a log level of 1, you will run:
1 2 3 4
kubectl -n rook-ceph deploy/rook-ceph-tools -- set-ceph-debug-level
Once you are done debugging, you can revert all the debug flag to their default value by running the following:
Activate log to file for a particular Ceph daemon¶
They are cases where looking at Kubernetes logs is not enough for diverse reasons, but just to name a few:
- not everyone is familiar for Kubernetes logging and expects to find logs in traditional directories
- logs get eaten (buffer limit from the log engine) and thus not requestable from Kubernetes
So for each daemon, dataDirHostPath is used to store logs, if logging is activated. Rook will bindmount dataDirHostPath for every pod. Let’s say you want to enable logging for mon.a , but only for this daemon. Using the toolbox or from inside the operator run:
This will activate logging on the filesystem, you will be able to find logs in dataDirHostPath/$NAMESPACE/log , so typically this would mean /var/lib/rook/rook-ceph/log . You don’t need to restart the pod, the effect will be immediate.
To disable the logging on file, simply set log_to_file to false .
A worker node using RBD devices hangs up¶
Symptoms¶
- There is no progress on I/O from/to one of RBD devices ( /dev/rbd* or /dev/nbd* ).
- After that, the whole worker node hangs up.
Investigation¶
This happens when the following conditions are satisfied.
- The problematic RBD device and the corresponding OSDs are co-located.
- There is an XFS filesystem on top of this device.
In addition, when this problem happens, you can see the following messages in dmesg .
1 2 3 4 5 6
dmesg /proc/sys/kernel/hung_task_timeout_secs" disables this message.
It’s so-called hung_task problem and means that there is a deadlock in the kernel. For more detail, please refer to the corresponding issue comment.
Solution¶
This problem will be solve by the following two fixes.
- Linux kernel: A minor feature that is introduced by this commit. It will be included in Linux v5.6.
- Ceph: A fix that uses the above-mentioned kernel’s feature. The Ceph community will probably discuss this fix after releasing Linux v5.6.
You can bypass this problem by using ext4 or any other filesystems rather than XFS. Filesystem type can be specified with csi.storage.k8s.io/fstype in StorageClass resource.
Too few PGs per OSD warning is shown¶
Symptoms¶
- ceph status shows «too few PGs per OSD» warning as follows.
1 2 3 4 5 6
ceph status
Solution¶
The meaning of this warning is written in the document. However, in many cases it is benign. For more information, please see the blog entry. Please refer to Configuring Pools if you want to know the proper pg_num of pools and change these values.
LVM metadata can be corrupted with OSD on LV-backed PVC¶
Symptoms¶
There is a critical flaw in OSD on LV-backed PVC. LVM metadata can be corrupted if both the host and OSD container modify it simultaneously. For example, the administrator might modify it on the host, while the OSD initialization process in a container could modify it too. In addition, if lvmetad is running, the possibility of occurrence gets higher. In this case, the change of LVM metadata in OSD container is not reflected to LVM metadata cache in host for a while.
If you still decide to configure an OSD on LVM, please keep the following in mind to reduce the probability of this issue.
Solution¶
- Disable lvmetad.
- Avoid configuration of LVs from the host. In addition, don’t touch the VGs and physical volumes that back these LVs.
- Avoid incrementing the count field of storageClassDeviceSets and create a new LV that backs an OSD simultaneously.
You can know whether the above-mentioned tag exists with the command: sudo lvs -o lv_name,lv_tags . If the lv_tag field is empty in an LV corresponding to the OSD lv_tags, this OSD encountered the problem. In this case, please retire this OSD or replace with other new OSD before restarting.
This problem doesn’t happen in newly created LV-backed PVCs because OSD container doesn’t modify LVM metadata anymore. The existing lvm mode OSDs work continuously even thought upgrade your Rook. However, using the raw mode OSDs is recommended because of the above-mentioned problem. You can replace the existing OSDs with raw mode OSDs by retiring them and adding new OSDs one by one. See the documents Remove an OSD and Add an OSD on a PVC.
OSD prepare job fails due to low aio-max-nr setting¶
If the Kernel is configured with a low aio-max-nr setting, the OSD prepare job might fail with the following error:
exec: stderr: 2020-09-17T00:30:12.145+0000 7f0c17632f40 -1 bdev(0x56212de88700 /var/lib/ceph/osd/ceph-0//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr
To overcome this, you need to increase the value of fs.aio-max-nr of your sysctl configuration (typically /etc/sysctl.conf ). You can do this with your favorite configuration management system.
Alternatively, you can have a DaemonSet to apply the configuration for you on all your nodes.
Unexpected partitions created¶
Symptoms¶
Users running Rook versions v1.6.0-v1.6.7 may observe unwanted OSDs on partitions that appear unexpectedly and seemingly randomly, which can corrupt existing OSDs.
Unexpected partitions are created on host disks that are used by Ceph OSDs. This happens more often on SSDs than HDDs and usually only on disks that are 875GB or larger. Many tools like lsblk , blkid , udevadm , and parted will not show a partition table type for the partition. Newer versions of blkid are generally able to recognize the type as «atari».
The underlying issue causing this is Atari partition (sometimes identified as AHDI) support in the Linux kernel. Atari partitions have very relaxed specifications compared to other partition types, and it is relatively easy for random data written to a disk to appear as an Atari partition to the Linux kernel. Ceph’s Bluestore OSDs have an anecdotally high probability of writing data on to disks that can appear to the kernel as an Atari partition.
Below is an example of lsblk output from a node where phantom Atari partitions are present. Note that sdX1 is never present for the phantom partitions, and sdX2 is 48G on all disks. sdX3 is a variable size and may not always be present. It is possible for sdX4 to appear, though it is an anecdotally rare event.
1 2 3 4 5 6 7 8 9 10 11
lsblk
Solution¶
Recover from corruption (v1.6.0-v1.6.7)¶
If you are using Rook v1.6, you must first update to v1.6.8 or higher to avoid further incidents of OSD corruption caused by these Atari partitions.
An old workaround suggested using deviceFilter: ^sd[a-z]+$ , but this still results in unexpected partitions. Rook will merely stop creating new OSDs on the partitions. It does not fix a related issue that ceph-volume that is unaware of the Atari partition problem. Users who used this workaround are still at risk for OSD failures in the future.
To resolve the issue, immediately update to v1.6.8 or higher. After the update, no corruption should occur on OSDs created in the future. Next, to get back to a healthy Ceph cluster state, focus on one corrupted disk at a time and remove all OSDs on each corrupted disk one disk at a time.
As an example, you may have /dev/sdb with two unexpected partitions ( /dev/sdb2 and /dev/sdb3 ) as well as a second corrupted disk /dev/sde with one unexpected partition ( /dev/sde2 ).
- First, remove the OSDs associated with /dev/sdb , /dev/sdb2 , and /dev/sdb3 . There might be only one, or up to 3 OSDs depending on how your system was affected. Again see the OSD management doc.
- Use dd to wipe the first sectors of the partitions followed by the disk itself. E.g.,
- dd if=/dev/zero of=/dev/sdb2 bs=1M
- dd if=/dev/zero of=/dev/sdb3 bs=1M
- dd if=/dev/zero of=/dev/sdb bs=1M
- Then wipe clean /dev/sdb to prepare it for a new OSD. See the teardown document for details.
- After this, scale up the Rook operator to deploy a new OSD to /dev/sdb . This will allow Ceph to use /dev/sdb for data recovery and replication while the next OSDs are removed.
- Now Repeat steps 1-4 for /dev/sde and /dev/sde2 , and continue for any other corrupted disks.
If your Rook cluster does not have any critical data stored in it, it may be simpler to uninstall Rook completely and redeploy with v1.6.8 or higher.
Operator environment variables are ignored¶
Symptoms¶
Configuration settings passed as environment variables do not take effect as expected. For example, the discover daemonset is not created, even though ROOK_ENABLE_DISCOVERY_DAEMON=»true» is set.
Investigation¶
Inspect the rook-ceph-operator-config ConfigMap for conflicting settings. The ConfigMap takes precedence over the environment. The ConfigMap must exist, even if all actual configuration is supplied through the environment.
Look for lines with the op-k8sutil prefix in the operator logs. These lines detail the final values, and source, of the different configuration variables.
Verify that both of the following messages are present in the operator logs:
rook-ceph-operator-config-controller successfully started rook-ceph-operator-config-controller done reconciling
Solution¶
If it does not exist, create an empty ConfigMap:
1 2 3 4 5 6
ConfigMap v1 rook-ceph-operator-config rook-ceph <>
If the ConfigMap exists, remove any keys that you wish to configure through the environment.
© Rook Authors 2022. Documentation distributed under CC-BY-4.0.
© 2022 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.