ElasticSearch logging stack consuming high disk usage

Issue

• Cluster in limited support due to ElasticSearch in high disk usage

Your cluster requires you to take action as its ElasticSearch cluster logging deployment has been detected as reaching a high disk usage threshold. Red Hat SRE strongly recommends reducing application logging on your cluster to ensure logging continues to function. If logging disk consumption exceeds 95%, data will be at risk of becoming unavailable or lost and the stability of your ElasticSearch deployment may be impacted.

Your cluster requires you to take action. Elasticsearch is at 100% load. Red Hat SRE strongly recommends immediately reducing application logging on your cluster to ensure logging continues to function in the near future. If logging stack stops functioning the logging stack will need reprovisioned to recover and application logs will be lost.

• Collector pods restarts several times.

$ oc get pods -n openshift-logging    
NAME                                            READY   STATUS      RESTARTS           AGE
cluster-logging-operator-xxxxxxxxx-xxxxx        1/1     Running     0                  177d
collector-xxxxx                                 1/1     Running     13801 (36m ago)    72d
collector-xxxxx                                 1/1     Running     3772 (140m ago)    85d
collector-xxxxx                                 1/1     Running     367 (32h ago)      154d
collector-xxxxx                                 1/1     Running     4953 (78m ago)     154d
collector-xxxxx                                 1/1     Running     385 (4h5m ago)     154d
collector-xxxxx                                 1/1     Running     11866 (67m ago)    154d
collector-xxxxx                                 1/1     Running     787 (166m ago)     154d
collector-xxxxx                                 1/1     Running     2594 (138m ago)    85d
collector-xxxxx                                 1/1     Running     431 (5d ago)       154d
collector-xxxxx                                 1/1     Running     395 (2d19h ago)    154d
collector-xxxxx                                 1/1     Running     338 (4d ago)       154d
collector-xxxxx                                 1/1     Running     223 (18h ago)      75d
collector-xxxxx                                 1/1     Running     514 (174m ago)     154d
collector-xxxxx                                 1/1     Running     6273 (90m ago)     154d
collector-xxxxx                                 1/1     Running     838 (21h ago)      154d
collector-xxxxx                                 1/1     Running     10331 (43m ago)    154d
collector-xxxxx                                 1/1     Running     512 (2d5h ago)     154d
collector-xxxxx                                 1/1     Running     611 (3d4h ago)     154d
collector-xxxxx                                 1/1     Running     4427 (56m ago)     154d
collector-xxxxx                                 1/1     Running     1159 (25h ago)     154d
collector-xxxxx                                 1/1     Running     7437 (136m ago)    154d
collector-xxxxx                                 1/1     Running     318 (45h ago)      154d
collector-xxxxx                                 1/1     Running     664 (122m ago)     154d
collector-xxxxx                                 1/1     Running     861 (61m ago)      154d
collector-xxxxx                                 1/1     Running     1334 (133m ago)    107d
collector-xxxxx                                 1/1     Running     2743 (10d ago)     154d
collector-xxxxx                                 1/1     Running     892 (93m ago)      154d
collector-xxxxx                                 1/1     Running     658 (9d ago)       154d
collector-xxxxx                                 1/1     Running     473 (4h24m ago)    154d
collector-xxxxx                                 1/1     Running     13431 (19m ago)    154d
collector-xxxxx                                 1/1     Running     670 (111m ago)     154d
collector-xxxxx                                 1/1     Running     11151 (21m ago)    154d
collector-xxxxx                                 1/1     Running     2621 (3h45m ago)   154d
collector-xxxxx                                 1/1     Running     10225 (94m ago)    72d
elasticsearch-xxx-xxxxxxxx-x-xxxxxxxxx-xxxxx    2/2     Running     6 (25h ago)        21d
elasticsearch-xxx-xxxxxxxx-x-xxxxxxxxx-xxxxx    2/2     Running     4 (2d22h ago)      21d
elasticsearch-xxx-xxxxxxxx-x-xxxxxxxxx-xxxxx    2/2     Running     0                  21d
elasticsearch-xx-xxx-xxxxxxxx-xxxxx             0/1     Completed   0                  41m
elasticsearch-xx-xxx-xxxxxxxx-xxxxx             0/1     Error       0                  11m
elasticsearch-xx-xxx-xxxxxxxx-xxxxx             0/1     Error       0                  8h
elasticsearch-xx-xxxxx-xxxxxxxx-xxxxx           0/1     Completed   0                  11m
elasticsearch-xx-xxxxx-xxxxxxxx-xxxxx           0/1     Error       0                  9h
elasticsearch-xx-xxxxx-xxxxxxxx-xxxxx           0/1     Completed   0                  11m
kibana-xxxxxxxxx-xxxxx                          2/2     Running     0                  21d