cancel
Showing results for 
Search instead for 
Did you mean: 

How to monitor ES (Elastic Search) health from controller logs and CLI?

Nayanendu
Moderator
Moderator

What is ES?

 

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real-time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements." — https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html

 

Starting from the 3.5 version, SZ/VSZ uses ES, especially for quick response for UI, needs and we have two data sources to store data. Some of them are in Cassandra (Configuration), some of them are in ES(status/stats). We rely on an underlying sync mechanism to make sure data consistency.

 

Why does the ES issue happen?

 

The problem can be separated into two parts.

  1. UI Not responding.

There are many reasons cause the ES cluster failed. In most cases, network partition (or unstable network connection between nodes). In such cases, we have a few ways to do recovery automatically. Configurer service can do some recovery but only try to restart the ES node.

  1. Data Out of sync.

Many reasons can probably fail ES, in this situation, we must check the log and deal with it in a different way. Reindex is the final solution to solve this problem. However, to confirm the problem is out of sync, usually need to manually check, not every case can be resolved by reindexing procedure.

 

Issues that you will notice on UI:

 

Pop-up error messages about ES keywords (For example: "all shards failed")

 

Image_ images_messages_61f798e72c844f7ca99369ad_21ddcf61f6ed0a4eb5839b9d7f715640_Allshardsfailed-f9e7da73-8889-4e73-991c-4fc1557ea26b-733799386.JPG

 

Pop-up error messages about "alias missing"

 

Image_ images_messages_61f798e72c844f7ca99369ad_c8e27c7585ecfd2da327de4561602a3a_AliasMissing-e53cbeb9-64c7-4cba-8c2e-2af5f1b0c5c4-1521986281.JPG

 

Show no result in WebGUI (For example: WLAN tab or AP tab show no result)

 

Image_ images_messages_61f798e72c844f7ca99369ad_4a48bd0f237d5039383149f05ca61f5e_wlanmissing-e9e05333-da61-4959-a7f7-99db6b08e2bc-215179319.JPG

 

AP Traffic column or radio (2.4 or 5GHz) column would show N/A.

 

Image_ images_messages_61f798e72c844f7ca99369ad_968872a55d02e47e99f2144e3341d60b_APtrafficNA-579e0fe8-d5d9-457c-bcf4-4d5fbada3294-1389817204.JPG

 

And sometimes, navigating between different tabs of the controller you would see “An Unknown error” occurred message.

 

How to identify from the logs if ES has gone bad?

 

Download Snapshot logs from the SZ/vSZ, by navigating to Diagnostics > Application logs

 

Image_ images_messages_61f798e72c844f7ca99369ad_1fd9d1286bf722ea1601346882f55061_Snapshotlogs-b56ac155-e5ca-4795-89a6-a6844e4515e6-847213798.JPG

 

  1. If Elastic Search service is offline, that could be verified by running the command “show service”
  2. Once you download the snapshot logs extract them using 7 zip. Then navigate to applogfiles folder, you would see all services folders of the controller.

Configurer/configurer.log

 

Configurer[pool-4-thread-1] INFO c.r.w.c.s.ElasticClientService - Start to initial ES client.
Configurer[pool-4-thread-1] INFO c.r.w.c.s.ElasticClientService - Failed to init ES client
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:86) ~[elasticsearch-1.7.2.jar:na]

 

Core/core.log

 

Caused by: org.elasticsearch.indices.IndexMissingException: [alias_apmeshstatus_all] missing
at org.elasticsearch.cluster.metadata.MetaData.convertFromWildcards(MetaData.java:884) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:692) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.<init>(TransportSearchTypeAction.java:118) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.<init>(TransportSearchDfsQueryThenFetchAction.java:76) ~[elasticsearch-1.7.2.jar:na]

Web/web.log

 

Web[localhost-startStop-2] ERROR c.r.s.d.s.r.ReIndexServiceImpl - init reindex [com.ruckuswireless.scg.domain.service.reindex.APReIndexCommand@47dda78] failed
org.elasticsearch.indices.IndexMissingException: [alias_apmeshstatus_all] missing
at org.elasticsearch.cluster.metadata.MetaData.convertFromWildcards(MetaData.java:884) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:692) ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.<init>(TransportSearchTypeAction.java:118) ~[elasticsearch-1.7.2.jar:na]

 

Configurer/es-monitor.log (search by “health” keyword), check if the health shows red or yellow.

 

Image_ images_messages_61f798e72c844f7ca99369ad_db6be6cff2192a1f6757765460d8920e_redyellow-f107512b-130f-47df-a7cf-b7756702040b-1185417880.JPG

 

In the same “es-monitor.log” (search by “UNASSIGNED” keyword)

 

index                           shard prirep state        docs   store ip           node
apcliscripthistory_20220127     1     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     1     r      UNASSIGNED
apcliscripthistory_20220127     0     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     0     r      UNASSIGNED

 

From version, 5.2 onwards, we can check the ES health status from the controller CLI:

 

vSZ-H-179#debug

vSZ-H-179(debug)# debug-tools

[Change to system]

Welcome to Debug CLI Framework!

(debug tool-set) system $ use sz

[Change to sz]

(debug tool-set) sz $ ?

Debug Tools (sz):

Command          Help

================ ================================
show-es-cat-aliases show ES cat aliases info
show-es-cat-health show ES cat health info
show-es-cat-indices show ES cat indices info
show-es-cat-master show ES cat master info
show-es-cat-nodes show ES cat nodes info
show-es-cat-shards show ES cat shards info
show-es-cluster-settings show ES cluster settings
show-es-folder-info show ES folder info

 

For example: If you want to check the ES health, you could run the below command. If it shows “green” and 100% then ES health is fine. If it shows “red” or “yellow”, then ES has gone bad.

 

(debug tool-set) sz $ show-es-cat-health
epoch      timestamp cluster      status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1643352441 06:47:21  vSZ-H-179_54 green           2         2    472 236    0    0        0             0                  -                100.0%

 

To check if the Indexes are in an UNASSIGNED state or not, if it shows like below then ES is OK.

 

(debug tool-set) sz $ show-es-cat-shards
*** Unknown syntax: show-es-cat-shards
index                           shard prirep state   docs  store ip            node
hccdclientconnection_20220128   1     r      STARTED    0   130b x.x.x.x bbf54faf-508b-44a3-8804-76a1b5da4b2d

 

If it shows like below then ES is not OK:

 

index                           shard prirep state        docs   store ip           node
apcliscripthistory_20220127     1     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     1     r      UNASSIGNED

 

A brief note on INDEX and SHARDs:

 

Index

An INDEX is a collection of documents that have somewhat similar characteristics. It is identified by a name and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it. In a single cluster, you can define as many indexes as you want.

 

Shards

An index can potentially store a large amount of data that can exceed the hardware limits of a single node. To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called SHARDS. Each shard is in itself a fully functional and independent "index" that can be hosted on any node in the cluster.

 

Troubleshooting:

 

If Elastic Search service is offline, then perform below: (plan a downtime)

 

Follow the below steps when the Controller is running on 3.5.1.x firmware

- Execute the below command in the debug mode to recover the Elastic Search service on the vSZ-E/SZ-100.

 

SZ-100(debug)#force-recover-escluster
SZ-100(debug)#reload now

 

Follow the below commands for controller running from 3.6.x and above

You don't have to go to debug mode in here.

 

SZ-100# force-recover-escluster
SZ-100# reload now

 

Once the Controller is online, check if all the data is available or not. If there is any data missing, execute the following command to recover the data.

 

SZ-100(debug)#reindex-elasticsearch-all

 

If Elastic Search service is not offline, and still you see some of the Data missing on UI or the errors as shown in the screenshot earlier, then only perform

 

SZ-100(debug)#reindex-elasticsearch-all

 

This command needs to be performed on individual nodes in the cluster one after another and does not require downtime. If the above-mentioned Troubleshooting steps still do not help in recovering the ES or the Data on UI, then reach out to TAC. There are certain commands that need to be run from the shell of the controller to recover the ES.

 

Hope the above article helps you in identifying the Elastic Search-related issues and resolving them.

 

Cheers!!

Happy Learning!!

3 REPLIES 3

eizens_putnins
Valued Contributor

Very useful article!

Of cause, the best is to maintain good network connections and all nodes in good health. But it is very important to know what to do when something out of our control happens and data are out of synch...

I had a few situation when ES re-indexing was really necessary and recovered the system data. TAC helped me with that, so I learned this commands and can now do similar proceeding myself. 

In one case after nasty power outage data were messed so far, that AP were missing from proper zones, instead were placed in wrong zones, and re-indexing fixed everything. Re-indexing spared me effort of restoring backups... 

But you still must have proper backups, as sometimes data can be damaged as well, not only indexes. 

@eizens_putnins​ You are absolutely correct!! Proper backups especially cluster backups should be there for the worst-case scenarios to recover. This article will help in doing initial triage.

abilashpr
Contributor III

Hi @nayanendu_mallick ,

Great Reference!! Thank you for sharing this one.

Regards,

Abilash