Showing results for 
Search instead for 
Did you mean: 

How to monitor ES (Elastic Search) health from controller logs and CLI?


What is ES?


Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real-time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements." —


Starting from the 3.5 version, SZ/VSZ uses ES, especially for quick response for UI, needs and we have two data sources to store data. Some of them are in Cassandra (Configuration), some of them are in ES(status/stats). We rely on an underlying sync mechanism to make sure data consistency.


Why does the ES issue happen?


The problem can be separated into two parts.

  1. UI Not responding.

There are many reasons cause the ES cluster failed. In most cases, network partition (or unstable network connection between nodes). In such cases, we have a few ways to do recovery automatically. Configurer service can do some recovery but only try to restart the ES node.

  1. Data Out of sync.

Many reasons can probably fail ES, in this situation, we must check the log and deal with it in a different way. Reindex is the final solution to solve this problem. However, to confirm the problem is out of sync, usually need to manually check, not every case can be resolved by reindexing procedure.


Issues that you will notice on UI:


Pop-up error messages about ES keywords (For example: "all shards failed")


Image_ images_messages_61f798e72c844f7ca99369ad_21ddcf61f6ed0a4eb5839b9d7f715640_Allshardsfailed-f9e7da73-8889-4e73-991c-4fc1557ea26b-733799386.JPG


Pop-up error messages about "alias missing"


Image_ images_messages_61f798e72c844f7ca99369ad_c8e27c7585ecfd2da327de4561602a3a_AliasMissing-e53cbeb9-64c7-4cba-8c2e-2af5f1b0c5c4-1521986281.JPG


Show no result in WebGUI (For example: WLAN tab or AP tab show no result)


Image_ images_messages_61f798e72c844f7ca99369ad_4a48bd0f237d5039383149f05ca61f5e_wlanmissing-e9e05333-da61-4959-a7f7-99db6b08e2bc-215179319.JPG


AP Traffic column or radio (2.4 or 5GHz) column would show N/A.


Image_ images_messages_61f798e72c844f7ca99369ad_968872a55d02e47e99f2144e3341d60b_APtrafficNA-579e0fe8-d5d9-457c-bcf4-4d5fbada3294-1389817204.JPG


And sometimes, navigating between different tabs of the controller you would see “An Unknown error” occurred message.


How to identify from the logs if ES has gone bad?


Download Snapshot logs from the SZ/vSZ, by navigating to Diagnostics > Application logs


Image_ images_messages_61f798e72c844f7ca99369ad_1fd9d1286bf722ea1601346882f55061_Snapshotlogs-b56ac155-e5ca-4795-89a6-a6844e4515e6-847213798.JPG


  1. If Elastic Search service is offline, that could be verified by running the command “show service”
  2. Once you download the snapshot logs extract them using 7 zip. Then navigate to applogfiles folder, you would see all services folders of the controller.



Configurer[pool-4-thread-1] INFO c.r.w.c.s.ElasticClientService - Start to initial ES client.
Configurer[pool-4-thread-1] INFO c.r.w.c.s.ElasticClientService - Failed to init ES client
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable( ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.client.transport.TransportClientNodesService.execute( ~[elasticsearch-1.7.2.jar:na]
at ~[elasticsearch-1.7.2.jar:na]




Caused by: org.elasticsearch.indices.IndexMissingException: [alias_apmeshstatus_all] missing
at org.elasticsearch.cluster.metadata.MetaData.convertFromWildcards( ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.cluster.metadata.MetaData.concreteIndices( ~[elasticsearch-1.7.2.jar:na]
at$BaseAsyncAction.<init>( ~[elasticsearch-1.7.2.jar:na]
at$AsyncAction.<init>( ~[elasticsearch-1.7.2.jar:na]



Web[localhost-startStop-2] ERROR c.r.s.d.s.r.ReIndexServiceImpl - init reindex [com.ruckuswireless.scg.domain.service.reindex.APReIndexCommand@47dda78] failed
org.elasticsearch.indices.IndexMissingException: [alias_apmeshstatus_all] missing
at org.elasticsearch.cluster.metadata.MetaData.convertFromWildcards( ~[elasticsearch-1.7.2.jar:na]
at org.elasticsearch.cluster.metadata.MetaData.concreteIndices( ~[elasticsearch-1.7.2.jar:na]
at$BaseAsyncAction.<init>( ~[elasticsearch-1.7.2.jar:na]


Configurer/es-monitor.log (search by “health” keyword), check if the health shows red or yellow.


Image_ images_messages_61f798e72c844f7ca99369ad_db6be6cff2192a1f6757765460d8920e_redyellow-f107512b-130f-47df-a7cf-b7756702040b-1185417880.JPG


In the same “es-monitor.log” (search by “UNASSIGNED” keyword)


index                           shard prirep state        docs   store ip           node
apcliscripthistory_20220127     1     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     1     r      UNASSIGNED
apcliscripthistory_20220127     0     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     0     r      UNASSIGNED


From version, 5.2 onwards, we can check the ES health status from the controller CLI:



vSZ-H-179(debug)# debug-tools

[Change to system]

Welcome to Debug CLI Framework!

(debug tool-set) system $ use sz

[Change to sz]

(debug tool-set) sz $ ?

Debug Tools (sz):

Command          Help

================ ================================
show-es-cat-aliases show ES cat aliases info
show-es-cat-health show ES cat health info
show-es-cat-indices show ES cat indices info
show-es-cat-master show ES cat master info
show-es-cat-nodes show ES cat nodes info
show-es-cat-shards show ES cat shards info
show-es-cluster-settings show ES cluster settings
show-es-folder-info show ES folder info


For example: If you want to check the ES health, you could run the below command. If it shows “green” and 100% then ES health is fine. If it shows “red” or “yellow”, then ES has gone bad.


(debug tool-set) sz $ show-es-cat-health
epoch      timestamp cluster      status shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1643352441 06:47:21  vSZ-H-179_54 green           2         2    472 236    0    0        0             0                  -                100.0%


To check if the Indexes are in an UNASSIGNED state or not, if it shows like below then ES is OK.


(debug tool-set) sz $ show-es-cat-shards
*** Unknown syntax: show-es-cat-shards
index                           shard prirep state   docs  store ip            node
hccdclientconnection_20220128   1     r      STARTED    0   130b x.x.x.x bbf54faf-508b-44a3-8804-76a1b5da4b2d


If it shows like below then ES is not OK:


index                           shard prirep state        docs   store ip           node
apcliscripthistory_20220127     1     p      STARTED         0    130b x.x.x.x 614c6b97-baa4-488e-8ace-e62b2f086855
apcliscripthistory_20220127     1     r      UNASSIGNED


A brief note on INDEX and SHARDs:



An INDEX is a collection of documents that have somewhat similar characteristics. It is identified by a name and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it. In a single cluster, you can define as many indexes as you want.



An index can potentially store a large amount of data that can exceed the hardware limits of a single node. To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called SHARDS. Each shard is in itself a fully functional and independent "index" that can be hosted on any node in the cluster.




If Elastic Search service is offline, then perform below: (plan a downtime)


Follow the below steps when the Controller is running on 3.5.1.x firmware

- Execute the below command in the debug mode to recover the Elastic Search service on the vSZ-E/SZ-100.


SZ-100(debug)#reload now


Follow the below commands for controller running from 3.6.x and above

You don't have to go to debug mode in here.


SZ-100# force-recover-escluster
SZ-100# reload now


Once the Controller is online, check if all the data is available or not. If there is any data missing, execute the following command to recover the data.




If Elastic Search service is not offline, and still you see some of the Data missing on UI or the errors as shown in the screenshot earlier, then only perform




This command needs to be performed on individual nodes in the cluster one after another and does not require downtime. If the above-mentioned Troubleshooting steps still do not help in recovering the ES or the Data on UI, then reach out to TAC. There are certain commands that need to be run from the shell of the controller to recover the ES.


Hope the above article helps you in identifying the Elastic Search-related issues and resolving them.



Happy Learning!!


Valued Contributor II

Very useful article!

Of cause, the best is to maintain good network connections and all nodes in good health. But it is very important to know what to do when something out of our control happens and data are out of synch...

I had a few situation when ES re-indexing was really necessary and recovered the system data. TAC helped me with that, so I learned this commands and can now do similar proceeding myself. 

In one case after nasty power outage data were messed so far, that AP were missing from proper zones, instead were placed in wrong zones, and re-indexing fixed everything. Re-indexing spared me effort of restoring backups... 

But you still must have proper backups, as sometimes data can be damaged as well, not only indexes. 

@eizens_putnins​ You are absolutely correct!! Proper backups especially cluster backups should be there for the worst-case scenarios to recover. This article will help in doing initial triage.

Contributor III

Hi @nayanendu_mallick ,

Great Reference!! Thank you for sharing this one.