VSZ cluster out of service

[igor_cherniienk]

HI! I have a problem after my VM Ruckus crash, because space in storage out, after that vm start but elastic search not starting, of course cluster out of service

some log 

elasticsearch-console

[2020-06-24 11:03:19,076][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initialized
[2020-06-24 11:03:19,077][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] starting ...
[2020-06-24 11:03:19,311][INFO ][transport                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9300]}, publish_address {inet[/172.16.55.55:9300]}
[2020-06-24 11:03:19,341][INFO ][discovery                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ruckus-controller/Edss4AbqTn2lU1zXlRtoag
[2020-06-24 11:03:22,421][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] new_master [79338a8c-a9af-4a2c-b1df-3c9995e48f35][Edss4AbqTn2lU1zXlRtoag][ltdc-ruckus][inet[/172.16.55.55:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2020-06-24 11:03:22,717][INFO ][http                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9200]}, publish_address {inet[/172.16.55.55:9200]}
[2020-06-24 11:03:22,717][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] started
[2020-06-24 11:03:22,968][INFO ][gateway                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] recovered [182] indices into cluster_state
[2020-06-24 11:03:26,114][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] added {[Arthur Parks][Qjxp6SgyRmCvMfSCLFFDYA][ltdc-ruckus][inet[/172.16.55.55:9306]]{client=true, data=false, master=false},[Helmut Zemo][kGWPSMNhQVek-XwotvV9QQ][ltdc-ruckus][inet[/172.16.55.55:9309]]{client=true, data=false, master=false},[Sack][8HaEo-MARW-aMgSmBXHKKg][ltdc-ruckus][inet[/172.16.55.55:9307]]{client=true, data=false, master=false},[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false},[Spyne][2vevL8--QrGYeDjx8QumBg][ltdc-ruckus][inet[/172.16.55.55:9303]]{client=true, data=false, master=false},[Infamnia][FFYEz1zDReW93gWvIjOwyQ][ltdc-ruckus][inet[/172.16.55.55:9304]]{client=true, data=false, master=false},[Plantman][gcNwzC23SJSE90dnmni9EQ][ltdc-ruckus][inet[/172.16.55.55:9301]]{client=true, data=false, master=false},[Hammerhead][lVNX2zcgRJ6Z6RrC0qVNiw][ltdc-ruckus][inet[/172.16.55.55:9308]]{client=true, data=false, master=false},[Baron Samedi][BBywHDwhSNa89EpsVVhLBw][ltdc-ruckus][inet[/172.16.55.55:9302]]{client=true, data=false, master=false},}, reason: zen-disco-receive(join from node[[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false}])
[2020-06-24 11:12:00,009][DEBUG][action.admin.cluster.health] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] observer: timeout notification from cluster service. timeout setting [6m], time since start [6m]

ElasticSearch_gc


OpenJDK 64-Bit Server VM (25.91-b14) for linux-amd64 JRE (1.8.0_91-b14), built on May 11 2016 09:56:20 by "mockbuild" with gcc 4.4.7 20120313 (Red Hat 4.4.7-17)
Memory: 4k page, physical 16467324k(1092376k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:ErrorFile=/opt/ruckuswireless/wsg/log/elasticsearch/elasticsearch-jvm_error%p.log -XX:GCLogFileSize=10485760 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/ruckuswireless/wsg/log/elasticsearch/elasticsearch-dump.hprof -XX:InitialHeapSize=1610612736 -XX:+ManagementServer -XX:MaxHeapSize=1610612736 -XX:MaxNewSize=160432128 -XX:MaxTenuringThreshold=6 -XX:NewSize=160432128 -XX:NumberOfGCLogFiles=10 -XX:OldPLABSize=16 -XX:/opt/ruckuswireless/wsg/cli/bin/elasticsearch.py -f oomstop -XX:+PrintFlagsFinal -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC 
2020-06-24T11:03:15.127+0000: 0.186: Total time for which application threads were stopped: 0.0117804 seconds, Stopping threads took: 0.0116780 seconds

ElasticSearch_ruckus-controller

[2020-06-24 11:03:15,646][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] version[1.7.2], pid[23204], build[e43676b/2015-09-14T09:49:53Z]
[2020-06-24 11:03:15,646][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initializing ...
[2020-06-24 11:03:15,784][INFO ][plugins                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] loaded [], sites []
[2020-06-24 11:03:15,840][INFO ][env                      ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] using [1] data paths, mounts [[/data (/dev/mapper/vg00-lv00)]], net usable_space [72.1gb], net total_space [87.4gb], types [ext4]
[2020-06-24 11:03:17,131][WARN ][monitor.jvm              ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ignoring gc_threshold for [young], missing warn/info/debug values
[2020-06-24 11:03:17,131][WARN ][monitor.jvm              ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ignoring gc_threshold for [old], missing warn/info/debug values
[2020-06-24 11:03:17,670][WARN ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] deprecated setting [script.disable_dynamic] is set, replace with fine-grained scripting settings (e.g. script.inline, script.indexed, script.file)
[2020-06-24 11:03:17,672][INFO ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] compiling script file [/opt/ruckuswireless/3rdparty/elasticsearch/config/scripts/acknowledge-alarm.groovy]
[2020-06-24 11:03:18,203][INFO ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] compiling script file [/opt/ruckuswireless/3rdparty/elasticsearch/config/scripts/clear-alarm.groovy]
[2020-06-24 11:03:19,076][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initialized
[2020-06-24 11:03:19,077][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] starting ...
[2020-06-24 11:03:19,311][INFO ][transport                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9300]}, publish_address {inet[/172.16.55.55:9300]}
[2020-06-24 11:03:19,341][INFO ][discovery                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ruckus-controller/Edss4AbqTn2lU1zXlRtoag
[2020-06-24 11:03:22,421][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] new_master [79338a8c-a9af-4a2c-b1df-3c9995e48f35][Edss4AbqTn2lU1zXlRtoag][ltdc-ruckus][inet[/172.16.55.55:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2020-06-24 11:03:22,717][INFO ][http                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9200]}, publish_address {inet[/172.16.55.55:9200]}
[2020-06-24 11:03:22,717][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] started
[2020-06-24 11:03:22,968][INFO ][gateway                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] recovered [182] indices into cluster_state
[2020-06-24 11:03:26,114][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] added {[Arthur Parks][Qjxp6SgyRmCvMfSCLFFDYA][ltdc-ruckus][inet[/172.16.55.55:9306]]{client=true, data=false, master=false},[Helmut Zemo][kGWPSMNhQVek-XwotvV9QQ][ltdc-ruckus][inet[/172.16.55.55:9309]]{client=true, data=false, master=false},[Sack][8HaEo-MARW-aMgSmBXHKKg][ltdc-ruckus][inet[/172.16.55.55:9307]]{client=true, data=false, master=false},[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false},[Spyne][2vevL8--QrGYeDjx8QumBg][ltdc-ruckus][inet[/172.16.55.55:9303]]{client=true, data=false, master=false},[Infamnia][FFYEz1zDReW93gWvIjOwyQ][ltdc-ruckus][inet[/172.16.55.55:9304]]{client=true, data=false, master=false},[Plantman][gcNwzC23SJSE90dnmni9EQ][ltdc-ruckus][inet[/172.16.55.55:9301]]{client=true, data=false, master=false},[Hammerhead][lVNX2zcgRJ6Z6RrC0qVNiw][ltdc-ruckus][inet[/172.16.55.55:9308]]{client=true, data=false, master=false},[Baron Samedi][BBywHDwhSNa89EpsVVhLBw][ltdc-ruckus][inet[/172.16.55.55:9302]]{client=true, data=false, master=false},}, reason: zen-disco-receive(join from node[[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false}])
[2020-06-24 11:12:00,009][DEBUG][action.admin.cluster.health] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] observer: timeout notification from cluster service. timeout setting [6m], time since start [6m]

[syamantakomer]

Hi Igor,

Please open a case with Ruckus support and they will help you to fix it.

Support may have to clean some corrupted data from temp folder and force start ElasticSearch.

Regards,
Syamantak Omer