cancel
Showing results for 
Search instead for 
Did you mean: 

VSZ cluster out of service

igor_cherniienk
New Contributor
HI! I have a problem after my VM Ruckus crash, because space in storage out, after that vm start but elastic search not starting, of course cluster out of service

some log 

elasticsearch-console

[2020-06-24 11:03:19,076][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initialized
[2020-06-24 11:03:19,077][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] starting ...
[2020-06-24 11:03:19,311][INFO ][transport                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9300]}, publish_address {inet[/172.16.55.55:9300]}
[2020-06-24 11:03:19,341][INFO ][discovery                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ruckus-controller/Edss4AbqTn2lU1zXlRtoag
[2020-06-24 11:03:22,421][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] new_master [79338a8c-a9af-4a2c-b1df-3c9995e48f35][Edss4AbqTn2lU1zXlRtoag][ltdc-ruckus][inet[/172.16.55.55:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2020-06-24 11:03:22,717][INFO ][http                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9200]}, publish_address {inet[/172.16.55.55:9200]}
[2020-06-24 11:03:22,717][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] started
[2020-06-24 11:03:22,968][INFO ][gateway                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] recovered [182] indices into cluster_state
[2020-06-24 11:03:26,114][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] added {[Arthur Parks][Qjxp6SgyRmCvMfSCLFFDYA][ltdc-ruckus][inet[/172.16.55.55:9306]]{client=true, data=false, master=false},[Helmut Zemo][kGWPSMNhQVek-XwotvV9QQ][ltdc-ruckus][inet[/172.16.55.55:9309]]{client=true, data=false, master=false},[Sack][8HaEo-MARW-aMgSmBXHKKg][ltdc-ruckus][inet[/172.16.55.55:9307]]{client=true, data=false, master=false},[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false},[Spyne][2vevL8--QrGYeDjx8QumBg][ltdc-ruckus][inet[/172.16.55.55:9303]]{client=true, data=false, master=false},[Infamnia][FFYEz1zDReW93gWvIjOwyQ][ltdc-ruckus][inet[/172.16.55.55:9304]]{client=true, data=false, master=false},[Plantman][gcNwzC23SJSE90dnmni9EQ][ltdc-ruckus][inet[/172.16.55.55:9301]]{client=true, data=false, master=false},[Hammerhead][lVNX2zcgRJ6Z6RrC0qVNiw][ltdc-ruckus][inet[/172.16.55.55:9308]]{client=true, data=false, master=false},[Baron Samedi][BBywHDwhSNa89EpsVVhLBw][ltdc-ruckus][inet[/172.16.55.55:9302]]{client=true, data=false, master=false},}, reason: zen-disco-receive(join from node[[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false}])
[2020-06-24 11:12:00,009][DEBUG][action.admin.cluster.health] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] observer: timeout notification from cluster service. timeout setting [6m], time since start [6m]

ElasticSearch_gc


OpenJDK 64-Bit Server VM (25.91-b14) for linux-amd64 JRE (1.8.0_91-b14), built on May 11 2016 09:56:20 by "mockbuild" with gcc 4.4.7 20120313 (Red Hat 4.4.7-17)
Memory: 4k page, physical 16467324k(1092376k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:ErrorFile=/opt/ruckuswireless/wsg/log/elasticsearch/elasticsearch-jvm_error%p.log -XX:GCLogFileSize=10485760 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/ruckuswireless/wsg/log/elasticsearch/elasticsearch-dump.hprof -XX:InitialHeapSize=1610612736 -XX:+ManagementServer -XX:MaxHeapSize=1610612736 -XX:MaxNewSize=160432128 -XX:MaxTenuringThreshold=6 -XX:NewSize=160432128 -XX:NumberOfGCLogFiles=10 -XX:OldPLABSize=16 -XX:/opt/ruckuswireless/wsg/cli/bin/elasticsearch.py -f oomstop -XX:+PrintFlagsFinal -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC 
2020-06-24T11:03:15.127+0000: 0.186: Total time for which application threads were stopped: 0.0117804 seconds, Stopping threads took: 0.0116780 seconds

ElasticSearch_ruckus-controller

[2020-06-24 11:03:15,646][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] version[1.7.2], pid[23204], build[e43676b/2015-09-14T09:49:53Z]
[2020-06-24 11:03:15,646][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initializing ...
[2020-06-24 11:03:15,784][INFO ][plugins                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] loaded [], sites []
[2020-06-24 11:03:15,840][INFO ][env                      ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] using [1] data paths, mounts [[/data (/dev/mapper/vg00-lv00)]], net usable_space [72.1gb], net total_space [87.4gb], types [ext4]
[2020-06-24 11:03:17,131][WARN ][monitor.jvm              ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ignoring gc_threshold for [young], missing warn/info/debug values
[2020-06-24 11:03:17,131][WARN ][monitor.jvm              ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ignoring gc_threshold for [old], missing warn/info/debug values
[2020-06-24 11:03:17,670][WARN ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] deprecated setting [script.disable_dynamic] is set, replace with fine-grained scripting settings (e.g. script.inline, script.indexed, script.file)
[2020-06-24 11:03:17,672][INFO ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] compiling script file [/opt/ruckuswireless/3rdparty/elasticsearch/config/scripts/acknowledge-alarm.groovy]
[2020-06-24 11:03:18,203][INFO ][script                   ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] compiling script file [/opt/ruckuswireless/3rdparty/elasticsearch/config/scripts/clear-alarm.groovy]
[2020-06-24 11:03:19,076][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] initialized
[2020-06-24 11:03:19,077][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] starting ...
[2020-06-24 11:03:19,311][INFO ][transport                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9300]}, publish_address {inet[/172.16.55.55:9300]}
[2020-06-24 11:03:19,341][INFO ][discovery                ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] ruckus-controller/Edss4AbqTn2lU1zXlRtoag
[2020-06-24 11:03:22,421][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] new_master [79338a8c-a9af-4a2c-b1df-3c9995e48f35][Edss4AbqTn2lU1zXlRtoag][ltdc-ruckus][inet[/172.16.55.55:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2020-06-24 11:03:22,717][INFO ][http                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] bound_address {inet[/172.16.55.55:9200]}, publish_address {inet[/172.16.55.55:9200]}
[2020-06-24 11:03:22,717][INFO ][node                     ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] started
[2020-06-24 11:03:22,968][INFO ][gateway                  ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] recovered [182] indices into cluster_state
[2020-06-24 11:03:26,114][INFO ][cluster.service          ] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] added {[Arthur Parks][Qjxp6SgyRmCvMfSCLFFDYA][ltdc-ruckus][inet[/172.16.55.55:9306]]{client=true, data=false, master=false},[Helmut Zemo][kGWPSMNhQVek-XwotvV9QQ][ltdc-ruckus][inet[/172.16.55.55:9309]]{client=true, data=false, master=false},[Sack][8HaEo-MARW-aMgSmBXHKKg][ltdc-ruckus][inet[/172.16.55.55:9307]]{client=true, data=false, master=false},[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false},[Spyne][2vevL8--QrGYeDjx8QumBg][ltdc-ruckus][inet[/172.16.55.55:9303]]{client=true, data=false, master=false},[Infamnia][FFYEz1zDReW93gWvIjOwyQ][ltdc-ruckus][inet[/172.16.55.55:9304]]{client=true, data=false, master=false},[Plantman][gcNwzC23SJSE90dnmni9EQ][ltdc-ruckus][inet[/172.16.55.55:9301]]{client=true, data=false, master=false},[Hammerhead][lVNX2zcgRJ6Z6RrC0qVNiw][ltdc-ruckus][inet[/172.16.55.55:9308]]{client=true, data=false, master=false},[Baron Samedi][BBywHDwhSNa89EpsVVhLBw][ltdc-ruckus][inet[/172.16.55.55:9302]]{client=true, data=false, master=false},}, reason: zen-disco-receive(join from node[[Hero for Hire][6g4HtR6VSAaBOPpTIjBLAA][ltdc-ruckus][inet[/172.16.55.55:9305]]{client=true, data=false, master=false}])
[2020-06-24 11:12:00,009][DEBUG][action.admin.cluster.health] [79338a8c-a9af-4a2c-b1df-3c9995e48f35] observer: timeout notification from cluster service. timeout setting [6m], time since start [6m]


1 REPLY 1

syamantakomer
Moderator
Moderator
Hi Igor,

Please open a case with Ruckus support and they will help you to fix it.

Support may have to clean some corrupted data from temp folder and force start ElasticSearch.

Regards,
Syamantak Omer
Regards,
Syamantak Omer
Official Rep | Staff TSE | CWNA | CCNA | RASZA | RICXI