all services offline after reboot

[jason_hilt]

SZ100.  2 unit cluster.  Just upgraded from 5.1.1.0.598 to 5.1.2.0.302
After upgrade all services stay offline after reboot on one unit.  Both units rebooted at the same time.  Even tried rebooting the offline unit again with no change.
  The offline unit I can log into the CLI using admin account. Obviously can't get to GUI in the offline unit since no services are running.  The other unit I can login to the GUI with my account, but not the admin account, and can't login into the CLI with either admin or my account.  In the unit that I can get into the GUI, the administrators list is empty, but the accounts still show in the Groups tab.
Are the CLI logins separate from the GUI logins?  Many of the commands in the offline unit don't work as I get a "Database is not ready to access. Please try later" reply.
I did 'service start' which did nothing.  So far a "service restart" has done nothing.
I have had both units working through the GUI.  A 'service start' fixed it previously. After the latest power down of both units, one unit doesn't want to work.

Any ideas on what is going on?  We had no issues on the previous version.

[javier_valdes_c]

Hi Jason,
Regarding the CLI credentials. CLI login credentials are different from the GUI ones.
Usually, when setting the SZ for the first time using the GUI wizard, the system asks for a user and password, at this moment the credentials you entered are the same for GUI and CLI access.

Later, usually the admin changes this credentials using the GUI, but the credentials only changes for the GUI access, the CLI remains the same, the ones you entered when doing the wizard. So, if you haven't access CLI since the beginning, probably the CLI login credentials are the ones you entered for the first time.

CLI admin password can only be changed from the CLI itself.

Hope this helps in any way.
Best regards,

[jason_hilt]

I am the admin for the units.

I can get into the CLI using the admin login, but it only works on the CLI and only if the GUI is not working.  Right now unit 2 is working.  I can log into the GUI using my account only, but I can't login to the CLI on unit 2 with the admin account or my account.  Unit 1 is not working correctly.  I can login to the CLI using the admin account, but since no services are running and wont start, there is no GUI.
I know the admin login works on the CLI for both units as I have logged in that way previously.

I can't change the admin password in the GUI, because the administrators list isn't showing any users.  I can't try changing the admin account on unit 1 because I get the database error.
It's been 2 hours and there are no services running on unit 1.  Service start, service restart and a full power unplugged reboot doesn't start them.
Just did another service restart and now 'show service' shows 'All services are down'.  After a minute or so, it now shows all services offline.
No idea right now other than to do a full factory reset as I can't even downgrade the unit since CLI commands don't work because of the database error.

[javier_valdes_c]

Definitely sounds like database corruption issue from unit 1 and unit 2 not syncing correctly. Did you tried rebooting unit 2 while unit 1 is off the network? But I have to mentioned that it could be risky that after a reboot from unit 2 encounters same issue of services down.
Probably the best here is to contact TAC, maybe they can provide a CLI command to help restore any corrupted data.
I'm sorry of not being able to provide more useful help.