What version of code are you running on the AP?
Is this AP running as standalone or is it under control of a SZ, ZD or unleashed.
If you make changes via AP CLI on a controlled AP these will get overwritten by the controller.
The basic command to set encryption key is:
set encryption wlanx (note replace x with correct wlan interface number)
To determine which wlan interface to use first do: get wlanlist
Note there is a different interface for 2.4GHz (wlan0 -wlan8) and 5GHz (wlan32-wlan40)
This is a interactive command that you will need to provide multiple replies to set up each interface.
rkscli: set encryption wlan0
Wireless Encryption Type:  quit,  OPEN,  WEP, or  WPA
Wireless Encryption Type: 3
WPA Protocol Version:  quit,  WPA,  WPA2, or  AUTO
WPA Protocol Version: 2
WPA Authentication Type:  quit,  OPEN (PSK),  EAP (.1X), or  AUTO
WPA Authentication Type: 1
WPA Cipher Type:  quit,  TKIP,  AES-CCMP, or  AUTO
WPA Cipher Type: 2
WPA PassPhrase: "YouAreNotReal"
Enter A New PassPhrase [8-63 letters], or Press "Enter" to Accept :
WPA no error
To verify the present settings use
get encryption wlanx (replace x with correct wlan interface number)
Please note - it is much easier to configure standalone AP's using the WEB-UI. If the AP was previously managed by a controller you may need to enable this using the command: set https enable.
I hope this information is helpful,
Thanks for selecting Ruckus Networks products - a Commscope company