ICX6610 10G Port Flapping Problems

marin_raskovic — Thu, 22 Feb 2018 17:03:03 GMT

Hi,
We are using 2xICX6610 in a stack for vmware esx hosts, storage and other servers.
1Gb ports are used for "normal" networking (client-server), and 10Gb ports are used for storage networking and vmware vmotion.
We are observing problems on network interfaces connected to 10G ports. At esx hosts we see port down/up situations (vmnic goes down and then up in same second), same on backup server (ms windows), and on storage ports we see link flapping situations. Also, we are faced sometimes with APD (all paths down) state for nfs datastores mounted from storage.
Jumbo frames are enabled at vmware side (vmkernel ports, vswitch), switch and on storage ports - 9000.
At vmware hosts side we do not use any kind of teaming - only virtual port based load balancing.
At storage side we use teaming, and ports on switch side are in LAG configuration.
Also, backup server 2 ports are in LAG, and teamed on server side.
No problems at 1Gb ports are found or logged, no vmnic down/up states.
To the rest of network only one port is used - 1/1/24.
FastIron version used is 08.0.10g.
Brief desciption
ESX host1 is connected to ports 1/3/3 and 2/3/3
ESX host2 is connected to ports 1/3/4 and 2/3/4
ESX host3 is connected to ports 1/3/5 and 2/3/5
backup server is connected to ports 1/3/6 and 2/3/6
storage is connected to ports 1/3/1 and 1/3/2
VLANs 112 (storage network) and 113 (vmotion) suffers problems - only ports in this vlan-s have port down/up state changes.

Here is copy-paste of our running config from the switch. Can anyone see any mistake, problem, have a hint?

ICX6610-S-1>show configuration
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.10gT7f1
!
stack unit 1
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 1/2/1 to 1/2/2
stack-trunk 1/2/6 to 1/2/7
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 2/2/1 to 2/2/2
stack-trunk 2/2/6 to 2/2/7
stack-port 2/2/1 2/2/6
stack enable
stack mac 609c.9f47.ee38
!
!
!
lag "storage2_10g_ports" static id 6
ports ethernet 1/3/2 ethernet 2/3/2
primary-port 1/3/2
deploy
!
lag "storage1_10g_ports" static id 2
ports ethernet 1/3/1 ethernet 2/3/1
primary-port 1/3/1
deploy
!
lag "BackupServer_10g_ports" static id 1
ports ethernet 1/3/6 ethernet 2/3/6
primary-port 1/3/6
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan 6 name DataNetwork by port
tagged ethe 1/1/4 to 1/1/9 ethe 1/1/24 ethe 2/1/4 to 2/1/9
!
vlan 7 DataNetwork2 by port
tagged ethe 1/1/4 to 1/1/9 ethe 1/1/24 ethe 2/1/4 to 2/1/9
untagged ethe 1/1/19 to 1/1/20 ethe 2/1/19 to 2/1/20
!
vlan 110 name ManagementNetwork by port
tagged ethe 1/1/4 to 1/1/9 ethe 1/1/24 ethe 2/1/4 to 2/1/9
untagged ethe 1/1/1 to 1/1/3 ethe 1/1/10 to 1/1/12 ethe 2/1/1 to 2/1/3 ethe 2/1/10
!
vlan 111 name ServersNetwork by port
tagged ethe 1/1/4 to 1/1/9 ethe 1/1/24 ethe 2/1/4 to 2/1/9
untagged ethe 1/1/13 ethe 2/1/11 to 2/1/13
!
vlan 112 name StorageNetwork by port
tagged ethe 1/3/1 to 1/3/5 ethe 2/3/1 to 2/3/5
untagged ethe 1/3/6 ethe 2/3/6
!
vlan 113 name vMotionNetwork by port
tagged ethe 1/3/3 to 1/3/5 ethe 2/3/3 to 2/3/5
!
vlan 155 name Servers2network by port
tagged ethe 1/1/4 to 1/1/9 ethe 1/1/24 ethe 2/1/4 to 2/1/9
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
enable super-user-password .....
enable aaa console
hostname ICX6610-S-1
ip address xxx.xxx.xxx.xxx 255.255.255.0
no ip dhcp-client enable
ip default-gateway xxx.xxx.xxx.xxx
!
logging host xxx.xxx.xxx.x
logging persistence
no telnet server
telnet server enable vlan 111
username admin password .....
cdp run
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
server x.x.x.x minpoll 5 burst
!
!
web-management https
hitless-failover enable
interface ethernet 1/1/4
port-name host01 slot2 port0 vmnic6
!
interface ethernet 1/1/5
port-name host01 slot3 port0 vmnic8
!
interface ethernet 1/1/6
port-name host02 slot2 port0 vmnic6
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/7
port-name host02 slot3 port0 vmnic8
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/8
port-name host03 slot2 port0 vmnic6
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/9
port-name host03 slot3 port0 vmnic8
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/11
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/1/22
port-name "VLAN 110 uplink to cisco port X - disconnected cable"
!
interface ethernet 1/1/23
port-name "VLAN 111 uplink to cisco port X - disconnected cable"
!
interface ethernet 1/1/24
port-name "Uplink port to central Cisco"
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/3/3
port-name saphost01 slot0 port0 vmnic2
speed-duplex 10G-full
!
interface ethernet 1/3/4
port-name host02 slot0 port0 vmnic2
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/3/5
port-name host03 slot0 port0 vmnic2
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/3/6
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ethernet 2/1/4
port-name host01 slot2 port1 vmnic7
!
interface ethernet 2/1/5
port-name host01 slot3 port1 vmnic9
!
interface ethernet 2/1/6
port-name host02 slot2 port 1 vmnic7
spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/7
port-name host02 slot3 port1 vmnic9
spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/8
port-name host03 slot2 port1 vmnic7
spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/1/9
port-name host03 slot3 port1 vmnic9
spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/3
port-name host01 slot1 port1 vmnic5
speed-duplex 10G-full
!
interface ethernet 2/3/4
port-name host02 slot1 port1 vmnic5
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
interface ethernet 2/3/5
port-name host03 slot1 port1 vmnic5
speed-duplex 10G-full
spanning-tree 802-1w admin-edge-port
!
!
!
!
lldp run
!
!
!
!
end

ICX6610-S-1>

topic ICX6610 10G Port Flapping Problems in ICX Switches

ICX6610 10G Port Flapping Problems