ICX 6430 and 7x50: mgmt IP/VLAN change issue

[andres_zimmerma]

hi,

i'm having issues with new management IP/VLAN change with ICX switches. It's a large manufacturing, all well spread out. I've "inherited" a substantional working ICX installation with multiple switches. Pure L2 (all switching licenses). L3 (old mgmt and customer traffic) is done on Palo Alto FW and Cisco IOS router (new mgmt)

topology:

• core stack of ICX7450-s (connecting aggregation switchstacks, few servers, PA firewall, L3VPN router and few access switches that are in its building).
• 2 separate aggregation switching stacks (one 7450 and one 7250, separate buildings). Each stack is connected to the core with 2x10G (LAG). They are not connected to each other.
• around ~15 ICX6430-s as access switches. they are connecting to one of the aggregation layer switches (but only one of the stacks). uplinks to aggr are mostly 1G fiber, many of them are using 2x1G LAG
• there are few non ICX L2 switches too as access switches (few HP/Aruba 2530G-s, two cisco 3650-s)

Now, i have to change management vlan and IP addressing. New management is originating from L3VPN router (Cisco IOS 892): STP is disabled, physical untagged port on both sides (core 7450 and l3vpn router). I see router IP in Core switch ARP table and switch mac/IP on router ARP table, but cannot ping each other.

Strangely enough, few switches (including one HP) behind core 7450 itself are working okay, so VLAN spanning is allright. i can access them over L3VPN and are under new mgmt.
Also, everything behind one aggregation switch 7450 (and aggr 7450 itself) are also working okay.

But, with same config (besides unique IP addressing), second aggregation stack switch 7250 itself and anything behind it: no go. Also, i'm seeing 7250 MAC and IP in l3vpn router ARP table, but no traffic flow. 

all core and aggre switches are running Version:10.1.06T215 image. Uptime is around 130 days.

vlan config (on all switches) is all very simple:
unique IP address from that subnet and

vlan 510 name newmgmt by port
 tagged ethe xxx to xxxx
 no spanning-tree
 management-vlan
 default-gateway x.x.x.x 1

only exception is core switch, where there is also one untagged port towards cisco router.
they are in default mstp 0 instance (created by adding vlan)

As soon as i change back old IP/managment vlan, management works. 

The kick here is old MSTP config. All vlans on core switches have "no spanning tree" under their config, but MSTP has few manually added instances (old management vlan is instance 2, and instances are persistent of other switches). I really-really don't want to touch old MSTP config, as most probably any change will bring down the whole network (have bad experience). I personally hate STP and try to avoid it like fire, but it's already configured.

So, what am i doing wrong? I'm not that new in networking, i'm managing (as Service Provider) thousands of devices (mostly IOS/Junos/Aruba though) for many good years, but now i've run out of ideas. I'm presuming something with MSTP config, but i see all ports in forwarding state
show mstp under not working 7250/7450 as well as working 7450 shows every port designate/forwarding state (and core 7450 uplink ports as root on aggr switches) in instance 0.

Or is this some sorf of software bug. I don't think the switches (purchased by customer) are covered by additional support services, though they might be covered under warranty.