ICX7750 route-map help

[jeremiah_yeary]

Currently we are using a Cisco for our core switch, and we are trying to migrate over to a ICX7750. 

On the Cisco it allows a vlan interface to have an ACL, and a route-map applied to it, but I am unable to configure the ICX to do the same thing, it tells me they can not coexist in the interface.

as an example of the infrastructure, a crude map is:

we have 4 sites that are interconnected to 1 shared server farm with our own fiber. Each site has their own internet connection we need to keep separate due to static IPs assigned at the WANs.

site 1 ip subnet 10.10.1.0/24
site 2 subnet 10.10.2.0/24
site 3 subnet 10.10.3.0/24
site 4 subnet 10.10.4.0/24
shared server farm subnet 10.10.100.0/24

so the Cisco is configured with the associated vlans to the subnets, which have ACLs in place to prevent the sites to communicate to each other for security reasons other than a few exceptions, and they can share server resources. then any traffic that is not part of our internal network is routed though their designated WANs.

here is an example of the config:


--------------------------------------

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0
ip policy route-map Internet-Vlan1

interface Vlan2
ip access-group 102 in
ip address 10.10.2.1 255.255.255.0
ip policy route-map Internet-Vlan2

interface Vlan3
ip access-group 103 in
ip address 10.10.3.1 255.255.255.0
ip policy route-map Internet-Vlan3

interface Vlan4
ip access-group 104 in
ip address 10.10.4.1 255.255.255.0
ip policy route-map Internet-Vlan4

interface Vlan100
ip access-group 110 in
ip address 10.10.100.1 255.255.255.0
ip policy route-map Internet-Vlan100

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any

access-list 102 permit ip 10.10.2.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 102 deny any any

access-list 103 permit ip 10.10.3.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 103 permit ip host 10.10.3.50 host 10.10.4.50
access-list 103 deny any any

access-list 104 permit ip 10.10.4.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 104 permit ip host 10.10.4.50 host 10.10.3.50
access-list 104 deny any any

access-list 110 permit ip 10.10.100.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 110 deny ip any any

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

access-list 152 deny ip 10.10.2.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 152 permit ip any any

access-list 153 deny ip 10.10.3.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 153 permit ip any any

access-list 154 deny ip 10.10.4.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

access-list 160 deny ip 10.10.100.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 160 permit ip any any


route-map Internet-Vlan1 permit 10
match ip address 151
set ip next-hop 10.10.1.254

route-map Internet-Vlan2 permit 10
match ip address 152
set ip next-hop 10.10.2.254

route-map Internet-Vlan3 permit 10
match ip address 153
set ip next-hop 10.10.3.254

route-map Internet-Vlan4 permit 10
match ip address 154
set ip next-hop 10.10.4.254

route-map Internet-Vlan100 permit 10
match ip address 160
set ip next-hop 10.10.100.254


--------------------------------------------------------------

so on the Cisco the route map policy takes precedence, then if it is not part of the allowed "set" then it moves on to the ACL assigned to the vlan interface

this way we can keep the internal from hitting our wan routers, and have our ACLs handle the permissions and restrictions for vlan cross communication.

This is a crude example, but if we can get some guidance on how to do this on the ICX7750 with the L3 firmware, that would be great!
 

[hashim_bharooc1]

Hey Jeremiah,
Here is an example from the Security config guide of ICX:

deviceBrocade(config)#access-list 101 permit tcp 10.10.10.0 0.0.0.255 eq http 5.5.5.0 0.0.0.255

device(config)#route-map net10web permit 101

device(config-routemap net10web)#match ip address 101

device(config-routemap net10web)#set ip next-hop 1.1.1.1

device(config-routemap net10web)#set ip next-hop 2.2.2.2

device(config-routemap net10web)#exit

device(config)#vlan 10

device(config-vlan-10)#tagged ethernet 1/1/1 to 1/1/4

device(config-vlan-10)#router-interface ve 1

device(config)#interface ve 1

device(config-vif-1)#ip policy route-map net10web

Document link:
https://support.ruckuswireless.com/documents/2128-fastiron-08-0-70-security-configuration-guide

[jeremiah_yeary]

Thank you Hashim response. The problem with that config example is that it still does not apply to my restrictions of vlans talking to each other. my sites are hooked up to 2 7750s that are stacked to each other, and they have 2 10gig SFPs (one on each switch) in a lag for failover/redundancy to each site. each lag has multiple vlans tagged.

the above example does not address the issue I am trying to address.

i only have 1 hop that will be applied to the "set ip next-hop" for it is the firewall that provides the WAN at each of my sites. the multiple next-hop only is for failover use if the primary WAN went down and I want a secondary. This part is not needed in my environment.

the major issue I need to figure out is how to get the 7750 to forward each vlan internet traffic to their respective WANs as in my first example, and still be able to block them from communicating cross vlan except for the permissions that I want them to. 

Currently on the 7750 "ip access-group xxx in" ingress restrictions are not able to co-exist with "ip policy route-map" on an "interface ve xxx"  I need to figure out how to get past this software restriction so I dont have to add a whole bunch of hardware just to maintain the infrastructure flow that is working now. 

I did look at trying to create 2 virtual interfaces with the route-maps daisy chained to try to bounce the ACLs off of each other, but that just got messy. I also tried applying a global route-map, and then ones on the virtual interfaces, but on the global one, it can not work with my multiple WAN issue.

Thanks again for your insight! 

[netwizz]

Good Morning:

I think what Op is saying  "cannot have a route-map and a ACL co-existing on the same Interface in the ICX world."

This is not exactly true, but there are constraints pertaining to direction etc... on both the ICX platform and Cisco, too.

Please correct me if I am wrong reading any of this because I am making a lot of assumptions trying to read-between-the lines for what most of it does and what the it is intended to do.   This is most likely not an actual working configuration snippet from a Cisco switch in that I do not think it does what the Op intends it to do...

Background:
Where a normal static-route is GET TO:  Network/Mask via Next-Hop (taking no consideration where the traffic is sourced from), generally route-maps are extended ACLs; since they are often used to match the source... making them more useful than simple static routing tables for certain complicated tasks.

This is referred to as Policy-Based-Routing (PBR).

Looking at this:
http://docs.ruckuswireless.com/fastiron/08.0.90/fastiron-08090-securityguide/GUID-8D819DAB-C84F-4339...

• PBR is not supported together with ingress ACLs on the same port.

This is indicating you cannot have an access-group ### in and an 
ip policy route-map on the same port.


******

This does not mean you cannot achieve your Goals on the ICX7750 though!
Please continue...

******






If I look at OPs VLAN 1 it is an SVI with assigned IP 10.10.1.1/24 serving as the default-gateway of that VLAN.  This makes a directly-connected route for 10.10.1.0/24 in the default routing table (since no ip VRF details are available).


Then the OP is doing an ip next-hop (not an ip default net-hop), so op is preempting the routing table for what otherwise would be a directly connected route for 10.10.1.0/24

Specifically, OPs route-map is overriding the routing table for everything with the permit ip any any statement... except that which is denied first not being processed by the route-map...

Looking at the ACL for OPs route-map:

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

OP Probably Intends:

deny subnet 10.10.1.0/24 to 10.10.0.0/16 (anything internal)
permit anything else (i.e. let it get to the Internet)

Intended Outcome:  For devices that attempt to communicate internally with inter-VLAN routing send the next-hop to some other device (that probably does not exist being each IP is different (10.10.1.254, 10.10.2.254, etc.) Op is setting the next-hop).  Do not interfere otherwise (i.e. traffic to Internet)



If this was the intent, I probably would have written it more as:

access-list 151 deny ip any 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

This is because any traffic going out from VLAN 1 will be sourced from 10.10.1.x.  If someone sets the IP wrong the default-gateway won't be on its same subnet, so it won't be found, and even if it was, the traffic wouldn't come back.  Hence I changed 10.10.1.0 0.0.0.255 to any above.


****

However, I suspect the route-map ACLs have the permits and denies backwards on the Cisco side, too if my understanding of what the OP is trying to do is allow Internet Traffic but no local traffic.


Simply put for something to be routed via PBR using a route-map it must be Permitted in.  That is denying does not drop the traffic like it would if applied as an access-group but rather denied entries are not policy routed.  It is best to think of ACLs as being processed top to bottom, and once a match is hit the list stops processing.  If it matches a Permit, it returns a True (Boolean) and if it matches a Deny it returns False (Boolean).  This is much more important a concept when dealing with QoS an Class-Maps which have Boolean operators such as AND or OR can be used to have multiple criteria to allow something into a Class.


There is also an implicit deny if no matches are made the processing terminates returning FALSE, so there is really no need to end with something like deny ip any any.,

For a simple route-map that overrides the default routing table for anything internally but does not impact anything else like traffic to the Internet the statement would be succinctly written something more like:

access-list 151 permit ip any 10.10.0.0 0.0.255.255


route-map Internet-Vlan1 permit 10
match ip address 151
set ip next-hop 10.10.1.254


interface Vlan1
ip address 10.10.1.1 255.255.255.0
ip policy route-map Internet-Vlan1


************************

This does not seem right on the Cisco side either:

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any


Reasoning, this is a Layer-3 SVI (Interface).  How could something externally be coming into the Interface sourced from the Inside?

Devices within VLAN 1 will presumably already be addressed 10.10.1.x/24 and external devices would be sourced from different subnets.  Internal devices will not hit the SVI for internal communications; since, they are on the same Layer-2, so this does not provide any internal controls if that is what you are attempting.

To get the SVI involved, something must come from VLAN 1 sourced from 10.10.1.x/24 and destined to some other subnet.  Then the device will actually forward to its default gateway 10.10.1.1 at which point, ACLs and route-maps can be applied.

If you are trying to impose internal controls, please check out PVLANS:

http://docs.ruckuswireless.com/fastiron/08.0.60/fastiron-08060-l2guide/GUID-6A7FE111-9F4A-4AC6-957F-...



Recommendations:

I am not even certain Policy-Based-Routing is appropriate for this use case in that they are generally not used to break routing tables to stop traffic. 

Access lists should most likely be extended ACLs controlling outbound traffic anyway... In which case if you do need to combine them with route-maps on the ICX they should work.


If you are trying to accomplish a specific use case, please let us know, and I am sure someone can hammer out a snippit of the ICX configuration for you.


Best Regards

NETWizz

[jeremiah_yeary]

Thanks NetWizz for the response. I do see what you are implying. So my example is a crude copy of an actual working config on our old Cisco 4507R-E core.

I appreciate your thoroughness on reviewing my setup. I did simplify my vlan count and ACL just added the "deny any any" just to show the ACL stop in our example, it is implicit on the Cisco.

So the example is not a working config on my 7750 though, since it handles the rules a little different.

so going backwards on your responses:
------------------------------------

This does not seem right on the Cisco side either:

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any

-------------------------------------------------------

So the "interface Vlan1" is similar to "Interface ve 1" on fastiron, to compare. 

so by default any device that targets 10.10.1.1 as their gateway without the ACL applied can talk to all other vlans that are known to the router.

so vlan 1 = devices targeting 10.10.1.1 as gateway 
vlan 2 = devices targeting 10.10.2.1 as gateway
vlan 100 = devices targeting 10.10.100.1 as gateway

with no ACL in place they automatically route between, and any device can potentially communicate to another device on the other vlan with no restrictions in between.

once i build the the access-list of 101 and apply it to vlan 1 on the ingress using the command "ip access-group 101 in", it automatically cuts vlan 2 traffic out, which is intended because they are completely different organizations that should not intrude on each other, but both organizations help fund shared server space in the farm, so they get access to vlan 100. Shared services like file servers, ip phones, active directory, end user management servers, exchange, etc. We do use a combination of ACL and Active Directory to restrict to appropriate use and access to the services.

On the Cisco, by adding the "ip policy route-map Internet-Vlan1" to the interface this forwards the non internal ip address ranges off to their firewall/WAN. due to the organizations funding and policies, they have to have their own internet installed at each site showing the connection and bill in their name at their address.

so the way the rules are in effect are not in the order as they are listed when doing a show run.

the route-map is applied first using the ACL:

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

this works and is in use this way, for first of all their firewall at each site handles DHCP rules and static assignments, so if a device is mis-configured on the vlan with the wrong ip and/or subnet then it just does not work. I would rather have something not work than have a vulnerability.

so as the "permit route-map" is implemented, it is the same on a cisco as it is on the fastiron. The "denied" range does not get sent to the "set ip next-hop" but continues on to the normal core routing. the "permit" range of everything else not of our ip gets sent to the firewall to deal with.

On the Cisco, the above mentioned "denied" items in the "route-map" then move on to the rule of "ip access-group 101 in" on the interface, which then helps regulate what ips or ranges or protocols they can communicate to and with. plug one end of the straw, and no fluid moves through, since it requires bi-directional traffic to function in our environment. if the access-group does not define the traffic, then it moves on to the default routing tables on the switch that sends it off to another firewall for analysis.


to make a remark about the ACL and route-map co-existence, that is the issue I am having. tried to "copy" within the best of my ability the Cisco config to the ICX on the 08.0.70d router firmware i get errors.

build the "ip access-group 101 in", on the ve 1 interface, then apply the route-map  policy "ip policy route-map Internet-Vlan1" to the interface ve 1, and it does not apply with the error that a route-map policy cannot co-exist with an ACL. I dont have the stack fired up at the moment, but that in general  words is the error the ICX gives me.  If I have "ip access-group 101 in" on an interface, and try to apply a global "ip policy route-map" instead of on the interface, I get the same error telling me ve 1 cannot have an ACL.

**Edit** 
I did notice that the ICX does permit the "ip access-group 101 out" command and work in conjunction with the policy route-map but not the "ip access-group 101 in". Not sure how the packets are handled in comparison between the IN and the OUT to maybe use this effectively by modifying the access-list to effectively work the way i hope.

So I guess I just need a fake example, not even of my own, on how I can restrict and deny or permit individual ips or ranges to cross talk vlans, since all the traffic shares this one core switch for management, then the ability to forward each vlan their out of scope ips to their respective firewalls for internet use.

the rules that I have work on the Catalyst, i just need to get them to work, so i can swap it with the ICX.

Thanks again for your insight!