nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

ICX7750 route-map help

unknown

Currently we are using a Cisco for our core switch, and we are trying to migrate over to a ICX7750.

On the Cisco it allows a vlan interface to have an ACL, and a route-map applied to it, but I am unable to configure the ICX to do the same thing, it tells me they can not coexist in the interface.

as an example of the infrastructure, a crude map is:

we have 4 sites that are interconnected to 1 shared server farm with our own fiber. Each site has their own internet connection we need to keep separate due to static IPs assigned at the WANs.

site 1 ip subnet 10.10.1.0/24
site 2 subnet 10.10.2.0/24
site 3 subnet 10.10.3.0/24
site 4 subnet 10.10.4.0/24
shared server farm subnet 10.10.100.0/24

so the Cisco is configured with the associated vlans to the subnets, which have ACLs in place to prevent the sites to communicate to each other for security reasons other than a few exceptions, and they can share server resources. then any traffic that is not part of our internal network is routed though their designated WANs.

here is an example of the config:

--------------------------------------

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0
ip policy route-map Internet-Vlan1

interface Vlan2
ip access-group 102 in
ip address 10.10.2.1 255.255.255.0
ip policy route-map Internet-Vlan2

interface Vlan3
ip access-group 103 in
ip address 10.10.3.1 255.255.255.0
ip policy route-map Internet-Vlan3

interface Vlan4
ip access-group 104 in
ip address 10.10.4.1 255.255.255.0
ip policy route-map Internet-Vlan4

interface Vlan100
ip access-group 110 in
ip address 10.10.100.1 255.255.255.0
ip policy route-map Internet-Vlan100

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any

access-list 102 permit ip 10.10.2.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 102 deny any any

access-list 103 permit ip 10.10.3.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 103 permit ip host 10.10.3.50 host 10.10.4.50
access-list 103 deny any any

access-list 104 permit ip 10.10.4.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 104 permit ip host 10.10.4.50 host 10.10.3.50
access-list 104 deny any any

access-list 110 permit ip 10.10.100.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 110 deny ip any any

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

access-list 152 deny ip 10.10.2.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 152 permit ip any any

access-list 153 deny ip 10.10.3.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 153 permit ip any any

access-list 154 deny ip 10.10.4.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

access-list 160 deny ip 10.10.100.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 160 permit ip any any

route-map Internet-Vlan1 permit 10
match ip address 151
set ip next-hop 10.10.1.254

route-map Internet-Vlan2 permit 10
match ip address 152
set ip next-hop 10.10.2.254

route-map Internet-Vlan3 permit 10
match ip address 153
set ip next-hop 10.10.3.254

route-map Internet-Vlan4 permit 10
match ip address 154
set ip next-hop 10.10.4.254

route-map Internet-Vlan100 permit 10
match ip address 160
set ip next-hop 10.10.100.254

--------------------------------------------------------------

so on the Cisco the route map policy takes precedence, then if it is not part of the allowed "set" then it moves on to the ACL assigned to the vlan interface

this way we can keep the internal from hitting our wan routers, and have our ACLs handle the permissions and restrictions for vlan cross communication.

This is a crude example, but if we can get some guidance on how to do this on the ICX7750 with the L3 firmware, that would be great!

Find more posts tagged with

Accepted answers

All comments

hashim_bharooc1

Hey Jeremiah,
Here is an example from the Security config guide of ICX:

deviceBrocade(config)#access-list 101 permit tcp 10.10.10.0 0.0.0.255 eq http 5.5.5.0 0.0.0.255

device(config)#route-map net10web permit 101

device(config-routemap net10web)#match ip address 101

device(config-routemap net10web)#set ip next-hop 1.1.1.1

device(config-routemap net10web)#set ip next-hop 2.2.2.2

device(config-routemap net10web)#exit

device(config)#vlan 10

device(config-vlan-10)#tagged ethernet 1/1/1 to 1/1/4

device(config-vlan-10)#router-interface ve 1

device(config)#interface ve 1

device(config-vif-1)#ip policy route-map net10web

Document link:
https://support.ruckuswireless.com/documents/2128-fastiron-08-0-70-security-configuration-guide

unknown

Thank you Hashim response. The problem with that config example is that it still does not apply to my restrictions of vlans talking to each other. my sites are hooked up to 2 7750s that are stacked to each other, and they have 2 10gig SFPs (one on each switch) in a lag for failover/redundancy to each site. each lag has multiple vlans tagged.

the above example does not address the issue I am trying to address.

i only have 1 hop that will be applied to the "set ip next-hop" for it is the firewall that provides the WAN at each of my sites. the multiple next-hop only is for failover use if the primary WAN went down and I want a secondary. This part is not needed in my environment.

the major issue I need to figure out is how to get the 7750 to forward each vlan internet traffic to their respective WANs as in my first example, and still be able to block them from communicating cross vlan except for the permissions that I want them to.

Currently on the 7750 "ip access-group xxx in" ingress restrictions are not able to co-exist with "ip policy route-map" on an "interface ve xxx" I need to figure out how to get past this software restriction so I dont have to add a whole bunch of hardware just to maintain the infrastructure flow that is working now.

I did look at trying to create 2 virtual interfaces with the route-maps daisy chained to try to bounce the ACLs off of each other, but that just got messy. I also tried applying a global route-map, and then ones on the virtual interfaces, but on the global one, it can not work with my multiple WAN issue.

Thanks again for your insight!

unknown

Good Morning:

I think what Op is saying "cannot have a route-map and a ACL co-existing on the same Interface in the ICX world."

This is not exactly true, but there are constraints pertaining to direction etc... on both the ICX platform and Cisco, too.

Please correct me if I am wrong reading any of this because I am making a lot of assumptions trying to read-between-the lines for what most of it does and what the it is intended to do. This is most likely not an actual working configuration snippet from a Cisco switch in that I do not think it does what the Op intends it to do...

Background:
Where a normal static-route is GET TO: Network/Mask via Next-Hop (taking no consideration where the traffic is sourced from), generally route-maps are extended ACLs; since they are often used to match the source... making them more useful than simple static routing tables for certain complicated tasks.

This is referred to as Policy-Based-Routing (PBR).

Looking at this:
http://docs.ruckuswireless.com/fastiron/08.0.90/fastiron-08090-securityguide/GUID-8D819DAB-C84F-4339-A013-9ADD6F5400C0.html

PBR is not supported together with ingress ACLs on the same port.

This is indicating you cannot have an access-group ### in and an
ip policy route-map on the same port.

******

This does not mean you cannot achieve your Goals on the ICX7750 though!
Please continue...

******

If I look at OPs VLAN 1 it is an SVI with assigned IP 10.10.1.1/24 serving as the default-gateway of that VLAN. This makes a directly-connected route for 10.10.1.0/24 in the default routing table (since no ip VRF details are available).

Then the OP is doing an ip next-hop (not an ip default net-hop), so op is preempting the routing table for what otherwise would be a directly connected route for 10.10.1.0/24

Specifically, OPs route-map is overriding the routing table for everything with the permit ip any any statement... except that which is denied first not being processed by the route-map...

Looking at the ACL for OPs route-map:

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

OP Probably Intends:

deny subnet 10.10.1.0/24 to 10.10.0.0/16 (anything internal)
permit anything else (i.e. let it get to the Internet)

Intended Outcome: For devices that attempt to communicate internally with inter-VLAN routing send the next-hop to some other device (that probably does not exist being each IP is different (10.10.1.254, 10.10.2.254, etc.) Op is setting the next-hop). Do not interfere otherwise (i.e. traffic to Internet)

If this was the intent, I probably would have written it more as:

access-list 151 deny ip any 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

This is because any traffic going out from VLAN 1 will be sourced from 10.10.1.x. If someone sets the IP wrong the default-gateway won't be on its same subnet, so it won't be found, and even if it was, the traffic wouldn't come back. Hence I changed 10.10.1.0 0.0.0.255 to any above.

****

However, I suspect the route-map ACLs have the permits and denies backwards on the Cisco side, too if my understanding of what the OP is trying to do is allow Internet Traffic but no local traffic.

Simply put for something to be routed via PBR using a route-map it must be Permitted in. That is denying does not drop the traffic like it would if applied as an access-group but rather denied entries are not policy routed. It is best to think of ACLs as being processed top to bottom, and once a match is hit the list stops processing. If it matches a Permit, it returns a True (Boolean) and if it matches a Deny it returns False (Boolean). This is much more important a concept when dealing with QoS an Class-Maps which have Boolean operators such as AND or OR can be used to have multiple criteria to allow something into a Class.

There is also an implicit deny if no matches are made the processing terminates returning FALSE, so there is really no need to end with something like deny ip any any.,

For a simple route-map that overrides the default routing table for anything internally but does not impact anything else like traffic to the Internet the statement would be succinctly written something more like:

access-list 151 permit ip any 10.10.0.0 0.0.255.255

route-map Internet-Vlan1 permit 10
match ip address 151
set ip next-hop 10.10.1.254

interface Vlan1
ip address 10.10.1.1 255.255.255.0
ip policy route-map Internet-Vlan1

************************

This does not seem right on the Cisco side either:

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any

Reasoning, this is a Layer-3 SVI (Interface). How could something externally be coming into the Interface sourced from the Inside?

Devices within VLAN 1 will presumably already be addressed 10.10.1.x/24 and external devices would be sourced from different subnets. Internal devices will not hit the SVI for internal communications; since, they are on the same Layer-2, so this does not provide any internal controls if that is what you are attempting.

To get the SVI involved, something must come from VLAN 1 sourced from 10.10.1.x/24 and destined to some other subnet. Then the device will actually forward to its default gateway 10.10.1.1 at which point, ACLs and route-maps can be applied.

If you are trying to impose internal controls, please check out PVLANS:

http://docs.ruckuswireless.com/fastiron/08.0.60/fastiron-08060-l2guide/GUID-6A7FE111-9F4A-4AC6-957F-A84DE36CB325.html

Recommendations:

I am not even certain Policy-Based-Routing is appropriate for this use case in that they are generally not used to break routing tables to stop traffic.

Access lists should most likely be extended ACLs controlling outbound traffic anyway... In which case if you do need to combine them with route-maps on the ICX they should work.

If you are trying to accomplish a specific use case, please let us know, and I am sure someone can hammer out a snippit of the ICX configuration for you.

Best Regards

NETWizz

unknown

Thanks NetWizz for the response. I do see what you are implying. So my example is a crude copy of an actual working config on our old Cisco 4507R-E core.

I appreciate your thoroughness on reviewing my setup. I did simplify my vlan count and ACL just added the "deny any any" just to show the ACL stop in our example, it is implicit on the Cisco.

So the example is not a working config on my 7750 though, since it handles the rules a little different.

so going backwards on your responses:
------------------------------------

This does not seem right on the Cisco side either:

interface Vlan1
ip access-group 101 in
ip address 10.10.1.1 255.255.255.0

access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255
access-list 101 deny ip any any

-------------------------------------------------------

So the "interface Vlan1" is similar to "Interface ve 1" on fastiron, to compare.

so by default any device that targets 10.10.1.1 as their gateway without the ACL applied can talk to all other vlans that are known to the router.

so vlan 1 = devices targeting 10.10.1.1 as gateway
vlan 2 = devices targeting 10.10.2.1 as gateway
vlan 100 = devices targeting 10.10.100.1 as gateway

with no ACL in place they automatically route between, and any device can potentially communicate to another device on the other vlan with no restrictions in between.

once i build the the access-list of 101 and apply it to vlan 1 on the ingress using the command "ip access-group 101 in", it automatically cuts vlan 2 traffic out, which is intended because they are completely different organizations that should not intrude on each other, but both organizations help fund shared server space in the farm, so they get access to vlan 100. Shared services like file servers, ip phones, active directory, end user management servers, exchange, etc. We do use a combination of ACL and Active Directory to restrict to appropriate use and access to the services.

On the Cisco, by adding the "ip policy route-map Internet-Vlan1" to the interface this forwards the non internal ip address ranges off to their firewall/WAN. due to the organizations funding and policies, they have to have their own internet installed at each site showing the connection and bill in their name at their address.

so the way the rules are in effect are not in the order as they are listed when doing a show run.

the route-map is applied first using the ACL:

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 151 permit ip any any

this works and is in use this way, for first of all their firewall at each site handles DHCP rules and static assignments, so if a device is mis-configured on the vlan with the wrong ip and/or subnet then it just does not work. I would rather have something not work than have a vulnerability.

so as the "permit route-map" is implemented, it is the same on a cisco as it is on the fastiron. The "denied" range does not get sent to the "set ip next-hop" but continues on to the normal core routing. the "permit" range of everything else not of our ip gets sent to the firewall to deal with.

On the Cisco, the above mentioned "denied" items in the "route-map" then move on to the rule of "ip access-group 101 in" on the interface, which then helps regulate what ips or ranges or protocols they can communicate to and with. plug one end of the straw, and no fluid moves through, since it requires bi-directional traffic to function in our environment. if the access-group does not define the traffic, then it moves on to the default routing tables on the switch that sends it off to another firewall for analysis.

to make a remark about the ACL and route-map co-existence, that is the issue I am having. tried to "copy" within the best of my ability the Cisco config to the ICX on the 08.0.70d router firmware i get errors.

build the "ip access-group 101 in", on the ve 1 interface, then apply the route-map policy "ip policy route-map Internet-Vlan1" to the interface ve 1, and it does not apply with the error that a route-map policy cannot co-exist with an ACL. I dont have the stack fired up at the moment, but that in general words is the error the ICX gives me. If I have "ip access-group 101 in" on an interface, and try to apply a global "ip policy route-map" instead of on the interface, I get the same error telling me ve 1 cannot have an ACL.

**Edit**
I did notice that the ICX does permit the "ip access-group 101 out" command and work in conjunction with the policy route-map but not the "ip access-group 101 in". Not sure how the packets are handled in comparison between the IN and the OUT to maybe use this effectively by modifying the access-list to effectively work the way i hope.

So I guess I just need a fake example, not even of my own, on how I can restrict and deny or permit individual ips or ranges to cross talk vlans, since all the traffic shares this one core switch for management, then the ability to forward each vlan their out of scope ips to their respective firewalls for internet use.

the rules that I have work on the Catalyst, i just need to get them to work, so i can swap it with the ICX.

Thanks again for your insight!

unknown

Q: So the "interface Vlan1" is similar to "Interface ve 1" on fastiron, to compare.

A: Yes. And Cisco calls it a Software Virtual Interface (SVI). On the ICX side they are Virtual Router Interfaces (VRI)... but they are the same in functionality

Q: so by default any device that targets 10.10.1.1 as their gateway without the ACL applied can talk to all other vlans that are known to the router.

A: YES! NO by technicality... Yes, by functionality... This is generally referred to as inter-VLAN routing, but it is really talking to other subnets known to the layer-3 switch's routing table not the different VLANS. Layer-2 Traffic (PDU type is "FRAMES" like Ethernet_II frames), is addressed via its source and destination pysical "MAC" addresses, which are not routeable! They are switchable within a Local-Area-Network segment...

However, these Layer-2 frames contain "encapsulate within them" IP packets ("Layer-3") with contain source and destination IP addresses... When a device triest to talk to a device on another subnet (a different network), the device does an ARP request for the IP of the default-gateway and sets the Layer-2 frames to be destined to the default-gateway's physical "MAC" address, but the IP packet actually has the destination IP address (belonging to the differnet network).

When the frame hits that Layer-2 boundary, the switch completely strips off the Layer-2 frame discarding and removing it. It then compares the packet to its routing table to figure out what to do with it. Presuming it sees this is a directly-conencted network on a different SVI or VRI, it re-wraps "encapsulates" that Layer-3 packet back into a brand-new Layer-2 Frame and drops it into the proper VLAN. ARP ensues resolving the physical addresses and ultimately it gets correctly delivered to the proper device with the return being that device's default-gateway.

Q:
so vlan 1 = devices targeting 10.10.1.1 as gateway
vlan 2 = devices targeting 10.10.2.1 as gateway
vlan 100 = devices targeting 10.10.100.1 as gateway

with no ACL in place they automatically route between, and any device can potentially communicate to another device on the other vlan with no restrictions in between.

A: YES, and they can also communicate within their own VLAN even if you place Layer-3 controlls such as access-lists.

Q: once i build the the access-list of 101 and apply it to vlan 1 on the ingress using the command "ip access-group 101 in", it automatically cuts vlan 2 traffic out, which is intended because they are completely different organizations that should not intrude on each other, but both organizations help fund shared server space in the farm, so they get access to vlan 100. Shared services like file servers, ip phones, active directory, end user management servers, exchange, etc. We do use a combination of ACL and Active Directory to restrict to appropriate use and access to the services.

A: Understood, but an ACL like this is going to cause massive problems:
access-list 101 permit ip 10.10.1.0 0.0.0.255 10.10.100.0 0.0.0.255

It will allow communication out, but it will not allow return traffic back. You can track TCP states for such things as "established" bits being set, but at the very least if you use probably want to look into setting up reflexive Access-Lists if this is your control mechanism. What it basically does is if a device A talks to B to request a web page from port 80 it might look like this

TCP 10.10.1.123:48293 ==> 10.10.2.111:80

Basically, the OS generates an ephemeral port such as 44829 shown above. When the web server responds, it sets the destination port back to that device's ephemeral port. This is what allows large numbbers of concurrently connnected connections to servers and for a firewall to do NAT translation sharing an Internet IP between hundreds or thousands of computers yet be able to return the packets back to the correct devcie.

That said, I do not think Access-Lists are the appropriate answer.

For different organizations, I hihly recmmend at least utilizing VRFs, which are diffent virtual routing tables.

http://docs.ruckuswireless.com/fastiron/08.0.80/fastiron-08080-l3guide/GUID-FA7A07AF-9EBB-4DDC-81B6-...

Realistically speaking separating traffic is best done with different firewall zones...

I will read and answer more tomorrow...

hashim_bharooc1

Hey All,

Thanks for all the good info.

Yes you can make PBR so it will send to selected routes based on your criteria. So you should be able to do routing differently for each router interface/Ve:

Configuring a PBR policy

To configure PBR, you define the policies using IP ACLs and route maps, then enable PBR globally or on individual interfaces. The

device programs the ACLs into the packet processor on the interfaces and routes traffic that matches the ACLs according to the

instructions in the route maps.

From the guide we have 3 simple steps:

To configure a PBR policy:

• Configure ACLs that contain the source IP addresses for the IP traffic you want to route using PBR.

• Configure a route map that matches on the ACLs and sets the route information.

• Apply the route map on untagged interface or on virtual interface.

If you are still having issues maybe you should open a Case with our TAC team and work with them.

Thanks

Best Regards

Hashim

unknown

Q: On the Cisco, by adding the "ip policy route-map
Internet-Vlan1" to the interface this forwards the non internal ip address
ranges off to their firewall/WAN. due to the organizations funding and
policies, they have to have their own internet installed at each site showing
the connection and bill in their name at their address.

A:
Makes sense

Q: so the way the
rules are in effect are not in the order as they are listed when doing a show
run.

the route-map is applied first using the ACL:

access-list 151 deny ip 10.10.1.0 0.0.0.255 10.10.0.0
0.0.255.255

access-list 151 permit ip any any

this works and is in use this way, for first of all their
firewall at each site handles DHCP rules and static assignments, so if a device
is mis-configured on the vlan with the wrong ip and/or subnet then it just does
not work. I would rather have something not work than have a vulnerability.

so as the
"permit route-map" is implemented, it is the same on a cisco as it is
on the fastiron. The "denied" range does not get sent to the
"set ip next-hop" but continues on to the normal core routing. the
"permit" range of everything else not of our ip gets sent to the
firewall to deal with.

A: If a device is misconfigured on a VLAN with the wrong
subnet, it will not have any routing even without a route-map. It will not be able to even ARP for the gateway. Your ACL above puts everything into the
route-map except 10.10.1.0/24 to 10.10.0.0/16

Q: On the Cisco,
the above mentioned "denied" items in the "route-map" then
move on to the rule of "ip access-group 101 in" on the interface,
which then helps regulate what ips or ranges or protocols they can communicate
to and with. plug one end of the straw, and no fluid moves through, since it
requires bi-directional traffic to function in our environment. if the
access-group does not define the traffic, then it moves on to the default
routing tables on the switch that sends it off to another firewall for analysis.

A: Good to know. Thanks.
On the ICX, you can probably apply “ip access-group 101 out” but not “in”
at least the documentation is specific the “in” direction does not work.

...

Q: the
rules that I have work on the Catalyst, i just need to get them to work, so i
can swap it with the ICX.

A: You may need to
reverse your access lists taking into consideration all possible sources.

Realistically, your ACL most likely acts like a one-way check-valve not allowing return traffic. For that there are reflexive Access Lists that dynamically add return entries that age-out upon completion of return traffic. This is the most straight forward resource I found on those:

http://packetlife.net/blog/2008/nov/25/reflexive-access-lists/

BTW, I doubt reflexive access lists will work with route-maps; since, the dynamically created list that allows traffic to flow the opposite direction most likely needs to be applied inbound.

.............

I would probably revamp this and try VRFs to visualize the routing table, so in essence
each organization has a separate virtual router. Here is an unrelated example:

You might need more device memory set asside to run VRFs depending upon Fastiron build family:

system-max ip-route 15168

vrf Organization46

rd 46:1

address-family ipv4

ip route 0.0.0.0/0
192.168.255.1

exit-address-family

exit-vrf

interface ve 46

vrf forwarding Organization46

ip address
192.168.46.1 255.255.255.0

ip helper-address 1
192.168.255.1

Just remember if you do that you would need to look at the
routing table by name:

SSH@TEST#sh ip route vrf organization46

Total number of IP routes: 3

Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost -
Dist/Metric

BGP Codes - i:iBGP
e:eBGP

OSPF Codes - i:Inter Area 1:External Type 1 2:External Type
2

Destination Gateway Port Cost Type Uptime

1 0.0.0.0/0 192.168.255.1 ve 101
1/1 S 27d20h

2
192.168.46.0/24 DIRECT ve 46 0/0 D
255d

3
192.168.255.0/30 DIRECT ve 101 0/0 D
27d20h