Sometimes you will have some L2 domains (Bridge Domains – BD) in your datacenter that will be used with hardware appliances like F5 NLB or something like an additional firewall, WAF or something similar. That is the case where ACI will not route or bridge but the only L3 point of exit from that kind of segment would be on actual hardware appliance outside ACI Fabric – connected to the Leaf port.
We will take an example here and use it throughout the article where BIG IP F5 NLB is used as an L3 termination of L2 BD 10.10.10.0/24.
F5 is directly connected to ACI Leaf and routing from 10.10.10.0/24 subnet (L2 BD) is done directly on F5 device which is default gateway for that subnet endpoints.
In those cases for some particular implementations when you decide not to use PBR or Service graphs, it will happen that appliances like our F5 would become L3 termination for some ACI L2 BD like the 10.10.10.0/24 from my beautiful image above.
When an appliance like F5 NLB becomes default gateway for some L2 BD you need to enable flooding in that DB because you need flooding on all BDs which have an outside appliance mapped to them (for ARP packets to get from the outside appliance to VMs and vice-versa).
IP unicast routing can be disabled on that BD because with no L3 point in ACI (no subnet configured on BD) the possibility for ACI to do “smart” bridging and routing of that traffic is lost. ACI will not learn IP addresses of endpoints (VMs) in that bridge domain nor it will do anything more for that traffic than normal L2 switch would do. He will only know MAC addresses and bridge the traffic towards F5 interface which will respond to ARP when some endpoint sends an ARP request asking for the default gateway.
The orange traffic flow
On the other side, when your datacenter has L3OUT connection to the outside world (in order to be able to receive traffic from clients on the Internet or somewhere), you are in a situation that all ACI learned subnets (ACI datacenter network segments or BDs with L3 configured) can be advertised outside through L3OUT.
On the other side, endpoints in L2 BD segments (like VMs in L2 BD behind F5 NLB appliance in subnet 10.10.10.0/24 in our example) will not be learned by ACI and they will not be advertised outside even if we configure that DB to advertise outside (because ACI is not aware that some IPv4 or IPv6 addresses exist in that BD).
For our L3 BD (F5-OUT 10.10.20.0/24) NLB’s outside interface IPs (SelfIP or VIP IP), ACI will know they exist and will be able to route towards them from any point in ACI Datacenter.
If we configure F5 device to load-balance some service that is running on VMs in L2 BD (VMs 10.10.10.10 and 10.10.10.20) that service will become available on a VIP address from 10.10.20.0/24 subnet as a VIP address and it will be reachable from ACI fabric and advertised outside ACI if so configured for that L3 BD.
We are talking about orange traffic flow from the image above in case you are wondering.
The purple traffic flow
When administrators request direct access to those same VMs behind NLB for administrative operations, they will want to access those VMs by their own IP and not VIP address on NLB (VIP address on NLB receives traffic for some service and load-balances that traffic towards more VMs behind in server pool). One of the reasons could be that by accessing VIP, you are not sure on what VM you will end up.
okay okay, you can control that on NLB device but we need direct access to VMs for administrative reasons to work all the time without the need to touch NLB config every time.
From the ACI perspective, traffic towards IP addresses like 10.10.10.10 and 10.10.10.20 inside L2 BD and originated within ACI will get to those VMs because F5 device will respond with his IP address of outside interface when someone asks (ARP requests) those addresses. All IPs from endpoints in BD 10.10.10.0/24 will be seen by ACI on F5 outside interface and they will be shown in L3 BD 10.10.20.0/24
F5 needs to be additionally configured as a forwarder in order to enable any IP traffic to get to those VMs by acting as a router and not load balancer.
Advertise Routes to Those Endpoints Outside ACI
The thing is, those IPs are only learned and known inside ACI and are not advertised outside 10.10.20.0/24 BD even when that BD has “Advertise Externally“ routing setting enabled.
We have an option to solve that by generating host routes inside EPG that represents F5 outside BD with IP addresses of those endpoints which will then (with defined proper next-hop to NLB appliance outside interface) be advertised outside ACI datacenter effectively enabling inbound traffic to get the information where to send the traffic to in order to get to the VMs in question (10.10.10.10 and .20).
L2BD Host Route to outside advertisement
When configuring host routes to those endpoints (in L2BD-F5IN) we need to configure host routes under outside EPG (L3BD-F5OUT).
In this case, like from the example just above, a host route will be generated and advertised outside ACI from L3 BD. This route will tell to outside clients that 10.10.10.10 IP exists inside ACI and that is reachable by sending traffic towards L3 BD IP address 10.10.20.10 which is actually F5 outside interface IP. F5 will receive that traffic and will be able to route it towards final destination 10.10.10.10
READ MORE ABOUT CISCO ACI:
- Google Jupiter Data Center Network Fabric – New Way of Building Data Center Network Underlay
- Switch vSphere Enterprise Plus license to vSphere Standard on a NSX-T enabled cluster
- NSX-T Edge Transport Node Packet Capture
- VMware NSX-T Install Tips & Tricks
- VMware TKGI – Deployment of Harbor Container Registry fails with error
- Software-defined data center and what’s the way to do it
- What is Cisco ACI?
- CLOS Topology
- Setting up Cisco ACI From Scratch
- New ACI deployment? Watch out when connecting APICs to Leafs
- ACI MultiPod and how to build MultiDatacenter with Cisco ACI
- Cisco ACI – API Calls vs JSON POST
- Cisco ACI – Configuring by POSTing JSON
- How to Advertise a Route from ACI Layer2 BD Outside the Fabric?
- ACI MultiPod – Enable Standby APIC
Needs a bit of clear diagram. but looks like a good note you put here. thanks
Would you be able to advertise a subnet or these are only host routes? I have a use case where I need to avertise off L3out a network that reside inside a VM.
Hi oq, only host routes unfortunately. I’ve used JSON post to get 254 host routes into config for that use case