It’s one of those articles aimed at the people with Cisco ACI experience who don’t bother with reading all the install and other guides again while going through n’th time of building and ACI fabric, like me. When it comes to Cisco ACI, you really should.
There’s a small change with the physical build of the third generation of APIC server where 10G SFP interfaces from APIC towards the Leaf switches (used for fabric discovery and later for the in-band controller to fabric communication) where 4x10G card is built in the server and not like 2x10G on M2/L2 and other first and second generation of APICs.
When you see those 4x10G ports on the server, the logical thing to do will be to use the first two ports on each APIC and connect them to two Leafs (for redundancy and stuff). It ended up being that is not really how Cisco intended those interfaces to be used and it will end up blowing your fabric stability and management. I was able to discover the fabric and register the fabric leaf and spines. It was even possible to configure the whole thing up to the functional fabric and L2-L3 functions but the APIC cluster was always unstable and going in and out of configuration stale and data diverged statuses on cluster view.
The main issue was that we were unable to upgrade APIC cluster successfully.
Cisco ACI Fabric discovery went out fine
.. but upgrade process failed every time
APIC state was always showing some errors
Cluster instability basically didn’t allow for the upgrade process to get through the first APIC communication checks that need to pass in order for the upgrade process to start. We noticed that APICs are not able to ping between themselves and with few rounds of TAC troubleshooting sessions we ended up with the info that we used APIC 10G uplink to Leafs in the wrong way.
The right way of connecting APIC server to two Leaf switches is to use the first and third SPF interfaces on 4x10G card and not the first two interfaces as it was in the GEN1 and GEN2 APIC cluster.
If you read your manuals before doing stuff it can save you and your TAC engineer a lot of time
This is the way GEN2 APIC server was connected towards the Leafs:
GEN2 APIC to Leafs connection
ANd this is the right way to use available interfaces on APIC server towards two Leaf switches:
GEN3 APIC to Leafs connection
So the whole idea of this article was to give you an early notice on the things that can go south when working with the latest gen of ACI fabric and controllers. Always give a look at the manuals, things are very dynamic at Cisco ACI departments and things are changing in every new generation of hardware as well as in each new software version.