New ACI deployment? Watch out when connecting APICs to Leafs
It’s one of those articles aimed at the people with Cisco ACI experience who don’t bother with reading all the install and other guides again while going through n’th time of building and ACI fabric, like me. When it comes to Cisco ACI, you really should.
There’s a small change with the physical build of the third generation of APIC server where 10G SFP interfaces from APIC towards the Leaf switches (used for fabric discovery and later for the in-band controller to fabric communication) where 4x10G card is built in the server and not like 2x10G on M2/L2 and other first and second generation of APICs.
When you see those 4x10G ports on the server, the logical thing to do will be to use the first two ports on each APIC and connect them to two Leafs (for redundancy and stuff). It ended up being that is not really how Cisco intended those interfaces to be used and it will end up blowing your fabric stability and management. I was able to discover the fabric and register the fabric leaf and spines. It was even possible to configure the whole thing up to the functional fabric and L2-L3 functions but the APIC cluster was always unstable and going in and out of configuration stale and data diverged statuses on cluster view.