Tag: troubleshooting

NSX-T v3.0.1 and v3.1.3 were used to try the stuff described below

As always with network engineers, even when working with SDN/SSDC solutions, sooner or later you will be asked to troubleshoot connectivity across your hops. And if working with VMware NSX-T platform, your next-hop for the North-South Datacenter traffic will almost always be NSX-T EDGE Transport Node VM. It will be really useful then to be able to get some packet traces out of that box in order to troubleshoot the traffic issues in detail.

One of the examples would be simple routing or some sort of Loadbalancing traffic that seems not to reach the backend hosts behind NSX-T edge.

On the NSX-T EDGE VM it’s fairly simple to capture traffic directly. It’s possible to get the output out on the console or to save it to the file on the EDGE and then pull it out with SCP.

If you have an EDGE Cluster, normally build out of 2 VMs, first, you need to see on which node the T0 or T1 router you want the traffic to be captured is active.

Let’s say we want to capture traffic on “T0-router” shown in the image below. You can go to that T0 router from the UI and check the High Availability Mode output:

APIC Controller Cluster

You actually need three APIC controller servers to get the cluster up and running in complete and redundant ACI system. You can actually work with only two APICs and you will still have a cluster quorum and will be able to change ACI Fabric configuration.

Loosing One Site

In the MultiPod, those three controllers need to be distributed so that one of them is placed in the secondary site. The idea is that you still have a chance to keep your configuration on one remaining APIC while losing completely primary site with two APICs. On the other hand, if you lose secondary site, two APICs in the first site will still enable you to do configuration and management of ACI Fabric as nothing happened.

Losing DCI Interconnect

The second type of MultiPod fail is when you lose DCI (datacenter interconnect). In that case, both sites will keep working but will alert that the other side is down. The secondary site with one APIC will be in read-only mode and the primary site will be fully functional with two remaining APICs on that site. If some changes are made on the primary site, those changes will be replicated to the third controller on the secondary site when DCI recovers and configuration relating site B POD will be then pushed to POD 2 fabric.

DCI issues are not a good time for APIC replacement, just wait for DCI to start working normally and continue to use ACI APIC controllers as before the issues. You will still have the option to manage primary site if DCI fails and after DCI starts working again changes will be replicated to secondary site APICs and Fabric.

Please note that temporary DCI issues are not a good time to replace the APIC. If you are experiencing just a DCI outage the second site still works but it cannot be configured. Think about it, perhaps the best thing to do in this case is not to change the configuration of your fabric on either side while DCI doesn’t get back up and running. That way you are sure your configuration does not affect the MultiPod stability once DCI gets back up and sites start to communicate again.

EIGRP internals and getting hands dirty in debugging routing adjacency and solving EIGRP neighboring issues.

What is sequence TLV and Conditional Receive CR-mode and CR flag

Couple of days ago I got a strange network behavior in my CCIE lab. Something was wrong between a router and L3 switch connection and there was EIGRP neighbor relationship reset every few minutes. It was happening all the time so I decided to debug a little.

It was looking something like this and I was very confused about it:

*Mar 1 01:00:32.135: EIGRP: Received Sequence TLV from 155.1.1.2
*Mar 1 01:00:32.135: 155.1.1.1
*Mar 1 01:00:32.139: address matched
*Mar 1 01:00:32.139: clearing CR-mode
*Mar 1 01:00:32.139: EIGRP: Received CR sequence TLV from 155.1.1.2, sequence 5
*Mar 1 01:00:32.139: EIGRP: Received UPDATE on FastEthernet0/0 nbr 155.1.1.2
*Mar 1 01:00:32.139: AS 1, Flags 0xA, Seq 5/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1, not in CR-mode, packet discarded
………………………….
*Mar 1 01:10:00.123: EIGRP: FastEthernet0/0 multicast flow blocking cleared

The thing that was happening here is basically just some EIGRP internals doing their job. To be precise this was reliable routing information delivery over both multicast and unicast doing his reliable update delivery.

Normal EIGRP synchronization is done using multicast.

If we have two EIGRP routers that are trying to make initial sync and become neighbors they will intentionally see each other as “laggard” in the beginning. The term laggard is important here as it symbolize a neighbor to witch we are sending EIGRP updates in separate unicast communication.

Why?

If one EIGRP router is sending updates to, let’s say, 5 other neighbors. It will send that update to address 224.0.0.10 and it will include the update sequence number inside. Let’s say Seq=25.

When it sends the update Seq=25 to the network segment it will get ready to send next update with Seq=26 and wait for the acknowledgement of sequence 25 update. The problem is that the router will put newly prepared update Seq=26 onto the transmission lists for all 5 neighbors and it will not send it out to anybody until he acknowledges sequence 25 update from all 5 neighbors. That means that if one of 5 routers does not send back the acknowledgement for Seq=25 update our router will not continue sending multicast update Seq=26 to anybody and he will lose the neighbors after hold timer expires.

This useful troubleshooting command for cmd provides information about network speed and network loss at every node between a source and destination. Pathping sends Echo Requests to every router between a source and destination and then calculates the results based on the packets returned from every device. By displaying the percentage of packet loss at any given node, we can see which router have some network problems. Pathping is making the same thing like tracert command but with more information about every router on the way.