TCAM and CAM memory usage inside networking devices

As this is networking blog I will focus mostly on the usage of CAM and TCAM memory in routers and switches. I will explain TCAM role in router prefix lookup process and switch mac address table lookup.

However, when we talk about this specific topic, most of you will ask: how is this memory made from architectural aspect?

How is it made in order to have the capability of making lookups faster than any other hardware or software solution? That is the reason for the second part of the article where I will try to explain in short how are the most usual TCAM memory build to have the capabilities they have.

CAM and TCAM memory

When using TCAM – Ternary Content Addressable Memory inside routers it’s used for faster address lookup that enables fast routing.

In switches CAM – Content Addressable Memory is used for building and lookup of mac address table that enables L2 forwarding decisions. By implementing router prefix lookup in TCAM, we are moving process of Forwarding Information Base lookup from software to hardware.

When we implement TCAM we enable the address search process not to depend on the number of prefix entries because TCAM main characteristic is that it is able to search all its entries in parallel. It means that no matter how many address prefixes are stored in TCAM, router will find the longest prefix match in one iteration. It’s magic, right?

Image 1 shows how FIB lookup functions and points to an entry in the adjacency table. Search process goes through all entries in TCAM table in one iteration.

Router

In routers, like High-End Cisco ones, TCAM is used to enable CEF – Cisco Express Forwarding in hardware. CEF is building FIB table from RIB table (Routing table) and Adjacency table from ARP table for building pre-prepared L2 headers for every next-hop neighbour.
TCAM finds, in one try, every destination prefix inside FIB. Every prefix in FIB points to adjacency table’s pre-prepared L2 header for every outgoing interface. Router glues the header to packet in question and send it out that interface. It seems fast to do it that way? It is fast!

Switch

In Layer 2 world of switches, CAM memory is most used as it enables the switch to build and lookup MAC address tables. MAC address is always unique and so CAM architecture and ability to search for only exact matches is perfect for MAC address lookup. That gives the switch ability to go over all MAC addresses of all host connected to all ports in one iteration and resolve where to send received packets.

CAM is so perfect here as the architecture of CAM provides the result of two kinds 0 or 1. So then we make the lookup on CAM table it will only get us with true (1) result if we searched for the exact same bits. L2 forwarding decisions are the one using this fast magical electronics!

More than plain routing and switching

Besides Longest-Prefix Matching, TCAM in today’s Routers and Multilayer Switch devices are used to store ACL, QoS and other things from upper-layer processing. TCAM architecture and the ability of fast lookup enables us to implement Access-Lists without an impact on router/switch performance.

Devices with this ability mostly have more TCAM memory modules in order to implement Access-List in both directions and QoS at the same time at the same port without any performance impact. All those different functions and their lookup process towards a decision is made in parallel.

More on TCAM

TCAM is basically a special version of CAM constructed for rapid table lookups. Not mentioned before, TCAM can get Us with three different results when doing lookups: 0, 1, and X (I don’t care state).

With this strange third state, TCAM is perfect for building and searching tables for stored longest matches in IP routing tables.

There is just one condition that  IP prefixes need to be sorted before they are stored in TCAM so that longest prefixes are on upper position with higher priority (lower address location) in a table. This enables us to always select the longest prefix from given results an thus enables Longest-Prefix Matching.

TCAM Architecture

In the Image 2 here below I showed, (please disregard my style), one of the simplest CEF Explanations I could find in scientific articles around. It is basically showing you usage of FIB on the left and Adjacency table on the right. FIB stored in TCAM table and Adjacency table stored in RAM memory. Great, it shows without words what we spoked about before in “ROUTER” section.

Image 2 FIB implemented in TCAM, adjacency table implemented in RAM

Ok, Here you must know that IP addresses in the example are smaller that real ones. Here we have addresses of 5 bits not 32 like IPv4, all other is the same as the real stuff.

We are looking on the left side now at the CAM part, it is basically explained for TCAM.

So in TCAM world in order to get the longest match like in the Image 2 above here, before populating the TCAM we need to sort the entries so that longer prefixes are always situated on higher priority places. As the lookup goes from top downwards it means that higher priority is higher in the table, closer to the top. OK, now that we solved this it is easy to see that TCAM here is searching in parallel from left to right all four address entries.

Entries here in TCAM are numbered 00,01,02,03 from top to bottom. Not like in Routing table above where they are numbered 1,2,3,4. Don’t let that confuse you.

Second and third entry (01 and 02 entry) are the same as the one we search in first three bits. When it comes to the fourth bit, he is “X” for entry 02.

X means don’t care or the third possible solution that can come out of TCAM table query. In the situation above, if we look at the second and third line of TCAM table, this search will make a match for both of entries. The fourth bit of “01” is matched and the fifth bit does not care. For “02” it will show true value at the encoder entrance as a fourth and fifth place do not care!

Based on the priority order from above, line “01” is the longest-prefix match and it is selected and send to encoder who will link that entry to Adjacency table entry for making the packet L2 ready. Remember, on this image, “01” is sent to Adjacency table as a pointer. It is pointing to Adjacency table entry 01 which will then be used use for this packet creation.

L2 header will be added to that packet and the packet will be sent out on port B to the neighbour.

TCAM parallel search process inside circuitry

Actually with CAM and TCAM chips the logic is slightly different that you might think.

For all entries that are matching the searched one, encoder entry will get “true” signal, and all not matched entries will show “false” output, no problems there. The catch is in the beginning of the process. Before search begins all entries when entered inside TCAM are closing the circuitry on TCAM word entry and show “true” at encoder side. All entries are temporarily in the match state. When parallel search is done it will brake all entries that have at least one bit that does not match the searched entry.

Here is the explanation of the “don’t care bit”, in the search process when the search gets to X bit (“don’t care bit”) it will not change the state of that matchline. That’s why No 2 and No3 lines made a match, and that’s why TCAM is perfect for longest-prefix lookup.

This also explains why TCAM memory is so power hungry. It needs to power on all circuits to be able to make a search not only the matched ones. Limited memory space and power consumption associated with a large amount of parallel active circuitry are the main issues with TCAM.

If we look at the right side of the Image 2 now, we see that adjacency table is built in RAM memory. Adjacency table uses ARP table and Routing table data for building pre-prepared L2 headers for every next-hop neighbour. As described before in “Router” section it will prepare the packet to be sent to Layer 1 and out the interface in a flash. Entries need to keep L2 data and this data does not change often. RAM memory is consequently perfect fit for adjacency table. Quick, not expensive, not space limited and not so power hungry.

Conclusion

TCAMs were on my post CCIE learning list so I gathered almost all this information by myself researching different materials online. It’s new to me too, so please correct me in comments if I made some mistakes…

The thing can go really ugly from here on so I will stop for now. The function and components of TCAM like CAM core cells and their different architecture like 9-T NAND or 10-T NOR will probably be too much even for me at this time. If I will ever be ready to try and explain further this amazing circuitry, it will be written here.