Tag: Switch

Google’s Datacenter Optical Circuit Switches and Jupiter network fabric

Google’s data centers are unlike any other. It seems they have windows like normal houses because as from the last SIGCOMM’22 presentation, they took their SPINE switches and threw them right out of that window.

Google worked on the Micro Electro Mechanical Systems (MEMS) for years in order to build an Optical Circuit Switch (OCS) that would enable dynamic reconfiguration of optical connections between switches in the data center. Optical Circuit Switch enables on-the-fly data center fabric aggregation block switch connections reconfiguration without the need for physical rewiring. And most interestingly, the usage of Software-Defined Networking (SDN) traffic engineering, enables aggregation block switches to be directly connected and completely removes the need for those bulky Spine switches that were connecting aggregation block switches in CLOS topologies.

OCS MEMS mirror array can redirect light from any input port to any output port by slightly moving one of its mirrors and changing dynamically optical connections between aggregation blocks

Spine switch roles are basically replaced with OCS devices for smart, dynamic and direct interconnections of Leafs. SDN is used for BUM traffic handling (Broadcast, unknown-unicast and multicast traffic) that was the other important spine role.

Google paper (Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking) that was presented at the SIGCOMM’22 conference describes how getting rid of Spines, and smart traffic handling with SDN, enabled the Jupiter fabric to get up to 5x higher speed and capacity and 40% reduction in power consumption in relation to similar CLOS topology datacenter fabrics.

Well… It will reboot your whole switch stack at once, In case you were wondering. But it has a neat feature of automatic rollback to the previous IOS XE version if something goes south with the newly upgraded switches.

The same goes for non-stacked Cisco Catalyst C9200 and C9300 switches, but the question was, and the answer is hard to find if the stack would reload members sequentially or it would just reload all members at once. The answer is of course the least good option which makes the upgrade impossible without network outage even if other devices are connected to the stack redundantly (to two or more stack members).

The whole procedure is fairly simple:

Switch Upgrade install mode

New Cisco switches are now usually shipped with install mode configuration for software installation. The other (older) bundle mode was simply a boot variable on the switch config that stated where the .bin files is saved on the switch flash and if multiple .bin files which one to load upon switch reboot.

Media Access Control Security or MACSec is the Layer 2 hop to hop network traffic protection.

Just like IPsec protects network layer, and SSL protects application data, MACSec protects traffic at data link layer (Layer 2).

MACSec is standardized IEEE 802.1AE hop-by-hop encryption that enables confidentiality and integrity of data at layer 2.

It encrypts entire Ethernet packet except Source and Destination MAC addresses on any device-to-device, switch-to-switch, router-to-switch, host-to-switch directly connected wired L2 connection.

If we compare MACSec with, for example, IPsec, MACsec provides same security but on layer 2 for each hop separately. On each hop, packets are received and decrypted enabling other features like QoS and load balancing to function. Same data is again encrypted before being sent out to next device on the way towards the destination. MACsec works on host-to-switch connection and on switch-to-switch connection enabling line-rate up to 10 GigabitEthernet fast encryption.

It uses, especially for that occasion developed and optimised Galois/Counter Mode Advanced Encryption Standard 128 (GCM-AES-128) which makes line-rate encryption/decryption possible.

There is some difference in MACSec on the connection between host and switch, and MACSec on the connection from switch to switch

MACsec uses MACsec Key Agreement (MKA) which is standardized protocol to provide the required session keys and manages the required encryption keys on a MACSec connection between hosts (PC, server, etc.) and switch.

This one is long. Do not be afraid though, I made it just to give you the fastest way to deploy functional dot1x to your company HQ without reading even more documentation and searching for those little timer default settings.

I the article prior to this I showed you how to setup your environment with simple dot1x and make it as simple as possible. I will not repeat again the part about setting up Radius Clients on server side, everything else is here once again just more complex. Now is time for a more complex example that will make your implementation work out-of-the-box for end users and they will probably not even know that you completed the implementation of one nice but fairly complex network security enhancement.

So, as I said, this is a better way to do it because it will be less intrusive and people in the office will start to use it but without prompt starting to show on everybody’s PC.

What needs to be done:

Here are the steps needed on all systems so you do not forget some of them. Of course you will not forget, but I know people who did forget, for example, the whole client computer part.

1. We will configure the switch for dot1x but with much more options now.
2. We will create Radius NPS policy to enable our Windows machines to authenticate using user or computer certificate. (This will enable us to skip boring credentials prompts mentioned above)
   Here the Radius config skips radius client configuration mentioned in previous article about dot1x.
3. We will setup a new GPO object that will automatically setup all PC’s for dot1x.

What will we get using dot1x:

It’s give you all descriptions of all possible options that you would need and the example in the end will show configuration of all those components together. You will see, it will be easy to recognise which command does what. If there would be some thing not clear enough, I am fairly quick with my comment replies 😉

Basic

Dot1x will allow access to network only to authenticated users on your wired LAN. It’s also used to authenticate users on Corporate WiFi network but we will skip that part now.

Give Internet only to unauthenticated users

Don’t be cruel, maybe you have some guests in your meeting room. If they connect and get denied they will tell you your network jacks are not working and blame you that you are not doing your job as network admin.

“Hey, network is not working in meeting room! Some guest are complaining, is bad publicity :)”

If the users are not authenticated switch port where they connected the cable does not get them access to the network. There is an option to use VLAN auto-configuration in conjunction to basic security feature to get unauthenticated users access to the network but only to some isolated or Internet only VLAN. Cool.

If your LAN is extending to some places where unauthorised people can just plug in and gain access to your protected network, it’s time to implement some security on your access switch. The best thing to do is to implement IEEE 802.1X port-based authentication which will enable users/machine authentication and prevent unauthorized devices from getting access switch port running when connected. IEEE 802.1X port-based authentication is mostly called simply as dot1x.

In this article I will show you how to configure some basic dot1x stuff on switch side. I will also include Windows machine side of configuration as this is something most people presume it’s working out-of-the-box but of course that’s not the case. Radius server policy is fairly simple so a screenshot of the policy will get you going. So as you see, to get dot1x running you need to configure: