Author: Valter Popeskic

It’s been almost exactly 11 years since I passed the R&S CCIE lab in Brussels, and now it was time to go with something more cloudy 🙂

I just passed the AWS Advanced Networking Specialty, and unlike CCIE I did this one on my first attempt.

This blog has been running for 16 years now. While the last few years have been quiet here, my professional life has been anything but. I’ve transitioned from a CCIE-focused, Cisco datacenter role into a fast-paced, startup-like environment. My focus has shifted from building physical cloud underlays to designing automated, high-performance cloud-native architectures.

Recently, I decided to validate this four-year journey into AWS by taking the AWS Certified Advanced Networking – Specialty exam. It’s widely considered one of the toughest in the AWS lineup, and I can confirm it truly tests your depth. Interestingly, this is where the “good old” networking fundamentals from my Cisco days became my greatest asset. Networking remains the heart of the cloud; only the tools have changed.

Today, my work is less about manual configuration and more about writing Infrastructure as Code (IaaC). It’s been a busy few years, and I have a backlog of drafts to prove it. It’s time to start sharing them.

Just got the news—I’ve been selected as a Cisco Champion for the 8th year in a row! Truly honored and thankful to be part of such an inspiring community.

This year is full of milestones. While Cisco celebrates its 40th birthday, I’m marking 10 years as a CCIE and 15 years from my CCNA—a journey that’s brought endless learning, challenges, and growth throughout my career.

Being a Cisco Champion year after year has been a great way to stay connected with amazing people, share knowledge, and keep up with the ever-evolving tech world.

Here’s to Cisco’s 40 years of networking—and to all of us continuing to grow and build what’s next!

Google’s Datacenter Optical Circuit Switches and Jupiter network fabric

Google’s data centers are unlike any other. It seems they have windows like normal houses because as from the last SIGCOMM’22 presentation, they took their SPINE switches and threw them right out of that window.

Google worked on the Micro Electro Mechanical Systems (MEMS) for years in order to build an Optical Circuit Switch (OCS) that would enable dynamic reconfiguration of optical connections between switches in the data center. Optical Circuit Switch enables on-the-fly data center fabric aggregation block switch connections reconfiguration without the need for physical rewiring. And most interestingly, the usage of Software-Defined Networking (SDN) traffic engineering, enables aggregation block switches to be directly connected and completely removes the need for those bulky Spine switches that were connecting aggregation block switches in CLOS topologies.

OCS MEMS mirror array can redirect light from any input port to any output port by slightly moving one of its mirrors and changing dynamically optical connections between aggregation blocks

Spine switch roles are basically replaced with OCS devices for smart, dynamic and direct interconnections of Leafs. SDN is used for BUM traffic handling (Broadcast, unknown-unicast and multicast traffic) that was the other important spine role.

Google paper (Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking) that was presented at the SIGCOMM’22 conference describes how getting rid of Spines, and smart traffic handling with SDN, enabled the Jupiter fabric to get up to 5x higher speed and capacity and 40% reduction in power consumption in relation to similar CLOS topology datacenter fabrics.

After initiating Cisco DNA Appliance version 2.1.2.4 and starting an upgrade towards 2.2.2.8 in order to get to 2.2.3.5 I got a strange issue where the appliance system update went fine but the switch to 2.2.2.8 was disabled until Application Updates did not finish.

The real issue here was that Application Updates of Cloud Connectivity – Data Hub got stuck on 12% for 4 days without timing out or finishing. Tried several appliance reboots from CIMC which didn’t help.

Below are the steps that helped sort out Application Updates issues with container pods being stuck at the point of pooling images from the cloud and creation on the Cisco DNA appliance. Of course, this was a long troubleshoot and here I listed only maglev CLI commands that were useful.

I SSH into the appliance and started digging the maglev CLI tool.

Well… It will reboot your whole switch stack at once, In case you were wondering. But it has a neat feature of automatic rollback to the previous IOS XE version if something goes south with the newly upgraded switches.

The same goes for non-stacked Cisco Catalyst C9200 and C9300 switches, but the question was, and the answer is hard to find if the stack would reload members sequentially or it would just reload all members at once. The answer is of course the least good option which makes the upgrade impossible without network outage even if other devices are connected to the stack redundantly (to two or more stack members).

The whole procedure is fairly simple:

Switch Upgrade install mode

New Cisco switches are now usually shipped with install mode configuration for software installation. The other (older) bundle mode was simply a boot variable on the switch config that stated where the .bin files is saved on the switch flash and if multiple .bin files which one to load upon switch reboot.