SDN in action: Hands-on with Cumulus Linux

Imagine being able to manage scores of network switches as easily as scores of servers; Cumulus Linux makes it happen

Data matrix networking connections system

The stage is set for SDN (software-defined networking) to change the way we push data through our infrastructures, with the promises of more agile network provisioning and management, as well as more affordable network hardware. But for many, the SDN concept is still amorphous. What does SDN look like in practice? 

To shed light on this question, I sat down with a few Dell Networking S6000 switches running Cumulus Linux 2.2.1. There are many approaches to an SDN solution, but one of the most significant is the advent of white-box switches and à la carte switch firmware. This is the essence of the solution offered by Cumulus Networks.

Decidedly Debian

Cumulus Linux is based on the Debian distribution. You may be SSHing into a 1U box with 52 physical interfaces, but from the inside, a Cumulus Networks switch is like any other Linux box. I ran switches built on a Freescale PowerPC base with 2GB of RAM and a 394MB read-write overlay root partition with 390MB free, or what you might expect from a small but complete Linux system.

As with any other Debian Linux box, if you want to configure an Ethernet interface, you either do it from the shell prompt with the various ip tools, or add an entry to the /etc/network/interfaces file that provides details about that physical interface, then use ifup/ifdown to control the interface. If you know how to do standard bridging, routing, forwarding, and firewalling with Debian, then you know how to configure Cumulus Linux for those tasks -- there is no real learning curve.

Cumulus makes use of Quagga for advanced routing protocols like OSPF and BGP; if you’ve used Quagga, you’ll be in familiar territory. Further, Quagga offers a Cisco-like interface that will make veteran Cisco admins feel right at home.

The upshot is that network admins who have Linux experience will be way ahead of the game, while network admins light on Linux experience will need to put in some time to get acclimated to a very different way of working with a network device.

However, keep in mind that the completely open framework of Cumulus Linux makes it possible to develop a custom command-line interface that can be used to manipulate the switch. This CLI could be as simple as some Bash scripts, a Python tool, or any other programming framework. It’s only Linux after all.

Behind the scenes

Although the configuration of the switch is handled entirely within Linux, the kernel itself is not directly involved in packet switching, routing, or firewalling. Instead, Cumulus has developed a translation service called switchd that communicates network configuration changes made in Linux to the hardware Ethernet controllers driving the physical ports and handle the internal switching and routing.

Running as a daemon on the switch, switchd monitors the Netlink communication channels for configuration changes to any part of the network stack, such as assigning an IP address, adding a port to a bridge group, and configuring a firewall rule. Whenever switchd detects a change, it configures the Ethernet controllers to match. Thus, the Linux layer is used as a template of sorts applied to the hardware.

Likewise, interface statistics and other data coming from the physical ports and switching internals are communicated back up through the chain and presented within the Linux layer. Thus, Linux interface inspection commands such as ip link show and ifconfig work as you'd expect.

That is the essence of Cumulus Linux. From the operator's standpoint, it’s merely Linux. The net result is the pairing of wire-speed network operation with a simple configuration layer.

Booting, provisioning, and configuration

The first task with a new switch is to boot it and install an OS. Cumulus uses ONIE (Open Network Install Environment) for this, and it boots looking for an install image either on a local USB device or on the network via standard DHCP and TFTP options. Once the OS is loaded, it runs initial boot automation procedures that use DHCP options to locate further instructions -- by downloading and running a Python script, for example. Thus, with a properly configured network boot configuration, switch provisioning could be completely automated. Then, to configure the switch, we SSH into it.

That script could perform any type of specific configuration tasks, including adding local APT sources and installing custom packages, installing data center orchestration clients for Puppet, or distributing pre-shared SSH keys -- you name it. If you can script it, it can run. This procedure can include the application of the required license file, as the switch will not allow the use of any switching ports unless a valid license file is installed.

The Dell Networking S6000-ON switches that formed my test bed are built to a familiar spec. The S6000-ON has 32 ports of 40G Ethernet or 96 ports of 10G and eight line-rate 40G Ethernet ports in breakout mode, plus a single out-of-band Ethernet management port and a serial console port. The ports are referred to internally as swp1 through swp32, with a loopback at lo.

Let's say we want to create a two-link LACP-bonded connection to another switch, then place a few ports into VLAN10 with this switch. We do this by defining the bonds and bridge specifications in the /etc/network/interfaces file. For the bonded ports, we create the bond and assign the ports we need as follows:

auto swp1
iface swp1

auto swp2
iface swp2

auto uplinkbond0
iface uplinkbond0
            bond-slaves swp1 swp2
            bond-mode 802.3ad
            bond-miimon 100
            bond-lacp-rate 1
            bond-use-carrier 1
            bond-min-links 1
            bond-xmit-hash-policy layer3+4

The above lines will place interfaces swp1 and swp2 into an LACP bond using 802.3ad. A similar configuration on a switch connected to those ports will bring up two bonded interfaces.

Of course, adding these lines to the file doesn’t enable the ports themselves. We need to bring up the ports and the bond from the command line:

$ sudo ifup swp1
$ sudo ifup swp2
$ sudo ifup uplinkbond0

This highlights a conceptual difference between Cumulus Linux and a traditional switch. The configuration file is simply a file, and changes made within the file do not happen immediately -- unlike changes made using a standard switch command-line interface. You can make changes that are immediately applied to the switch with the Linux ip tools, but those will not be retained in the interfaces file. Thus, admins new to a Linux-driven switch will need to understand the configuration persistence rules.

If we have a large number of ports to enable, we don’t need to do them one at a time. Instead, we use the shell because it’s Linux. The following command enables ports 1 through 24:

$ for i in {1..24}; do sudo ifup swp$i; done

We can check the status of the bond as you might expect, by viewing /proc/net/bonding/uplinkbond0.

Now that we have our bonded interface, let’s create a bridge, or VLAN, and assign our ports:

auto swp10
iface swp10

auto swp11
iface swp11

auto vlan128
iface vlan128
       bridge-ports swp10 swp11
       bridge-stp on

Here we have created VLAN128, placing ports swp10 and swp11 on that VLAN, with their traffic untagged. If I wanted to add a tagged port, such as a trunk interface, I could add swp12.128, which would include traffic from port swp12 that was tagged 128.

So far, everything we've seen here is standard procedure for using interface bonding and bridging on a Debian Linux system.

To look at the interfaces, we use the Linux ip tools:

$ ip -s link show dev swp1
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master uplinkbond0 state UP mode DEFAULT qlen 500
   link/ether 6c:64:1a:00:2e:16 brd ff:ff:ff:ff:ff:ff
   RX: bytes packets errors dropped overrun mcast  
   39442     318     0       8       0       318  
   TX: bytes packets errors dropped carrier collsns
   37952     29      0       0       0       0    

Likewise, we can use brctl to show bridging information:

$ brctl show
bridge name   bridge id           STP enabled   interfaces
vlan128       8000.6c641a002e1e   yes           swp10

When working with routing protocols, the rules are a little different. As noted above, Cumulus uses Quagga to handle the configuration of these protocols, and several methods can be used to manipulate the configuration. In addition to the standard Quagga vtysh utility that drives like Cisco IOS, Cumulus-specific commands such as cl-ospf and cl-bgp achieve the same results without the need to work with the command-line interface. These allow for one-line configuration changes and easy scripting.

As an example, we could specify an OSPF area thusly:

$ sudo cl-ospf network add area

Or we could do the same through vtysh:

$ sudo vtysh

Hello, this is Quagga (version 0.99.21).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

q# configure terminal

q(config)# router ospf

q(config-router)# network area

The companion to cl-ospf is cl-bgp, which manipulates the BGP daemon. Routing protocol availability via Quagga is handled as usual, by specifying which protocols to enable in the Quagga daemons file. Here it's important to note that changes made in the Quagga.conf file persist over reboots but are not implemented immediately unless Quagga is restarted, whereas changes made via the cl-ospf and cl-bgp commands take place immediately but do not persist across reboots. Changes made in the Quagga command-line interface are made immediately and retained if a wr mem command is issued. Thus, you can drive the routing protocols either way for the best of both worlds.

There is also LLDP and CDP support via lldpcli:

$ sudo lldpcli show neighbors ports swp1
LLDP neighbors:
Interface:   swp1, via: LLDP, RID: 3, Time: 0 day, 00:16:54
    ChassisID:   mac 6c:64:1a:00:2a:8f
    SysName:     leaf1
    SysDescr:     Cumulus Linux
    Capability:   Bridge, off
    Capability:   Router, on
    PortID:       ifname swp1
    PortDescr:   swp1

Switch internals

Aside from networking functions, general switch maintenance is simple. There are provisions for two firmware images -- one primary, one alternate -- and command-line options to swap between them if needed. The file system layouts differ depending on the architecture of the switch. PowerPC-based switches have an overlay file system visible to each image, while x86-based switches have separate logical volumes. The separate file systems are nevertheless available to each image, so you can copy files to and from active and alternate image file systems if needed.

Logging is handled with syslog as you would expect, with logs present in /var/log.

From a Linux admin perspective, Cumulus Linux is standard stuff, and most of what has been outlined in this article will seem remedial, basic operations. That’s the underlying principle and value of Cumulus Linux -- it's really just Linux. It can be scripted, adapted, adjusted, tweaked, and customized to suit nearly any network or administrative requirement, using commodity hardware switches.

The brilliance of Cumulus Linux is in the simplicity. For a significant number of seasoned Linux administrators and use cases, Cumulus Linux switches will carry no substantive learning curve to integrate into production networks. Paired with data center orchestration tools, Cumulus Linux offers a combination of flexibility, compatibility, and ease of use that will shake up traditional networks everywhere.


Copyright © 2014 IDG Communications, Inc.