Estimated reading time: 4 minutes
By Chris Wilkinson, Internet2 Network Services Director of Planning and Architecture
Internet2 marks the beginning of this week having deployed its first operational 400 Gbps Ethernet connections for its new Next Generation Infrastructure (NGI) platform. Cisco 8201 routers in adjacent cities are now connected using high capacity coherent optical links provided by Ciena Waveserver 5 transponders. This work builds on the successful deployment of the first 800 Gbps single-carrier optical circuit between Phoenix and Tucson that was announced at the end of February.
The seven completed packet network links as of March 22, 2021, include: Albany, NY to Hartford, CT; Ashburn, VA to Pittsburgh, PA; Dallas, TX to Kansas City, MO; El Paso, TX to Tucson, AZ; Las Vegas, NV to Salt Lake City, UT; Phoenix, AZ to Tucson, AZ; Reno, NV to Salt Lake City, UT.
Current Deployment Status of 400G packet backbone as of March 22, 2021
400G Intra-city Ethernet Links Technical Deployment
Each intra-city link represented in the map above connects 400Gbps native interfaces on adjacent Cisco 8201 routers via Ciena Waveserver 5 modules; 400 Gbps active optical cables (AOCs) are used to connect the key devices. The Waveserver 5 encapsulates the 400G Ethernet frames within a 600-800 Gbps single-carrier optical circuit which traverses the Ciena 6500 flex-grid optical line system.
An example block diagram of an inter-metro 400G link
On the optical platform, single carrier optical performance between cities has consistently performed above the rates in the original pre-deployment models. For example, 800 Gbps of Ethernet capacity was achieved on the 518 kilometer amp chain between El Paso and Tucson; the projected performance on this segment as originally modeled was 650 Gbps.
A sample of the original model compared to rates delivered by the Waveserver 5 platform
| Link | Pre-deployment Modeled Line Rate | Validated Ethernet Performance | Day 1 Operational Configuration |
Albany – Hartford
549 km | N/A1 | 700 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
Ashburn – Pittsburgh
453 km | 600 Gbps | 800 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
Dallas – Kansas City
928 km | 550 Gbps | 700 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
El Paso – Tucson
518 km | 650 Gbps | 800 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
Phoenix – Tucson
202 km | 700 Gbps | 800 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
Las Vegas – Salt Lake
731 km | 550 Gbps | 700 Gbps | 600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4 |
Reno – Salt Lake
876 km | 550 Gbps | 700 Gbps | 700 Gbps, Ethernet-only
400GE + 100GE + 100GE + 100GE |
1 No model was run for the specified route
As the Internet2 optical system provides 400GE, 100GE, and 10GE services using a uniform transport layer, Mixed OTN-Ethernet mapping is required for the majority of the Waveserver 5 links being deployed. While this does reduce the delivered capacity on a given link, the deployed solution meets NGI 400 Gbps packet design requirements while also providing 10 Gbps and 100 Gbps wave services; additionally, this combined strategy reduces overall power consumption.
Next Steps: Router Configuration, Testing and Connector Integration
The NGI packet platform deployment will proceed with a focus on east to west connectivity at 400 Gbps. The deployment will also include the movement of router configurations from the lab to the future production network platform, as well as the initial use of Cisco Network Services Orchestrator (NSO) for system deployment.
In the coming weeks, the team plans to build a linear prototype path to stress test the baseline backbone hardware and software, culminating in the launch of 400 Gbps of test traffic into the platform.
Example of planned linear prototype path and 400G+ perfSONAR test configuration
Deployment, testing, and system validation will expand to the entire footprint in April, followed shortly by connector integration in May and June.
The above activities are the culmination of thousands of hours of effort from both the Internet2 community and teams in many organizations, including Internet2, GlobalNOC at Indiana University, Ciena, Cisco, Lumen, and GDT.