I’ve been at GlobalNOC since 2002, and our automation journey started before I arrived! One of my early tasks as GlobalNOC’s first full-time software developer was to maintain a set of scripts written by some of our network engineers that automated the configuration of our network monitoring systems based on scraping configuration of the routers we were supporting.
More recently, as part of our GlobalNOC Renewal Program that is designed to help set up GlobalNOC for growth and success for the next 20 years, we focused on network configuration automation as one of the foundational pillars of the program. Our goal was to work with our network owner partners to build and deploy systems to automate the day-to-day configuration of the network devices we support on their behalf.
What are some of the biggest wins GlobalNOC has had in the network automation space and how are those successes benefiting the community you serve?
Luke Fowler: When we embarked on our network configuration automation effort, we already had a good base established to build on. We had a source of truth that we had been developing and enhancing for many years: GlobalNOC Database. We also had been using open-source automation tools like Ansible to manage our network management servers for a long time.
Around the same time, the network engineering community began to pick up these open tools such as Ansible and git/Github – already very popular in the systems administration community – and apply them to network configuration automation.
While we did evaluate commercial solutions, we ended up taking what we already had developed and our experience with these open-source systems, and we built some “glue” applications around them to enable our engineering teams to begin automating configuration of the devices they supported. Today, we have two automation applications – built on top of Ansible/AWX, git/Github, and GlobalNOC Database.
GlobalNOC Network Automation Tool (GNAT) is an application that allows networks we support to get started with automation. Often they will first use it to baseline their configurations, making sure that all the pieces of configuration that should be the same across all network devices (of each class) in their network are in sync and stay consistent over time.
GlobalNOC Service Configuration System (GSCS) is the other tool we developed, used for more sophisticated automation. This tool allows network engineers to automate deployment and maintenance of configuration at a service level, across whatever set of devices need to be touched to deploy and maintain a particular instance of a network service.
Think of GNAT as a way to keep the things that should be the same across the network the same everywhere and GSCS as a way to holistically manage each individual network service deployed on the network – across multiple network devices – from one tool.
What are two of the hardest lessons GlobalNOC has learned about network automation along the way?
Luke Fowler: I think the biggest lesson we learned – and a value that we try to instill in our culture across GlobalNOC – was “progress over perfection.” Many of the networks we support are large and complex, with hundreds of devices and many, many thousands of lines of configuration. None of the deployments that we were working to automate were greenfield networks. They were all networks with a long configuration history, and we had to keep them running well while we brought automation into the picture. We adopted a “just get started and automate more and more each week, month, and year” strategy in our efforts. This is one of the main reasons why we use the two different tools (GNAT and GSCS), depending on where each network we support is in their automation journey.
We also learned that we needed to be flexible and willing to change and adapt over time. We couldn’t possibly define all the requirements for our automation environments at the beginning. We’re big fans of agile-like development practices and tried to bring this ethos to our automation projects. While we knew it was important to have a solid architecture to build on, it was also critical for us to get going quickly with something simple and learn over time how to evolve our tools and processes to be more powerful as the demands on them grew.
We also had to clearly communicate this to our stakeholders. Some had no existing experience with network automation, and the idea of fully automating their network configurations seemed like a huge task. It was important to communicate how we had been successful with incremental change in other areas and how we could use the same model to be successful as we worked through automating their network device configurations. By being able to show small and consistent progress frequently, our users became more comfortable with automation and confident in our ability to achieve the end goal.
What’s next for GlobalNOC in the network automation space?
Luke Fowler: I mentioned our source of truth – GlobalNOC Database. We’re currently in the middle of a project to totally redevelop the user interface using a modern technology stack based on best-of-breed user interface toolkits, including the React component library. This will provide a streamlined, speedy user interface for the engineers and operations technicians that use and maintain our data. We have a preview release deployed on every instance of GlobalNOC Database we operate. Every week we are deploying new functionality and collecting feedback from our users – and doing it all again the next week. Our goal is to have this new UI completed around the end of this summer.
We also are continuing to make progress on fully automating the configuration of each network we’re supporting automation for. We’re taking it step by step until we’re able to get everything automated. We’re also using these automation projects to drive the feedback loop for our GNAT and GSCS tools. As we automate more, we learn more about how to automate, and then we integrate what we learn into our tools. Over time, we hope that as our networks become more automated, the tools we use to automate them also will become more capable, powerful, and easy to use – accelerating new automation projects as time goes on.
What advice would you give to peers across the R&E community on how to embrace network automation at their organizations?
Luke Fowler: The best advice I can give is to just get started somewhere. You don’t have to have everything figured out to begin your automation journey and to begin reaping benefits from automation.
Every environment has different challenges, different pieces of low-hanging fruit, and different existing capabilities. Seek out your low-hanging fruit and look at how you can build on what tools, scripts, or skills you already have to introduce a little bit of automation. That little bit will start the snowball rolling down the hill and will give you ideas on how to automate more and more and more over time.
Read the other Q&As in the “Embracing Network Automation” series.
ICYMI