Backing Up Cisco IOS/XE Configurations With The Archive Command And Azure Functions

Cisco IOS/XE has for a very long time now had the archive command which allows the configuration of the device to be sent off to a remote system for backup. This is an awesome feature but it hasn’t seen many improvements over the years and it’s quite rare that I’ve seen it in production.

Many times other tools like RANCID or Oxidized are used to pull (rather than push) configs into a version control system or some other repository. These do work well, however, there is a key disadvantage I mentioned, they pull the config from the device, rather than the device initiating the connection and pushing it out.

In many environments this won’t be an issue, but for IT Services companies or MSPs you’re not likely to have direct access deep into customer networks where you can pull switch configs from a single location. Routers, firewalls and other internet facing devices would be doable with a static IP and ACL, but anything past the NAT boundary would be painful - you would be relying on some agent inside the network. This is where the archive feature shines and becomes incredibly useful.

The following is an exploration into some considerations and findings while planning to build a small system to accept configs hosted in Azure (App Service or Azure Functions).

Cisco archive command

The archive command supports a number of protocols for copying the config to another system, and as we’re wanting to ship it offsite there are a few protocol features to look for. Firstly, it should support encryption, and secondly it should be something relatively easy to set up.

Here is what is supported on IOS XE:

starburst(config-archive)#path ?
  bootflash:  Write archive on bootflash: file system
  flash:      Write archive on flash: file system
  ftp:        Write archive on ftp: file system
  http:       Write archive on http: file system
  https:      Write archive on https: file system
  rcp:        Write archive on rcp: file system
  scp:        Write archive on scp: file system
  sftp:       Write archive on sftp: file system
  tftp:       Write archive on tftp: file system

HTTPS fits the bill, it is ubiquitous and provides encryption during transport. I did look at SFTP but Azure charges over $200 per month just to have SFTP enabled on a storage account, and I have no desire to run my own, so that, and all of the others are out.

Let’s explore further.

Cisco archive with HTTPS

There were no docs i could find on how the archive command works with HTTPS, and very little in the way of debug commands, so a bit of experimentation was done with two key takeaways.

Firstly, it makes a PUT request, and secondly, it must trust the certificate on the site accepting the data.

Adding CAs to the Trusted Root

Azure uses DigiCert for their *.azurewebsites.net namespace, and it was just a matter of installing the “Global Root G2” certificate on the router to allow to it successfully connect to the function.

I did not need to install the intermediate certificate, which is great as the root certificate expires in 2038.

starburst(config)# crypto pki trustpoint DigiCertGlobalRootG2
starburst(ca-trustpoint)# enrollment terminal
starburst(ca-trustpoint)# exit

starburst(config)#crypto pki authenticate DigiCertGlobalRootG2

Enter the base 64 encoded CA certificate.
End with a blank line or the word "quit" on a line by itself

-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIQAzrx5qcRqaC7KGSxHQn65TANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBH
MjAeFw0xMzA4MDExMjAwMDBaFw0zODAxMTUxMjAwMDBaMGExCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IEcyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuzfNNNx7a8myaJCtSnX/RrohCgiN9RlUyfuI
2/Ou8jqJkTx65qsGGmvPrC3oXgkkRLpimn7Wo6h+4FR1IAWsULecYxpsMNzaHxmx
1x7e/dfgy5SDN67sH0NO3Xss0r0upS/kqbitOtSZpLYl6ZtrAGCSYP9PIUkY92eQ
q2EGnI/yuum06ZIya7XzV+hdG82MHauVBJVJ8zUtluNJbd134/tJS7SsVQepj5Wz
tCO7TG1F8PapspUwtP1MVYwnSlcUfIKdzXOS0xZKBgyMUNGPHgm+F6HmIcr9g+UQ
vIOlCsRnKPZzFBQ9RnbDhxSJITRNrw9FDKZJobq7nMWxM4MphQIDAQABo0IwQDAP
BgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUTiJUIBiV
5uNu5g/6+rkS7QYXjzkwDQYJKoZIhvcNAQELBQADggEBAGBnKJRvDkhj6zHd6mcY
1Yl9PMWLSn/pvtsrF9+wX3N3KjITOYFnQoQj8kVnNeyIv/iPsGEMNKSuIEyExtv4
NeF22d+mQrvHRAiGfzZ0JFrabA0UWTW98kndth/Jsw1HKj2ZL7tcu7XUIOGZX1NG
Fdtom/DzMNU+MeKNhJ7jitralj41E6Vf8PlwUHBHQRFXGU7Aj64GxJUTFy8bJZ91
8rGOmaFvE7FBcf6IKshPECBV1/MUReXgRPTqh5Uykw7+U0b6LJ3/iyK5S9kJRaTe
pLiaWN0bfVKfjllDiIGknibVb63dDcY3fe0Dkhvld1927jyNxF1WW6LZZm6zNTfl
MrY=
-----END CERTIFICATE-----

Certificate has the following attributes:
       Fingerprint MD5: E4A68AC8 54AC5242 460AFD72 481B2A44
      Fingerprint SHA1: DF3C24F9 BFD66676 1B268073 FE06D1CC 8D4F82A4

% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.
% Certificate successfully imported

Authentication with Cisco archive HTTPS method

There is no support for API Keys or any HTTP headers with the Cisco archive feature. Azure functions do provide authentication options via either a header or query string, but neither work well here. App Service also provides built in authentication but again the limitations of the archive command are the issue.

We can’t add headers to the request, and adding a query string is downright painful as it requires some escaping using the ASCII Synchronous Idle control character due to the ? being a special input character with IOS/XE. I want to be able to easily copy/paste configs and this did not play well, so query strings are also out. Another option is Basic auth by adding credentials to the archive config:

starburst(config)# archive
starburst(config-archive)# path https://username:password@ciscobackups.azurewebsites.net/api/config/...

But this requires Azure functions / App service to support Basic auth, which they don’t. You could implement this yourself using a reverse proxy, but again, I don’t want to have to manage anything.

The simplest way I saw around these limitations was to include an “api key” in the path and doing the authentication in the app - something along the lines of [Function("Config/{authKey}/{groupId}")]. The authKey is the “api key” for which we will need to implement our own authentication, and the groupId is a string under which we will group configurations of several devices - this may be a site name, or customer name, etc.

This means the Azure function which receives the config will be “public” (AuthorizationLevel.Anonymous), and for this reason I don’t recommend using the ‘free’ consumption plan. If you’re building something similar run it under an App Service Plan and not a consumption model.

DotNet and Kestrel with Expect: 100-continue

When the Cisco devices makes the PUT request it will first send a Expect: 100-continue header, to which the server will need to reply before the Cisco device sends the config. Here is what that first request looks like:

PUT /Config/{authKey}/xkln/starburst HTTP/1.1
User-Agent: cisco-IOS
Host: 10.250.1.99:5171
Expect: 100-continue
Connection: Keep-Alive
Content-Type: application/octet-stream; charset=utf-8

Our server then needs to reply with HTTP/1.1 100 Continue, at which point the device will send the config.

I made the mistake of trying to do things step by step and attempted to work out how to manually send this response without terminating the connection (as HTTP is stateless) before dealing with the config.

This was a waste of time, Kestrel will automatically send this response on your behalf, as long as you attempt to read the request body.

// Read config from request body stream
StreamReader reader = new(req.Request.Body, Encoding.ASCII);
string config = await reader.ReadToEndAsync();

Redacting sensitive content from Cisco configs

Cisco configurations can have all kinds of sensitive details in them: crypto PSKs, RADIUS keys, PPPoE CHAP/PAP creds, etc.

Here’s a C# class for redacting those fields. I’d recommend testing this with your own configs in case anything has been missed.

public static class Redactor
{
    internal static string RedactSensitiveStrings(string input)
    {
        // Ensure input string has data
        if (string.IsNullOrEmpty(input)) throw new ArgumentNullException("input");

        // Passwords, secrets, and keys
        string secrets = "(?<=.*(secret|password|key|key-string) [0-9] ).*";
        string redacted = Regex.Replace(input, secrets, "<redacted>", RegexOptions.IgnoreCase);

        // Crypto Key
        string plaintext = "(?<=crypto isakmp key ).*(?= address)";
        redacted = Regex.Replace(redacted, plaintext, "<redacted>", RegexOptions.IgnoreCase);

        // SNMP RW Strings
        string snmp = "(?<=snmp-server community ).*(?= RW)";
        redacted = Regex.Replace(redacted, snmp, "<redacted>", RegexOptions.IgnoreCase);

        return redacted;
    }
}

Cleaning up the config to remove inconsequential changes

If the configuration is to be stored in a version control system you want to ensure only consequential changes are written. Simply commiting an IOS XE confing with wr mem or copy run start with no changes will update the Last configuration change and NVRAM config last updated strings, and will show as a change in the version control system.

Here’s a C# class to clean up the config and remove some unnecessary lines that frequently cause unwanted diffs.

public static class Cleaner
{
    internal static string Clean(string input)
    {
        // Ensure input string has data
        if (string.IsNullOrEmpty(input)) throw new ArgumentNullException("input");

        // Remove headers that update on every config save
        string pattern = "! Last configuration change.*\n";
        string cleaned = Regex.Replace(input, pattern, "", RegexOptions.IgnoreCase);

        pattern = "! NVRAM config last updated.*\n";
        cleaned = Regex.Replace(cleaned, pattern, "", RegexOptions.IgnoreCase);

        // Remove Empty Lines
        pattern = "^[ |\\t|\\s]*\n";
        cleaned = Regex.Replace(cleaned, pattern, "", RegexOptions.IgnoreCase);

        return cleaned;
    }
}

That’s about it for now. This was written while exploring how I would eventually build the system so only snippets of code have been provided. At some point in the future the whole project may be available on GitHub, but for the time being it’s sitting in the unfinished hobby project pile 🙃.