YAML
Jinja2
Python
Netmiko
All of these came together to help solve a problem for me. What inspired me to create this mess of code?
I had a job to update the configuration and firmware on over 250 VyOS routers. Coming soon is another job to build 40+ switch stacks (on about 180 switches) so I know that I’m going to be using this again.
In the past I’ve done configuration by hand and it has worked for a long time. Most of the time text configurations don’t change too much, I take a technically working configuration and modify it for a new job and then slap on whatever security template I think makes sense.
I’ve become proficient at scripting within Excel and I can generate single-shot configurations that follow strict rules, it’s helpful but not really practical for big jobs. I needed a tool that could pump out configurations quickly.
Excel Tricks
This post isn’t supposed to be about Excel tricks, but I figured I throw in the few things I’ve used in the past. It might help.
- Concatenate – merge a string with a excel value:
=CONCATENATE(“interface “, E1) - Vlookup – search a column of data for a match.
=VLOOKUP(lookup_value, lookup_range, column, match_type) - Index/Match – search any data for a match. This is far faster and more flexible than vlookup.
=INDEX(range, MATCH(lookup_value, lookup_range, match_type))
Once you’ve got your config output into one line per-row, then copy all that into a text file and swap the TABs with spaces and you’re done.
You should also watch You Suck at Excel by Joel Spolsky.
YAML
YAML Ain’t Markup Language is a human friendly data serialization standard for all programming languages. It’s basically a data storage system that you can read, but is easily parsed by Python into a data structure like a dictionary.
The hierarchy in the file is defined by indents which makes it easy to see the relationships between values.
Here’s a quick example of a YAML file:
SERVER: 1.1.1.1
OUTSIDE_NET: 2.2.2.0/22
INSIDE_NET: 192.168.0.0/24
PORTGROUP:
- name: RTP
description: SIP-RTP
port: 10000-65535
- name: SIP
description: SIP
port: 5060
- name: RTSP
description: RTSP
port: 10000-65535
POLICIES:
- Client-LAN:
- rule: 1
action: accept
description: SIP
destAdd: SERVER
destGroup: SIP
log: disable
protocol: tcp_udp
- rule: 2
action: accept
description: SIP-RTP
destAdd: OUTSIDE_NET
destGroup: RTP
log: disable
protocol: udp
You can see that even just looking at this, you can get a sense for what the data represents.
Jinja2
This Python module programmatically generates text blocks, from within a text file. It is like a programming language in a config file – so you can build up logical structures in a template file that gets interpreted and spits out a sensible configuration.
Jinja2 Text Blocks
Jinja2 lets you create blocks of text and silently insert programming control structures within that text.
Jinja2 control structures are marked using {% stuff %} and variables are accessed using {{ variable_name }}.
Use the hyphen like this {%- stuff %} so your jinja2 commands don’t mess up your line breaks or white space.
Jinja2 For/endfor
There’s other loop types, but this is all I used. If you need different tools then you can dig into the documentation.
Here’s an example:
firewall {
group {
{%- for PG in PORTGROUP %}
port-group name {
description "give me all your cash”
port 12345
}
{%- endfor %}
}
So that section after the “firewall” statement could potentially repeat many times. In the context of this VyOS configuration it is building each firewall rule one at a time.
Indentation doesn’t matter for jinja2, but you can and probably should use it to keep your code readable.
Jinja2 If/else/elif/endif
{%- if ITEM.destAdd == "SERVER" %}
address 10.10.10.10
{%- elif ITEM.destAdd %}
address 30.30.30.30
{%- endif %}
Jinja2 Variables
firewall {
group {
{%- for PG in PORTGROUP %}
port-group {{ PG.name }} {
description "{{ PG.description }}"
port {{ PG.port }}
}
{%- endfor %}
}
Here the variable PORTGROUP references the YAML definition PORTGROUP. The for loop cycles through the upper level entries of each group stored in the temporary variable (jinja2 only) PG.
The variable PG contains the data for ONE entry from YAML, and its own data is accessed with dot notation also taken from YAML.
Python
It is a bit of an understatement to say that Python is very popular. But it is and there are many code examples on the Internet. Many modules are available and can be easily installed with ‘pip’ while others can be downloaded from github.
Python Import objects
from jinja2 import Environment, FileSystemLoader
import yaml
import csv
from netmiko import ConnectHandler, SCPConn
from netmiko.vyos import VyOSSSH
import os
import scp
import subprocess
This pulls modules into play, allowing your script to access the objects and code within them.
Python Load data structures
with open("config.yaml") as _:
config = yaml.load(_)
offices = csv.DictReader(open("offices.csv"))
This loads the YAML file into memory, and interprets it into a dictionary called ‘config’. I also imported the office room numbers/IP addresses into Python too. These two data sources are referenced by Jinja2 to build the configurations themselves.
Python Loops
offices = csv.DictReader(open("offices.csv"))
for row in offices:
s_file = row['IPA'] + '-config.boot'
d_file = '/config/' + s_file
There are multiple loop types, but I only used the for loop in my script.
Python Error handling
for row in offices:
response = os.system("ping -c 1" + row['IPA'])
if response == 0:
print "Success, code goes here"
else:
print row['Office'], ' is down!'
I used the ‘os.system’ library to make sure I can ping a router before I attempt to connect to it and load a configuration file.
Python File writing and transfer
with open("config.yaml") as _:
yamlfile = yaml.load(_)
for row in offices:
s_file = row['IPA'] + '-config.boot'
d_file = '/config/' + s_file
configfile = open(s_file, 'w')
rowdict['Office'] = row['Office']
rowdict['IPA'] = row['IPA']
rowdict['IPB'] = row['IPB']
yamlfile.update(rowdict)
template = ENV.get_template("jinja2.text")
configfile.write(template.render(yamlfile))
configfile.close()
Once I generated the configuration, I wanted to save it to a unique file for future reference.
Netmiko
Netmiko was based on paramiko (a ssh library for Python) but it simplifies using SSH with Python. It creates an object that is referenced by ssh functions within your program and allows your script to be less vendor specific because you don’t have to deal with the madness of screen scraping to get the job done.
Netmiko object
Before you can do anything with netmiko, you have to create the object:
from netmiko.vyos import VyOSSSH
for row in offices:
rtr = {
'device_type': 'vyos',
'ip': row['IPA'],
'username': 'admin',
'password': 'strong-password,
}
In my example, as the loop iterates through the list of offices it builds an object that netmiko can use for SSH or SCP.
Netmiko SCP
for row in offices:
s_file = row['IPA'] + '-config.boot'
d_file = '/config/' + s_file
ssh_conn = ConnectHandler(**rtr)
scp_conn = SCPConn(ssh_conn)
scp_conn.scp_transfer_file(s_file, d_file)
scp_conn.close()
}
First Netmiko transfers the configuration file to the router.
Netmiko SSH
for row in offices:
s_file = row['IPA'] + '-config.boot'
d_file = '/config/' + s_file
ssh_conn = ConnectHandler(**rtr)
vsum = ssh_conn.send_command(
'md5sum config.boot').split()
hsum = subprocess.check_output(
['md5sum', s_file]).split()
if vsum[0] == hsum[0]:
print row['IPA'] + " success!
else:
print "Configuration transfer failed"
}
Then it uses SSH to log into the router to do a MD5 hash on the file to make sure it transferred correctly.
Useful Links
wozneyenterprises.wordpress.com 