ntc-templates 1.3.0

Package to return structured data from the output of network devices.

NTC TEMPLATES

Repository of TextFSM Templates for Network Devices, and Python wrapper for TextFSM's CliTable.

TextFSM is a project built by Google that takes CLI string output and passes each line through a series of regular expressions until it finds a match. The regular expressions use named capture groups to build a text table out of the significant text. The names of the capture groups are used as column headers, and the captured values are stored as rows in the table.

This project provides a large collection of TextFSM Templates (text parsers) for a variety of Networking Vendors. In addition to the templates, there is a function that will convert the CLI output into a CliTable object; the resulting text table is converted into a list of dictionaries mapping the column headers with each row in the table.

Installation and Usage

The project can be installed using either Git or PyPI; if you would like to use the templates outside of this project, then Git is the recommended approach.

Git

$ git clone [e-mail]:networktocode/ntc-templates.git
$ 
# Optional steps to install ntc-templates as a python package
$ pip install -e ntc-templates/
$ 

PyPI

$ pip install ntc_templates
$ 

Usage

>>> from ntc_templates.parse import parse_output
>>> vlan_output = (
        "VLAN Name                             Status    Ports\n"
        "---- -------------------------------- --------- -------------------------------\n"
        "1    default                          active    Gi0/1\n"
        "10   Management                       active    \n"
        "50   VLan50                           active    Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5,\n"
        "                                                Fa0/6, Fa0/7, Fa0/8\n"
    )
>>> vlan_parsed = parse_output(platform="cisco_ios", command="show vlan", data=vlan_output)
>>> vlan_parsed
[
    {
        'vlan_id': '1',
        'name': 'default',
        'status': 'active',
        'interfaces': ['Gi0/1']
    },
    {
        'vlan_id': '10',
        'name': 'Management',
        'status': 'active',
        'interfaces': []
    },
    {
        'vlan_id': '50',
        'name': 'VLan50', 'status': 'active',
        'interfaces': ['Fa0/1', 'Fa0/2', 'Fa0/3', 'Fa0/4', 'Fa0/5', 'Fa0/6', 'Fa0/7', 'Fa0/8']
    }
]
>>> 

Contributing

Pull requests are welcomed and automatically built and tested through TravisCI.

New Templates

To contribute new templates, each new pull request must include the following:

• TextFSM template
• Modified version of the index file
• Tests
  • Raw version of text to be parsed
  • YAML file containing the expected parsed dictionary

TextFSM Template

TextFSM templates should be placed in the ./templates directory and should adhere to the following NTC-Templates style. The TextFSM template name should be in the following format:

Naming

The template should be named using: {{ vendor_os }}_{{ command_with_underscores }}.template

Ex: cisco_ios_show_cdp_neighbors.template

Note: The vendor name must be valid from the os_choices tests, which is primarily based on Netmiko list of supported vendors. New vendors added should adhere to vendor_os.

Ex: vmware_nsx

Values

The capture group names should be in UPPERCASE.

An example of the proper format is shown below.

Value TIME (\d+:\d+:\d+)
Value TIMEZONE (\S+)
Value DAYWEEK (\w+)
Value MONTH (\d+)
Value DAY (\d+)
Value YEAR (\d+)

Start
  ^${TIME}\s+${TIMEZONE}\s+${DAYWEEK}\s+${DAY}/${MONTH}/${YEAR} -> Record
  ^. -> Error

States

If the raw output has a heading, the Start state should match on the column headings and then transition to another state that will match the device's output table with the capture groups. This helps ensure the regex patterns for the capture groups are attempting to match the correct information, and allows templates to easily add additional States for tables that have different headings. Example:

Raw Output

... omitted
Network Next Hop Metric LocPrf Weight Path
*> 111.111.111.111/32 112.112.112.112 4294967295 4294967295 65535 1000 1000 1000 i

Sample Template

Start
# Checking for header
^\s*Network\s+Next(?:\s+|-)[Hh]op\s+Metric\s+LocPrf\s+Weight\s+Path\s*$$ -> BGPTable

BGPTable
 ... omitted

Each state should end with ^. -> Error. This helps to ensure we're accounting for every line within the raw output for the command. This doesn't mean we have to capture all the data as a Value, but we do have to account for it. In addition, it is also good to provide an expression to match blank lines, ^\s*$$

An example would be the following raw output:

NAME: "3640 chassis", DESCR: "3640 chassis"
PID: , VID: 0xFF, SN: FF1045C5

The template would be the following:

Value NAME (.*)
Value DESCRIPTION (.*)

Start
  ^NAME:\s+"${NAME}",\s*DESCR:\s+"${DESCRIPTION}"
  ^PID:\s*,\s*VID:\s*\S+,\s*SN:\s*\S+
  ^\s*$$
  ^. -> Error

Taking a look at the example template above, you notice that we're using \s* and \s+. These are the preferred way to match space characters, and should be used over the literal space character. For example, This\s+is\s+preferred\s*$$ vs This is not preferred$$

• \s* accounts for zero or more spaces (use when the output may or may not have a space between characters)
• \s+ accounts for one or more spaces (use when output will have a space, but could have more than one space)

Index File

The Index file binds the templates to the commands being run. Special care has been taken on ordering, as there is potential for issues. e.g. show ip route picking up for show ip router vrf <vrf-name>. We have used a combination of ordering, as defined:

• OS in alphabetical order
• Template name in length order
• When length is the same, use alphabetical order of command name
• Keep space between OS's

Example:

Template, Hostname, Platform, Command

# same os, same length, used alphabetical order of command name
arista_eos_show_mlag.template, .*, arista_eos, sh[[ow]] ml[[ag]]
arista_eos_show_vlan.template, .*, arista_eos, sh[[ow]] vl[[an]]

# os in alphabetical order and space between cisco_asa and arista_eos
cisco_asa_dir.template,  .*, cisco_asa, dir

# same os, template name length different and space between cisco_asa and cisco_ios
cisco_ios_show_capability_feature_routing.template,  .*, cisco_ios, sh[[ow]] cap[[ability]] f[[eature]] r[[outing]]
cisco_ios_show_interface_transceiver.template, .*, cisco_ios, sh[[ow]] int[[erface]] trans[[ceiver]]
cisco_ios_show_cdp_neighbors_detail.template, .*, cisco_ios, sh[[ow]] c[[dp]] neig[[hbors]] det[[ail]]

Tests

Tests will be located in ./tests with the following hierarchy:

• ./tests/{{ vendor_os }}/{{ command_name }}/

The {{ command_name }} directory should include the .raw file that includes the raw output of the command to be parsed, and the .parsed file of the returned structured data.

$ ls tests/cisco_ios/show_clock/
cisco_ios_show_clock.parsed
cisco_ios_show_clock.raw
$ 

Raw version of input text

The raw text file should contain only the output of the CLI command to be parsed. It should not contain the CLI command itself.

An example of the proper format is shown below:

$ cat tests/cisco_ios/show_clock/cisco_ios_show_clock.raw
*18:57:38.347 UTC Mon Oct 19 2015
$ 

YAML file containing expected parsed dictionary

The parsed file should match the data that is returned from the parse_output function discussed in the beginning. Dictionary keys should be in lowercase.

The parsed text file should be placed in a directory in the ./tests directory with the same name as the template file but replace .template file extension with .parsed. The raw text file and the parsed text file should be in the same directory. ex. ./tests/cisco_ios/show_clock/

There is an available helper that uses Ansible and ntc-ansible custom modules to create the parsed file automatically into the correct format. Helpers are located within ./helpers/.

An example of the proper format is shown below:

$ cat ./tests/cisco_ios/show_clock/cisco_ios_show_clock.parsed
---
parsed_sample:
  - time: "18:57:38.347"
    timezone: "UTC"
    dayweek: "Mon"
    month: "Oct"
    day: "19"
    year: "2015"
$ 

Multiple raw and parsed files are supported per directory, and are encouraged, as there are differences depending on version, length, etc. Additional test files and more real life data helps ensure backwards compatibility is maintained as each template is updated and merged into the repo.

Updating/Fixing Existing Templates

When either fixing a bug within a template or adding additional Values to be captured, additional test files should be added to ensure backwards compatibility and that the new data is being parsed correctly.

To add additional raw/parsed tests for a command:

• Navigate to ./tests/{{ vendor_os }}/{{ command_name }}/
• Create new .raw and .parsed files within the directory, preferrably with a name identifying why the data is unique:
  • Existing raw: ./tests/cisco_ios/show_version/cisco_ios_show_version.raw
  • New raw: ./tests/cisco_ios/show_version/cisco_ios_show_version_stack_platforms.raw
  • Existing parsed: ./tests/cisco_ios/show_version/cisco_ios_show_version.parsed
  • New parsed: ./tests/cisco_ios/show_version/cisco_ios_show_version_stack_platforms.parsed

Testing

You can test your changes locally within your Git branch before submitting a PR. If you do not have tox already installed, you can do that using pip or your systems package manager. Tox should be ran inside the ntc-templates root directory. The tox file is configured to run against python3.6, so either python3.6 needs to be available, or the tox.ini file will need to be updated with an available Python version.

$ tox
GLOB sdist-make: /home/admin/ntc-templates/setup.py
py36 inst-nodeps: /home/admin/ntc-templates/.tox/dist/ntc_templates-1.1.0.zip
py36 installed: atomicwrites==1.3.0,attrs==19.1.0,importlib-metadata==0.18,more-itertools==7.1.0,ntc-templates==1.1.0,packaging==19.0,pkg-resources==0.0.0,pluggy==0.12.0,py==1.8.0,pyparsing==2.4.0,pytest==5.0.1,PyYAML==5.1.1,six==1.12.0,terminal==0.4.0,textfsm==0.4.1,wcwidth==0.1.7,zipp==0.5.2
py36 runtests: PYTHONHASHSEED='1913863515'
py36 runtests: commands[0] | pytest
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/admin/ntc-templates
collected 364 items

tests/test_index_order.py .                                                                                                                                                  [  0%]
tests/test_structured_data_against_parsed_reference_files.py ............................................................................................................... [ 30%]
............................................................................................................................................................................ [ 78%]
...............................................................................                                                                                              [ 99%]
tests/test_testcases_exists.py .                                                                                                                                             [100%]

=========================================================================== 364 passed in 15.69 seconds ============================================================================
_____________________________________________________________________________________ summary ______________________________________________________________________________________
  py36: commands succeeded
  congratulations :)

Questions

For any questions or comments, please feel free to swing by the networktocode slack channel.

Sign up here

[1.3.0]

Fixed for any bug fixes

• [#401] ASA show route: Fix UPTIME to account for additional output formats
• [#445] IOS show ip eigrp topology: Fix FD to allow value to be "Inaccessible"
• [#465] ASA show failover: Add line to match FirePOWER module
• [#466] PAN show arp: Fix MAC capture group to account for entries that are "incomplete"
• [#471] Procurve show arp: Fix IP capture group to account for IP Adresses that are "Incomplete"
• [#471] Procurve show arp: Fix PORT capture group to account for interfaces that contain more than just digits
• [#474] IOS show ip mroute: Fix Value UP_TIME, EXPIRATION_TIME, OUTGOING_MULTICAST_UP_TIME (\S+), and OUTGOING_MULTICAST_EXPIRATION_TIME to account for additional time formats
• [#479] NXOS show interface brief: Fix capturing MODE values of "pvlan" and "fabric"
• [#480] IOS show cdp neighbors: Allow for table data to have leading spaces but not require it
• [#485] IOS show mac-address-table: Allow for table data to have leading spaces but not require it (VLAN IDs are left adjusted so VLANs from 1000 on do not have a leading space)
• [#487] IOS show authentication sessions: Add match for lines separating output with a "-" in order to avoid capturing them as part of normal table data
• [#488] EOS show interfaces status: Fix NAME and STATUS capture groups to support disabled interfaces
• [#494] IOS show ip interface: Add matches for additional output lines
• [#495] IOS show ip interface: Add matches for additional output lines
• [#497] ASA show version: Fix MAX_INTF capture group to account for Unlimited
• [#497] ASA show version: Fix HARDWARE capture group to account for trailing commas
• [#499] ASA show inventory: Add matches for additional output lines
• [#503] IOS show standby: Fix output that spread across two lines
• [#505] NXOS show ip bgp neighbors: Fix matching entries accurately when neighbors are in a "down" state; move to Continue.Record to record on new entries instead of "last expected line" of output data
• [#516] IOS show ip interface: Add matches for additional output lines
• [#517] EOS show ip bgp summary: Fix capturing entries with more explicit match line
• [#519] XR show version - Fix VERSION, UPTIME, and HARDWARE to account for CRS syntax
• [#525] IOS: Added matches for vty timestamp lines
• [#532] IOS show snmp user: Fix capture groups to support all non-whitespace characters (e.g. "-" in string values)
• [#538] IOS show interfaces switchport: Added matches for additional output lines

Added for new features

• [#378] XR show ip route: Added VRF capture group
• [#391] WLC show client detail: Added new template
• [#419] ASA dir: Added TOTAL_PERCENT_FREE capture group for capturing percentage of bytes free
• [#425] WLC show exclusionlist: Added new template
• [#445] IOS show ip eigrp topology: Added SOURCE capture group
• [#446] ASA show asp drop: Added FLOW_DROP_IPSEC_SELECTOR_FAILURE, FLOW_LAST_CLEARED, MP_SVC_BAD_LENGTH, and SSL_FIRST_RECORD_INVALID capture groups
• [#464] Comware display counters bound interface: Added new template
• [#468] Firebox show arp: Added new template
• [#476] IOS show ip bgp summary: Added UP_DOWN capture group
• [#478] NXOS show ip interface brief: Added new template
• [#481] NXOS show ip bgp neighbor: Added INBOUND_ROUTEMAP and OUTBOUND_ROUTEMAP capture groups
• [#482] XR admin show inventory: Added new template
• [#486] NXOS show route-map: Added new template
• [#489] NXOS show forwarding ipv4 route: Added new template
• [#491] IOS show switch detail: Added new template
• [#492] XR show ipv6 neighbors: Added new template
• [#493] Fastiron show arp: Added new template
• [#496] EOS show ip helper-address: Added new template
• [#501] WLC show 802.11a|b: Added new template
• [#510] WLC show 802.11 cleanair config: Added new template
• [#512] VRP display interface brief: Added new template
• [#512] VRP display temperature: Added new template
• [#513] WLC show rf profile-summary: Added new template
• [#514] IOS show process memory sorted: Added new template
• [#515] WLC show inventory: Added new template
• [#518] Ciena software show: Added new template
• [#521] IOS show ip route: Added VRF capture group
• [#524] EOS show vrf: Added new template
• [#526] WLC show advanced 802.11 channel: Added new template
• [#527] IOS show interfaces switchport: Added VOICE_VLAN capture group
• [#527] NXOS show interfaces switchport: Added VOICE_VLAN capture group
• [#529] Fortios get router info bgp summary: Added new template
• [#531] NXOS show interface: Added LAST_LINK_FLAPPED capture group
• [#533] IOS show license: Added new template
• [#539] WLC show interface summary: Added new template
• [#540] IOS show environment temperature: Added new template

Changed for changes in existing functionality

• [#378] XR show ip route: Update index to allow syntax of "show ip route" or "show route"
• [#497] ASA show version: Change SERIAL to be a list for clusters
• [#503] IOS show standby: Update index to allow syntax including specifying an interface
• [#520] IOS show ip bgp summary: Update template and index to support VRF syntax
• [#520] EOS show ip bgp summary: Update template and index to support VRF syntax
• [#521] IOS show ip route: Add Error for unmatched lines to ensure accurate data collection
• [#522] EOS show bgp summary: Update index to allow syntax of "show bgp evpn summary"
• [#523] IOS show mac-address-table: Add Error for unmatched lines to ensure accurate data collection
• [#528] IOS show interfaces switchport: Update template to use Continue.Record syntax to record on new entries instead of "last expected line" of output data
• [#528] NXOS show interfaces switchport: Update template to use Continue.Record syntax to record on new entries instead of "last expected line" of output data

Deprecated for soon-to-be removed features

Removed for now removed features

[1.2.1]

Fixed

• Bumped __version__ in __init__ file

Added

Changed

Deprecated

Removed

[1.2.0]

Fixed

• [#389] IOS-XR show interfaces: Fixed matching speed and duplex for bundle-ethernet interfaces
• [#442] IOS-XR show version: Fixed matching build_host field that uses both "Build" and "Built" keywords
• [#455][#456] Template files were inaccessable when installing from local directory; now works when using pip -e

Added

• [#470] ASA show version: Add capturing compiled_date field