This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

IP Block Lists Aggregator

A Python tool that downloads IP block lists from various sources and builds configurations for network equipments and firewalls.

Installation

Installation using pip:

$ pip install blocklistsaggregator

Editable Installation using your GitHub forked repository and virtualenv:

$ mkdir blocklistsaggregator
$ cd blocklistsaggregator
$ virtualenv venv
$ source venv/bin/activate
$ pip install -e git+https://github.com/YOUR_USERNAME/blocklistsaggregator.git#egg=blocklistsaggregator

Usage

It’s a command line tool, the --help is your friend! Some examples are worth a thousand words.

  • Download and display entries from all the configured block lists:

    $ blocklistsaggregator.py
    
  • Only from Ransomware Tracker RW_IPBL and DROP:

    $ blocklistsaggregator.py --lists rw_ipbl drop
    
  • Download entries from all the configured lists and save them in JSON format into all.json:

    $ blocklistsaggregator.py -f json -o all.json
    
  • Read the previously saved entries from all.json and display them in a Cisco IOS prefix-list style:

    $ blocklistsaggregator.py -i all.json -f cisco-ios
    
  • From the previously saved entries, filter out those falling in 6.0.0.0/8 and those with a prefix-len shorter than /24 and save them into cisco.acl in a Cisco ACL style with name BADGUYS:

    $ blocklistsaggregator.py -i all.json --exclude 6.0.0.0/8 --exclude-ipv4-shorter-than 24 -o cisco.acl -f cisco-ios --cisco-cfg-element acl_source --cisco-cfg-element-name BADGUYS
    
  • Prepare an ip route <network> <mask> null0 command for each IPv4 entry in DROP:

    $ blocklistsaggregator.py --lists drop -4 --lines-format "ip route {network} {netmask} null0"
    
  • Download standard block lists and output them in a Mikrotik address-list format into addMalwareIPs.rsc; save lists into /tmp and, in case of failure during one of the next executions, reuse them to build the output:

    $ blocklistsaggregator --output addMalwareIPs.rsc --output-format mikrotik --mikrotik-address-list-name addressListMalware --lists-storage-dir /tmp/ --recover-from-file
    

Logging

Error logging and reporting can be configured in order to have feedback about BlockListsAggregator’s activity. The --logging-config-file option can be set to the path of a configuration file in Python’s logging.fileConfig() format. An example is provided within the distrib/log.ini file (here the file hosted on GitHub).

Source block lists

The following block lists are currenly implemented:

Warning for RW_DOMBL and RW_URLBL: the program extracts the domain names reported into these lists to resolve the IP addresses and uses them for the output. This may result in an overblocking behaviour because these filters should be applied with a more granular level than layer-3 addresses. These lists are not used by default unless explicitly given via the command line –lists or –lists-include arguments.

A list of block-lists can be found on http://iplists.firehol.org/

Output options

The following output formats are currenly implemented:

  • JSON
  • lines (with macros)
  • Cisco IOS prefix-list
  • Cisco IOS ACL (source-based, destination-based, permit/deny actions)
  • Mikrotik RouterOS address-list

Status

This tool is currently in beta: some field tests have been done but it needs to be tested deeply and on more scenarios.

Moreover, contributions (fixes to code and to grammatical errors, typos, new features) are very much appreciated.

Bug? Issues?

But also suggestions? New ideas?

Please create an issue on GitHub at https://github.com/pierky/blocklistsaggregator/issues

Author

Pier Carlo Chiodi - https://pierky.com

Blog: https://blog.pierky.com Twitter: @pierky

Changelog

0.5.0

  • Better empty lines detection for RW_IPBL.

  • Add –lists-include and –lists-exclude arguments.

  • Add rw_dombl and rw_urlbl lists (Ransomware Tracker RW_DOMBL and RW_URLBL).

    Warning: the program extracts the domain names reported into these lists to resolve the IP addresses and uses them for the output. This may result in an overblocking behaviour because these filters should be applied with a more granular level than layer-3 addresses. These lists are not used by default unless explicitly given via the command line –lists or –lists-include arguments.

0.4.1

  • Fix issue with RW_IPBL entries counter.

    It seems that RW_IPBL is having some issues with the number of entries reported in the last line. If an empty line is found it’s counted as an entry, so last line’s counter reports a wrong number. Trying to mitigate this behaviour.

0.4.0

0.3.0

  • Add –lists-storage-dir and –recover-from-file arguments to save lists into files and reuse them in case of failure of next updates.

0.2.0

Please note: JSON files saved with the previous version are not compatible with this one; blocklists must be downloaded and saved again to work.

  • Keep track of source blocklist for each entry.
  • Add bl_ids and bl_names macros to the lines formatter.
  • Add a comment containing the source blocklist to each Mikrotik RouterOS address-list entry.

0.1.0

First release (beta)

Release History

Release History

0.5.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.4.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.4.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
blocklistsaggregator-0.5.0.tar.gz (29.0 kB) Copy SHA256 Checksum SHA256 Source Aug 4, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting