Skip to main content

Downloads BGPStream info solely for research purposes

Project description

Python 3.10 Python 3.11 Tests

bgpstream_website_collector

Package Description

This package downloads information from bgpstream.com and stores it in a CSV

As a caveat, I wrote this a very long time ago when I was still an undergrad. I've cleaned it up a bit, but this has a long way to go before it's good code quality PRs welcome. Also, almost everything is type ignored because mypy spazzes out with bs4 Also, the tests could really use a good update

Usage

from a script:

from pathlib import Path
from typing import Any

from bgpstream_website_collector import BGPStreamWebsiteCollector


def main():
    # Leave csv_path as None to not write CSV
    csv_path: Path = Path.home() / "Desktop" / "bgpstream_website.csv"
    rows: list[dict[str, Any]] = BGPStreamWebsiteCollector(csv_path=csv_path).run()


if __name__ == "__main__":
    main()

From the command line:

bgpstream_website_collector

Installation

Install python and pip if you have not already.

Then run:

pip3 install pip --upgrade
pip3 install wheel

For production:

pip3 install bgpstream_website_collector

This will install the package and all of it's python dependencies.

If you want to install the project for development:

git clone https://github.com/jfuruness/bgpstream_website_collector.git
cd bgpstream_website_collector
pip3 install -e .[test]
pre-commit install

To test the development package: Testing

Testing

To test the package after installation:

cd bgpstream_website_collector
pytest bgpstream_website_collector
ruff bgpstream_website_collector
black bgpstream_website_collector
mypy bgpstream_website_collector

If you want to run it across multiple environments, and have python 3.10 and 3.11 installed:

cd bgpstream_website_collector
tox

Credits

Huge contributions to original version in lib_bgp_data to the testing from Tony Zheng

History

1.1.2 Fixed event date bug 1.1.1 Fixed start_time -> start_date type 1.1.0 Added option to pass None to csv_path 1.0.0 First major release

Development/Contributing

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Test it
  4. Run tox
  5. Commit your changes: git commit -am 'Add some feature'
  6. Push to the branch: git push origin my-new-feature
  7. Ensure github actions are passing tests
  8. Email me at jfuruness@gmail.com if it's been a while and I haven't seen it

License

BSD License (see license file)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bgpstream_website_collector-1.1.2.tar.gz (13.3 kB view hashes)

Uploaded Source

Built Distribution

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page