Scratchpad for scraper development and general utilities.
Project description
cdp-scrapers
Scratchpad for scraper development and general utilities.
Council Data Project
Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable.
For more information about Council Data Project, please visit our website.
About
cdp-scrapers
is a collection of utilities and in-progress or actively maintained
CDP instance event scrapers. The purpose of this library is to help new CDP instance
maintainers have a quick plethora of examples for getting started on developing their
event scraper functions.
Quick Start
Legistar
General Legistar utility functions.
from cdp_scrapers.legistar_utils import get_legistar_events_for_timespan, LegistarScraper
from datetime import datetime
# Get all events (and minutes item and voting details)
# for a provided timespan for a legistar client
# Returns List[Dict]
seattle_legistar_events = get_legistar_events_for_timespan(
client="seattle",
start=datetime(2021, 7, 12),
end=datetime(2021, 7, 14),
)
# Or parse and convert to CDP EventIngestionModel
seattle_scraper = LegistarScraper("seattle")
seattle_cdp_parsed_events = seattle_scraper.get_events(
begin=datetime(2021, 7, 12),
end=datetime(2021, 7, 14),
)
Scrapers
In-progress or completed scrapers.
If you would like to deploy a CDP instance or would like to use this library as a method for retrieving formatted legislative data, please feel free to contribute a new custom municipality scraper!
Custom Scrapers
If it isn't possible to use our generalized Legistar tooling to write your scraper, we welcome the addition of custom scrapers, however please see our documentation on the minimum data required for CDP event ingestion.
From there, begin with our empty custom scraper function template and fill in your scraper.
Installation
Stable Release: pip install cdp-scrapers
Development Head: pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git
Documentation
For full package documentation please visit councildataproject.org/cdp-scrapers.
Development
Refer to CONTRIBUTING.md for information related to developing the code.
MIT license
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdp_scrapers-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a833d716b463343d58d23f1a0cd17843248fa754d0eaaf612a6e8a4dcf7b57e2 |
|
MD5 | 27fadc2dcc72c9e7337452dc143b7b57 |
|
BLAKE2b-256 | 6ca0c67a00a8a70d6b3b811ef9671b769213c32861f1bc996a356b5dbcd3299e |