Skip to main content

This project attempts to map the organization of the US Federal Government by gathering and consolidating information from various directories.

Project description

Overview

This project attempts to map the organization of the US Federal Government by gathering and consolidating information from various directories.

PyPI License PyPI Version PyPI Downloads

Current sources:

Each source is scraped (see out directory) in raw JSON format, including fields for the organizational unit name/parent (if any), unique ID/parent-ID fields (if the names are not unique) as well as any other attribute data for that organization available from that source.

A normalized name (still WIP) is then added, which corrects letter case, spacing and expands acronyms. Acronyms are selected and verified manually using data from USCD GovSpeak and the DOD Dictionary of Military and Associated Terms as well as manual entry when needed.

Each source is them imported into a tree and exported into the following formats for easy consumption:

  • Plain text tree
  • JSON flat format (with path to each element)
  • JSON nested tree format
  • CSV format (with embedded JSON attributes)
  • Wide CSV format (with flattened attributes)
  • DOT file (does not include attributes)
  • GEXF graph file (includes flattened attributes)
  • GraphQL graph file (includes flattened attributes)
  • Cytoscape.js JSON format (includes flattened attributes)

To merge the lists, each tree is merged into a selected base tree by comparing the normalized names of each node in the tree to the names of each node in the base tree using a fuzzy matching algorithm. Similarity scores between each pair of parents are incorporated into the score to more correctly identify cases where the same/similar office or program name is used for different organizations.

Note that the fuzzy matching is imperfect and may have some inaccurate mappings (although most appear OK) and will certainly have some entries which actually should be merged, but aren't.

The final merged dataset is written in the above formats to the data/merged directory.

Setup

Requirements

Installation

Check out this repository, then from the repository root, install dependencies:

$ poetry install

See command line usage:

poetry run allusgov --help

Run a complete scrape and merge:

poetry run allusgov

Project details


Release history Release notifications | RSS feed

This version

0.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

allusgov-0.0.tar.gz (424.3 kB view hashes)

Uploaded Source

Built Distribution

allusgov-0.0-py3-none-any.whl (432.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page