Python Scripts for BNIA-JFIs Vital Signs Data
Project description
Vital Signs
Scripts to create our annual, publicly-available, community-focused datasets; for Baltimore City.
Hi! We are BNIA-JFI.
This is BNIA-JFI's principal repository for creating Vital Signs Indicators.
Included
- IPYNB/ Google Colab notebooks with indicator creation notes and scripts.
- Online documentation and PyPi libraries created from the notebooks.
ACS Scripts
These scripts will download and clean ACS data for Baltimore and then construct indicators from the data.
AcsDownload.ipynb Mount notebook to google drive Read in ACS Meta Data from XLSX and the crosswalk data from a csv Add path to python script that performs download function Enter a year and Run the download function for every record in XLSX sheet For each dataset, remove columnID’s then save it as Raw Then, Append Community Names using a crosswalk and save again in as Clean
AcsIndicators.ipynb Mount notebook to google drive Read in ACS Meta Data from XLSX Prepare the Compare Historic Data
- For Each Indicator
- Grab its Meta Data
- If the Indicator is valid for the year, and uses ACS Data, and method exists
- retrieve the Python ACS indicator
- Put Baltimore City at the bottom of the list
- Write the results back into the XL dataframe
- Save the Dataset
- drop columns with any empty values
- Save the Data xlsx Do comparison to historic year if exists. Write xlsx.
Install
The code is on PyPI so you can install the scripts as a python library using the command:
!pip install VitalSigns geopandas
{% include important.html content='Contributers should follow the maintanance instructions and will not need to run this step. ' %}>
Their modules will be retrieved from the VitalSigns-GDrive repo they have mounted into their Colabs Enviornment.
Then...
Examples
- Import the installed module into your code:
from VitalSigns.acsDownload import retrieve_acs_data
- use it
retrieve_acs_data(state, county, tract, tableId, year, saveAcs)
Now you could do something like merge it to another dataset!
from dataplay.merge import mergeDatasets
mergeDatasets(left_ds=False, right_ds=False, crosswalk_ds=False, use_crosswalk = True, left_col=False, right_col=False, crosswalk_left_col = False, crosswalk_right_col = False, merge_how=False, interactive=True)
You can get information on the package by using the help command.
import VitalSigns
help(VitalSigns)
Help on package VitalSigns:
NAME
VitalSigns
PACKAGE CONTENTS
BCPSS
HUD
_nbdev
acsDownload
bidbaltimore
bpd
citistat
cityfinance
closecrawl
create
dhr
enoch
fares
fdic
indicators
infousa
liquor
mdprop
rbintel
tidyaddr
treebaltimore
utils
VERSION
0.0.5
FILE
/content/drive/My Drive/Software Development Documents/VitalSigns/VitalSigns/__init__.py
help(VitalSigns.acsDownload)
Help on module VitalSigns.acsDownload in VitalSigns:
NAME
VitalSigns.acsDownload - # AUTOGENERATED! DO NOT EDIT! File to edit: notebooks/90_ACS_Explore_and_Download.ipynb (unless otherwise specified).
FUNCTIONS
retrieve_acs_data(state, county, tract, tableId, year, save)
DATA
__all__ = ['retrieve_acs_data']
__warningregistry__ = {'version': 352, ('Passing a negative integer is...
FILE
/content/drive/My Drive/Software Development Documents/VitalSigns/VitalSigns/acsDownload.py
help(VitalSigns.acsDownload.retrieve_acs_data)
Help on function retrieve_acs_data in module VitalSigns.acsDownload:
retrieve_acs_data(state, county, tract, tableId, year, save)
Here an example:
from VitalSigns.acsDownload import retrieve_acs_data
# Our download function will use Baltimore City's tract, county and state as internal paramters
# Change these values in the cell below using different geographic reference codes will change those parameters
tract = '*'
county = '510'
state = '24'
# Specify the download parameters the function will receieve here
tableId = 'B19001'
year = '17'
saveAcs = False
df = retrieve_acs_data(state, county, tract, tableId, year, saveAcs)
df.head()
Number of Columns 17
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
B19001_001E_Total | B19001_002E_Total_Less_than_$10,000 | B19001_003E_Total_$10,000_to_$14,999 | B19001_004E_Total_$15,000_to_$19,999 | B19001_005E_Total_$20,000_to_$24,999 | B19001_006E_Total_$25,000_to_$29,999 | B19001_007E_Total_$30,000_to_$34,999 | B19001_008E_Total_$35,000_to_$39,999 | B19001_009E_Total_$40,000_to_$44,999 | B19001_010E_Total_$45,000_to_$49,999 | B19001_011E_Total_$50,000_to_$59,999 | B19001_012E_Total_$60,000_to_$74,999 | B19001_013E_Total_$75,000_to_$99,999 | B19001_014E_Total_$100,000_to_$124,999 | B19001_015E_Total_$125,000_to_$149,999 | B19001_016E_Total_$150,000_to_$199,999 | B19001_017E_Total_$200,000_or_more | state | county | tract | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
NAME | ||||||||||||||||||||
Census Tract 2710.02 | 1510 | 209 | 73 | 94 | 97 | 110 | 119 | 97 | 65 | 36 | 149 | 168 | 106 | 66 | 44 | 50 | 27 | 24 | 510 | 271002 |
Census Tract 2604.02 | 1134 | 146 | 29 | 73 | 80 | 41 | 91 | 49 | 75 | 81 | 170 | 57 | 162 | 63 | 11 | 6 | 0 | 24 | 510 | 260402 |
Census Tract 2712 | 2276 | 69 | 43 | 41 | 22 | 46 | 67 | 0 | 30 | 30 | 80 | 146 | 321 | 216 | 139 | 261 | 765 | 24 | 510 | 271200 |
Census Tract 2804.04 | 961 | 111 | 108 | 61 | 42 | 56 | 37 | 73 | 30 | 31 | 106 | 119 | 74 | 23 | 27 | 24 | 39 | 24 | 510 | 280404 |
Census Tract 901 | 1669 | 158 | 124 | 72 | 48 | 108 | 68 | 121 | 137 | 99 | 109 | 191 | 160 | 141 | 28 | 88 | 17 | 24 | 510 | 90100 |
From there, you can go on to do even greater things using our dataplay library. Like these visuals:
Have Fun!
MISC
This section is not definite but provides a good idea of how our scripts are made.
Basic Indicator Creation Outline
- ? Count = 1
- Create the num and denom
- filter num denom
- ? sum/ median = ungrouped.median
- group by csa
- ? bcity = median or sum
- perform the calculation
- compare years
Miscellaneous things I should have for every notebook
- Module/filenames need to be fixed.
- RB Intel has the best prelim analysis script. The others are messed up a bit?
- include links indicator Esri and Bnia pages details on category, name, description, years
- Don’t drop columns at end, but keep selected at beginning.
- Merge on CSA for ordering
- Bcity Median gets calculated before aggregation. Appended after
- Add Years in header. Use denom and numerator as var names.
- Code to compare past years
FOR CONTRIBUTERS
Dev Instructions
From a local copy of the git repo: 0. Clone the repo local onto GDrive
- Via Direct-DL&Drive-Upload or Colab/Terminal/Git
git clone https://github.com/BNIA/VitalSigns.git
- Update the the IPYNB
- From the GDrive VitalSigns folder via Colabs
- Build the new libraries from these NBs
- Using this index.ipynb
-
- Mount the Colabs Enviornment (and navigate to) the GDrive VitalSigns folder
-
- run
!nbdev_build_lib
to build .py modules.
- run
- Test the Library/ Modules
- Using the same runtime as step 2's index.ipynb.
-
- Do not install the module from PyPi (if published) and then...
-
- Import your module ( from your VitalSigns/VitalSigns)
-
- If everything runs properly, go to step 5.
- Edit modules directly
- Within the same runtime as step 2/3's index.ipynb...
-
- Locate the VitalSigns/VitalSigns using the colab file nav
-
- double-click the .py modules in the file nav to open them in an in-browser editor
- Make changes and return to step 3 with the following caveat:
-
- Use the hot module reloading to ensure updates are auto-re-imported
-
%load_ext autoreload %autoreload 2
- Then when finished, persist the changes from the .py modules back to the .ipynb docs
-
- via
!nbdev_update_lib
and!relimport2name
- via
- Create Docs, Push to Github, and Publish to PyPI
- All done via nbdev
- Find more notes I made on that here: dataplay > nbdev notes
!nbdev_build_docs --force_all True --mk_readme True
!git commit -m ...
%%capture ! pip install twine
!nbdev_bump_version
! make pypi
# https://nbdev.fast.ai/tutorial.html#Set-up-prerequisites
# settings.ini > requirements = fastcore>=1.0.5 torchvision<0.7
# https://nbdev.fast.ai/tutorial.html#View-docs-locally
# console_scripts = nbdev_build_lib=nbdev.cli:nbdev_build_lib
# https://nbdev.fast.ai/search
Dev Scripts
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for VitalSigns-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4eea2089c0b838dc1c29b9ef08f961d3a5f7de5d8e23f7919962eb44f082bc0 |
|
MD5 | 6bd9cc9c60876b6caccd6345ed425923 |
|
BLAKE2b-256 | 77130b6d206e4c63c4a514190327f297b290dc6a92e2c8ea77b950ebff081ff5 |