A python package to help facilitate the collection and analysis of education-related datasets.
This is a python package aiming to analyze IPEDS and other education datasets using the data science tools available within python. Dare I say, enrollment science tools?
WARNING: This is a very young project with exploration around the API and the toolkits included.
Install and Test Notes
pypeds is as simple as
pip install pypeds.
It is highly recommended that you leverage the use of environments when coding in python; at least that is my opinion anyway. I prefer to use conda for my environment management. Assuming that you have conda setup properly, this could be as simple as:
conda create -n pypeds python=3.6 conda activate pypeds pip install pypeds
Alternatively, if you want to use the development version of this package, you can review this Google Colab Notebook which provides an installation overview to install from this repository. You can replicate this process anytime you want code/analyze data using Google's fantastic notebook coding environment.
Last but not least, it's worth stating that you could just as easily install
pypeds via collab with:
! pip install colab
years argument is based on the survey year shown on the IPEDS website. When the datasets are built, this package adds both the survey year and the fall year (of the academic year) to the datasets. For example, the survey SFA (Student Financial Aid) for the survey year
2017 is reporting on data for the 1617 academic year, so I include
fall_year with a value of 2016 in this case.
Below is a simple session to get the last 2 years of the directory information surveys:
from pypeds import ipeds # generate a list of survey years -- this must be a python list # YEARS = list(range(2014, 2018, 1)) ## another way to create a range of years YEARS = [2016,2017] # instantiate the survey of interest hd = ipeds.HD(years=YEARS) # extract, or download the surveys hd.extract() # load the surveys as an explicit pandas dataframe in your session df = hd.load() # confirm that we have the data we expected df.survey_year.value_counts() 2016.0 7521 2017.0 7153
This package aims to be a go-to resource for those of us who analyze data in higher education, and perhaps more specifically, enrollment management. As such, I have chosen an API scheme that hopefully will make your transition to python easier, especially as you transition into machine learning with scikit-learn. The use of classes and methods is heavily inspired by that toolkit.
Moreover, this package attempts to remove the friction of data prep as much as possible. For that reason, and also borrowing from the tidyverse use of verbs for data munging, the API is built around the idea of ETL (Extract, Transform, Load) as well as common Exploratory Data Analysis (EDA) concepts. Below is an outline of the tools
- Instantiate the survey of interest with
ipeds.HD(). The years argument defaults to the current survey (2017) year but can setup with a list of years. For example,
- Next, we
.extractthe data, which collects 1 or more years of full survey datasets from IPEDS and keeps the data within our survey object.
- Optionally, you can
.transformthe data. This API yet built, but will supply functionality to modify the survey data by filtering rows, selecting columns, or deriving new columns using commonly accepted logic in order to establish standard definitions. Again, the aim is to make things easier.
.loadpulls out the survey datasets as a pandas dataframe, at which point you can analyze, visualize the survey dataset using the full suite of data analytics tools found within python.
This package is under heavy development and as noted at the top, is subject to breaking changes within the API. However, beyond the ETL verbs, this package will also include various methods for exploration, competitive benchmarking, and data visualization. While all of this work can be done by the analyst after the
.load method, the aim is facilitate learning and insight by extracting away the "how" for basic and common questions in the enrollment management space.
Surveys currently supported:
- HD: Directory Info [HD]
- IC: Institutional Characteristics, which also merges on the ADM survey which started in 2014 [IC]
- SFA: Student Financial Aid [SFA]
- EF_C: Residence and Migration of First-Time Freshmen [EFC]
- EF_D: Total entering class, retention rates, and student-to-faculty ratio [EFD]
- IC_AY: Charges for Academic Year Programs [ICAY]
- C_A: Awards/degrees conferred by program (6-digit CIP code), award level, race/ethnicity, and gender [C_A]
- OM: Award and enrollment data at four, six and eight years of entering degree/certificate-seeking undergraduate cohorts [OM]
- FF1: Finance, Public institutions - GASB [FF1]
- FF2: Private not-for-profit institutions or Public institutions using FASB [FF2]
The class names are in the brackets. For example, the
EF_C survey can be instantiated using
This package will also periodically add datasets that one might use when studying and analyzing data in higher education and enrollment management.
Most notably, once importing the
pypeds attempts to provide simple access to core education-related datasets. For example:
wichefor data, and projections, of high school graduates in the US by state.
scorecard_xdatasets related to the National Scorecard. There are currenly 4 supported. Review the documentation for details.
- In addition to above, there are other datasets to help with mapping code and geographies for easier reporting and visualization.
For example, let's play around with the WICHE projections below.
WICHE High School Projections
For example, you might want to look at the WICHE projections for High School Graduates. The data are currently included in this package via the datasets module.
# import the datasets module from pypeds import datasets # extract the wiche dataset wiche = datasets.wiche() # first few rows wiche.head() year status state demo grads 0 2000 actual Alabama grand_total 41316.0 1 2001 actual Alabama grand_total 40127.0 2 2002 actual Alabama grand_total 41412.0 3 2003 actual Alabama grand_total 41729.0 4 2004 actual Alabama grand_total 42644.0
You will notice that the data are not wide, but long. This is by design, and it allows us to reshape and aggregate as desired.
For example, plot the
projected (the status column) total high school graduates (the demo column) in the dataset.
# isolate total grads, but still for each state and year all_grads = wiche.loc[wiche.demo=='grand_total', ] # group by the year and aggregate (sum) of grads by year grad_totals = all_grads.groupby("year", as_index=False).sum() # generate the line plot import seaborn as sns import matplotlib.pyplot as plt sns.lineplot(x='year', y='grads', data=grad_totals) plt.show()
- include tooling to help analysts connect to Salesforce or Data Warehouses/Databases
- submit via pypeds
- additional higher education datasets
- It appears that the mission statement data is not part of the full surveys, but is included in the MS Access version. Obviously it's less than ideal that the data collected is not included within the full file option, which this package highly leverages.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pypeds-0.104-py3-none-any.whl (28.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size pypeds-0.104.tar.gz (93.0 kB)||File type Source||Python version None||Upload date||Hashes View hashes|