A Stata emulator for Python/pandas
Project description
"...succinctness is power... we take the trouble to develop high-level languages... so that we can say (and more importantly, think) in 10 lines of a high-level language what would require 1000 lines of machine language... -Paul Graham, Succinctness is Power
pdexplorer
pdexplorer is a Stata emulator for Python/pandas. In contrast to pandas, Stata syntax achieves succinctness by:
- Using spaces and commas rather than parentheses, brackets, curly braces, and quotes (where possible)
- Specifying a set of concise commands on the "current" dataset rather than cluttering the namespace with multiple datasets
- Being verbose by default i.e., displaying output that represents the results of the command
- Having sensible defaults that cover the majority of use cases and demonstrate common usage
- Allowing for namespace abbreviations for both commands and variable names
- Employing two types of column names: Variable name are concise and used for programming. Variable labels are verbose and used for presentation.
Installation
pdexplorer
is available on PyPI. Run pip
to install:
pip install pdexplorer
Then import the pdexplorer
commands with
from pdexplorer import *
Examples
Load Stata dataset and perform exploratory data analysis
webuse('auto')
browse()
See https://www.stata.com/manuals/dwebuse.pdf
Summarize Data by Subgroups
webuse('auto')
with by('foreign'):
summarize('mpg weight')
See https://www.stata.com/manuals/rsummarize.pdf
Ordinary Least Squares (OLS) Regression
webuse('auto')
regress('mpg weight foreign')
ereturnlist()
Return Values
In the last example, note the use of ereturnlist()
, corresponding to the Stata command ereturn list
. Additionally, a Python object may also be available as the command's return value. For example,
webuse('auto')
results = regress('mpg weight foreign')
Here, results
is a RegressionResultsWrapper object from the statsmodels package.
Similarly,
results = regress('mpg weight foreign', library='scikit-learn')
Now, results
is a LinearRegression object from the scikit-learn package.
Finally,
results = regress('mpg weight foreign', library='pytorch')
Here, results
is a torch.nn.Linear object from the PyTorch package.
My Story
I used Stata for 7 years for both data exploration and programming. After that, I used Python/pandas for 3 years and found that pandas is just too verbose and "explicit" for rapid data exploration. So I started working on this project on September 3, 2023.
Why not use Stata instead of pandas?
Stata is great, but Python/pandas is free and easier to integrate with other applications. For example, you can build a web server in Python, but not Stata; You can run Python in AWS SageMmaker, but not Stata.
Additionally, even for devout Stata users, there is utility in being able to run Stata commands through a Python stack for comparison purposes.
How pdexplorer
fulfills the Zen of Python (relative to pandas)
YES | NO |
---|---|
Beautiful is better than ugly. | Explicit is better than implicit. |
Simple is better than complex. | In the face of ambiguity, refuse the temptation to guess. |
Flat is better than nested. | |
Readability counts. | |
Although practicality beats purity. | |
There should be one-- and preferably only one --obvious way to do it. | |
Now is better than never. |
How pdexplorer
differs from Stata
- Commands are implemented as Python functions and hence require at least one set of parentheses
pdexplorer
uses Python libraries under the hood. (The result of a command reflects the output of those libraries. See above.)- There is no support for mata. Under the hood,
pdexplorer
is just the Python data stack. - The API for producing charts is based on Altair, not Stata.
Syntax summary
With few exceptions, the basic Stata language syntax (as documented here) is
[by varlist:] command [varlist] [=exp] [if exp] [in range] [weight] [, options]
where square brackets distinguish optional qualifiers and options from required ones. In this diagram, varlist denotes a list of variable names, command denotes a Stata command, exp denotes an algebraic expression, range denotes an observation range, weight denotes a weighting expression, and options denotes a list of options.
The by varlist:
prefix causes Stata to repeat a command for each subset of the data for which the
values of the variables in varlist are equal. When prefixed with by varlist:, the result of the command
will be the same as if you had formed separate datasets for each group of observations, saved them,
and then gave the command on each dataset separately. The data must already be sorted by varlist,
although by has a sort option.
In pdexplorer, this gets translated to
with by('varlist'):
command("[varlist] [=exp] [if exp] [in range] [weight] [, options]", *args, **kwargs)
where *args
, and **kwargs
represent additional arguments that are available in a pdexplorer
command but
not in the equivalent Stata command.
Sometimes, Stata commands are two words. In such cases, the pdexplorer
command is a concatenation of the two words. For example,
label data "label"
becomes
labeldata("label")
Dependencies
pdexplorer command |
package dependency |
---|---|
cf | ydata-profiling or sweetviz |
browse | dtale |
regress | statsmodels or scikit-learn or PyTorch |
Charts
pdexplorer
departs somewhat from Stata in producing charts. Rather than emulating Stata's chart syntax,
pdexplorer
uses Altair with some syntactic sugar.
Take the example "Simple Scatter Plot with Tooltips":
import altair as alt
from vega_datasets import data
source = data.cars()
alt.Chart(source).mark_circle(size=60).encode(
x="Horsepower",
y="Miles_per_Gallon",
color='Origin',
tooltip=["Name", "Origin", "Horsepower", "Miles_per_Gallon"],
)
In pdexplorer
, this becomes:
from pdexplorer import *
webuse("cars", "vega") # note that names are forced to be lower case by default
circlechart(size=60).encode(
x="horsepower",
y="miles_per_gallon",
color='origin',
tooltip=["name", "origin", "horsepower", "miles_per_gallon"]
)() # () indicates that the chart is complete and should be opened in a web browser
The ()
at the end tells Altair to open the chart in a web browser. This method is not available in alt.Charts
itself,
but we monkey patched this into the class for convenience. In this example, circlechart()
itself simply returns a alt.Chart
object.
pdexplorer
charts also support a varlist
parameter with y
/x
encodings and additional encodings via options e.g., the previous block can be written as
from pdexplorer import *
webuse("cars", "vega")
circlechart(
"miles_per_gallon horsepower, color(origin) \
tooltip(name origin horsepower miles_per_gallon)",
size=60,
)()
Here, pdexplorer
uses variable labels for the y- and x- axes.
Also, note that encode()
is optional here. So, for a quick scatterplot, we can write
from pdexplorer import *
webuse("cars", "vega")
circlechart("miles_per_gallon horsepower")()
Since pdexplorer
charts are regular alt.Chart
objects,
Layered and Multi-View Charts are also supported e.g.,
from pdexplorer import *
webuse("cars", "vega")
chartA = circlechart("miles_per_gallon weight_in_lbs")
chartB = circlechart("horsepower weight_in_lbs")
(chartA & chartB)() # Vertically concatenate chartA and chartB
Finally, pdexplorer
also offers syntactic sugar for charting multiple x
/y
variables. More specifically, the previous block can be
written as
from pdexplorer import *
webuse("cars", "vega")
circlechart("miles_per_gallon horsepower weight_in_lbs", stacked=True)()
Note that Stata's varlist
interpretation is used here by default i.e., var1 var2 var3
is assumed to represent
yvar1 yvar2 xvar
. We can change this interpretation with the optional argument yX
.
from pdexplorer import *
webuse("cars", "vega")
circlechart("miles_per_gallon horsepower weight_in_lbs", yX=True, stacked=True)()
Now var1 var2 var3
is assumed to represent yvar xvar1 xvar2
as it would be for the regress
command.
Moreover, the Stata default is to layer all variables onto a single chart. The stacked=True
option allows the graphs to be shown on
separate grids. If stacked=False
, the charts are all shown on the same grid i.e.,
from pdexplorer import *
webuse("cars", "vega")
circlechart("miles_per_gallon horsepower weight_in_lbs")() # stacked=False is the default option
Note that pdexplorer
uses Altair's transform_fold
method under the hood. For further customization, the Altair methods can be used explicitly e.g.,
import altair as alt
from pdexplorer import *
webuse("cars", "vega")
circlechart().transform_calculate_variable_labels().transform_fold(
["Miles_per_Gallon", "Horsepower"] # The variable labels are accessible
).encode(
y="value:Q", x=alt.X("weight_in_lbs", title="Weight_in_lbs"), color="key:N"
)()
Note that transform_calculate_variable_labels
is monkey patched into alt.Chart
to enable access to variable labels in the
transform_fold
method. transform_form
is Altair's version of pd.melt
. So another option is to first reshape the data using
pandas and then use Altair for charting.
import altair as alt
from pdexplorer import *
webuse("cars", "vega")
melt("miles_per_gallon horsepower, keep(weight_in_lbs)")
alt.Chart(current.df_labeled).mark_circle().encode(
y="value:Q", x='Weight_in_lbs', color="variable:N"
)()
References
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pdexplorer-0.0.19-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 484108c3dfbf0c6f62e5f41e8c97ea07090106264a1a00bbf1686b38a04bb148 |
|
MD5 | 514fcfde1f5faf846e605acb718448c8 |
|
BLAKE2b-256 | 215abd2ec1f68446e4aeb05a42106ff96f76f4f603c42b3747eaa55973e8809b |