pyoso

WARNING: THIS IS A WORK IN PROGRESS

pyoso is a Python package for fetching models and metrics from OSO. This package provides an easy-to-use interface to interact with oso and retrieve valuable data for analysis and monitoring.

Features

• Execute custom SQL queries for analyzing the OSO dataset.
• Inspect data dependencies and freshness with an analytics tree.
• Semantic modeling layer to build and execute complex queries (optional).

Installation

You can install pyoso using pip:

pip install pyoso

Optional Semantic Modeling

For semantic modeling capabilities, you can install with the semantic extra:

pip install pyoso[semantic]

This will include the oso_semantic package for building semantic models and queries.

Usage

Here is a basic example of how to use pyoso to fetch data directly into a pandas DataFrame:

import os
from pyoso import Client

# Initialize the client with an API key
os.environ["OSO_API_KEY"] = 'your_api_key'
client = Client()

# Fetch artifacts
query = "SELECT * FROM artifacts_v1 LIMIT 5"
artifacts = client.to_pandas(query)

print(artifacts)

Inspecting Data Dependencies

For more advanced use cases, the client.query() method returns a QueryResponse object that contains both the data and analytics metadata. This allows you to inspect the dependency tree of the data sources used in your query.

import os
from pyoso import Client

# Initialize the client
os.environ["OSO_API_KEY"] = "your_api_key"
client = Client()

# Execute a query to get a QueryResponse object
query = "SELECT * FROM artifacts_v1 LIMIT 5"
response = client.query(query)

# You can still get the DataFrame as before
df = response.to_pandas()
print("\n--- Query Data ---")
print(df)

# Now, inspect the analytics to see the dependency tree
print("\n--- Data Dependency Tree ---")
response.analytics.print_tree()

This will output a tree structure showing how the final artifacts_v1 table was constructed from its upstream dependencies, helping you understand the data's origin and freshness.

Documentation

For detailed documentation about the OSO dataset, please refer to the official documentation.

Future Plans

• Create DataFrame wrapper for creating SQL query from data transforms

Manually testing with pyodide

We need to add pyodide to CI, but for now to manually run tests do the following:

Get current pyodide version

You will need to do this from the pyoso directory.

PYODIDE_EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version)

Install emscripten

Choose a place to store the code and git clone emsdk:

cd some/base/directory
git clone https://github.com/emscripten-core/emsdk
cd emsdk

./emsdk install ${PYODIDE_EMSCRIPTEN_VERSION}
./emsdk activate ${PYODIDE_EMSCRIPTEN_VERSION}
source emsdk_env.sh

Build pyodide wheel

Now go back to the pyoso directory

cd oso/warehouse/pyoso
uv run pyodide build

This will generate a .whl file in dist

Download pyodide version

Download the recent pyodide version (at the time of writing is 0.27.2):

cd dist/
wget https://github.com/pyodide/pyodide/releases/download/0.27.2/pyodide-0.27.2.tar.bz2
tar xjf pyodide-0.27.2.tar.bz2

This will now have generated a dist/pyodide directory.

Run pytest

uv run pytest --run-in-pyodide . --runtime node --dist-dir=./dist