What is it?
Malloy is an experimental language for describing data relationships and transformations. It is both a semantic modeling language and a querying language that runs queries against a relational database. Malloy currently connects to BigQuery, and natively supports DuckDB. We've built a Visual Studio Code extension to facilitate building Malloy data models, querying and transforming data, and creating simple visualizations and dashboards.
Note: These APIs are still in development and are subject to change.
How do I get it?
Binary installers for the latest released version are available at the Python Package Index (PyPI).
python3 -m pip install malloy
Resources
- Malloy Language GitHub - Primary location for the malloy language source, documentation, and information
- Malloy Language - A quick introduction to the language
- eCommerce Example Analysis - A walkthrough of the basics on an ecommerce dataset (BigQuery public dataset)
- Modeling Walkthrough - An introduction to modeling via the Iowa liquor sales public data set (BigQuery public dataset)
- Malloy on YouTube - Watch demos / walkthroughs of Malloy
Join The Community
- Join our Malloy Slack Community! Use this community to ask questions, meet other Malloy users, and share ideas with one another.
- Use GitHub issues to provide feedback, suggest improvements, report bugs, and start new discussions.
Syntax Examples
Run a named query from a Malloy file
import asyncio
import malloy
from malloy.data.duckdb import DuckDbConnection
async def main():
home_dir = "/path/to/samples/duckdb/imdb"
with malloy.Runtime() as runtime:
runtime.add_connection(DuckDbConnection(home_dir=home_dir))
data = await runtime.load_file(home_dir + "/imdb.malloy").run(
named_query="genre_movie_map")
dataframe = data.to_dataframe()
print(dataframe)
if __name__ == "__main__":
asyncio.run(main())
Get SQL from an in-line query, using a Malloy file as a source
import asyncio
import malloy
from malloy.data.duckdb import DuckDbConnection
async def main():
home_dir = "/path/to/samples/duckdb/faa"
with malloy.Runtime() as runtime:
runtime.add_connection(DuckDbConnection(home_dir=home_dir))
[sql, connection
] = await runtime.load_file(home_dir + "/flights.malloy").get_sql(query="""
run: flights -> {
where: carrier ? 'WN' | 'DL', dep_time ? @2002-03-03
group_by:
flight_date is dep_time.day
carrier
aggregate:
daily_flight_count is flight_count
aircraft.aircraft_count
nest: per_plane_data is {
limit: 20
group_by: tail_num
aggregate: plane_flight_count is flight_count
nest: flight_legs is {
order_by: 2
group_by:
tail_num
dep_minute is dep_time.minute
origin_code
dest_code is destination_code
dep_delay
arr_delay
}
}
}
""")
print(sql)
if __name__ == "__main__":
asyncio.run(main())
Write an in-line Malloy model, and run a query
import asyncio
import malloy
from malloy.data.duckdb import DuckDbConnection
async def main():
home_dir = "/path/to/samples/duckdb/imdb/data"
with malloy.Runtime() as runtime:
runtime.add_connection(DuckDbConnection(home_dir=home_dir))
data = await runtime.load_source("""
source:titles is duckdb.table('titles.parquet') extend {
primary_key: tconst
dimension:
movie_url is concat('https://www.imdb.com/title/',tconst)
}
""").run(query="""
run: titles -> {
group_by: movie_url
limit: 5
}
""")
dataframe = data.to_dataframe()
print(dataframe)
if __name__ == "__main__":
asyncio.run(main())
Querying BigQuary tables
BigQuery auth via OAuth using gcloud.
gcloud auth login --update-adc
gcloud config set project {my_project_id} --installation
Actual usage is similar to DuckDB.
import asyncio
import malloy
from malloy.data.bigquery import BigQueryConnection
async def main():
with malloy.Runtime() as runtime:
runtime.add_connection(BigQueryConnection())
data = await runtime.load_source("""
source:ga_sessions is bigquery.table('bigquery-public-data.google_analytics_sample.ga_sessions_20170801') extend {
measure:
hits_count is hits.count()
}
""").run(query="""
run: ga_sessions -> {
where: trafficSource.`source` != '(direct)'
group_by: trafficSource.`source`
aggregate: hits_count
limit: 10
}
""")
dataframe = data.to_dataframe()
print(dataframe)
if __name__ == "__main__":
asyncio.run(main())
Development
Initial setup
git submodule init
git submodule update
python3 -m pip install -r requirements.dev.txt
scripts/gen-services.sh
Regenerate Protobuf files
scripts/gen-protos.sh
Tests
python3 -m pytest
Metadata
Release files for malloy 2024.1091
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| malloy-2024.1091.tar.gz | 86.8 MB | Without provenance | Details for malloy-2024.1091.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| malloy-2024.1091-py3-none-any.whl | Python 3 | none | any | Details for malloy-2024.1091-py3-none-any.whl |
Total release size: 173.2 MB
Release files / malloy-2024.1091.tar.gz
| Download URL | malloy-2024.1091.tar.gz |
|---|---|
| Size | 86.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3edeb00ddceab8b42523cc075f41af66ae1b6877f67700f1d4e9d535cfd3d6dc
|
|
BLAKE2b-256 checksum How to use checksums |
07f457b049f6db05147753ba92c9ea5e3430a9469d4fae1d55212942864e2d78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.19
|
Release files / malloy-2024.1091-py3-none-any.whl
| Download URL | malloy-2024.1091-py3-none-any.whl |
|---|---|
| Size | 86.4 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9cf7de9ff252b77bee7b4478a51dcc4a4f3c4d9687d715dddfb2458af49697f1
|
|
BLAKE2b-256 checksum How to use checksums |
601895e2cde4b22302c6920ea86bcdd55021991b1c8582deb299cb8666cf1038
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.19
|
Security for malloy 2024.1091
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.