Skip to main content

MicroDrill

Simple Apache Drill alternative using PySpark inspired by PyDAL

Setup

Run terminal command pip install microdrill

Dependencies

PySpark was tested with Spark 1.6

Usage

Defining Query Parquet Table

ParquetTable(table_name, schema_index_file=file_name)

  • table_name: Table referenced name.

  • file_name: File name to search for table schema.

Using Parquet DAL

ParquetDAL(file_uri, sc)

Connecting in tables

parquet_conn = ParquetDAL(file_uri, sc)
parquet_table = ParquetTable(table_name, schema_index_file=file_name)
parquet_conn.set_table(parquet_table)

Queries

Returning Table Object

parquet_conn(table_name)

Returning Field Object

parquet_conn(table_name)(field_name)

Basic Query

parquet_conn.select(field_object, [field_object2, ...]).where(field_object=value)
parquet_conn.select(field_object1, field_object2).where(field_object1==value1 & ~field_object2==value2)
parquet_conn.select(field_object1, field_object2).where(field_object1!=value1 | field_object1.regexp(reg_exp))

Grouping By

parquet_conn.groupby(field_object1, [field_object2, ...])

Ordering By

parquet_conn.orderby(field_object1, [field_object2, ...])
parquet_conn.orderby(~field_object)

Limiting

parquet_conn.limit(number)

Executing

df = parquet_conn.execute() execute() returns a PySpark DataFrame.

Returning Field Names From Schema

parquet_conn(table_name).schema()

Developers

Install latest jdk and run in terminal make setup

Metadata

Release files for microdrill 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for microdrill 0.0.3
File Size Uploaded Provenance Actions
microdrill-0.0.3.tar.gz 7.9 kB Without provenance Details for microdrill-0.0.3.tar.gz

Release files / microdrill-0.0.3.tar.gz

Download URL microdrill-0.0.3.tar.gz
Size 7.9 kB
Tags Source
SHA-256 checksum
How to use checksums
0ffa274b046a602225909f4cdaa7943be64f7dc99217583d37456468d303ca73
BLAKE2b-256 checksum
How to use checksums
4e9412aab5729bafac62bc3f036eaf60ae2a6e7be7dcbdd243817ee463e1d27a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for microdrill 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page