Objectiv Bach provides Pandas-like DataFrames backed by SQL
Project description
Objectiv Bach: Pandas-like DataFrames backed by SQL
Bach is a python-based data modeling library that enables you to use Pandas-like operations that run on your full dataset in the SQL database. Any dataframe or model built with Bach can be converted to an SQL statement with a single command. It includes a set of operations that enable effective feature creation for data sets that embrace the open analytics taxonomy.
Bach uses sql_models
under the hood, which makes it possible to easily build graphs of SQL models and generate SQL for the resulting composite sql-models
. See sql_models/README.md for more information.
Visit Objectiv Docs to learn more
Using Bach
To use Bach, use the following command:
pip install objectiv-bach
pip install objectiv-bach[bigquery] # for bigquery support
If you want the latest and greatest from your local checkout, install objectiv_bach in edit mode:
pip install -e .
pip install -e .[bigquery] # for bigquery support
This will install Bach in edit mode, meaning you get the latest version from the local checkout. For detailed installation & usage instructions, visit Objectiv Docs.
Running Functional and Unit Tests
In case you are interested on running tests, install all dev requirements through:
pip install -e .[dev,bigquery]
Setting up environmental variables
Functional tests require access to multiple databases. In order to run them you should define
the variables shown in the below table, depending on the engine you want to test.
Variables can be set as environment variables, or can be defined in the file .secrets/.test_env
.
Database | Variables | Description |
---|---|---|
Postgres | OBJ_DB_PG_TEST_URL |
Database URL |
Athena | OBJ_DB_ATHENA_TEST_URL |
Database URL |
Athena | OBJ_DB_ATHENA_AWS_ACCESS_KEY_ID |
Access key id* |
Athena | OBJ_DB_ATHENA_AWS_SECRET_ACCESS_KEY |
Secret key* |
Athena | OBJ_DB_ATHENA_REGION_NAME |
Region* |
Athena | OBJ_DB_ATHENA_SCHEMA_NAME |
Schema name* |
Athena | OBJ_DB_ATHENA_S3_STAGING_DIR |
S3 staging dir* |
Athena | OBJ_DB_ATHENA_WORK_GROUP |
Workgroup to run queries as* |
BigQuery | OBJ_DB_BQ_TEST_URL |
Database URL |
BigQuery | OBJ_DB_BQ_CREDENTIALS_PATH |
Credentials Path |
For Athena one can either specify the full connection string as OBJ_DB_ATHENA_TEST_URL
or set
all the fields marked with an asterisk. The asterisk marked fields are ignored if OBJ_DB_ATHENA_TEST_URL
is set.
For more information on creating the correct database setup, see the markdown files in the tests/
folder.
Running Tests for different databases
For running tests for Postgres, as well as database independent tests, run the following command:
make tests
For running tests for Athena, as well as database independent tests, run the following command:
make tests-athena
For running tests for BigQuery, as well as database independent tests, run the following command:
make tests-bigquery
In case you want to run all tests for all supported databases, as well as database independent tests, run the following command:
make tests-all
See Also
- Pandas: the inspiration for the API. Pandas has excellent documentation for its API.
- SQL-models: Sub-project that is used for generating the underlying sql-queries. Can be
found in the
sql_models
package
Support & Troubleshooting
If you need help using or installing Bach, join our Slack channel and post your question there.
Bug Reports & Feature Requests
If you’ve found an issue or have a feature request, please check out the Contribution Guide.
Security Disclosure
Found a security issue? Please don’t use the issue tracker but contact us directly. See SECURITY.md for details.
Custom development & contributing code
If you want to contribute to Objectiv or use it as a base for custom development, take a look at CONTRIBUTING.md. It contains detailed development instructions and a link to information about our contribution process and where you can fit in.
License
This repository is part of the source code for Objectiv, which is released under the Apache 2.0 License. Please refer to LICENSE.md for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for objectiv_bach-0.0.26-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4260e3d063e03d5c1c87844314f35bd48904b8de7fa2482bca5dc89025713a08 |
|
MD5 | bbd7f4b9433e171a1a69129c8d7d27a0 |
|
BLAKE2b-256 | d422057cf9e0e9e6b5c9f6a75beac95a245f2ec65b1eaea8903b42d7be1f7cac |