API to interact with a few AIND databases
Project description
aind-data-access-api
API to interact with a few AIND databases.
Usage
We have two primary databases. A Document store to keep unstructured json documents, and a relational database to store structured tables.
Document Store
We have some convenience methods to interact with our Document Store. You can create a client by explicitly setting credentials, or downloading from AWS Secrets Manager.
from aind_data_access_api.credentials import DocumentStoreCredentials
from aind_data_access_api.document_store import Client
# Method one assuming user, password, and host are known
ds_client = Client(
credentials=DocumentStoreCredentials(
username="user",
password="password",
host="host",
database="metadata",
),
collection_name="data_assets",
)
# Method two if you have permissions to AWS Secrets Manager
ds_client = Client(
credentials=DocumentStoreCredentials(
aws_secrets_name="aind/data/access/api/document_store/metadata"
),
collection_name="data_assets",
)
# To get all records
response = list(ds_client.retrieve_data_asset_records())
# To get a list of filtered records:
response = list(ds_client.retrieve_data_asset_records({"subject.subject_id": "123456"}))
RDS Tables
We have some convenience methods to interact with our Relational Database. You can create a client by explicitly setting credentials, or downloading from AWS Secrets Manager.
from aind_data_access_api.credentials import RDSCredentials
from aind_data_access_api.rds_tables import Client
# Method one assuming user, password, and host are known
ds_client = Client(
credentials=RDSCredentials(
username="user",
password="password",
host="host",
database="metadata",
),
collection_name="data_assets",
)
# Method two if you have permissions to AWS Secrets Manager
ds_client = Client(
credentials=RDSCredentials(
aws_secrets_name="aind/data/access/api/rds_tables"
),
)
# To retrieve a table as a pandas dataframe
df = ds_client.read_table(table_name="spike_sorting_urls")
# Can also pass in a custom sql query
df = ds_client.read_table(query="SELECT * FROM spike_sorting_urls")
# It's also possible to save a pandas dataframe as a table. Please check internal documentation for more details.
ds_client.overwrite_table_with_df(df, table_name)
Installation
To use the software, it can be installed from PyPI.
pip install aind-data-access-api
To develop the code, clone repo and run
pip install -e .[dev]
Contributing
Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the coverage library, which will run the tests and log a coverage report:
coverage run -m unittest discover && coverage report
- Use interrogate to check that modules, methods, etc. have been documented thoroughly:
interrogate .
- Use flake8 to check that code is up to standards (no unused imports, etc.):
flake8 .
- Use black to automatically format the code into PEP standards:
black .
- Use isort to automatically sort import statements:
isort .
Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:
<type>(<scope>): <short summary>
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- docs: Documentation only changes
- feat: A new feature
- fix: A bugfix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- test: Adding missing tests or correcting existing tests
Semantic Release
The table below, from semantic release, shows which commit message gets you which release type when semantic-release
runs (using the default configuration):
Commit message | Release type |
---|---|
fix(pencil): stop graphite breaking when too much pressure applied |
|
feat(pencil): add 'graphiteWidth' option |
|
perf(pencil): remove graphiteWidth option BREAKING CHANGE: The graphiteWidth option has been removed. The default graphite width of 10mm is always used for performance reasons. |
(Note that the BREAKING CHANGE: token must be in the footer of the commit) |
Documentation
To generate the rst files source files for documentation, run
sphinx-apidoc -o doc_template/source/ src
Then to create the documentation HTML files, run
sphinx-build -b html doc_template/source/ doc_template/build/html
More info on sphinx installation can be found here.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aind-data-access-api-0.7.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9231fec222d82c4ca0d4af2a78174c75750fb0d0e905ffcfaaf4e5bd8fcf9d3 |
|
MD5 | b69115c1d63e62bc4e01fe2aec9f8920 |
|
BLAKE2b-256 | dc34c95fcc64b1ee47a84084c315be4fc6dcff316d5a5734a3d236aeb910d63b |
Hashes for aind_data_access_api-0.7.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8eac00aa2658a0adfef194438976425aec0b4652b62f83921a8435305d628661 |
|
MD5 | e266686be46c9fe0c7d0f68ef1c640a0 |
|
BLAKE2b-256 | 9d7f29f0ace1563202d67e8a4786b6b04b38a2c92f30c38c8be57bd0115f120d |