aind-data-transformation
Usage
Please import this package and extend the abstract base class to define a new transformation job
from aind_data_transformation.core import (
BasicJobSettings,
GenericEtl,
JobResponse,
)
# An example JobSettings
class NewTransformJobSettings(BasicJobSettings):
# Add extra fields needed, for example, a random seed
random_seed: Optional[int] = 0
# An example EtlJob
class NewTransformJob(GenericEtl[NewTransformJobSettings]):
# This method needs to be defined
def run_job(self) -> JobResponse:
"""
Main public method to run the transformation job
Returns
-------
JobResponse
Information about the job that can be used for metadata downstream.
"""
job_start_time = datetime.now()
# Do something here
job_end_time = datetime.now()
return JobResponse(
status_code=200,
message=f"Job finished in: {job_end_time-job_start_time}",
data=None,
)
Contributing
The development dependencies can be installed with
pip install -e .[dev]
Adding a new transformation job
Any new job needs a settings class that inherits the BasicJobSettings class. This requires the fields input_source and output_directory and makes it so that the env vars have the TRANSFORMATION_JOB prefix.
Any new job needs to inherit the GenericEtl class. This requires that the main public method to execute is called run_job and returns a JobResponse.
Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the coverage library, which will run the tests and log a coverage report:
coverage run -m unittest discover && coverage report
- Use interrogate to check that modules, methods, etc. have been documented thoroughly:
interrogate .
- Use flake8 to check that code is up to standards (no unused imports, etc.):
flake8 .
- Use black to automatically format the code into PEP standards:
black .
- Use isort to automatically sort import statements:
isort .
Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:
<type>(<scope>): <short summary>
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- docs: Documentation only changes
- feat: A new feature
- fix: A bugfix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- test: Adding missing tests or correcting existing tests
Semantic Release
The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):
| Commit message | Release type |
|---|---|
fix(pencil): stop graphite breaking when too much pressure applied |
|
feat(pencil): add 'graphiteWidth' option |
|
perf(pencil): remove graphiteWidth optionBREAKING CHANGE: The graphiteWidth option has been removed.The default graphite width of 10mm is always used for performance reasons. |
(Note that the BREAKING CHANGE: token must be in the footer of the commit) |
Metadata
Release files for aind-data-transformation 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aind_data_transformation-0.1.3.tar.gz | 38.3 kB | Without provenance | Details for aind_data_transformation-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aind_data_transformation-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for aind_data_transformation-0.1.3-py3-none-any.whl |
Total release size: 44.0 kB
Release files / aind_data_transformation-0.1.3.tar.gz
| Download URL | aind_data_transformation-0.1.3.tar.gz |
|---|---|
| Size | 38.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8b00207c448354f2da1ec3702fe96857bcc1d3f39d316987a559b055c4729a27
|
|
BLAKE2b-256 checksum How to use checksums |
76ea8ef00724931443f329eba025ff0e843671688165680189a73da1e29dfb86
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / aind_data_transformation-0.1.3-py3-none-any.whl
| Download URL | aind_data_transformation-0.1.3-py3-none-any.whl |
|---|---|
| Size | 5.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9f0cdc63d5660526f87cf057aa45c34792fbf24fa1aec1ee10d74c6d194e2b1d
|
|
BLAKE2b-256 checksum How to use checksums |
c273c40883d5ccbe23a4a0d9423a495b1b0132b0c92cca30bf824d7fd0e3ca7d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for aind-data-transformation 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.