dds_py - Data driven software
Data-driven software (python implementation)
Introduction
The DDS package solves the synchronization problem between code and data. It allows programmers, scientists and data scientists to integrate code with data and data with code without fear of stale data, disparate storage frameworks or concurrency issues. DDS allows quick collaboration and data software reuse without the complexity. In short, you do not have to think about changes in your data pipelines.
How to use
This package is published on PyPI:
pip install dds_py
This package is known to work on python 3.8, 3.9, 3.10, 3.11. No other versions are officially supported. Python 3.4 and 3.5 might work but they are not supported.
Plotting dependencies If you want to plot the graph of data dependencies,
you must install separately the pydotplus package, which requires graphviz
on your system to work properly. Consult the documentation of the pydotplus
package for more details. The pydotplus package is only required with the
dds_export_graph option.
Databricks users: If you want to use this package with Databricks, some specific hooks for Spark are available. See this notebook for a complete example:
Documentation
API reference, tutorials and FAQs are located here: https://tjhunter.github.io/dds_py/
Example
Here is the Hello world example (using type annotations for clarity)
import dds
import requests
@dds.data_function("/hello_data")
def data() -> str:
url = "https://gist.githubusercontent.com/bigsnarfdude/515849391ad37fe593997fe0db98afaa/raw/f663366d17b7d05de61a145bbce7b2b961b3b07f/weather.csv"
return requests.get(url=url, verify=False).content.decode("utf-8")
data()
This example does the following:
- it defines a source of data, here a piece of weather data from the internet.
This source is defined as the function
data_creator - it assigns the data produced by this source into a variable (
data) and also to a path in a storage system (/hello_data)
The DDS library guarantees the following after evaluation of the code:
- the path
/hello_datacontains a copy of the data returned bydata_creator, as if the functiondata_creatorhad been called at this moment - the function
data_creatoris only evaluated when its inputs, or its code, are modified (referential transparency)
License
The dds package is published under the Affero General Public License.
1 maintainer
Release files for dds-py 0.13.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dds_py-0.13.1-py3-none-any.whl | Python 3 | none | any | Details |
Release files / dds_py-0.13.1-py3-none-any.whl
| Download URL | dds_py-0.13.1-py3-none-any.whl |
|---|---|
| Size | 74.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dadf366015938d984af516a7edd8a322373fa91b2fdd037906c2a6a12619f288
|
|
BLAKE2b-256 checksum How to use checksums |
32fcba9e9eafc2ee27c5efabc6dd9931859eca199c1c5df0bb89c68b4a8507d5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
