A Python package to read and write files in CDM format. Customized for SkyPoint use cases.
Project description
skypoint-python-cdm-connector
Python Spark CDM Connector by SkyPoint.
A prototype Spark data source for the Azure "Common Data Model". Reading and writing is supported, but
skypoint-python-cdm-connector is definitely a work in progress. Library is in early Alpha Stage. Please file issues for any bugs that you find. For more information about the Azure Common Data Model, check out this page.
We support Azure (ADLS), AWS (S3*) and Google Cloud (Cloud Storage*) Common Data Model (CDM) Deployments.
*Upcoming Support
Example
- Please look into the sample usage file skypoint_python_cdm.py
- Dynamically add/remove entities, annotations and attributes
- Pass Reader and Writer object for any storage account you like to write/read data to/from.
- Check out the below code for basic read and write examples.
# Initialize empty model
m = Model()
# Sample dataframe
df = {"country": ["Brazil", "Russia", "India", "China", "South Africa", "ParaSF"],
"currentTime": [datetime.now(), datetime.now(), datetime.now(), datetime.now(), datetime.now(), datetime.now()],
"area": [8.516, 17.10, 3.286, 9.597, 1.221, 2.222],
"capital": ["Brasilia", "Moscow", "New Dehli", "Beijing", "Pretoria", "ParaSF"],
"population": [200.4, 143.5, 1252, 1357, 52.98, 12.34] }
df = pd.DataFrame(df)
# Generate entity from the dataframe
entity = Model.generate_entity(df, "customEntity")
# Add generated entity to model
m.add_entity(entity)
# Add model level annotation
# Annotation can be added at entity level as well as attribute level
Model.add_annotation("modelJsonAnnotation", "modelJsonAnnotationValue", m)
# Create an ADLSWriter to write into ADLS
writer = ADLSWriter("ACCOUNT_NAME", "ACCOUNT_KEY",
"CONTAINER_NAME", "STORAGE_NAME", "DATAFLOW_NAME")
# Write data as well as model.json in ADLS storage
m.write_to_storage("customEntity", df, writer)
Contributing
This project welcomes contributions and suggestions.
References
A clean implementation for Python Objects from/to model.json file
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdm_connector-0.0.6.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c28d954f031fb9b265cc999d2645dcfb547eb2c8a65c912d2430d576e0d2afa |
|
MD5 | 6c00f186208a3edb5607f5b9b1da37a8 |
|
BLAKE2b-256 | f3b536db023964cbc5931ab4aed74d56bb260e671adbc41793c44ead99361a4e |