dataforge-sdk
SDK for creating DataForge extensions.
Example projects and usage patterns: https://github.com/dataforgelabs/dataforge-sdk
Postgres Utilities
The dataforge.pg module provides helper functions to execute SQL operations against the DataForge Postgres metastore:
from dataforge.pg import select, update, pull
# Execute a SELECT query and return a Spark DataFrame
df = select("SELECT * FROM my_table")
# Execute an UPDATE/INSERT/DELETE query
update("UPDATE my_table SET col = 'value'")
# Trigger a new data pull for source_id 123
pull(123)
IngestionSession
The IngestionSession class manages a custom data ingestion process lifecycle.
from dataforge import IngestionSession
# Initialize a session (production use)
session = IngestionSession()
# Initialize a session (optional source_name/project_name for testing)
session = IngestionSession(source_name="my_source", project_name="my_project")
# Ingest data
# pass a function returning a DataFrame (recommended to integrate logging with DataForge)
session.ingest(lambda: spark.read.csv("s3://bucket/path/input.csv"))
# pass a DataFrame (can be used for testing, not recommended for production deployment)
df = spark.read.csv("s3://bucket/path/input.csv")
session.ingest(df)
# ingest empty dataframe to create 0-record input
session.ingest()
# Fail the process with error message
session.fail("Error message")
# Retrieve latest tracking fields
tracking = session.latest_tracking_fields()
# Retrieve connection parameters for the current source
connection_parameters = session.connection_parameters()
# Retrieve custom parameters for the current source
custom_parameters = session.custom_parameters()
# Retrieve system configuration for the current session
system_configuration = session.system_configuration
ParsingSession
The ParsingSession class manages a custom parse process lifecycle.
from dataforge import ParsingSession
# Initialize a session (production use)
session = ParsingSession()
# Initialize a session (optional input_id for testing)
session = ParsingSession(input_id=123)
# Retrieve custom parameters
params = session.custom_parameters()
# Retrieve system configuration for the current session
system_configuration = session.system_configuration
# Get the path of file to be parsed
path = session.file_path
# Run parsing: pass a DataFrame, a function returning a DataFrame or None (0-record file)
session.run(lambda: spark.read.json(session.file_path))
# Fail the process with error message
session.fail("Error message")
PostOutputSession
The PostOutputSession class manages a custom post-output process lifecycle.
from dataforge import PostOutputSession
# Initialize a session (production use)
session = PostOutputSession()
# Initialize a session (optional names for testing)
session = PostOutputSession(output_name="report", output_source_name="my_source", project_name="my_project")
# Get the path of file generated by preceding output process
path = session.file_path()
# Retrieve connection parameters for the current output
connection_parameters = session.connection_parameters()
# Retrieve custom parameters for the current source
custom_parameters = session.custom_parameters()
# Retrieve system configuration for the current session
system_configuration = session.system_configuration
# Run post-output logic: pass a function encapsulating custom code
session.run(lambda: print(f"Uploading file from {path}"))
# Fail the process with error message
session.fail("Error message")
Release files for dataforge-sdk 10.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dataforge_sdk-10.3.1.tar.gz | 13.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dataforge_sdk-10.3.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 34.2 kB
Release files / dataforge_sdk-10.3.1.tar.gz
| Download URL | dataforge_sdk-10.3.1.tar.gz |
|---|---|
| Size | 13.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
34cddac96ceff194c5d67cab0cea74f3e744014ada9a82e2638139f55864fdc7
|
|
BLAKE2b-256 checksum How to use checksums |
08e45b00c07e11d4b5fe59f20621cf6dbec85fd5c362c746cb9a17f0b8347d1e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.21
|
Release files / dataforge_sdk-10.3.1-py3-none-any.whl
| Download URL | dataforge_sdk-10.3.1-py3-none-any.whl |
|---|---|
| Size | 20.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7757f4bb8768f0985577d9fd4b9c580f6b9aecf55fd359e2ddc3fb4282b56fb0
|
|
BLAKE2b-256 checksum How to use checksums |
50e5323d89d45317f1ec1f30b88783ca9583ea9c5eb8d9f811d81a0f40a782ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.21
|
