pam-python-data-plugin-framework
This repository provides the pam CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.
This README is a practical, step-by-step guide you can follow to create and run a real service.
What you get
- CLI to initialize a project and scaffold services
- Service lifecycle contract (start, data input, upload, exit)
- Temp file and SQLite helpers
- Managed Report Store V2 with typed Table, Metric, and ApexCharts-compatible builders
- Named
ResultBatchUploaderstreams for bounded DataFrame uploads - A monitoring loop for service timeouts and periodic cleanup
Table of Contents
- Prerequisites
- Install
- Initialize a Project
- Create a Service
- Understand the Lifecycle
- Using Temp Files Correctly
- Running the Server
- Testing a Service
- Configuration
- Project Structure
- Troubleshooting
Prerequisites
- Python 3.12
uv(recommended) orpip
Create a uv project
Install PAM inside each project instead of installing the CLI globally:
mkdir my_data_plugin
cd my_data_plugin
uv init --python 3.12
uv add pam-python
uv run pam init
pam init recognizes a fresh uv project, keeps its uv configuration, adds the
Flask and Gunicorn runtime dependencies, and adds the PAM scaffold. It refuses
projects whose generated source files have already been edited.
Pip remains supported for new projects that do not use uv:
mkdir my_data_plugin
cd my_data_plugin
python3 -m venv venv
source venv/bin/activate
python -m pip install pam-python
pam init
When run in an empty directory, pam init still asks whether to create a uv or
pip project. PAM never migrates an existing application.
Create a Service
uv run pam new service rfm_segment
The generator asks you to select Polars (default) or Pandas. Polars services use
lazy Parquet processing. Pandas services also receive Parquet and install
pandas plus pyarrow. The selected direct dependencies are added to the
project automatically.
This creates a new folder (e.g. rfm_segment/) with:
- a service class (
RfmSegmentSvc.py) functions.pyfor your logicservice.yamlfor registration- a test file
Understand the Lifecycle The runtime calls your service in two main phases.
on_start
- Called once at the beginning
- Read parameters from
self.request.runtime_parameters - Should return quickly (start a thread for long work)
on_data_input
- Called when CDP sends input files
req.input_filesis the ordered positional data-source contract configured in PAMreq.file_formatis the typed format sent by PAM- Should also return quickly (use a thread if needed)
For customer-aligned Data Plugin Input V2, use _request_data_v2(...) and
req.data_input_v2. The framework verifies the manifest and every physical
part before plugin code sees them, and provides lazy Polars scans for CSV and
Parquet (plus streaming row iterators for CSV).
See Data Plugin Input V2.
New Pandas and Polars service projects are generated on the V2 lane by default;
legacy source can continue using V1 unchanged. For materialized Pandas frames,
use data.read_population_pandas() and either
data.dataset_at(position).read_pandas() or
data.dataset_named(name).read_pandas().
When your service is done:
- Upload ordinary CDP result rows through
ResultBatchUploader - Publish managed reports through
self.reports; do not build report JSON manually - Call
self._exit()to signal completion
Using Temp Files Correctly
Temp storage is managed by the framework. Do not delete individual files
manually. V2 code may release one completed page with
req.data_input_v2.cleanup(); otherwise scheduled cleanup retains ownership.
Standard helpers:
TempfileUtils.get_temp_path_for_service(self, self.service_name)TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)TempfileUtils.remove_dataset_bundle(path)for one framework-owned page
Notes:
get_temp_path_for_service(...)returns a directory path without a trailing slash.- The temp path includes date/service/token in this structure:
TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>
Uploading Results in Batches
Polars LazyFrame/DataFrame and Pandas DataFrame results all pass through
ResultBatchUploader. It streams lazy results in bounded chunks, buffers incomplete
batches across calls, writes complete batches as CSV, and uploads them through PAM.
The framework does not prescribe page-local/global computation or intermediate
storage; submit rows when your business result is ready.
Recommended usage:
from pam.result_batch_uploader import ResultBatchUploader
batch_uploader = ResultBatchUploader(self, batch_size=50000)
batch_uploader.upload(df, name="main")
batch_uploader.flush()
batch_uploader.wait_for_uploader()
status = batch_uploader.get_status()
Notes:
request.runtime_parameters["batch_size"]overrides the constructor default when valid.nameseparates result streams and enforces one stable column schema per stream.optionsmay be passed toupload(...)and are forwarded to_upload_result.- Uploads run in a bounded background queue so processing can continue while PAM receives CSV batches.
flush()closes input and queues the final remainder; it does not wait for network completion.- Always call
wait_for_uploader()afterflush()and before_exit(). - Failed uploads retry with exponential backoff, then log/record the failed batch and continue.
get_status()includes buffered, uploaded, failed, and retried counts per stream.- Polars
LazyFrameis supported without collecting the complete result at once.
Managed Reports (Report Store V2)
Managed reports are static facts stored in framework-owned SQLite. Plugin code
selects a typed builder, writes source-derived DAY, MONTH, or NONE facts, and
publishes only after the full calculation succeeds.
fresh = self.request.get_runtime_bool("fresh", default=False)
store = (
self.reports.create("reports.sqlite")
if fresh
else self.reports.open("reports.sqlite")
)
# Choose the builder agreed in the customer requirement.
# See generated REPORTS.md for every supported declaration and write API.
Important rules:
- Use only typed builders documented in
REPORTS.md; there is no raw-report API. open()resumes the remote file or creates it only after an authoritative 404.- Network/5xx download failures never create a replacement store.
- Use
create()only for a human-requested fresh restart. - Delete a partition, write its recalculated facts, then call
store.publish()once. - Low-level
_request_sqlite/_upload_sqliteremain for custom plugin state, not reports.
Running the Server
The generated main.py runs the Flask server.
uv run python main.py # uv project
python main.py # pip project
By default it binds to 0.0.0.0:8000. You can override with:
export SERVER_HOST=0.0.0.0
export SERVER_PORT=8000
Testing a Service For uv projects, use one cross-platform command:
uv run python -m unittest discover -s rfm_segment -p "test_*.py"
Pip projects include shell-specific scripts:
macOS/Linux:
./run_unit_test.sh rfm_segment
Windows PowerShell:
.\run_unit_test.ps1 rfm_segment
Windows Command Prompt:
run_unit_test.bat rfm_segment
Place custom tests in the service folder and name them test_<service>.py.
Configuration Environment variables you can set:
SERVER_HOSTSERVER_PORTTEMP_BASE_PATH(default/app/data)TEMP_DATASOURCE_PATH(default/app/data/data_sources)TEMP_CLEAN_DAYS(default10)TEMP_CLEAN_INTERVAL_HOURS(default6, set empty to disable periodic cleanup)PAM_DATA_INPUT_V2_MAX_ZIP_BYTES(default1 GiB)PAM_DATA_INPUT_V2_MAX_UNCOMPRESSED_BYTES(default2 GiB)PAM_DATA_INPUT_V2_MAX_FILES(default10000)
Project Structure
After pam init and one service:
.
├── main.py
├── AGENT.md
├── REPORTS.md
├── Dockerfile
├── pyproject.toml
├── uv.lock
├── .python-version
├── rfm_segment/
│ ├── RfmSegmentSvc.py
│ ├── functions.py
│ ├── service.yaml
│ └── test_rfm_segment.py
└── .pam-project
Troubleshooting
- If
pamis missing, verify the uv tool installation or activate the pip virtualenv. - If uv initialization fails, verify
uvis onPATHand package indexes are reachable. - If
pam new servicefails, confirm the service name is provided. - If temp cleanup is too frequent or too slow, adjust
TEMP_CLEAN_INTERVAL_HOURSandTEMP_CLEAN_DAYS.
Next Steps
- Implement your logic in
functions.py. - Wire it into
on_startandon_data_inputin your service class. - Use the temp utilities to write intermediate files.
- Keep Polars transformations lazy and submit final results through
ResultBatchUploader. - Read
REPORTS.mdbefore implementing a managed report.
Metadata
Release files for pam-python 0.2.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pam_python-0.2.20.tar.gz | 142.6 kB | Without provenance | Details for pam_python-0.2.20.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pam_python-0.2.20-py3-none-any.whl | Python 3 | none | any | Details for pam_python-0.2.20-py3-none-any.whl |
Total release size: 284.1 kB
Release files / pam_python-0.2.20.tar.gz
| Download URL | pam_python-0.2.20.tar.gz |
|---|---|
| Size | 142.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9488765aadc93f9d0f3dc668541425b235834cb0d1940cdc588a7881331d0210
|
|
BLAKE2b-256 checksum How to use checksums |
931929801fe5fb866a6a20c0a20829bace0861f99c5c2cab9051fc4871a1fd4d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pam_python-0.2.20-py3-none-any.whl
| Download URL | pam_python-0.2.20-py3-none-any.whl |
|---|---|
| Size | 141.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1e4e2669f9d68ba79aaafaf5becf6d164e60d22937b4f4c4c47fb7ffc9f41d30
|
|
BLAKE2b-256 checksum How to use checksums |
089d520ea609da0b2da52e4532365bfa70c6c737876604f94d8b21add8530d9f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pam-python 0.2.20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.