pam-python-data-plugin-framework
This repository provides the pam CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.
This README is a practical, step-by-step guide you can follow to create and run a real service.
What you get
- CLI to initialize a project and scaffold services
- Service lifecycle contract (start, data input, upload, exit)
- Temp file and SQLite helpers
- Managed Report Store V2 with typed Table, Metric, and ApexCharts-compatible builders
- Named
ResultBatchUploaderstreams for bounded DataFrame uploads - A monitoring loop for service timeouts and periodic cleanup
Table of Contents
- Prerequisites
- Install
- Initialize a Project
- Create a Service
- Understand the Lifecycle
- Using Temp Files Correctly
- Running the Server
- Testing a Service
- Configuration
- Project Structure
- Troubleshooting
Prerequisites
- Python 3.12
uv(recommended) orpip
Install Recommended installation:
mkdir my_data_plugin
cd my_data_plugin
uv tool install pam-python
Pip remains supported:
python3 -m venv venv
source venv/bin/activate
python -m pip install pam-python
Initialize a Project
This creates a runnable project with templates, including AGENT.md for runtime
rules and REPORTS.md for the managed-report public API and copyable examples.
pam init
pam init asks you to select uv (default) or pip. A uv project contains
pyproject.toml, .python-version, and uv.lock; a pip project contains
requirements.txt. The two templates are independent and are never mixed.
Initialization only supports a new project directory and will not migrate or overwrite an existing project.
Create a Service
pam new service rfm_segment
The generator asks you to select Polars (default) or Pandas. Polars services use
lazy Parquet processing. Pandas services also receive Parquet and install
pandas plus pyarrow. The selected direct dependencies are added to the
project automatically.
This creates a new folder (e.g. rfm_segment/) with:
- a service class (
RfmSegmentSvc.py) functions.pyfor your logicservice.yamlfor registration- a test file
Understand the Lifecycle The runtime calls your service in two main phases.
on_start
- Called once at the beginning
- Read parameters from
self.request.runtime_parameters - Should return quickly (start a thread for long work)
on_data_input
- Called when CDP sends input files
req.input_filescontains ordered Parquet filesreq.file_formatis the typed format sent by PAM- Should also return quickly (use a thread if needed)
When your service is done:
- Upload ordinary CDP result rows through
ResultBatchUploaderor_upload_result(...) - Publish managed reports through
self.reports; do not build report JSON manually - Call
self._exit()to signal completion
Using Temp Files Correctly Temp storage is managed by the framework. Do not delete temp files manually.
Standard helpers:
TempfileUtils.get_temp_path_for_service(self, self.service_name)TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)
Notes:
get_temp_path_for_service(...)returns a directory path without a trailing slash.- The temp path includes date/service/token in this structure:
TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>
Uploading Results in Batches
Polars templates pass LazyFrame results directly to _upload_result(...), which
streams them to the CSV upload boundary. Do not collect or convert to Pandas first.
For eager Polars or Pandas DataFrames, use the batch uploader to handle chunking and flushing automatically.
Recommended usage:
from pam.result_batch_uploader import ResultBatchUploader
batch_uploader = ResultBatchUploader(self, batch_size=50000)
batch_uploader.upload(df, name="main")
batch_uploader.flush()
status = batch_uploader.get_status()
Notes:
request.runtime_parameters["batch_size"]overrides the constructor default when valid.nameseparates result streams and enforces one stable column schema per stream.optionsmay be passed toupload(...)and are forwarded to_upload_result.- Complete batches upload immediately;
flush()sends the remaining rows. get_status()returns buffered rows, uploaded rows, and uploaded batch counts per stream.ResultBatchUploaderintentionally rejects PolarsLazyFrame; upload it directly.
Managed Reports (Report Store V2)
Managed reports are static facts stored in framework-owned SQLite. Plugin code
selects a typed builder, writes source-derived DAY, MONTH, or NONE facts, and
publishes only after the full calculation succeeds.
fresh = self.request.get_runtime_bool("fresh", default=False)
store = (
self.reports.create("reports.sqlite")
if fresh
else self.reports.open("reports.sqlite")
)
# Choose the builder agreed in the customer requirement.
# See generated REPORTS.md for every supported declaration and write API.
Important rules:
- Use only typed builders documented in
REPORTS.md; there is no raw-report API. open()resumes the remote file or creates it only after an authoritative 404.- Network/5xx download failures never create a replacement store.
- Use
create()only for a human-requested fresh restart. - Delete a partition, write its recalculated facts, then call
store.publish()once. - Low-level
_request_sqlite/_upload_sqliteremain for custom plugin state, not reports.
Running the Server
The generated main.py runs the Flask server.
uv run python main.py # uv project
python main.py # pip project
By default it binds to 0.0.0.0:8000. You can override with:
export SERVER_HOST=0.0.0.0
export SERVER_PORT=8000
Testing a Service For uv projects, use one cross-platform command:
uv run python -m unittest discover -s rfm_segment -p "test_*.py"
Pip projects include shell-specific scripts:
macOS/Linux:
./run_unit_test.sh rfm_segment
Windows PowerShell:
.\run_unit_test.ps1 rfm_segment
Windows Command Prompt:
run_unit_test.bat rfm_segment
Place custom tests in the service folder and name them test_<service>.py.
Configuration Environment variables you can set:
SERVER_HOSTSERVER_PORTTEMP_BASE_PATH(default/app/data)TEMP_DATASOURCE_PATH(default/app/data/data_sources)TEMP_CLEAN_DAYS(default10)TEMP_CLEAN_INTERVAL_HOURS(default6, set empty to disable periodic cleanup)
Project Structure
After pam init and one service:
.
├── main.py
├── AGENT.md
├── REPORTS.md
├── Dockerfile
├── pyproject.toml
├── uv.lock
├── .python-version
├── rfm_segment/
│ ├── RfmSegmentSvc.py
│ ├── functions.py
│ ├── service.yaml
│ └── test_rfm_segment.py
└── .pam-project
Troubleshooting
- If
pamis missing, verify the uv tool installation or activate the pip virtualenv. - If uv initialization fails, verify
uvis onPATHand package indexes are reachable. - If
pam new servicefails, confirm the service name is provided. - If temp cleanup is too frequent or too slow, adjust
TEMP_CLEAN_INTERVAL_HOURSandTEMP_CLEAN_DAYS.
Next Steps
- Implement your logic in
functions.py. - Wire it into
on_startandon_data_inputin your service class. - Use the temp utilities to write intermediate files.
- Keep Polars transformations lazy and upload the resulting
LazyFramedirectly. - Use
ResultBatchUploaderonly for eager Polars/Pandas frames that need row batching. - Read
REPORTS.mdbefore implementing a managed report.
Metadata
Release files for pam-python 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pam_python-0.2.1.tar.gz | 97.5 kB | Without provenance | Details for pam_python-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pam_python-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for pam_python-0.2.1-py3-none-any.whl |
Total release size: 203.9 kB
Release files / pam_python-0.2.1.tar.gz
| Download URL | pam_python-0.2.1.tar.gz |
|---|---|
| Size | 97.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9319b2bd57d344cbd5ae811a0b7e709820e4f7535972a1de9a51a223d9a8dc46
|
|
BLAKE2b-256 checksum How to use checksums |
d14286b01e30935e069d3987b331459ef254048728b2948ac38decadfc051a33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pam_python-0.2.1-py3-none-any.whl
| Download URL | pam_python-0.2.1-py3-none-any.whl |
|---|---|
| Size | 106.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
08436b21eb8a0b9a99caf0db2ee088cd3039820118189534590cfe4ffe371be9
|
|
BLAKE2b-256 checksum How to use checksums |
d034034637fb480f94203f16e31281033d69faab831c72eb5c1850095c502567
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pam-python 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.