pam-python-data-plugin-framework
This repository provides the pam CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.
This README is a practical, step-by-step guide you can follow to create and run a real service.
Current release target: 0.2.0
What you get
- CLI to initialize a project and scaffold services
- Service lifecycle contract (start, data input, upload, exit)
- Temp file and SQLite helpers
- Managed Report Store V2 with typed Table, Metric, and ApexCharts-compatible builders
- Named
ResultBatchUploaderstreams for bounded DataFrame uploads - A monitoring loop for service timeouts and periodic cleanup
Table of Contents
- Prerequisites
- Install
- Initialize a Project
- Create a Service
- Understand the Lifecycle
- Using Temp Files Correctly
- Running the Server
- Testing a Service
- Configuration
- Project Structure
- Troubleshooting
Prerequisites
- Python 3.9+
pipand a working virtual environment
Install Create a project folder and a virtual environment.
mkdir my_data_plugin
cd my_data_plugin
python3 -m venv venv
source venv/bin/activate
Install the framework:
pip install pam-python
Initialize a Project
This creates a runnable project with templates, including AGENT.md for runtime
rules and REPORTS.md for the managed-report public API and copyable examples.
pam init
When requirements.txt already exists, you will be prompted to choose how to proceed:
- overwrite
- keep
- merge
pam init now writes a project baseline requirements.txt. It pins pam-python to the scaffold version and includes core runtime/test dependencies used by the generated project. It does not copy the current machine's pip freeze.
Create a Service Generate a service scaffold. Do not hand-create service templates.
pam new service rfm_segment
This creates a new folder (e.g. rfm_segment/) with:
- a service class (
RfmSegmentSvc.py) functions.pyfor your logicservice.yamlfor registration- a test file
Understand the Lifecycle The runtime calls your service in two main phases.
on_start
- Called once at the beginning
- Read parameters from
self.request.runtime_parameters - Should return quickly (start a thread for long work)
on_data_input
- Called when CDP sends input files
req.input_filescontains ordered CSV files- Should also return quickly (use a thread if needed)
When your service is done:
- Upload ordinary CDP result rows through
ResultBatchUploaderor_upload_result(...) - Publish managed reports through
self.reports; do not build report JSON manually - Call
self._exit()to signal completion
Using Temp Files Correctly Temp storage is managed by the framework. Do not delete temp files manually.
Standard helpers:
TempfileUtils.get_temp_path_for_service(self, self.service_name)TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)
Notes:
get_temp_path_for_service(...)returns a directory path without a trailing slash.- The temp path includes date/service/token in this structure:
TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>
Uploading Results in Batches If your service produces too many rows (or too few per event), use the batch uploader to handle chunking and flushing automatically.
Recommended usage:
from pam.result_batch_uploader import ResultBatchUploader
batch_uploader = ResultBatchUploader(self, batch_size=50000)
batch_uploader.upload(df, name="main")
batch_uploader.flush()
status = batch_uploader.get_status()
Notes:
request.runtime_parameters["batch_size"]overrides the constructor default when valid.nameseparates result streams and enforces one stable column schema per stream.optionsmay be passed toupload(...)and are forwarded to_upload_result.- Complete batches upload immediately;
flush()sends the remaining rows. get_status()returns buffered rows, uploaded rows, and uploaded batch counts per stream.
Managed Reports (Report Store V2)
Managed reports are static facts stored in framework-owned SQLite. Plugin code
selects a typed builder, writes source-derived DAY, MONTH, or NONE facts, and
publishes only after the full calculation succeeds.
fresh = self.request.get_runtime_bool("fresh", default=False)
store = (
self.reports.create("reports.sqlite")
if fresh
else self.reports.open("reports.sqlite")
)
# Choose the builder agreed in the customer requirement.
# See generated REPORTS.md for every supported declaration and write API.
Important rules:
- Use only typed builders documented in
REPORTS.md; there is no raw-report API. open()resumes the remote file or creates it only after an authoritative 404.- Network/5xx download failures never create a replacement store.
- Use
create()only for a human-requested fresh restart. - Delete a partition, write its recalculated facts, then call
store.publish()once. - Low-level
_request_sqlite/_upload_sqliteremain for custom plugin state, not reports.
Running the Server
The generated main.py runs the Flask server.
python main.py
By default it binds to 0.0.0.0:8000. You can override with:
export SERVER_HOST=0.0.0.0
export SERVER_PORT=8000
Testing a Service
Run tests through the generated script for your current shell. This keeps the
active terminal's Python environment and project context instead of launching
test discovery through the pam CLI.
macOS/Linux:
./run_unit_test.sh rfm_segment
Windows PowerShell:
.\run_unit_test.ps1 rfm_segment
Windows Command Prompt:
run_unit_test.bat rfm_segment
Place custom tests in the service folder and name them test_<service>.py.
Configuration Environment variables you can set:
SERVER_HOSTSERVER_PORTTEMP_BASE_PATH(default/app/data)TEMP_DATASOURCE_PATH(default/app/data/data_sources)TEMP_CLEAN_DAYS(default10)TEMP_CLEAN_INTERVAL_HOURS(default6, set empty to disable periodic cleanup)
Project Structure
After pam init and one service:
.
├── main.py
├── AGENT.md
├── REPORTS.md
├── Dockerfile
├── rfm_segment/
│ ├── RfmSegmentSvc.py
│ ├── functions.py
│ ├── service.yaml
│ └── test_rfm_segment.py
├── requirements.txt
└── run_unit_test.sh
Troubleshooting
- If
pamcommand is missing, ensure your virtualenv is activated. - If
pam new servicefails, confirm the service name is provided. - If temp cleanup is too frequent or too slow, adjust
TEMP_CLEAN_INTERVAL_HOURSandTEMP_CLEAN_DAYS.
Next Steps
- Implement your logic in
functions.py. - Wire it into
on_startandon_data_inputin your service class. - Use the temp utilities to write intermediate files.
- Use
ResultBatchUploaderto return ordinary output to CDP. - Read
REPORTS.mdbefore implementing a managed report.
Metadata
Release files for pam-python 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pam_python-0.2.0.tar.gz | 93.7 kB | Without provenance | Details for pam_python-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pam_python-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for pam_python-0.2.0-py3-none-any.whl |
Total release size: 194.7 kB
Release files / pam_python-0.2.0.tar.gz
| Download URL | pam_python-0.2.0.tar.gz |
|---|---|
| Size | 93.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9e9a95b8199ec0b8911c54f672d93cc6808e4cf958524b091e1e70e0a68e9675
|
|
BLAKE2b-256 checksum How to use checksums |
f753115fd3661a25577a1aa546ec0ed30b581af47890580819dff1c91ab2acbc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Release files / pam_python-0.2.0-py3-none-any.whl
| Download URL | pam_python-0.2.0-py3-none-any.whl |
|---|---|
| Size | 101.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
92172a7d46ed95355e21df5e8ebd8fb5b0db9a25564b633a3ca28a7a7f39076d
|
|
BLAKE2b-256 checksum How to use checksums |
91d75509a1f62a20f7c5d2f6a0706cc3119785ac48664552acccdbb74f9c8e66
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Security for pam-python 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.