Skip to main content

pam-python-data-plugin-framework

This repository provides the pam CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.

This README is a practical, step-by-step guide you can follow to create and run a real service.

Current release target: 0.2.0

What you get

  • CLI to initialize a project and scaffold services
  • Service lifecycle contract (start, data input, upload, exit)
  • Temp file and SQLite helpers
  • Managed Report Store V2 with typed Table, Metric, and ApexCharts-compatible builders
  • Named ResultBatchUploader streams for bounded DataFrame uploads
  • A monitoring loop for service timeouts and periodic cleanup

Table of Contents

  1. Prerequisites
  2. Install
  3. Initialize a Project
  4. Create a Service
  5. Understand the Lifecycle
  6. Using Temp Files Correctly
  7. Running the Server
  8. Testing a Service
  9. Configuration
  10. Project Structure
  11. Troubleshooting

Prerequisites

  • Python 3.9+
  • pip and a working virtual environment

Install Create a project folder and a virtual environment.

mkdir my_data_plugin
cd my_data_plugin
python3 -m venv venv
source venv/bin/activate

Install the framework:

pip install pam-python

Initialize a Project This creates a runnable project with templates, including AGENT.md for runtime rules and REPORTS.md for the managed-report public API and copyable examples.

pam init

When requirements.txt already exists, you will be prompted to choose how to proceed:

  • overwrite
  • keep
  • merge

pam init now writes a project baseline requirements.txt. It pins pam-python to the scaffold version and includes core runtime/test dependencies used by the generated project. It does not copy the current machine's pip freeze.


Create a Service Generate a service scaffold. Do not hand-create service templates.

pam new service rfm_segment

This creates a new folder (e.g. rfm_segment/) with:

  • a service class (RfmSegmentSvc.py)
  • functions.py for your logic
  • service.yaml for registration
  • a test file

Understand the Lifecycle The runtime calls your service in two main phases.

  1. on_start
  • Called once at the beginning
  • Read parameters from self.request.runtime_parameters
  • Should return quickly (start a thread for long work)
  1. on_data_input
  • Called when CDP sends input files
  • req.input_files contains ordered CSV files
  • Should also return quickly (use a thread if needed)

When your service is done:

  • Upload ordinary CDP result rows through ResultBatchUploader or _upload_result(...)
  • Publish managed reports through self.reports; do not build report JSON manually
  • Call self._exit() to signal completion

Using Temp Files Correctly Temp storage is managed by the framework. Do not delete temp files manually.

Standard helpers:

  • TempfileUtils.get_temp_path_for_service(self, self.service_name)
  • TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)

Notes:

  • get_temp_path_for_service(...) returns a directory path without a trailing slash.
  • The temp path includes date/service/token in this structure: TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>

Uploading Results in Batches If your service produces too many rows (or too few per event), use the batch uploader to handle chunking and flushing automatically.

Recommended usage:

from pam.result_batch_uploader import ResultBatchUploader

batch_uploader = ResultBatchUploader(self, batch_size=50000)
batch_uploader.upload(df, name="main")
batch_uploader.flush()
status = batch_uploader.get_status()

Notes:

  • request.runtime_parameters["batch_size"] overrides the constructor default when valid.
  • name separates result streams and enforces one stable column schema per stream.
  • options may be passed to upload(...) and are forwarded to _upload_result.
  • Complete batches upload immediately; flush() sends the remaining rows.
  • get_status() returns buffered rows, uploaded rows, and uploaded batch counts per stream.

Managed Reports (Report Store V2)

Managed reports are static facts stored in framework-owned SQLite. Plugin code selects a typed builder, writes source-derived DAY, MONTH, or NONE facts, and publishes only after the full calculation succeeds.

fresh = self.request.get_runtime_bool("fresh", default=False)
store = (
    self.reports.create("reports.sqlite")
    if fresh
    else self.reports.open("reports.sqlite")
)

# Choose the builder agreed in the customer requirement.
# See generated REPORTS.md for every supported declaration and write API.

Important rules:

  • Use only typed builders documented in REPORTS.md; there is no raw-report API.
  • open() resumes the remote file or creates it only after an authoritative 404.
  • Network/5xx download failures never create a replacement store.
  • Use create() only for a human-requested fresh restart.
  • Delete a partition, write its recalculated facts, then call store.publish() once.
  • Low-level _request_sqlite/_upload_sqlite remain for custom plugin state, not reports.

Running the Server The generated main.py runs the Flask server.

python main.py

By default it binds to 0.0.0.0:8000. You can override with:

export SERVER_HOST=0.0.0.0
export SERVER_PORT=8000

Testing a Service Run tests through the generated script for your current shell. This keeps the active terminal's Python environment and project context instead of launching test discovery through the pam CLI.

macOS/Linux:

./run_unit_test.sh rfm_segment

Windows PowerShell:

.\run_unit_test.ps1 rfm_segment

Windows Command Prompt:

run_unit_test.bat rfm_segment

Place custom tests in the service folder and name them test_<service>.py.


Configuration Environment variables you can set:

  • SERVER_HOST
  • SERVER_PORT
  • TEMP_BASE_PATH (default /app/data)
  • TEMP_DATASOURCE_PATH (default /app/data/data_sources)
  • TEMP_CLEAN_DAYS (default 10)
  • TEMP_CLEAN_INTERVAL_HOURS (default 6, set empty to disable periodic cleanup)

Project Structure After pam init and one service:

.
├── main.py
├── AGENT.md
├── REPORTS.md
├── Dockerfile
├── rfm_segment/
│   ├── RfmSegmentSvc.py
│   ├── functions.py
│   ├── service.yaml
│   └── test_rfm_segment.py
├── requirements.txt
└── run_unit_test.sh

Troubleshooting

  • If pam command is missing, ensure your virtualenv is activated.
  • If pam new service fails, confirm the service name is provided.
  • If temp cleanup is too frequent or too slow, adjust TEMP_CLEAN_INTERVAL_HOURS and TEMP_CLEAN_DAYS.

Next Steps

  • Implement your logic in functions.py.
  • Wire it into on_start and on_data_input in your service class.
  • Use the temp utilities to write intermediate files.
  • Use ResultBatchUploader to return ordinary output to CDP.
  • Read REPORTS.md before implementing a managed report.

Metadata

Release files for pam-python 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pam-python 0.2.0
File Size Uploaded Provenance Actions
pam_python-0.2.0.tar.gz 93.7 kB Without provenance Details for pam_python-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pam-python 0.2.0
File Interpreter ABI Platform Actions
pam_python-0.2.0-py3-none-any.whl Python 3 none any Details for pam_python-0.2.0-py3-none-any.whl

Total release size: 194.7 kB

Release files / pam_python-0.2.0.tar.gz

Download URL pam_python-0.2.0.tar.gz
Size 93.7 kB
Tags Source
SHA-256 checksum
How to use checksums
9e9a95b8199ec0b8911c54f672d93cc6808e4cf958524b091e1e70e0a68e9675
BLAKE2b-256 checksum
How to use checksums
f753115fd3661a25577a1aa546ec0ed30b581af47890580819dff1c91ab2acbc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / pam_python-0.2.0-py3-none-any.whl

Download URL pam_python-0.2.0-py3-none-any.whl
Size 101.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
92172a7d46ed95355e21df5e8ebd8fb5b0db9a25564b633a3ca28a7a7f39076d
BLAKE2b-256 checksum
How to use checksums
91d75509a1f62a20f7c5d2f6a0706cc3119785ac48664552acccdbb74f9c8e66
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Security for pam-python 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page