Skip to main content

pam-python-data-plugin-framework

This repository provides the pam CLI and runtime framework to build Data Plugin services for PAM Real CDP. It generates a ready-to-run project, standardizes service lifecycle, and handles common tasks like input handling, temp storage, uploads, and service monitoring.

This README is a practical, step-by-step guide you can follow to create and run a real service.

What you get

  • CLI to initialize a project and scaffold services
  • Service lifecycle contract (start, data input, upload, exit)
  • Temp file and SQLite helpers
  • Managed Report Store V2 with typed Table, Metric, and ApexCharts-compatible builders
  • Named ResultBatchUploader streams for bounded DataFrame uploads
  • A monitoring loop for service timeouts and periodic cleanup

Table of Contents

  1. Prerequisites
  2. Install
  3. Initialize a Project
  4. Create a Service
  5. Understand the Lifecycle
  6. Using Temp Files Correctly
  7. Running the Server
  8. Testing a Service
  9. Configuration
  10. Project Structure
  11. Troubleshooting

Prerequisites

  • Python 3.12
  • uv (recommended) or pip

Create a uv project

Install PAM inside each project instead of installing the CLI globally:

mkdir my_data_plugin
cd my_data_plugin
uv init --python 3.12
uv add pam-python
uv run pam init

pam init recognizes a fresh uv project, keeps its uv configuration, adds the Flask and Gunicorn runtime dependencies, and adds the PAM scaffold. It refuses projects whose generated source files have already been edited.

Pip remains supported for new projects that do not use uv:

mkdir my_data_plugin
cd my_data_plugin
python3 -m venv venv
source venv/bin/activate
python -m pip install pam-python
pam init

When run in an empty directory, pam init still asks whether to create a uv or pip project. PAM never migrates an existing application.


Create a Service

uv run pam new service rfm_segment

The generator asks you to select Polars (default) or Pandas. Polars services use lazy Parquet processing. Pandas services also receive Parquet and install pandas plus pyarrow. The selected direct dependencies are added to the project automatically.

This creates a new folder (e.g. rfm_segment/) with:

  • a service class (RfmSegmentSvc.py)
  • functions.py for your logic
  • service.yaml for registration
  • a test file

Understand the Lifecycle The runtime calls your service in two main phases.

  1. on_start
  • Called once at the beginning
  • Read parameters from self.request.runtime_parameters
  • Should return quickly (start a thread for long work)
  1. on_data_input
  • Called when CDP sends input files
  • req.input_files is the ordered positional data-source contract configured in PAM
  • req.file_format is the typed format sent by PAM
  • Should also return quickly (use a thread if needed)

For customer-aligned Data Plugin Input V2, use _request_data_v2(...) and req.data_input_v2. The framework verifies the manifest and every physical part before plugin code sees them, and provides lazy Polars scans for CSV and Parquet (plus streaming row iterators for CSV). See Data Plugin Input V2.

New Pandas and Polars service projects are generated on the V2 lane by default; legacy source can continue using V1 unchanged. For materialized Pandas frames, use data.read_population_pandas() and either data.dataset_at(position).read_pandas() or data.dataset_named(name).read_pandas().

When your service is done:

  • Upload ordinary CDP result rows through ResultBatchUploader
  • Publish managed reports through self.reports; do not build report JSON manually
  • Call self._exit() to signal completion

Using Temp Files Correctly Temp storage is managed by the framework. Do not delete individual files manually. V2 code may release one completed page with req.data_input_v2.cleanup(); otherwise scheduled cleanup retains ownership.

Standard helpers:

  • TempfileUtils.get_temp_path_for_service(self, self.service_name)
  • TempfileUtils.get_temp_file_name_for_service(self, self.service_name, prefix, extension)
  • TempfileUtils.remove_dataset_bundle(path) for one framework-owned page

Notes:

  • get_temp_path_for_service(...) returns a directory path without a trailing slash.
  • The temp path includes date/service/token in this structure: TEMP_DATASOURCE_PATH/YYYY_MM_DD/<service>/<token>

Uploading Results in Batches Polars LazyFrame/DataFrame and Pandas DataFrame results all pass through ResultBatchUploader. It streams lazy results in bounded chunks, buffers incomplete batches across calls, writes complete batches as CSV, and uploads them through PAM. The framework does not prescribe page-local/global computation or intermediate storage; submit rows when your business result is ready.

Recommended usage:

from pam.result_batch_uploader import ResultBatchUploader

batch_uploader = ResultBatchUploader(self, batch_size=50000)
batch_uploader.upload(df, name="main")
batch_uploader.flush()
batch_uploader.wait_for_uploader()
status = batch_uploader.get_status()

Notes:

  • request.runtime_parameters["batch_size"] overrides the constructor default when valid.
  • name separates result streams and enforces one stable column schema per stream.
  • options may be passed to upload(...) and are forwarded to _upload_result.
  • Uploads run in a bounded background queue so processing can continue while PAM receives CSV batches.
  • flush() closes input and queues the final remainder; it does not wait for network completion.
  • Always call wait_for_uploader() after flush() and before _exit().
  • Failed uploads retry with exponential backoff, then log/record the failed batch and continue.
  • get_status() includes buffered, uploaded, failed, and retried counts per stream.
  • Polars LazyFrame is supported without collecting the complete result at once.

Managed Reports (Report Store V2)

Managed reports are static facts stored in framework-owned SQLite. Plugin code selects a typed builder, writes source-derived DAY, MONTH, or NONE facts, and publishes only after the full calculation succeeds.

fresh = self.request.get_runtime_bool("fresh", default=False)
store = (
    self.reports.create("reports.sqlite")
    if fresh
    else self.reports.open("reports.sqlite")
)

# Choose the builder agreed in the customer requirement.
# See generated REPORTS.md for every supported declaration and write API.

Important rules:

  • Use only typed builders documented in REPORTS.md; there is no raw-report API.
  • open() resumes the remote file or creates it only after an authoritative 404.
  • Network/5xx download failures never create a replacement store.
  • Use create() only for a human-requested fresh restart.
  • Delete a partition, write its recalculated facts, then call store.publish() once.
  • Low-level _request_sqlite/_upload_sqlite remain for custom plugin state, not reports.

Running the Server The generated main.py runs the Flask server.

uv run python main.py  # uv project
python main.py         # pip project

By default it binds to 0.0.0.0:8000. You can override with:

export SERVER_HOST=0.0.0.0
export SERVER_PORT=8000

Testing a Service For uv projects, use one cross-platform command:

uv run python -m unittest discover -s rfm_segment -p "test_*.py"

Pip projects include shell-specific scripts:

macOS/Linux:

./run_unit_test.sh rfm_segment

Windows PowerShell:

.\run_unit_test.ps1 rfm_segment

Windows Command Prompt:

run_unit_test.bat rfm_segment

Place custom tests in the service folder and name them test_<service>.py.


Configuration Environment variables you can set:

  • SERVER_HOST
  • SERVER_PORT
  • TEMP_BASE_PATH (default /app/data)
  • TEMP_DATASOURCE_PATH (default /app/data/data_sources)
  • TEMP_CLEAN_DAYS (default 10)
  • TEMP_CLEAN_INTERVAL_HOURS (default 6, set empty to disable periodic cleanup)
  • PAM_DATA_INPUT_V2_MAX_ZIP_BYTES (default 1 GiB)
  • PAM_DATA_INPUT_V2_MAX_UNCOMPRESSED_BYTES (default 2 GiB)
  • PAM_DATA_INPUT_V2_MAX_FILES (default 10000)

Project Structure After pam init and one service:

.
├── main.py
├── AGENT.md
├── REPORTS.md
├── Dockerfile
├── pyproject.toml
├── uv.lock
├── .python-version
├── rfm_segment/
│   ├── RfmSegmentSvc.py
│   ├── functions.py
│   ├── service.yaml
│   └── test_rfm_segment.py
└── .pam-project

Troubleshooting

  • If pam is missing, verify the uv tool installation or activate the pip virtualenv.
  • If uv initialization fails, verify uv is on PATH and package indexes are reachable.
  • If pam new service fails, confirm the service name is provided.
  • If temp cleanup is too frequent or too slow, adjust TEMP_CLEAN_INTERVAL_HOURS and TEMP_CLEAN_DAYS.

Next Steps

  • Implement your logic in functions.py.
  • Wire it into on_start and on_data_input in your service class.
  • Use the temp utilities to write intermediate files.
  • Keep Polars transformations lazy and submit final results through ResultBatchUploader.
  • Read REPORTS.md before implementing a managed report.

Metadata

Release files for pam-python 0.2.17

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pam-python 0.2.17
File Size Uploaded Provenance Actions
pam_python-0.2.17.tar.gz 138.9 kB Without provenance Details for pam_python-0.2.17.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pam-python 0.2.17
File Interpreter ABI Platform Actions
pam_python-0.2.17-py3-none-any.whl Python 3 none any Details for pam_python-0.2.17-py3-none-any.whl

Total release size: 279.0 kB

Release files / pam_python-0.2.17.tar.gz

Download URL pam_python-0.2.17.tar.gz
Size 138.9 kB
Tags Source
SHA-256 checksum
How to use checksums
04eee3627279fb39799c8b3a7b780f442d842197bcbd9d68a2b85603f0706eb7
BLAKE2b-256 checksum
How to use checksums
1c4e1479ad74099df8437932f62d7a52a835cb5b3688f72cbf1b747a87497a92
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / pam_python-0.2.17-py3-none-any.whl

Download URL pam_python-0.2.17-py3-none-any.whl
Size 140.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dedaf0c2c1f74bfb69cd7d7125c4795cc0e8d12694755f0f9a57a11903e4e175
BLAKE2b-256 checksum
How to use checksums
ce966d3abd3689b4d2864335e3bc6e92217362b3b0abdad76a44fc0339a6aa69
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for pam-python 0.2.17

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.17 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page