analytic-schema 1.0.0

I/O schema loader + validator for analytics notebooks

Analytic Schema

Analytic Schema is a lightweight Python package for loading, validating, and building standardized I/O documents for analytics notebooks based on a single, versioned JSON contract that drives both input parsing and output construction.

Table of Contents

• Description
• Dependencies
• Installation
• Usage
• Project structure
• Background and Motivation
• Contributing
• Contributors
• License

Description

Analytic Schema centralizes your notebook or script I/O definitions into one authoritative JSON file. From that contract it automatically:

• Generates a complete command-line interface (via argparse) for every input field.
• Parses parameters provided as CLI flags, JSON dictionaries, or JSON files.
• Injects sensible defaults for all optional inputs.
• Performs deep schema validation (type checks, enums, date-time formats, oneOf branches, and no extra fields).
• Builds structured output documents with embedded metadata, SHA-256 hashes for auditability, and a built-in logging mechanism.

With zero dependencies beyond the standard library and pandas, Analytic Schema is ideal for air-gapped notebooks, CI pipelines, or any environment where you need a robust, self-contained I/O layer for cybersecurity analytics.

Dependencies

This project depends only on the Python standard library (>=3.8) and pandas (>=1.0).

Installation

pip install analytic-schema

In your code:

import analytic_schema

Usage

Below is a minimal end-to-end example showing how to go from raw inputs to a validated output file:

from analytic_schema import parse_input, validate_input, OutputDoc
import time
 # 1) Parse input parameters (CLI string, list, JSON, or file)
raw = parse_input(
    “—input-schema-version 1.0.0 “
    “—start-dtg 2025-06-01T00:00:00Z “
    “—end-dtg   2025-06-02T00:00:00Z “
    “—data-source-type file “
    “—data-source /tmp/log.csv”
)
 # 2) Validate against the JSON contract and fill defaults
params = validate_input(raw)
 # 3) Run your analytic logic…
start = time.perf_counter()

# ... your detection code here ...

findings = [
    {
        “finding_id”: “uuid-v4”,
        “title”: “Suspicious pattern”,
        “description”: “Detected anomalous traffic...”,
        “event_dtg”: “2025-06-07T12:34:56Z”,
        “severity”: “high”,
        “confidence”: “0.92”,
        “observables”: [“1.2.3.4”, “bad.example.com”],
        “mitre_attack_tactics”: [“TA0001”],
        “mitre_attack_techniques”: [“T1001”],
        “recommended_actions”: “Block IP and review logs”,
        “recommended_pivots”: “Check DNS logs”,
        “classification”: “U”
    }
]
elapsed_ms = (time.perf_counter() - start) * 1000
 # 4) Build the structured output document
out = OutputDoc(
    input_data_hash=params[“input_data_hash”],
    inputs=params,
    findings=findings,
    records_processed=len(findings)
)
out.add_message(“INFO”, “Analysis completed in %.2f ms” % elapsed_ms)
out.finalise()
 # 5) Serialize to JSON
out.save(“analysis_output.json”)

This example demonstrates how Analytic Schema handles all the I/O boilerplate—CLI parsing, default injection, validation, metadata, logging, hashing, and final serialization—so you can focus on the core analytic logic.

Project structure

analytic-schema/ # Project repository
├── analytic_schema/ # Package
│   ├── __init__.py
│   ├── loader.py
│   ├── parser.py
│   ├── validator.py
│   ├── output.py
│   └── analytic_schema.json
│
├── tests/
│   └── test_analytic_schema.py
│
├── example_usage.py
│
├── README.md      # This file
├── LICENSE.md     # Project license
├── Makefile       # Project makefile
└── pyproject.toml

Background and Motivation

In cybersecurity analytics, consistency and auditability are paramount. Analysts and automation pipelines often spin up dozens of scripts and notebooks, each rolling its own argument parsing, validation, and output formatting. This fragmentation leads to subtle bugs, schema drift, and integration headaches.

Analytic Schema addresses these challenges by elevating your I/O contract to one single JSON schema. This contract drives:

• Uniformity: All analytics share the same field names, types, and defaults.
• Reliability: Fail-fast validation prevents runtime surprises from missing or mistyped parameters.
• Traceability: Inputs and findings are hashed, and logs are captured inline, enabling full audit trails.
• Simplicity: With only the standard library plus pandas, it works in air-gapped environments and keeps your dependencies minimal.

By abstracting away boilerplate, you can focus on detecting and investigating threats, while ensuring your pipelines remain robust, maintainable, and easily integrated.

Contributing

Contributions are welcome from all, regardless of rank or position.

There are no system requirements for contributing to this project. To contribute via the web:

1. Click GitLab’s “Web IDE” button to open the online editor.
2. Make your changes. Note: limit your changes to one part of one file per commit; for example, edit only the “Description” section here in the first commit, then the “Background and Motivation” section in a separate commit.
3. Once finished, click the blue “Commit...” button.
4. Write a detailed description of the changes you made in the “Commit Message” box.
5. Select the “Create a new branch” radio button if you do not already have your own branch; otherwise, select your branch. The recommended naming convention for new branches is first.middle.last.
6. Click the green “Commit” button.

You may also contribute to this project using your local machine by cloning this repository to your workstation, creating a new branch, committing and pushing your changes, and creating a merge request.

Contributors

This section lists project contributors. When you submit a merge request, remember to append your name to the bottom of the list below. You may also include a brief list of the sections to which you contributed.

• Creator: Zachary Szewczyk

License

This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. You can view the full text of the license in LICENSE.md. Read more about the license at the original author’s website. Generally speaking, this license allows individuals to remix this work provided they release their adaptation under the same license and cite this project as the original, and prevents anyone from turning this work or its derivatives into a commercial product.