Quick start · How it works · Self-host · Frameworks · Scope · Explore
Open-source, self-hosted compliance automation. Connect your stack, continuously test controls, collect evidence, and hand auditors proof — running in your own cloud or VPC, on your own data lake.
- Your cloud, your evidence. Evidence stays in storage you run. Data leaves only through the connectors, sinks, and model integrations you configure.
- Deterministic and API-first. Rules decide pass or fail, and every result links to its evidence; models may summarize or propose, never decide. The console, API, CLI, MCP server, and CI gates share one engine.
Quick start
With Docker Compose v2:
git clone https://github.com/msaad00/trustops-security-data-lake.git
cd trustops-security-data-lake
docker compose up
Open localhost:8787/console/dashboard/.
It serves the bundled sample company with authentication off, on 127.0.0.1
only. The 5-minute tutorial continues to a real cloud
account, a mapping review, and an OSCAL export.
Other setup paths — source, pip, CLI only, and MCP
From source (Python 3.11+, uv, Node 22+):
uv sync --frozen --extra dev --extra server
make demo-local
From PyPI, as a server or CLI only:
pip install "trustops-security-data-lake[server]"
security-lakehouse fixtures load --company golden --out ./lake --rebase-times
security-lakehouse assessment status --lake ./lake
security-lakehouse serve --server --allow-insecure-no-auth --lake ./lake --port 8787
The same lake over MCP (stdio); see headless GRC for the trust boundary:
pip install 'trustops-security-data-lake[mcp]'
TRUSTOPS_LAKE=./lake trustops-mcp
How it works
| Step | What you do | What you get |
|---|---|---|
| Collect | Connect a source with read-only access. | Evidence with source, freshness, and provenance. |
| Evaluate | Apply deterministic control rules. | Results tied to evidence and the evaluated catalog. |
| Resolve | Assign findings, track fixes, and review exceptions. | Ownership and a record of follow-up decisions. |
| Export | Freeze an assessment and share reports. | Evidence and assessment history for reviewers. |
Evidence arrives in one of two modes; both feed the same rules and assessments.
| Mode | How | Sources |
|---|---|---|
| Ingest | Read-only connectors pull evidence into a lake you own. No existing lake needed. | AWS, Azure, GCP, GitHub, GitLab, Okta, Google Workspace, Jira, Intune, BambooHR, Rippling, Workday, Jamf*, CrowdStrike Falcon*, Kubernetes*, KnowBe4* |
| Existing lake | Read-only queries against the lake you already run; a lake mapping maps your existing tables. | Snowflake, ClickHouse, Databricks*, Iceberg/Parquet* (including Amazon Security Lake through OCSF presets), BigQuery*, S3 object evidence, SIEM exports |
* Preview: implemented and fixture-tested, not yet verified against a live tenant. Lake mappings are experimental. The connector catalog lists all 28 contracts: 25 executable (18 generally available, 7 in preview).
Self-host
| Path | Use it for |
|---|---|
| Docker Compose | One host: docker compose up -d trustops-server, with authentication on. |
| Helm chart | Kubernetes, with a persistent /lake volume and the scheduler. |
| EKS Terraform | Reference infrastructure for the chart on Amazon EKS. |
| AWS · Azure · GCP | Read-only posture roles for each cloud (see Connector credentials below). |
| Snowflake · Databricks · ClickHouse | Schema and bootstrap SQL for each existing-lake reader. |
Server mode requires authentication (OIDC, SAML, or API keys). In hosted mode each tenant's connector secrets and cloud access stay tenant-scoped: hosted connector credentials. Start with the deployment guide.
Frameworks and common controls
20 framework packs · 94 reusable safeguards · 21 control families in 10 categories · 2,245 catalogued requirements. 2 more registry entries are planned or superseded and hold no requirements.
1,415 requirements have safeguard mappings; 350 have reviewed mappings. Catalog coverage and evaluated customer posture are separate measures.
Control families by category:
- Governance and risk: Risk management · Governance
- Identity and access: Identity and access
- Data protection and privacy: Data protection · Privacy
- Secure engineering: Change management · Secure development · Secure architecture
- Infrastructure security: Configuration management · Vulnerability management · Network security
- Detection and response: Detection · Audit logging · Incident response
- Resilience and integrity: Availability and recovery · System maintenance · Processing integrity
- Third-party and supply chain: Third-party risk
- People and physical: People security · Physical security
- AI governance: AI governance
SOC 2 |
ISO 27001 · 27017 · 27701 · 42001 |
NIST CSF 2.0 |
NIST AI RMF |
CIS Controls · CIS AWS |
CMMC 2.0 |
EU AI Act |
NIST 800-53 · 800-171 Rev 3 · RMF FedRAMP · HIPAA · PCI DSS GDPR · NIS2 · DORA |
A reviewed mapping has been confirmed by a person; a proposed one has not, and neither is a certification. Some packs are limited: PCI DSS v4.0.1 covers its 12 principal requirements, ISO/IEC 27701:2025 seeds 10 of its 78 Annex A controls, and every NIST RMF, NIST 800-171 Rev 3, NIS2, and DORA mapping is proposed. SOC 1 is planned. The AI governance family's 13 safeguards map 80 requirements across NIST AI RMF, ISO/IEC 42001, the EU AI Act, and GDPR, mostly proposed. The coverage matrix has the exact boundary per framework. Your own reviewers can approve or reject mappings for your tenant, with an attributable audit trail: mapping review.
How the Common Control Framework evaluates
| Layer | What it represents |
|---|---|
| Control families | Risk domains that organize reusable safeguards. |
| Safeguards | Evidence requirements, ownership, review frequency, and executable evaluation rules. |
| Framework mappings | Links from safeguards to framework requirements, reviewed or proposed. |
| Assessment results | Pass, fail, stale, or not evaluated, from the collected evidence. |
One safeguard can serve several frameworks. A requirement passes only when its reviewed safeguards pass for the applicable observed assets. Missing bindings, stale or unknown evidence, and pending reviews prevent a passing result; an unmapped requirement stays unmapped. Details: Common Control Framework · executable catalog. From the CLI:
security-lakehouse frameworks safeguards --format table
Scope
| Area | Status |
|---|---|
| License | Apache-2.0; no per-seat license. You run the infrastructure. |
| Audit workflow | Audit room, readiness, frozen assessments, trust-center shares, access reviews, OSCAL export. |
| Policies and vendor risk | MVP: policy templates with attestation, vendor questionnaires. |
| Automation | Versioned API, CLI, MCP server, CI posture gate, webhooks. |
| Integrations | The read-only connectors above; add your own as a separately installed Python package. |
| Not offered | A managed service, or certification. Results are evidence for your auditor. |
Details: product status · roadmap.
Assessment and scale boundaries
CCF mapping coverage describes the catalog. Operational safeguard results require explicit evidence bindings, reviewed mappings, and evidence for applicable observed assets. The Frameworks page shows those results separately from framework-control readiness; complete inventory and organizational compliance remain separate claims. See CCF evaluation.
For a technical review, start with these reproducible contracts:
| Question | Implementation and evidence |
|---|---|
| Which evidence produced this result? | Sealed generations and historical exports |
| How does a large assessment behave? | 100k-event pipeline measurements and indexed CCF reads |
| What happens when an export fails? | Atomic DuckDB refresh and recovery |
| When should a deployment receive traffic? | Readiness probes and release gates |
Evaluation runs in local Python and publishes immutable, hashed generations. Warehouse sinks export completed results; they do not remove local memory limits. The supported Helm deployment has one writable application replica. See the architecture, benchmark evidence, and HA boundary for measured and unverified scope.
Explore
01 · Product tour
Images show the bundled demo fixture, not live customer evidence. The tour follows an assessment from overview to evidence, mapping review, and owned findings.
Overview
Frameworks
Evidence
Mapping review
Control detail
Triage
Walkthrough · Control families · Crosswalk · Connections · Findings · Graph · Remediation · Policies · Vendor risk · Audit room · Workflows · Trust center · Phone
02 · Connector credentials
In the console: Connections → choose a source → Test → Enable → Sync. For automation, use the headless setup playbook. Cloud connectors use short-lived or workload identity credentials. Settings keep a credential reference (an environment variable name or mounted secret file), not the secret itself.
- AWS uses STS AssumeRole, one External ID per deployed role, short-lived session credentials, and read-only IAM posture APIs. Temporary credentials expire after each session; TrustOps stores no long-lived access keys. Roll out with CloudFormation StackSets or Terraform workspaces; bulk account import is planned. See the cloud setup guide and the credential lifecycle.
- Azure uses a customer-owned Entra application, managed identity, or federated workload identity with Reader scope.
- GCP uses Application Default Credentials or workload identity; a service-account key file also works.
- Snowflake uses a read-only service identity with a key-pair or OAuth token reference. TrustOps stores identifiers, not passwords or private-key contents. Snowflake is one of the existing-lake readers.
- GitHub uses a GitHub App installation token, which expires within an hour.
- SaaS sources use scoped API tokens or an integration-user login.
Ship your own connector as a Python package: adding connectors.
03 · Architecture and storage
Source → Raw evidence → Normalized facts → Control evaluation → Assessment
↓ ↓
Owned findings Review / export
| Layer | Boundary |
|---|---|
| Evidence and evaluation | Local JSONL and verified assessment generations; one writer per lake. |
| Analytics and state | SQLite mart (DuckDB optional) and an application database. |
| Portable evidence | Optional Parquet export and Iceberg REST publication. |
Existing-lake readers are evidence sources; they do not host TrustOps. Architecture · assessment generations · continuous ingestion
04 · API, agents, and CI
| Surface | Purpose |
|---|---|
| API | Versioned /api/v1 access for integrations. |
| MCP | Read assessments and propose actions through governed tools. |
| CI | Posture and control-test thresholds in delivery workflows. |
| OSCAL | Component-definition and assessment-results JSON. |
| Webhooks | Signed event delivery to your systems. |
Develop and verify
Checks and repository layout
make smoke # backend, contracts, docs, brand, pipeline, API
make web-ci # install, typecheck, production build
make security # dependency audits and pre-commit checks
| Directory | Contents |
|---|---|
src/security_lakehouse/ |
Assessment engine, API, auth, connectors, and MCP. |
app/web/ |
Next.js console. |
controls/, frameworks/, mappings/ |
Rules, framework catalogs, and mappings. |
deploy/ |
Deployment and infrastructure examples. |
docs/ |
Product, architecture, operations, and API guides. |
Contributing · Security policy · Benchmarks · Third-party assets · Apache-2.0 license
Metadata
Release files for trustops-security-data-lake 0.2.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| trustops_security_data_lake-0.2.20.tar.gz | 3.0 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| trustops_security_data_lake-0.2.20-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.1 MB
Release files / trustops_security_data_lake-0.2.20.tar.gz
| Download URL | trustops_security_data_lake-0.2.20.tar.gz |
|---|---|
| Size | 3.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
29eec1d8f5786d46f64218f19f0660a17a7f8b6929ca51d5a11e0688e3109eaf
|
|
BLAKE2b-256 checksum How to use checksums |
e15664d1dabb9c55be4b00d41c60cf65983e565ac47b1fcd089389070a79e4e2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency logRelease files / trustops_security_data_lake-0.2.20-py3-none-any.whl
| Download URL | trustops_security_data_lake-0.2.20-py3-none-any.whl |
|---|---|
| Size | 3.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a8ff2fc3ad628fd818d774df43e8ad8455607aa2b148a70b0416e3bcd35db624
|
|
BLAKE2b-256 checksum How to use checksums |
60e3b27aeb7a512dbfdca0712abdb8c5d5c6518de3a74afee0b28ef60ea3eabb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency log