Skip to main content

Document layer

The document and OCR layers now use native readers, versioned artifacts and Hugpy-managed OCR. They no longer depend on abstract_ocr or abstract_pdfs. See DOCUMENTS.md for current APIs, limits and migration details. The remaining facade layers retain the architecture documented below.

media_intelligence — Abstract Intelligence Platform

A unified, layered facade that turns raw media — PDFs, images, and video — into structured, searchable, SEO-ready data. It does not reimplement any engine: it selects the best function of each sibling package and exposes it behind one clean, lazy API, plus an orchestrated pipeline.

Raw Media (PDF / Image / Video / URL)
   │
   ▼
ingest  → extract → structure → enrich → persist → publish
(webtools) (ocr/    (typed     (hugpy)  (FS / DB) (react/
            pdfs/    metadata)                      nginx)
            videos)

Layers → canonical owners

Layer Owner package What it does
ingest abstract_webtools scrape pages, download video (yt-dlp/ffmpeg)
ocr Hugpy HTTP adapter managed vision transcription
documents document_engine native structure, artifacts, page retrieval
video abstract_videos registry pipeline: download/frames/transcribe
transcribe hugpy (→ abstract_ocr fallback) Whisper speech-to-text
enrich hugpy summaries, keywords, vision captioning, SEO
persist filesystem (DB-pluggable) typed JSON/JSONB manifests
publish abstract_react + abstract_nginx SEO/OG metadata + static HTML

Overlapping capabilities are resolved to one owner (Whisper → hugpy; video download → webtools; summarize/keywords → hugpy).

Install

media_intelligence is just this src/ facade — it contains none of the engines. Each layer's owner is its own PyPI package, declared as an optional extra, so you install only what you use:

pip install media_intelligence              # facade plus hugpy-resources
pip install "media_intelligence[ocr,enrich]"  # just those layers
pip install "media_intelligence[all]"       # the full platform

The base package depends on hugpy-resources; heavyweight backing packages remain optional. Each sibling is imported lazily, only when its layer is actually called; a missing one raises a clear MissingDependency naming the extra to install.

Check what's usable in the current environment without importing anything:

import media_intelligence as mi
mi.available()            # {'ingest': True, 'ocr': True, 'publish': False, ...}
mi.available("enrich")    # True / False

Usage

Direct namespace access

import media_intelligence as mi

text = mi.ocr.image_to_text("page.png")
kw   = mi.enrich.keywords(text)
mi.documents.process_pdf("doc.pdf")
mi.ingest.download_video("https://site.com/v.mp4", download_directory="/data")

Orchestrated pipeline (idempotent + resumable)

from media_intelligence import MediaPipeline

pipe = MediaPipeline("https://site.com/video.mp4", out_root="/data")
pipe.ingest().extract().structure().enrich().persist().publish()
print(pipe.report.summary)
#   ... or simply:
pipe.run()

The pipeline autodetects media kind, dispatches each stage accordingly, skips stages already satisfied (idempotent), and rehydrates from a prior manifest on re-run (resumable). Results land in out_root/<media_id>/manifest.json.

Persistence (DB-pluggable, two records)

Each item is persisted as two records so indexing stays cheap while aggregation stays simple:

  • manifest.json — lean index: ids, counts, text_chars, summary, keywords, SEO, asset pointers. (The JSONB metadata row.)
  • document.json — canonical content: full text, pages/segments, transcript. The single source of truth for search / aggregation / LLM datasets — one read per item, no re-stitching of per-owner on-disk files.
store = mi.persist.FileStore("/data")
store.save_manifest(item.media_id, manifest)   # lean index
store.save_document(item.media_id, document)   # full body
doc = store.load_document(item.media_id)        # aggregation reads this

# later, identical interface, JSONB backend:
# store = mi.persist.PgStore(dsn=...)   # planned (abstract_database)
#   -> metadata in JSONB, body text in a full-text-indexed column

MediaPipeline.persist() writes both. On re-run, the body is rehydrated from document.json, so extract/enrich skip (no re-OCR / re-transcribe).

Metadata

Release files for media-intelligence 0.1.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for media-intelligence 0.1.6
File Size Uploaded Provenance Actions
media_intelligence-0.1.6.tar.gz 48.2 kB Without provenance Details for media_intelligence-0.1.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for media-intelligence 0.1.6
File Interpreter ABI Platform Actions
media_intelligence-0.1.6-py3-none-any.whl Python 3 none any Details for media_intelligence-0.1.6-py3-none-any.whl

Total release size: 98.8 kB

Release files / media_intelligence-0.1.6.tar.gz

Download URL media_intelligence-0.1.6.tar.gz
Size 48.2 kB
Tags Source
SHA-256 checksum
How to use checksums
013a2fa98eaef10c98604b18e8548c6da42e901cb2563433576fd6313745f1e3
BLAKE2b-256 checksum
How to use checksums
1f09957b94e4569f536a170fed77b30b3e090289580321bfc3feac40b7aeb0b2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / media_intelligence-0.1.6-py3-none-any.whl

Download URL media_intelligence-0.1.6-py3-none-any.whl
Size 50.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ff25fdb2d82ce7a2ede1b85e57d93f55f26ef70b0036e0132467b6054ee3b85e
BLAKE2b-256 checksum
How to use checksums
e5830f265dbb7359fd955047a7328f89c74ca099e7fb56a5cf3556f72d281365
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for media-intelligence 0.1.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page