porosdata-designer 0.1.0

Structured delivery toolkit for converting MinerU parses into full_text, datamining, and multimodal outputs.

PorosData-Designer

PorosData-Designer converts MinerU-generated document parses into three stable deliverables: full_text, datamining, and multimodal.

It is designed for scientific data processing, structure-aware training preparation, and atomic document design centered on paragraphs, formulas, chemical expressions, and figure assets.

What It Does

• Builds a structure-aware full-text view from *_content_list.json.
• Maps document sections, formulas, chemical expressions, and asset references into a datamining view.
• Extracts image-caption-mention relationships into a multimodal view with copied assets and Markdown outputs.

Install

pip install porosdata-designer

Quick Start

--input_dir should point to a directory tree that contains MinerU outputs. In practice, Designer expects:

• one or more *_content_list.json files
• image assets that remain resolvable relative to those input files

For package usage, it is recommended to pass explicit output and log directories.

Run the full pipeline:

porosdata-designer run all --input_dir "path/to/input_dir" --output_dir "path/to/output_dir" --log_dir "path/to/log_dir"

Equivalent module mode:

python -m porosdata_designer run all --input_dir "path/to/input_dir" --output_dir "path/to/output_dir" --log_dir "path/to/log_dir"

Stage Commands

Run text structuring only:

porosdata-designer run text --input_dir "path/to/input_dir" --output_dir "path/to/output_dir"

Run multimodal extraction only:

porosdata-designer run multimodal --input_dir "path/to/input_dir" --output_dir "path/to/output_dir"

Outputs

Designer produces three delivery views by default:

• full_text: for structure-aware training and human review.
• datamining: for retrieval, extraction, indexing, and downstream storage.
• multimodal: for image-text linking, multimodal indexing, and asset delivery.

Typical output layout:

path/to/output_dir/
├── full_text/{doc_id}/
│   ├── {doc_id}_structured.json
│   └── {doc_id}_structured.txt
├── datamining/{doc_id}/
│   └── {doc_id}_datamining.json
└── multimodal/{doc_id}/
    ├── {doc_id}_index.json
    ├── fig_n.md
    └── assets/

Validation

Audit structured outputs:

porosdata-designer audit structured --root_dir "path/to/output_dir"

Validate text outputs:

porosdata-designer validate structured --output_dir "path/to/output_dir/full_text"

Validate multimodal outputs:

porosdata-designer validate multimodal --output_dir "path/to/output_dir/multimodal"

Run final acceptance validation:

porosdata-designer validate acceptance --output_dir "path/to/output_dir/multimodal"

Python Usage

You can also use the package directly in Python:

from porosdata_designer import DataMiningMapper, MultimodalInterleaver, TextAggregator

aggregator = TextAggregator()
mapper = DataMiningMapper()
interleaver = MultimodalInterleaver()

A more complete text-side example:

from porosdata_designer import DataMiningMapper, TextAggregator

content_list = [
    {"type": "text", "text_level": 1, "text": "Abstract", "page_idx": 0},
    {"type": "text", "text": "This work studies a Cu-Zr metallic glass system.", "page_idx": 0},
    {"type": "text", "text_level": 1, "text": "Results and Discussion", "page_idx": 1},
    {"type": "text", "text": "Figure 1 shows the microstructure evolution at 700 K.", "page_idx": 1},
]

aggregator = TextAggregator()
structured_text = aggregator.aggregate(content_list)

mapper = DataMiningMapper()
datamining_view = mapper.map(structured_text, {"doc_id": "demo-0001"})

print(structured_text)
print(datamining_view.pure_text_stream)
print(datamining_view.structured_json["sections"])

Expected outcome:

• structured_text contains Poros tags such as <poros_doc>, <poros_section_*>, and <poros_paragraph>.
• pure_text_stream removes the structure tags while keeping readable text.
• structured_json exposes mined fields such as sections, formulas, chemical_formulas, and asset_refs.