suur-data 1.0.5

Intelligent data ingestion and tokenization pipeline

Suur Data

Intelligent data ingestion, filtering, and tokenization pipeline.

Installation

pip install suur-data

See It In Action

Single Source

from suur_data import suur_data

tokens = suur_data("https://en.wikipedia.org/wiki/Neural_network", topic="neural networks")
print(tokens)

Multiple Sources

from suur_data import suur_data

tokens = suur_data(
    [
        "data.txt",
        "research_paper.pdf",
        "https://en.wikipedia.org/wiki/Deep_learning",
        "https://en.wikipedia.org/wiki/Artificial_neural_network",
    ],
    topic="neural networks",
    threshold=0.05,
)
print(f"Total tokens: {len(tokens)}")

All sources are downloaded, merged, filtered together, and tokenized in one call.

---

Full Documentation

All Installation Options

# Core — supports .txt, .csv, .json, .html, URLs
pip install suur-data

# Add PDF support
pip install suur-data[pdf]

# Add Word document support
pip install suur-data[docx]

# Add EPUB support
pip install suur-data[epub]

# Add HuggingFace pretrained tokenizers
pip install suur-data[hf]

# Everything
pip install suur-data[all]

Supported Input Formats

Format	Notes
.txt .md .rst	Plain text, auto encoding detection
.pdf	Requires suur-data[pdf]
.docx	Requires suur-data[docx]
.csv .tsv	All cells joined as text
.json	Recursively flattened key-value pairs
.html .htm	Scripts and styles stripped automatically
.epub	Requires suur-data[epub]
HTTP/HTTPS URL	Auto-downloaded, parsed by extension

Python API

from suur_data import suur_data

# From a URL
tokens = suur_data("https://en.wikipedia.org/wiki/Neuroscience", topic="brain neurons")

# From a local file
tokens = suur_data("data.txt", topic="machine learning")

# Multiple sources at once
tokens = suur_data(
    ["data.txt", "paper.pdf", "https://en.wikipedia.org/wiki/Deep_learning"],
    topic="neural networks"
)

# Custom BPE tokenizer trained on your data
tokens = suur_data("data.txt", topic="machine learning", tokenizer="custom", vocab_size=4000)

# Strict filter — only highly relevant chunks survive
tokens = suur_data("data.pdf", topic="quantum computing", threshold=0.15)

# Save tokenizer to disk for reuse
tokens = suur_data("data.txt", topic="biology", save_dir="./my_tokenizer")

# Skip filter entirely
tokens = suur_data("data.txt", no_filter=True)

All Parameters

Parameter	Type	Default	Description
data_location	str or List[str]	required	URL, file path, or list of multiple sources
topic	str	""	Subject for relevance filtering. Empty skips filter
tokenizer	str	"pretrained"	"pretrained" or "custom"
model	str	"gpt2"	HuggingFace model name or Hub ID
vocab_size	int	8000	BPE vocab size for custom tokenizer
threshold	float	0.05	Relevance cutoff between 0.0 and 1.0
save_dir	str	None	Directory to save tokenizer files
no_filter	bool	False	Skip the relevance filter
verbose	bool	True	Show progress output

Pretrained Model Shortcuts

Shortcut	Model
gpt2	GPT-2 (OpenAI)
bert	BERT base uncased
roberta	RoBERTa base
distilbert	DistilBERT base uncased
t5	T5 small

You can also pass any HuggingFace Hub model ID directly:

tokens = suur_data("data.txt", model="facebook/opt-125m")

How the Filter Works

The filter splits text into paragraph chunks, converts each chunk and the topic into TF-IDF vectors, then scores them using cosine similarity. Chunks below the threshold are deleted. If the threshold is too strict and everything gets dropped, it automatically relaxes and keeps the top 30 percent.

tokens = suur_data("data.txt", topic="AI", threshold=0.10)  # strict
tokens = suur_data("data.txt", topic="AI", threshold=0.02)  # loose

Saving and Loading Tokens

import json

tokens = suur_data("data.txt", topic="neural networks")
with open("tokens.json", "w") as f:
    json.dump(tokens, f)

with open("tokens.json", "r") as f:
    tokens = json.load(f)

Decoding Tokens Back to Text

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
text = tok.decode(tokens)
print(text)

---

Architecture

Source (URL or file or list of sources)
        |
        v
Stage 1 — Ingest
Handles 8 file types and HTTP download.
Merges all sources into one text string.
        |
        v
Stage 2 — Neural Filter
Splits text into paragraph chunks.
Scores each chunk against topic via TF-IDF cosine similarity.
Shows progress bar while scoring.
Drops chunks below the relevance threshold.
        |
        v
Stage 3 — Tokenize
Pretrained: HuggingFace AutoTokenizer (GPT-2, BERT, etc.)
Custom: trains a BPE tokenizer on the filtered corpus.
        |
        v
List[int] — token IDs

---

License

MIT