Shadow Tools

Train on regular text

from shadowtools import predict_text, train_text

texts = ["I love this game", "this is terrible", "really fun game", "I hate it"]
labels = ["positive", "negative", "positive", "negative"]

model, history = train_text(texts, labels, epochs=20)
print(predict_text(model, ["this game is fun", "I really hate this"]))
# ['positive', 'negative']

train_text learns a vocabulary from your examples. Its tokenizer keeps words, contractions, URLs, punctuation, and common two-word phrases, then converts each text into a TF-IDF number vector before training. Use SmartTokenizer yourself when you want to fit once and reuse the exact same vocabulary later.

Train a tokenizer for your own Transformer

from shadowtools import BPETokenizer

tokenizer = BPETokenizer(vocab_size=8_000).train(my_training_texts)
batch = tokenizer.batch_encode(["Hello, new world!"], max_length=32)

# Pass these directly to your Transformer.
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
tokenizer.save("my_tokenizer.json")

This is byte-level BPE: it learns reusable chunks instead of treating each full word as a separate thing. Install it with pip install "shadow-tools[nlp]".

Train and generate text from one package

from shadowtools import learn, say

bot = learn("corpus.txt", epochs=20)
print(say(bot, "The future of technology", max_tokens=80))
bot.save("my_text_model")

Training prints the tokenizer status plus each epoch and loss. Use learn("corpus.txt", epochs=20, verbose=False) when you do not want updates.

Train a question-and-answer bot

from shadowtools import ask, teach

bot = teach({
    "What is ChatGPT?": "ChatGPT is an AI system that predicts useful next tokens to answer a prompt.",
    "How does a tokenizer work?": "A tokenizer splits text into smaller tokens a model can process.",
    "What is machine learning?": "Machine learning is when a model learns patterns from examples.",
}, epochs=20)

print(ask(bot, "What is ChatGPT?"))

Add many more real Q&A pairs for better answers.

Install it with pip install "shadow-tools[ml,nlp]". The package trains the tokenizer and Transformer together, then converts predicted token IDs back to normal text for you.

shadow-tools is a lightweight Python toolbox for everyday apps and scripts. It keeps the basics in one clean package without trying to replace specialized libraries.

Install

pip install shadow-tools
pip install "shadow-tools[websocket]"  # optional WebSocket client

What is included

• HTTP requests with no required third-party dependency
• WebSocket connections through an optional extra
• Async execution for blocking jobs
• File and directory helpers
• JSON and CSV reading/writing
• Email, URL, and required-key checks
• DNS lookup, subdomain parsing, and documentation-only test IPs
• Provider-neutral AI prompt and tool-schema helpers
• File and text hashing
• UUIDs, secure tokens, and timestamp IDs
• Slugs, snake/camel case, chunks, environment variables, and retry logic
• Defensive security checks, secret redaction, password generation, and file verification
• Optional PyTorch training for new MLPs or supplied local models
• shadow command-line tool

Quick start

from shadowtools import get, write_json, is_email

response = get("https://api.github.com")
print(response.status, response.json())
write_json("output.json", {"ready": True})
assert is_email("person@example.com")

DNS, subdomains, and test IPs

from shadowtools import generate_random_ips, generate_test_ips, make_subdomain, resolve_dns

print(make_subdomain("api", "example.com"))  # api.example.com
print(resolve_dns("example.com"))
print(generate_test_ips(2))  # safe documentation-only addresses
print(generate_random_ips(5))  # globally routable public IPv4 addresses only
print(generate_random_ips(3, family=6))
print(generate_random_ips(3, include_private=True, include_special=True))

Reverse-DNS lookup for IP addresses:

from shadowtools import generate_random_ips, reverse_dns

for result in reverse_dns(generate_random_ips(5)):
    print(result)  # only IPs with hostnames are shown

# Or look up every generated IP at once.
print(reverse_dns(generate_random_ips(5)))
print(reverse_dns(generate_random_ips(5), only_hostnames=False))  # include misses

AI app helpers

from shadowtools.ai import prompt, tool_schema

message = prompt("Explain {topic} in {words} words.", topic="DNS", words=50)
search_tool = tool_schema(
    "search_docs", "Search internal documentation",
    {"query": {"type": "string"}}, required=["query"]
)

Everyday helpers

from shadowtools import hash_text, random_token, slugify, to_bool, uuid4

print(hash_text("hello"))
print(random_token())
print(uuid4())
print(slugify("Shadow Tools: Version 3!"))  # shadow-tools-version-3
print(to_bool("yes"))  # True

shadow hash settings.json
shadow uuid

Security and training

from shadowtools.security import audit_headers, redact_secrets

print(redact_secrets("Authorization: Bearer secret-value"))
print(audit_headers({"X-Content-Type-Options": "nosniff"}))

Install ML support, then train a small classifier from scratch:

pip install "shadow-tools[ml]"

from shadowtools.training import Trainer, TrainingConfig

config = TrainingConfig(
    mode="from_scratch",
    architecture={"type": "mlp", "input_size": 8, "hidden_sizes": [32, 16], "num_classes": 2},
    epochs=5,
)
trainer = Trainer(config)
history = trainer.fit(train_loader, validation_loader)

For normal lists, use the short root-level API—no PyTorch imports or loaders:

from shadowtools import train

model, history = train(
    features=[[0, 1], [1, 1], [0, 0], [1, 0]],
    labels=[1, 1, 0, 0],
    epochs=10,
)
print(history)

shadow get https://api.github.com --json
shadow json-format settings.json

WebSockets

from shadowtools.websocket import connect

socket = connect("wss://example.com/socket")
socket.send("hello")
print(socket.recv())
socket.close()

Release

Build locally with python -m build, then publish only after reviewing the generated distributions and confirming the PyPI project name is available.

License

MIT