foo-py 0.1.1

foo

Features · Modes · Architecture · Install · Windows Installer · Run · Loaders · Scraping · Retrievers · Fetchers · AI · Requirements · Examples ·

Foo is a data loading, scraping, retrieval, geospatial, environmental, astronomical, demographic, generative-AI, and data-processing workspace. It is designed to give users explicit, hands-on control over how content is loaded, extracted, queried, fetched, cleaned, analyzed, visualized, and routed into downstream machine-learning or agentic workflows.

Foo is modular by design. Loaders, scrapers, fetchers, generators, databases, and external APIs can operate independently while remaining composable inside one cohesive interface. The application supports local files, web pages, public archives, Google services, government data sources, geospatial APIs, environmental APIs, astronomical APIs, demographic APIs, and multiple LLM providers.

🎥 Demo

☁️ Cloud

✨ Features

🕹️ Application Modes

🏛️ Architecture

📥 Loader → 🧹 Text Processing → 🧠 Session State → 🔍 Retrieval / Analysis / Generation
      │                  │                    │
      ├── 🕸️ Scraper      ├── 📊 Metrics        ├── 🗄️ SQLite
      ├── 🌐 Fetcher      ├── 🧩 Documents      ├── 📈 Visualization
      └── ☁️ Cloud        └── 📝 Raw Text       └── 🤖 LLM Providers

Foo uses a Streamlit UI over modular Python classes. The application imports loader classes from loaders.py, provider classes from generators.py, and API/data-source wrappers from fetchers.py. Shared working state is coordinated through st.session_state, allowing loaded documents, raw text, processed text, tokens, metrics, and database results to flow between controls.

🗂️ Directory Structure

foo/
├── app.py                 # Streamlit user interface
├── config.py              # App title, mode map, defaults, labels, API references, and constants
├── core.py                # Optional package core abstractions
├── data.py                # Data helpers and shared data abstractions
├── embedders.py           # Hosted and local LangChain embedding implementations
├── fetchers.py            # External API, archive, geospatial, environmental, and science fetchers
├── generators.py          # ChatGPT, Claude, Grok, Mistral, and Gemini wrappers
├── loaders.py             # File, web, cloud, repository, and corpus loaders
├── scrapers.py            # HTML extraction helpers
├── requirements.txt       # Python dependencies
├── stores/
│   ├── vector.py          # Chroma and Pinecone lifecycle implementations
│   └── sqlite/            # SQLite database storage
└── resources/
    └── images/            # README and UI image assets

🔑 API Set-up

• Science APIs
• OpenAI
• Gemini AI
• Grok AI
• Mistral AI
• Claude AI

🛡️ Installation

git clone https://github.com/is-leeroy-jenkins/Foo.git
cd Foo
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For Linux or macOS:

git clone https://github.com/is-leeroy-jenkins/Foo.git
cd Foo
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

🪟 Windows Desktop Installer

Foo can be packaged as a Windows desktop application using PyInstaller (onedir) and Inno Setup. The installed Foo.exe runs Streamlit on the local computer and displays the interface in a dedicated Microsoft Edge WebView2 window, without requiring the user to open a browser tab. The installer provides Start Menu shortcuts, an optional desktop shortcut, and an uninstaller. Writable application data is intended to live under %LOCALAPPDATA%\Foo.

To build an installer, run the Windows desktop installer workflow from GitHub Actions, then download its Foo-Windows-Installer artifact after a successful run. Windows builds can also be created locally using the PyInstaller specification and Inno Setup script. Browser runtimes, models, and some native dependencies require separate validation before distribution.

Windows installation guide · Desktop launcher · Installer script · Build workflow · Desktop build notes

▶️ Running the Streamlit App

streamlit run app.py

The application initializes Streamlit in wide layout, loads the configured mode map from config.py, and displays the active mode selector in the sidebar under 🕹️ Mode.

🚀 Quick Start

Run the Application

streamlit run app.py

Load a Document

1. Open Loading mode.
2. Expand a loader such as PDF Loader, Excel Loader, Web Loader, or GitHub Loader.
3. Select or enter the source.
4. Click Load.
5. Configure recursive chunking, embedding, and vector storage in the same source expander.
6. Review the Document, Chunks, and Embeddings tabs. Chunking occurs first, followed by word tokenization of every resulting chunk; embeddings use the original chunk text.

Scrape a Web Page

1. Open Scraping mode.
2. Enter a target URL.
3. Select core output and structured extraction options.
4. Optionally enable recursive crawl controls.
5. Click Run Scraper.
6. Use the adjoining processing controls and right-side tabs to inspect, chunk, tokenize, embed, and store the scraped documents.

Query a Public Source

1. Open Retrieval mode.
2. Expand a source such as ArXiv, Google Search, Gov Info, or US Congress.
3. Enter the query and parameters.
4. Click Submit.
5. Review rendered summaries, rows, and raw results.

Retrieval, Geospatial, Environmental, Astronomical, and Demographic source expanders use the same processing workflow while retaining their provider-specific maps, tables, images, metrics, and raw results.

📤 Loaders

🕸️ Scraping

🏛️ Retrieval Sources

🌎 Domain Fetchers

Geospatial

Environmental

Astronomical

Demographic and Health

🤖 Generation Providers

📦 Requirements

The table below reflects the requirements implied by the active imports, loaders, fetchers, and UI surface in app.py. Some provider-specific loaders/fetchers may require additional credentials or cloud SDKs depending on deployment.

🔑 Configuration

🔍 Example Usage

Scrape Web Page Paragraphs

from foo.scrapers import WebExtractor

extractor = WebExtractor()
paragraphs = extractor.scrape_paragraphs("https://example.com")
print(paragraphs)

Load and Chunk a PDF

from foo.loaders import PdfLoader

loader = PdfLoader()
documents = loader.load("docs/report.pdf")
chunks = loader.split(documents, chunk=1000, overlap=100)
print(chunks)

Query a Fetcher

from foo.fetchers import Wikipedia

fetcher = Wikipedia(language="en", max_documents=5)
documents = fetcher.fetch("Natural language processing")
for document in documents:
    print(document.metadata)
    print(document.page_content[:500])

Run a Read-Only SQLite Query

import sqlite3
import pandas as pd

with sqlite3.connect("stores/sqlite/data.db") as connection:
    df_results = pd.read_sql_query("SELECT * FROM Prompts LIMIT 10;", connection)

print(df_results)

⚙️ Technical Notes

🔑 AI API Key

Data Services

📝 License

• MIT License here
• Copyright © 2022–2025 Terry D. Eppler