stpstone 2.0.0

Solid financial ETL, analytics and utils with support to global markets.

stpstone

stpstone (short for stepping stone) is a Python framework designed for ingesting, processing, and analyzing structured and unstructured financial data. It provides tools for ETL (Extract, Transform, Load), quantitative analysis, and derivatives pricing, optimized for financial market applications.

✨ Key Features

🌍 Data Extraction

🇧🇷 Brazilian Markets (B3)

Equities & Derivatives

• BOVESPA Cash Equities
• Futures/Options Metadata
• Consolidated Trades
• After-Market Trading
• Futures Closing Adjustments
• Historical Volatility (Sigma)
• BMF Interest Rates
• Trading Hours & Sessions
• Securities Registry
• Volume Analytics
• Warranty Information

Fixed Income

• ANBIMA 550 Bonds
• ANBIMA Indexes
• ANBIMA Index Data
• Brazilian Corporate Bonds
• Sovereign Debt Pricing
• ANBIMA Theoretical Portfolios
• ANBIMA Site Information

Macroeconomic Data

• Central Bank Data (BCB Olinda)
• FX Rates (PTAX)
• Economic Time Series (SGS BCB)
• IBGE Statistics (SIDRA)
• B3 Financial Indicators
• ANBIMA PMI Forecasts
• Investing.com Brazil Data
• Yahoo Finance Brazil Rates
• Yahoo Finance Other Data

Registries & Regulatory

• CVM Securities Data
• CVM Web Data
• CVM Direct Data
• B3 Trading Securities
• ANBIMA Funds Data
• ANBIMA Debentures Data
• Brazilian Banks Registry
• Mais Retorno Instruments
• Investment Funds Bylaws
• Tax Records (IRS Brazil)

🇺🇸 US Markets

Equities & ETFs

• Tiingo EOD + Fundamentals
• Alpha Vantage (VIX/CBOE)
• S&P 500 Constituents
• ETF Database

Macroeconomic Data

• FRED Economic Series
• US Treasury Yields

🌐 Worldwide Markets

Cryptocurrencies

• CoinMarketCap Top-500
• CoinCap Market Data
• CoinPaprika Analytics

Global Markets

• FMP Financial Markets
• ADVFN Global Data
• Investing.com Global
• Yahoo Finance WebService

Global Fixed Income

• World Government Bonds
• Global Interest Rates
• Trading Economics Data

Credit Ratings & Risk

• S&P Global Corporate Ratings

---

🔄 Data Transformation

Data Cleaning

• Data Cleaning Utilities
• Exploratory Data Analysis
• Feature Selection

Data Standardization

• DataFrame Standardization

Data Validation

• Type Enforcement
• Brazilian Docs Validation
• DataFrame Validation
• Bracket Balance Checker

---

📥 Data Loading

Database Connectors

• PostgreSQL
• MongoDB
• Databricks
• MySQL
• SQL Server
• SQLite
• Redis
• Generic Database Interface

Cloud Storage

• AWS S3
• MinIO
• SendGrid Email

---

📊 Analytics

Quantitative Analysis

• Financial Mathematics
• Statistical Inference
• Statistical Description
• Portfolio Allocation
• Regression Analysis
• Classification Models
• Probability Distributions
• Linear Algebra
• Calculus Operations
• Interpolation Methods
• Root Finding
• Sequences Analysis
• Statistical Charts
• Fit Assessment
• Josephus Problem Solver

Performance Metrics

• Data Deltas
• Earnings Quality
• ROE Decomposition

Pricing Models

• Derivatives Pricing
  • American Options
  • European Options
  • Forward Contracts
  • Futures Contracts
• Bond Valuation
• Debentures Pricing

Risk Management

• Capital Risk
• Liquidity Risk
• Market Risk
• Yield Risk

Computer Arithmetic

• Binary Operations
  • Binary Comparator
  • Binary Converter
  • Binary Arithmetic
  • Bit Operations
  • Logic Gates
  • Fraction Operations

---

⚙️ Utilities

Microsoft Office Integration

• Excel Automation
• Outlook Email Handling
• Windows OS Utilities
• OneDrive Cloud Storage
• Command Line Tools

Data Parsers

Structured Data

• JSON/YAML Parsing
• YAML Configuration
• XML/HTML Parsing
• HTML Processing
• LXML Processing
• DataFrame Processing

Unstructured Data

• PDF Text Extraction
• Image Processing
• Text File Handling
• Color Processing

Data Structure Parsers

• Array Processing
• Dictionary Operations
• List Processing
• String Operations
• Number Processing
• Object Handling

Specialized Formats

• Archive Files (TGZ)
• Object Serialization
• Folder Operations

Data Pipelines

• Parallel Processing
• Asynchronous Workflows
• Streaming Data Flows
• Conditional Execution
• Logging Integration
• Generic Pipeline
• Multiprocessing Helper

Network Operations

• Network Diagnostics
• Proxy Management
  • Free Proxy Sources
  • Proxy Load Testing
  • Proxy Testing
• Web Scraping
  • Scrape Validation
  • User Agent Management

Calendar & Date Utilities

• Brazilian Business Calendar
• US Business Calendar
• Date Handling

Geographic Data

• Brazilian Geography
• Worldwide Geography

Data Providers

• Brazilian Providers
  • ANBIMA Data API
  • INOA Provider
  • B3 Line API
  • B3 Margin Simulator
• Global Providers
  • Reuters Data

Trading & Automation

• MetaTrader 5
• Playwright WebDriver
• Selenium WebDriver

Communication & Notifications

• Slack Webhooks
• Microsoft Teams

Other Utilities

• Airflow Plugins
• Database Logging
• Log Creation
• Initial Setup
• Data Conversions
  • Base Converter
  • Expression Converter
• Security & Cryptography
• System Drive Management
• LLM Integration

---

🏗️ Data Structures & Algorithms

Queue Implementations

• Priority Queues
• Simple Deque
• Simple Queue

Stack Implementations

• Simple Stack

Tree Structures

• AVL Tree
• B-Tree

🚀 Getting Started

Prerequisites:

• Python ^3.12

• Pyenv

• Make (optional, below are some benefits of usage)

  Category	Example Targets	Benefit
  Package Lifecycle	build_package_pypi_org, clean_dist	Automated PyPI publishing
  Git Workflows	git_pull_force, gh_protect_main	Enforces branch policies
  CI/CD	gh_actions_local_tests	Local pipeline validation
  Code Generation	ingestion_concrete_creator	Factory pattern automation
  Dev Environment	vscode_install_extensions	Consistent IDE setup

Installation Guide:

• Option 1: Pip Install (Recommended) 📌 Available on PyPI

  
   #!/bin/bash
   pip install stpstone
   poetry install --no-root
   poetry shell
   

• Option 2: Build from Source

  
    #!/bin/bash
    git clone https://github.com/guilhermegor/stpstone.git
    cd stpstone
    pyenv install 3.12.8
    pyenv local 3.12.8
    poetry install --no-root
    poetry shell
    

• Setting Up Make (Optional for Build Automation)

  Platform	Instructions
  Windows	Install via MinGW 1. Download MinGW installer 2. Select mingw32-make during installation 3. Add C:\MinGW\bin to PATH 4. mingw32-make --version
  MacOS	Pre-installed or via Xcode CLI tools: xcode-select --install
  Linux	Install via package manager: sudo apt-get install build-essential

• Python Kernel Versioning (Pyenv): pyenv instructions to installation

🧪 Running Tests

Execute unit and integration tests:

#!/bin/bash
poetry run python -m unittest discover -s tests/unit -p "*.py" -v
poetry run python -m unittest discover -s tests/integration -p "*.py" -v

📂 Project Structure

stpstone/
│
├── 📁 .github/
│   ├── 📁 workflows/         # GitHub Actions CI/CD pipelines
│   ├── 📜 CODEOWNERS         # Code ownership definitions
│   └── 📜 PULL_REQUEST_TEMPLATE.md  # PR template
│
├── 📁 .vscode/               # VSCode configuration
│   └── ⚙️ settings.json      # Editor preferences and extensions
│
├── 📁 cli/                   # Command Line Interface components
│
├── 📁 data/                  # Data storage and management
│
├── 📁 docs/                  # Project documentation
│
├── 📁 examples/              # Example implementations
│
├── 📁 img/                   # Visual assets
│
├── 📁 stpstone/              # Core Python package
│   ├── 📁 _config/           # Configuration management
│   ├── 📁 analysis/          # Analytical components
│   ├── 📁 dsa/               # Data structures & algorithms
│   ├── 📁 ingestion/         # Data ingestion pipelines
│   ├── 📁 transformations/   # Data transformation logic
│   ├── 📁 utils/             # Shared utilities
│   └── 🐍 __init__.py        # Package initialization
│
├── 📁 tests/                 # Test suites
│   ├── 📁 unit/              # Unit tests
│   ├── 📁 integration/       # Integration tests
│   └── 📁 performance/       # Performance benchmarking
│
├── 📜 .gitignore             # Git ignore patterns
├── ⚙️ .pre-commit-config.yaml # Pre-commit hook configurations
├── � .python-version        # Pyenv version specification
├── 📜 LICENSE               # MIT License file
├── ⚙️ Makefile              # Automation tasks
├── 📦 poetry.lock           # Exact dependency versions
├── ⚙️ pyproject.toml        # Project metadata and dependencies
├── 📖 README.md             # Project overview
├── 📦 requirements.txt      # Production dependencies
├── 🔧 requirements-dev.txt  # Development dependencies
└── 💻 requirements-venv.txt # Virtual environment setup

👨‍💻 Authors

Guilherme Rodrigues

📜 License

This project is licensed under the MIT License - see LICENSE for details.

🙌 Acknowledgments

• Inspired by open-source financial libraries and tools

• This documentation follows a structure inspired by PurpleBooth's README-Template.md.

• Special thanks to Python community

🔗 Useful Links

• GitHub Repository

• Issue Tracker