fx-bharat 0.4.1

Ingestion utilities for downloading and storing RBI forex reference rates

fx-bharat

---

FxBharat is an end-to-end Python package that automatically retrieves foreign-exchange reference rates published by the Reserve Bank of India (RBI), normalizes the downloaded Excel/HTML workbooks, and stores them in your database of choice (PostgreSQL, MySQL/MariaDB, MongoDB, or SQLite when explicitly configured). SBI historical PDFs are sourced from the public GitHub archive at sahilgupta/sbi-fx-ratekeeper, while RBI and LME data are fetched live from their respective endpoints.

⚠️ Breaking change (0.4.0+): The bundled SQLite snapshot/resources have been removed. You must provide a database URL, and historical seeding now pulls SBI PDFs from the public GitHub archive while RBI/LME data are fetched live. Daily updates now use ingestion checkpoints to backfill any missed days up to today.

Historical data is ingested live from:

RBI reference rate archive (starting 01/04/2022)

SBI Forex PDFs sourced from the public GitHub archive sahilgupta/sbi-fx-ratekeeper (starting 01/04/2022)

LME (COPPER and ALUMINUM) scraped live (data generally available from 2008)

Provide your own database URL; no bundled database is shipped.

---

📌 Table of Contents

• Overview
• Data Source
• Installation
• Package Layout
• Database Schema
• Usage
• Quick Start
• Source Selection (RBI vs SBI)
• Connecting to your database
  • Checking Database Connectivity
  • PostgreSQL Example
  • MySQL/MariaDB Example
  • MongoDB Example
  • Ingestion Controls
• Backend Requirements
• Running Tests
• Design Philosophy
• Contributing
• License

---

Overview

FxBharat provides:

• 🔄 Automated Selenium workflow to download daily reference rates
• 📑 Parsing of RBI Excel/HTML into clean pandas DataFrames
• 💾 Storage via PostgreSQL, MySQL/MariaDB, MongoDB, or SQLite (when explicitly configured)
• 📈 Easy APIs to fetch latest rates or historical rollups
• 🧩 A clean façade (FxBharat) to simplify ingestion and queries
• 📦 Type-annotated, structured, and production-ready ingestion pipeline

All of this works default-first: install the package → start querying FX rates instantly.

---

Data Source

FxBharat retrieves daily reference exchange rates from:

• 👉 RBI Reference Rate Archive — https://www.rbi.org.in/Scripts/ReferenceRateArchive.aspx
• 👉 SBI Forex Card Rates PDF — https://sbi.bank.in/documents/16012/1400784/FOREX_CARD_RATES.pdf
• 👉 LME Copper & Aluminum (Westmetall) — LME Copper / LME Aluminum

Coverage:

• Historical seeding fetches from 01/04/2022 → today based on what you ingest into your DB (no bundled data is shipped).

Publication cadence (IST):

• RBI updates the reference rates on the archive page around 5:00 PM.
• SBI refreshes the Forex Card PDF around 10:00 AM.

If you seed RBI data before the daily publish window, the package exits early when the site returns "No Reference Rate Found."—re-run after the data is available.

Please respect the RBI/SBI website terms of service when running ingestion. The default retry and incremental settings are tuned to minimise load on the upstream providers.

Workflow:

1. Selenium downloads the RBI Excel/HTML reference rate workbook
2. BeautifulSoup4 + pandas parse and normalize the data
3. pypdf parses SBI's Forex Card PDF when you opt into the SBI source
4. SQLAlchemy or PyMongo persist these rows into your configured backend

The resulting dataset mirrors the RBI reference rates or SBI Forex card tables while keeping a source column to distinguish entries.

---

Installation

Install from PyPI

pip install fx-bharat

The installation includes:

• Selenium
• pandas
• BeautifulSoup4
• SQLAlchemy
• SQLite support

For local development

pip install -r requirements.txt
pip install -e .

---

Package Layout

fx_bharat/
    __init__.py               # FxBharat façade
    db/
        base_backend.py       # Unified DB backend interface
        relational_backend.py # SQLAlchemy helpers
        sqlite_backend.py     # SQLite adapter (explicit DB_URL required)
        postgres_backend.py   # PostgreSQL adapter
        mysql_backend.py      # MySQL/MariaDB adapter
        mongo_backend.py      # MongoDB adapter via PyMongo
        sqlite_manager.py     # SQLite utilities + schema creation
    ingestion/
        rbi_selenium.py       # Selenium automation
        rbi_workbook.py       # HTML/Excel → DataFrame converter
        rbi_csv.py            # Intermediate CSV helpers
        sbi_pdf.py            # SBI Forex Card PDF parser
        models.py             # Dataclasses for parsed rates
    seeds/
        populate_rbi_forex.py # Programmatic seeding logic
        populate_sbi_forex.py # SBI seeding logic (backfills PDFs into SQLite)
    scripts/
        populate_rbi_forex.py # Legacy CLI
        populate_sbi_forex.py # SBI CLI helper
    utils/
        date_range.py         # Date interval utilities
        logger.py             # Structured logging
        rbi.py                # RBI parsing constants
    py.typed                 # PEP 561 type hinting marker

---

Database Schema

FxBharat stores RBI and SBI snapshots in two normalized tables. A generated resources/schema.sql mirrors the schema for external databases.

erDiagram
    forex_rates_rbi {
        DATE rate_date PK
        TEXT currency PK
        REAL rate
        TIMESTAMP created_at
    }
    forex_rates_sbi {
        DATE rate_date PK
        TEXT currency PK
        REAL rate
        REAL tt_buy
        REAL tt_sell
        REAL bill_buy
        REAL bill_sell
        REAL travel_card_buy
        REAL travel_card_sell
        REAL cn_buy
        REAL cn_sell
        TIMESTAMP created_at
    }
    lme_copper_rates {
        DATE rate_date PK
        REAL price
        REAL price_3_month
        INTEGER stock
        TIMESTAMP created_at
    }
    lme_aluminum_rates {
        DATE rate_date PK
        REAL price
        REAL price_3_month
        INTEGER stock
        TIMESTAMP created_at
    }
    ingestion_metadata {
        TEXT source PK
        DATE last_ingested_date
        TIMESTAMP updated_at
    }
    forex_rates_sbi ||--|| forex_rates_rbi : "aligned by rate_date/currency"
    lme_copper_rates ||--|| lme_aluminum_rates : "daily LME cash seller"

---

Usage

Quick Start (External DB Required)

from datetime import date
from fx_bharat import FxBharat

# Provide your database URL (Postgres/MySQL/Mongo/SQLite)
fx = FxBharat(db_config="postgresql://user:pwd@localhost/forex")

# Seed historical RBI + SBI + LME (copper, aluminum) from 2022-04-01 to today
fx.seed()

# Get latest available snapshots (SBI first, then RBI)
latest = fx.rate()
print(latest)

# Get a specific day's snapshots (optional `rate_date`)
print(fx.rate(rate_date=date(2025, 11, 1)))

# Fetch a historical window
history = fx.history(date(2025, 10, 1), date(2025, 10, 31), frequency="weekly")

for snapshot in history:
    print(snapshot["rate_date"], snapshot["source"], snapshot["rates"].get("USD"))

Prefer SQLite? Supply your own path, e.g. FxBharat(db_config="sqlite:///./forex.db").

What these methods do:

• .seed(start_date, end_date) → Downloads & inserts missing entries directly into your DB (no bundled SQLite)

• .rate(rate_date=None) → Returns latest available SBI and RBI observations (or specific rate_date snapshots) with SBI first

• .history(start, end, frequency) → Supports

  • "daily"
  • "weekly"
  • "monthly"
  • "yearly"

• .history_lme(start, end, frequency, source_filter=None) → Returns LME snapshots for COPPER/ALUMINUM with the same frequency options

Seeding LME Copper & Aluminum

Run the new helpers to ingest daily LME cash seller prices (data is available from 2008 onwards):

fx.seed_lme("COPPER")
fx.seed_lme("ALUMINUM")

All seeding writes directly to your configured database; there is no bundled SQLite datastore.

LME History (Copper & Aluminum)

from datetime import date

# Daily snapshots for both metals
lme_history = fx.history_lme(
    from_date=date(2024, 1, 1),
    to_date=date(2024, 2, 1),
    frequency="daily",
)

# Monthly snapshots for just copper
copper_history = fx.history_lme(
    from_date=date(2024, 1, 1),
    to_date=date(2024, 6, 1),
    frequency="monthly",
    source_filter="COPPER",
)

Legacy note: the former .rates() helper now lives on as a deprecated alias of .history(); new code should prefer .history() or .historical().

Daily update shortcut

from fx_bharat import FxBharat

fx = FxBharat(db_config="postgresql://user:pwd@localhost/forex")

# Pull any missing RBI/SBI/LME data since the last ingested day up to today
fx.update_daily()

Source Selection (RBI vs SBI)

FxBharat now stores RBI and SBI data in separate tables/collections. Query helpers always return SBI snapshots first (when present) followed by RBI snapshots. Use the unified seed(from_date=..., to_date=..., source=...) helper to ingest targeted ranges; calling seed() with no arguments replays data for both sources (and LME) from 2022-04-01 through today and stores downloaded SBI PDFs under your chosen resource_dir.

---

Ingestion Controls

• source_filter on rate, history, and rates lets you restrict output to "rbi" or "sbi" while keeping blended ordering.
• source_filter on history_lme accepts "COPPER" or "ALUMINUM" (case-insensitive).
• Historical seeding defaults to the window from 2022-04-01; set incremental=True if you want to resume from the database ingestion checkpoints instead.
• Pass dry_run=True to seed, seed_sbi_historical, or seed_rbi_forex to validate connectivity without writing rows.
• Yearly aggregations now select the most recent snapshot per calendar year for each source.
• seed accepts optional from_date, to_date, and source parameters to restrict ingestion. When you omit them, FxBharat ingests 2022-04-01 → today for both sources and LME.

2. Connecting to Your Own Database

You can use PostgreSQL, MySQL, MongoDB, or your own SQLite file.

Checking Database Connectivity (External)

When using an external DB (PostgreSQL/MySQL/MongoDB), you may want to verify that the connection is valid before running migrate() or seed().

FxBharat provides:

success, error = fx.connection()

• success → True/False
• error → The raw exception message from the database driver

Example: Failed Connection Output

success, error = fx.connection()

if not success:
    print("Connection failed:")
    print(error)

Typical output if the database does not exist:

(psycopg2.OperationalError) connection to server at "localhost" (127.0.0.1), port 5432 failed:
FATAL: database "forex-db" does not exist

This helps diagnose DSN, credentials, port issues, or missing databases before ingestion begins.

Example: PostgreSQL

from datetime import date

from fx_bharat import FxBharat

fx = FxBharat(db_config='postgresql://postgres:postgres@localhost/forex')

success, error = fx.connection()
if not success:
    print(error)
    exit(1)

fx.migrate()
# =>  will migrate the data from Sqlite to PostgreSQL (forex + LME)
# fx.migrate(from_date=date(2024, 1, 1), to_date=date(2024, 12, 31))

# Latest Forex entry
rate = fx.rate()
print(rate)
# => {'rate_date': datetime.date(2025, 11, 18), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}

# Specific Forex entries by date (optional rate_date)
historical_rates = fx.rate(rate_date=date(2025, 11, 1))
print(historical_rates)

# Weekly/daily Forex entries (SBI first, then RBI)
rates = fx.history(from_date=date(2025, 11, 1), to_date=date.today(), frequency='daily')
print(rates[:2])

# Monthly Forex entries
rates = fx.history(from_date=date(2025, 9, 1), to_date=date.today(), frequency='monthly')
print(rates)

# Yearly Forex entries
rates = fx.history(from_date=date(2023, 9, 1), to_date=date.today(), frequency='yearly')
print(rates)

# Seed SBI + RBI Forex rates into PostgreSQL as well
fx.seed()

Example: MySQL/MariaDB

from datetime import date

from fx_bharat import FxBharat

fx = FxBharat(db_config='mysql://user:pass@localhost:3306/forex')

success, error = fx.connection()
if not success:
    print(error)
    exit(1)

fx.migrate()
# =>  will migrate the data from Sqlite to MySQL (forex + LME)
# fx.migrate(from_date=date(2024, 1, 1), to_date=date(2024, 12, 31))

# Latest Forex entry
rate = fx.rate()
print(rate)
# => {'rate_date': datetime.date(2025, 11, 18), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}

# Specific Forex entry by date (optional rate_date)
historical_rate = fx.rate(rate_date=date(2025, 11, 1))
print(historical_rate)
# => {'rate_date': datetime.date(2025, 11, 1), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}

# weekly Forex entries
rates = fx.history(from_date=date(2025, 11, 1), to_date=date.today(), frequency='daily')
print(rates[:2])
# => [{'rate_date': datetime.date(2025, 11, 3), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# monthly Forex entries
rates = fx.history(from_date=date(2025, 9, 1), to_date=date.today(), frequency='monthly')
print(rates)
# => [{'rate_date': datetime.date(2025, 9, 30), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# yearly Forex entries
rates = fx.history(from_date=date(2023, 9, 1), to_date=date.today(), frequency='yearly')
print(rates)
# => [{'rate_date': datetime.date(2023, 12, 29), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# Seed SBI Forex Card rates into MySQL as well
fx.seed()
print(fx.rate())

fx.seed()

Example: MongoDB

from datetime import date

from fx_bharat import FxBharat

fx = FxBharat(db_config='mongodb://127.0.0.1:27017/forex')

success, error = fx.connection()
if not success:
    print(error)
    exit(1)
    
fx.migrate()
# =>  will migrate the data from Sqlite to MongoDB (forex + LME)
# fx.migrate(from_date=date(2024, 1, 1), to_date=date(2024, 12, 31))

# Latest Forex entry
rate = fx.rate()
print(rate)
# => {'rate_date': datetime.date(2025, 11, 18), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}

# Specific Forex entry by date (optional rate_date)
historical_rate = fx.rate(rate_date=date(2025, 11, 1))
print(historical_rate)
# => {'rate_date': datetime.date(2025, 11, 1), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}

# weekly Forex entries
rates = fx.history(from_date=date(2025, 11, 1), to_date=date.today(), frequency='daily')
print(rates[:2])
# => [{'rate_date': datetime.date(2025, 11, 3), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# monthly Forex entries
rates = fx.history(from_date=date(2025, 9, 1), to_date=date.today(), frequency='monthly')
print(rates)
# => [{'rate_date': datetime.date(2025, 9, 30), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# yearly Forex entries
rates = fx.history(from_date=date(2023, 9, 1), to_date=date.today(), frequency='yearly')
print(rates)
# => [{'rate_date': datetime.date(2023, 12, 29), 'base_currency': 'INR', 'source': 'RBI', 'rates': {...}}, ...]

# Seed SBI Forex Card rates into MongoDB as well
fx.seed()
print(fx.rate())

fx.seed()

FxBharat internally sanitizes the DSN to satisfy PyMongo.

---

Backend Requirements

Optional dependency matrix

Backend	Required extra package(s)
PostgreSQL	psycopg2-binary
MySQL	mysqlclient or PyMySQL
MongoDB	pymongo

SQLite works with no external drivers.

Install extras manually when needed:

pip install psycopg2-binary
pip install PyMySQL
pip install pymongo

---

Running Tests

pytest

To collect coverage with the built-in settings:

pytest --cov=fx_bharat --cov-report=term-missing
coverage report

If you prefer coverage run, make sure you do not also pass --cov to pytest; running both simultaneously can zero-out the generated .coverage file and lead to the 0% report seen in CI.

---

Design Philosophy

FxBharat is built on the following principles:

🧱 Explicit configuration

Provide your database URL up front; ingestion streams directly into your DB of choice.

🔌 Plug-and-play backends

The same APIs work across SQLite, PostgreSQL, MySQL, or MongoDB.

🛠 Extensible architecture

All ingestion and persistence layers are modular and override-able.

🔁 Idempotent ingestion

seed() can be run safely multiple times without duplicate entries.

---

Migration Notes

See MIGRATIONS.md for upgrade guidance from 0.1.0 → 0.2.0 and 0.2.1, including the new ingestion metadata table and unified seed API.

---

Contributing

Pull requests are welcome! You can contribute to:

• New ingestion capabilities
• Error handling & retry logic
• Additional backends
• Documentation improvements
• Performance optimizations

Open an issue to discuss major changes before submitting a PR.

---

License

Copyright (c) 2025 Vindago Innovations LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.