pyetfdb-scraper 0.3

Scrape ETFs from ETFDB

pyetfdb_scraper: Free ETF data at your fingertips

pyetfdb_scraper is a Python library for extracting ETF data directly from ETFDB, a website providing one of the largest ETF Databases containing ETFs from a vast range of asset classes, industries, issuers, and investment styles.

Quick Start

Install with pip as a package pip. See the pip package here https://pypi.org/project/pyetfdb-scraper/.

pip install pyetfdb-scraper

from pyetfdb_scraper import etf

Example Usage

from pyetfdb_scraper.etf import ETF,load_etfs
# returns list of available ETFs.
etfs = load_etfs()

>>> ['SPY', 'IVV', 'VTI', 'VOO', 'QQQ', 'VEA', 'IEFA', ...]

Retrieve info about a single ticker

# Load ETF
ivv = ETF('IVV')

print(ivv.info)

Results

>>> {
    "vitals": {
        "etf_name": "iShares Core S&P 500 ETF",
        "issuer": "BlackRock, Inc.",
        "issuer_link": "/issuer/blackrock-inc/",
        "brand": "iShares",
        "brand_link": "/issuer/ishares/",
        "structure": "ETF",
        "structure_link": "",
        "expense_ratio": "0.03%",
        "hompage_link": "http://us.ishares.com/product_info/fund/overview/IVV.htm?qt=IVV",
        "inception": "May 15, 2000",
        "index_tracked": "S&P 500 Index",
        "index_tracked_link": "/index/sp-500-index/",
    },
    "dbtheme": {
        "category": "Large Cap Growth Equities",
        "category_link": "",
        "asset_class": "Equity",
        "asset_class_link": "/etfs/asset-class/equity/",
        "asset_class_size": "Large-Cap",
        "asset_class_size_link": "/etfs/size/large-cap/",
        "asset_class_style": "Blend",
        "asset_class_style_link": "/etfs/style/blend/",
        "general_region": "North America",
        "general_region_link": "/etfs/region/north-america/",
        "specific_region": "U.S.",
        "specific_region_link": "/etfs/country/us/",
    },
    "fact_set": {
        "segment": ["Equity: U.S.  -  Large Cap"],
        "category": ["Size and Style"],
        "focus": ["Large Cap"],
        "niche": ["Broad-based"],
        "strategy": ["Vanilla"],
        "weighting_scheme": ["Market Cap"],
    },
    "analyst_report": "Another alternative is VOO, which is slightly cheaper and is eligible for commission free trading within Vanguard accounts. Beyond the S&P 500, RSP may be another alternative worth a closer look; that ETF, which is a bit more expensive, holds all stocks in the S&P 500 but gives an equivalent weighting to each. As such, it might be attractive to investors looking to steer clear of the potential inefficiencies in market cap weighting methodologies.",
    "trade_data": {
        "open": "",
        "volume": "",
        "day_low": "",
        "day_high": "",
        "52_week_low": "$376.34",
        "52_week_high": "$491.10",
        "aum": "$416,620.0 M",
        "shares": "854.5 M",
    },
    "historical_trade_data": {
        "1_month_avg_volume": "5,672,082",
        "3_month_avg_volume": "5,182,910",
    },
    "alternative_etfs": [
        {
            "type": "Cheapest",
            "ticker": "SFY",
            "expense_ratio": "0.00%",
            "assets": "$622.0 M",
            "avg_daily_volume": "175,030",
            "ytd_return": "1.89%",
        },
        {
            "type": "Largest (AUM)",
            "ticker": "SPY",
            "expense_ratio": "0.09%",
            "assets": "$485.9 B",
            "avg_daily_volume": "79 M",
            "ytd_return": "2.68%",
        },
        {
            "type": "Most Liquid (Volume)",
            "ticker": "SPY",
            "expense_ratio": "0.09%",
            "assets": "$485.9 B",
            "avg_daily_volume": "79 M",
            "ytd_return": "2.68%",
        },
        {
            "type": "Top YTD Performer",
            "ticker": "WUGI",
            "expense_ratio": "0.75%",
            "assets": "$25.1 M",
            "avg_daily_volume": "2,590",
            "ytd_return": "8.00%",
        },
    ],
    "other_alternative_etfs": [
        {
            "type": "Cheapest",
            "ticker": "BKLC",
            "expense_ratio": "0.00%",
            "assets": "$2.1 B",
            "avg_daily_volume": "78,895",
            "ytd_return": "2.60%",
        },
        {
            "type": "Largest (AUM)",
            "ticker": "SPY",
            "expense_ratio": "0.09%",
            "assets": "$485.9 B",
            "avg_daily_volume": "79 M",
            "ytd_return": "2.68%",
        },
        {
            "type": "Most Liquid (Volume)",
            "ticker": "SPY",
            "expense_ratio": "0.09%",
            "assets": "$485.9 B",
            "avg_daily_volume": "79 M",
            "ytd_return": "2.68%",
        },
        {
            "type": "Top YTD Performer",
            "ticker": "AMOM",
            "expense_ratio": "0.75%",
            "assets": "$16.0 M",
            "avg_daily_volume": "4,800",
            "ytd_return": "7.15%",
        },
    ],
}

print(ivv.to_dict())

Results

>>> {
    "info": {
        "vitals": {
            "issuer": "BlackRock, Inc.",
            "issuer_link": "/issuer/blackrock-inc/",
            "brand": "iShares",
            "brand_link": "/issuer/ishares/",
            "structure": "ETF",
            "structure_link": "",
            "expense_ratio": "0.03%",
            "hompage_link": "http://us.ishares.com/product_info/fund/overview/IVV.htm?qt=IVV",
            "inception": "May 15, 2000",
            "index_tracked": "S&P 500 Index",
            "index_tracked_link": "/index/sp-500-index/",
            "etf_name": "iShares Core S&P 500 ETF",
        },
        "dbtheme": {
            "category": "Large Cap Growth Equities",
            "category_link": "",
            "asset_class": "Equity",
            "asset_class_link": "/etfs/asset-class/equity/",
            "asset_class_size": "Large-Cap",
            "asset_class_size_link": "/etfs/size/large-cap/",
            "asset_class_style": "Blend",
            "asset_class_style_link": "/etfs/style/blend/",
            "general_region": "North America",
            "general_region_link": "/etfs/region/north-america/",
            "specific_region": "U.S.",
            "specific_region_link": "/etfs/country/us/",
        },
        "fact_set": {
            "segment": ["Equity: U.S.  -  Large Cap"],
            "category": ["Size and Style"],
            "focus": ["Large Cap"],
            "niche": ["Broad-based"],
            "strategy": ["Vanilla"],
            "weighting_scheme": ["Market Cap"],
        },
        "analyst_report": "Another alternative is VOO, which is slightly cheaper and is eligible for commission free trading within Vanguard accounts. Beyond the S&P 500, RSP may be another alternative worth a closer look; that ETF, which is a bit more expensive, holds all stocks in the S&P 500 but gives an equivalent weighting to each. As such, it might be attractive to investors looking to steer clear of the potential inefficiencies in market cap weighting methodologies.",
        "trade_data": {
            "open": "",
            "volume": "",
            "day_low": "",
            "day_high": "",
            "52_week_low": "$376.34",
            "52_week_high": "$491.10",
            "aum": "$416,620.0 M",
            "shares": "854.5 M",
        },
        "historical_trade_data": {
            "1_month_avg_volume": "5,672,082",
            "3_month_avg_volume": "5,182,910",
        },
        "alternative_etfs": [
            {
                "type": "Cheapest",
                "ticker": "SFY",
                "expense_ratio": "0.00%",
                "assets": "$622.0 M",
                "avg_daily_volume": "175,030",
                "ytd_return": "1.89%",
            },
            {
                "type": "Largest (AUM)",
                "ticker": "SPY",
                "expense_ratio": "0.09%",
                "assets": "$485.9 B",
                "avg_daily_volume": "79 M",
                "ytd_return": "2.68%",
            },
            {
                "type": "Most Liquid (Volume)",
                "ticker": "SPY",
                "expense_ratio": "0.09%",
                "assets": "$485.9 B",
                "avg_daily_volume": "79 M",
                "ytd_return": "2.68%",
            },
            {
                "type": "Top YTD Performer",
                "ticker": "WUGI",
                "expense_ratio": "0.75%",
                "assets": "$25.1 M",
                "avg_daily_volume": "2,590",
                "ytd_return": "8.00%",
            },
        ],
        "other_alternative_etfs": [
            {
                "type": "Cheapest",
                "ticker": "BKLC",
                "expense_ratio": "0.00%",
                "assets": "$2.1 B",
                "avg_daily_volume": "78,895",
                "ytd_return": "2.60%",
            },
            {
                "type": "Largest (AUM)",
                "ticker": "SPY",
                "expense_ratio": "0.09%",
                "assets": "$485.9 B",
                "avg_daily_volume": "79 M",
                "ytd_return": "2.68%",
            },
            {
                "type": "Most Liquid (Volume)",
                "ticker": "SPY",
                "expense_ratio": "0.09%",
                "assets": "$485.9 B",
                "avg_daily_volume": "79 M",
                "ytd_return": "2.68%",
            },
            {
                "type": "Top YTD Performer",
                "ticker": "AMOM",
                "expense_ratio": "0.75%",
                "assets": "$16.0 M",
                "avg_daily_volume": "4,800",
                "ytd_return": "7.15%",
            },
        ],
    },
    "expense": {
        "tax_analysis": {
            "max_short_term_capital_gains_rate": ["39.60%"],
            "max_long_term_capital_gains_rate": ["20.00%"],
            "tax_on_distributions": ["Qualified dividends"],
            "distributes_k1": ["No"],
        },
        "expense_ratio_analysis": [
            {"ivv": "0.03%"},
            {"etf_database_category_average": "0.37%"},
            {"factset_segment_average": "0.59%"},
        ],
    },
    "holdings": {
        "top_holdings": [
            {
                "symbol": "AAPL",
                "holding": "Apple Inc.",
                "share": "7.18%",
                "url": "https://etfdb.com/stock/AAPL/",
            },
            {
                "symbol": "MSFT",
                "holding": "Microsoft Corporation",
                "share": "6.50%",
                "url": "https://etfdb.com/stock/MSFT/",
            },
            {
                "symbol": "AMZN",
                "holding": "Amazon.com, Inc.",
                "share": "3.32%",
                "url": "https://etfdb.com/stock/AMZN/",
            },
            {
                "symbol": "NVDA",
                "holding": "NVIDIA Corporation",
                "share": "2.95%",
                "url": "https://etfdb.com/stock/NVDA/",
            },
            {
                "symbol": "GOOGL",
                "holding": "Alphabet Inc. Class A",
                "share": "2.03%",
                "url": "https://etfdb.com/stock/GOOGL/",
            },
            {
                "symbol": "META",
                "holding": "Meta Platforms Inc. Class A",
                "share": "1.83%",
                "url": "https://etfdb.com/stock/META/",
            },
            {
                "symbol": "TSLA",
                "holding": "Tesla, Inc.",
                "share": "1.82%",
                "url": "https://etfdb.com/stock/TSLA/",
            },
            {
                "symbol": "GOOG",
                "holding": "Alphabet Inc. Class C",
                "share": "1.75%",
                "url": "https://etfdb.com/stock/GOOG/",
            },
            {
                "symbol": "BRK.B",
                "holding": "Berkshire Hathaway Inc. Class B",
                "share": "1.66%",
                "url": "https://etfdb.com/stock/BRK.B/",
            },
            {
                "symbol": "UNH",
                "holding": "UnitedHealth Group Incorporated",
                "share": "1.25%",
                "url": "https://etfdb.com/stock/UNH/",
            },
            {
                "symbol": "JPM",
                "holding": "JPMorgan Chase & Co.",
                "share": "1.22%",
                "url": "https://etfdb.com/stock/JPM/",
            },
            {
                "symbol": "JNJ",
                "holding": "Johnson & Johnson",
                "share": "1.17%",
                "url": "https://etfdb.com/stock/JNJ/",
            },
            {
                "symbol": "XOM",
                "holding": "Exxon Mobil Corporation",
                "share": "1.16%",
                "url": "https://etfdb.com/stock/XOM/",
            },
            {
                "symbol": "V",
                "holding": "Visa Inc. Class A",
                "share": "1.03%",
                "url": "https://etfdb.com/stock/V/",
            },
            {
                "symbol": "AVGO",
                "holding": "Broadcom Inc.",
                "share": "0.98%",
                "url": "https://etfdb.com/stock/AVGO/",
            },
        ],
        "holding_comparison": [
            {
                "number_of_holdings": "1000",
                "etf_database_category_average": "418",
                "factset_segment_average": "173",
            },
            {
                "pct_of_assets_in_top_10": "41.95%",
                "etf_database_category_average": "43.40%",
                "factset_segment_average": "60.51%",
            },
            {
                "pct_of_assets_in_top_15": "51.15%",
                "etf_database_category_average": "52.16%",
                "factset_segment_average": "65.06%",
            },
            {
                "pct_of_assets_in_top_50": "83.62%",
                "etf_database_category_average": "81.26%",
                "factset_segment_average": "81.64%",
            },
        ],
        "size_comparison": [
            {
                "large_(>12.9b)": "98.31%",
                "etf_database_category_average": "86.63%",
                "factset_segment_average": "46.30%",
            },
            {
                "mid_(>2.7b)": "1.49%",
                "etf_database_category_average": "5.88%",
                "factset_segment_average": "3.26%",
            },
            {
                "small_(>600m)": "0.00%",
                "etf_database_category_average": "0.58%",
                "factset_segment_average": "0.09%",
            },
            {
                "micro_(<600m)": "0.00%",
                "etf_database_category_average": "0.12%",
                "factset_segment_average": "0.01%",
            },
        ],
    },
    "holdings_analysis": [
        {"North, Central and South America": 99.87, "Other": 0.2},
        {
            "United States": 96.79,
            "Ireland": 1.61,
            "United Kingdom": 0.66,
            "Switzerland": 0.43,
            "Other": 0.2,
            "Netherlands": 0.14,
            "Canada": 0.13,
            "Bermuda": 0.11,
        },
        {
            "Technology Services": 21.08,
            "Electronic Technology": 18.45,
            "Finance": 12.34,
            "Health Technology": 9.51,
            "Retail Trade": 7.75,
            "Consumer Non-Durables": 4.48,
            "Producer Manufacturing": 3.61,
            "Consumer Services": 3.4,
            "Energy Minerals": 3.11,
            "Commercial Services": 2.92,
            "Utilities": 2.25,
            "Health Services": 2.21,
            "Consumer Durables": 1.91,
            "Process Industries": 1.78,
            "Transportation": 1.76,
            "Communications": 0.93,
            "Distribution Services": 0.92,
            "Industrial Services": 0.92,
            "Non-Energy Minerals": 0.54,
            "CASH": 0.2,
        },
        {"Large": 98.31, "Mid": 1.49, "Small": 0, "Micro": 0},
        {},
        {"Share/Common/Ordinary": 99.87, "CASH": 0.2},
        {
            "Technology Services": 21.08,
            "Electronic Technology": 18.45,
            "Finance": 12.34,
            "Health Technology": 9.51,
            "Retail Trade": 7.75,
            "Consumer Non-Durables": 4.48,
            "Producer Manufacturing": 3.61,
            "Consumer Services": 3.4,
            "Energy Minerals": 3.11,
            "Commercial Services": 2.92,
            "Utilities": 2.25,
            "Health Services": 2.21,
            "Consumer Durables": 1.91,
            "Process Industries": 1.78,
            "Transportation": 1.76,
            "Communications": 0.93,
            "Distribution Services": 0.92,
            "Industrial Services": 0.92,
            "Non-Energy Minerals": 0.54,
            "CASH": 0.2,
        },
    ],
    "performance": [
        {
            "1_month_return": "3.05%",
            "etf_database_category_average": "2.89%",
            "factset_segment_average": None,
        },
        {
            "3_month_return": "15.68%",
            "etf_database_category_average": "16.58%",
            "factset_segment_average": None,
        },
        {
            "ytd_return": "2.66%",
            "etf_database_category_average": "2.55%",
            "factset_segment_average": None,
        },
        {
            "1_year_return": "23.86%",
            "etf_database_category_average": "24.93%",
            "factset_segment_average": None,
        },
        {
            "3_year_return": "10.10%",
            "etf_database_category_average": "5.02%",
            "factset_segment_average": None,
        },
        {
            "5_year_return": "15.07%",
            "etf_database_category_average": "8.85%",
            "factset_segment_average": None,
        },
    ],
    "dividends": [
        {
            "dividend": "$ 1.93",
            "etf_database_category_average": "$ 0.35",
            "factset_segment_average": "$ 0.22",
        },
        {
            "dividend_date": "2023-12-20",
            "etf_database_category_average": "N/A",
            "factset_segment_average": "N/A",
        },
        {
            "annual_dividend_rate": "$ 6.90",
            "etf_database_category_average": "$ 0.92",
            "factset_segment_average": "$ 0.64",
        },
        {
            "annual_dividend_yield": "1.41%",
            "etf_database_category_average": "1.11%",
            "factset_segment_average": "1.42%",
        },
    ],
    "technicals": {
        "indicators": {
            "20_day_ma": "$478.81",
            "60_day_ma": "$461.74",
            "macd_15_period": "10.67",
            "macd_100_period": "40.06",
            "williams_%_range_10_day": "4.05",
            "williams_%_range_20_day": "3.43",
            "rsi_10_day": "77",
            "rsi_20_day": "71",
            "rsi_30_day": "68",
            "ultimate_oscillator": "65",
            "lower_bollinger_(10_day)": "$471.45",
            "upper_bollinger_(10_day)": "$492.45",
            "lower_bollinger_(20_day)": "$467.48",
            "upper_bollinger_(20_day)": "$489.54",
            "lower_bollinger_(30_day)": "$465.05",
            "upper_bollinger_(30_day)": "$488.05",
            "support_level_1": "n/a",
            "support_level_2": "$486.67",
            "resistance_level_1": "n/a",
            "resistance_level_2": "$492.45",
            "stochastic_oscillator_%d_(1_day)": "63.66",
            "stochastic_oscillator_%d_(5_day)": "86.50",
            "stochastic_oscillator_%k_(1_day)": "63.23",
            "stochastic_oscillator_%k_(5_day)": "80.88",
            "tracking_difference_median_(%)": "-0.03",
            "tracking_difference_max_upside_(%)": "-0.02",
            "tracking_difference_max_downside_(%)": "-0.06",
            "median_premium_discount_(%)": "0.01",
            "maximum_premium_discount_(%)": "0.07",
            "average_spread_(%)": "2.01",
            "average_spread_($)": "2.01",
        },
        "volatility": {
            "5_day_volatility": ["193.65%"],
            "20_day_volatility": ["8.87%"],
            "50_day_volatility": ["9.26%"],
            "200_day_volatility": ["11.69%"],
            "beta": ["1.0"],
            "standard_deviation": ["25.88%"],
        },
    },
    "realtime_rankings": [
        {
            "metric": "Liquidity",
            "metric_realtime_rating": "A",
            "a+_metric_rated_etf": "SPY",
        },
        {
            "metric": "Expenses",
            "metric_realtime_rating": "A",
            "a+_metric_rated_etf": "BKLC",
        },
        {
            "metric": "Performance",
            "metric_realtime_rating": "B",
            "a+_metric_rated_etf": "ESPO",
        },
        {
            "metric": "Volatility",
            "metric_realtime_rating": "B+",
            "a+_metric_rated_etf": "NUSI",
        },
        {
            "metric": "Dividend",
            "metric_realtime_rating": "A-",
            "a+_metric_rated_etf": "QYLD",
        },
        {
            "metric": "Concentration",
            "metric_realtime_rating": "A-",
            "a+_metric_rated_etf": "VT",
        },
    ],
}

Help Needed!

I am working full-time, and as such don't have much time to constantly push commits or updates. I will appreciate if some help can be provided, such as:

• Unit tests for the current code
• ETF Category has yet to be updated

Contributing

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

If you are simply looking to start working with the codebase, navigate to the GitHub "issues" tab and start looking through interesting issues. You can also triage issues which may include reproducing bug reports, or asking for vital information such as version numbers or reproduction instructions. To read more about contributing, you can refer to CONTRIBUTING

License

GPLv3

Disclaimer

This package is built with some reference to the existing pyetf package.