datamule 0.331

Making it easier to use SEC filings.

datamule

A Python package to work with SEC filings at scale. Also includes Mulebot, an open-source chatbot for SEC data that does not require storage. Integrated with datamule's APIs and datasets.

Articles: How to deploy a financial chatbot to the internet in 5 minutes

Features

• Monitor EDGAR for new filings
• Parse textual filings into simplified HTML, interactive HTML, or structured JSON
• Download SEC filings quickly and easily
• Access datasets such as every 10-K since 2001, 2024 MD&A, 2024 10-K converted to structured JSON, and more.
• Interact with SEC data using MuleBot

Table of Contents

• Installation
• Quick Start
• Usage
  • Downloader
  • Parsing
  • Filing Viewer
  • MuleBot
• Examples
• Known Issues
• Roadmap
• Contributing
• License
• Change Log
• Other Useful SEC Packages

Installation

Basic installation:

pip install datamule

Installation with additional features:

pip install datamule[filing_viewer]  # Install with filing viewer module
pip install datamule[mulebot]  # Install with MuleBot
pip install datamule[all]  # Install all extras

Available extras:

• filing_viewer: Includes dependencies for the filing viewer module
• mulebot: Includes MuleBot for interacting with SEC data
• mulebot_server: Includes Flask server for running MuleBot
• all: Installs all available extras

Quick Start

import datamule as dm

downloader = dm.Downloader()
downloader.download(form='10-K', ticker='AAPL')

Usage

Downloader

downloader = dm.Downloader()

Downloading Filings

Uses the EFTS API to retrieve filings locations, and the SEC API to download filings.

download(self, output_dir = 'filings',  return_urls=False,cik=None, ticker=None, form=None, date=None)

# Download all 10-K filings for Tesla using CIK
downloader.download(form='10-K', cik='1318605', output_dir='filings')

# Download 10-K filings for multiple companies using tickers
downloader.download(form='10-K', ticker=['TSLA', 'META'], output_dir='filings')

# Download every form 3 for a specific date
downloader.download(form='3', date='2024-05-21', output_dir='filings')

View the SEC Filing Glossary here or download the json file here.

Downloading Company Concepts XBRL

Uses the Company Concepts API to retrieve XBRL.

download_company_concepts(self, output_dir = 'company_concepts',cik=None, ticker=None)

View the XBRL Fact Glossary here or as a csv file here.

Changing Rate Limits

The SEC.gov officially supports 10 requests / second. In practice this is not the case. After heavy experimentation the downloader's default rate limit for sec.gov has been set to 7 requests / second. If you intend to download less than 1,000 filings at a time, setting the rate limit to 10 should be fine. If you need to download more than 10,000 filings, setting the rate limit to 5 will likely avoid rate limiting. Also, downloading at off-peak times will likely let you set higher rate-limits. Experiment Details

downloader.set_limiter('www.sec.gov', 10)

Datasets

Note: Dataset module is a WIP. Some downloads may be large.

Available datasets:

• 2024 10-K filings converted to JSON parsed_10k
• Management's Discussion and Analysis (MD&A) sections extracted from 2024 10-K filings mda
• Every Company Concepts XBRL xbrl (730mb)
• Every 10-K from 2001 to September 2024 10k_{year} e.g. 10k_2002. Takes ~ 2.5 minutes to download per year.

Also available on Dropbox and Zenodo

# Download all 2024 10-K filings converted to JSON
downloader.download_dataset('parsed_10k')

Note: I'm currently exploring ways to speed up dataset downloads. Zenodo has great hosting, but caps download speed at ~1-5mb/s. I can workaround this limit by uploading data in smaller chunks, but I'd like to find a better solution.

Monitoring for New Filings

print("Monitoring SEC EDGAR for changes...")
changed_bool = downloader.watch(1, silent=False, cik=['0001267602', '0001318605'], form=['3', 'S-8 POS'])
if changed_bool:
    print("New filing detected!")

Parsing

Parse SEC XBRL

Parses XBRL in JSON format to tables. SEC XBRL. See Parse every SEC XBRL to csv in ten minutes

from datamule import parse_company_concepts
table_dict_list = parse_company_concepts(company_concepts) # Returns a list of tables with labels

Parse Textual Filings into structured data

Parse textual filings into different formats. Uses datamule parser endpoint. If it is too slow for your use-case let me know. A faster endpoint is coming soon.

# Simplified HTML
simplified_html = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm', return_type='simplify')

# Interactive HTML
interactive_html = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm', return_type='interactive')

# JSON
json_data = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm', return_type='json')

Table Parser

Parses html tables into a useful form. This exists, mostly, as a placeholder. Links: Visual Example

table_parser = TableParser()

Filing Viewer

Convert parsed filing JSON into HTML with features like a table of contents sidebar:

from datamule import parse_textual_filing
from datamule.filing_viewer import create_interactive_filing

data = parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm', return_type='json')
create_interactive_filing(data)

Try out the Filings Viewer here. Note: This is an older version with bugs, that will be updated with the next release of the Parser API.

Mulebot

Interact with SEC data using MuleBot. Mulebot uses tool calling to interface with SEC and datamule endpoints.

from datamule.mulebot import MuleBot
mulebot = MuleBot(openai_api_key)
mulebot.run()

To use Mulebot you will need an OpenAI API Key.

Mulebot Server

Mulebot server is a customizable front-end for Mulebot. Example

Artifacts:

• Filing Viewer
• Company Facts Viewer
• List Viewer

from datamule.mulebot.mulebot_server import server

def main():
    # Your OpenAI API key
    api_key = openai_api_key
    server.set_api_key(api_key)

    # Run the server
    print("Starting MuleBotServer...")
    server.run(debug=True, host='0.0.0.0', port=5000)

if __name__ == "__main__":
    main()

Quickstart

from datamule.mulebot.mulebot_server import server

def main():
    # Your OpenAI API key
    api_key = "sk-<YOUR_API_KEY>"
    server.set_api_key(api_key)

    # Run the server
    print("Starting MuleBotServer...")
    server.run(debug=True, host='0.0.0.0', port=5000)

if __name__ == "__main__":
    main()

Known Issues

• Some SEC files are malformed, which can cause parsing errors. For example, this Tesla Form D HTML from 2009 is missing a closing </meta> tag.

  Workaround:

  from lxml import etree
  
  with open('filings/000131860509000005primary_doc.xml', 'r', encoding='utf-8') as file:
      html = etree.parse(file, etree.HTMLParser())
  

• SEC Endpoints have issues. e.g. The EFTS search returns the primary doc url for https://www.sec.gov/Archives/edgar/data/1036804/000095011601000004/ as https://www.sec.gov/Archives/edgar/data/1036804/000095011601000004/0001.txt, when it should send you to https://www.sec.gov/Archives/edgar/data/1036804/000095011601000004/0000950116-01-000004.txt.

This is currently a low priority issue. Let me know if you need the data, and I'll move it up the priority list.

Roadmap

• Downloader add option to save progress
• mulebot - look at adding summarization. Add some protections to too many tokens being used + add options to allow summarization etc.
• Make sure mulebot is stateless.
• Fix code debt.
• filing viewer band-aid fix. will wait until mule parser update to devote more effort
• Paths may be messed up on non windows devices. Need to verify.
• Analytics?

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT LICENSE.

Change Log

Change Log.

---

Other Useful SEC Packages

• Edgartools
• SEC Parsers