datamule 0.26

Making it easier to use SEC filings.

datamule

A python package to make using SEC filings easier. Integrated with datamule's APIs and datasets.

features

current:

• monitor edgar for new filings
• parse textual filings into simplified html, interactive html, or structured json.
• download sec filings quickly and easily
• download datasets such as every MD&A from 2024 or every 2024 10K converted to structured json

installation

pip install datamule

quickstart:

import datamule as dm
downloader = dm.Downloader()
downloader.download_using_api(form='10-K',ticker='AAPL')

documentation

indexer

indexer = dm.Indexer()

indexer.run() constructs the locations of filings for the downloader and stores it to 'data/submissions_index.csv'. If download is set to True, the indexer downloads the locations using the prebuilt indices from Dropbox. The prebuilt indices are typically updated every few days.

It uses the submissions endpoint, and submissions archive endpoint.

TODO: add indexer run option using EFTS endpoint

indexer.run(download=True)

indexer.watch() returns True when new filings are posted to edgar.

It uses the EFTS endpoint.

watching all companies and forms

print("Monitoring SEC EDGAR for changes...")
changed_bool = indexer.watch(1,silent=False)
if changed_bool:
    print("New filing detected!")

watching specific companies and forms

print("Monitoring SEC EDGAR for changes...")
changed_bool = indexer.watch(1,silent=False,cik=['0001267602','0001318605'],form=['3','S-8 POS'])
if changed_bool:
    print("New filing detected!")

TODO: add args for company and ticker.

downloader

downloader = dm.Downloader()

downloads and downloads_using_api

downloader.download() downloads filings using the indices.

# Example 1: Download all 10-K filings for Tesla using CIK
downloader.download(form='10-K', cik='1318605', output_dir='filings')

# Example 2: Download 10-K filings for Tesla and META using CIK
downloader.download(form='10-K', cik=['1318605','1326801'], output_dir='filings')

# Example 3: Download 10-K filings for Tesla using ticker
downloader.download(form='10-K', ticker='TSLA', output_dir='filings')

# Example 4: Download 10-K filings for Tesla and META using ticker
downloader.download(form='10-K', ticker=['TSLA','META'], output_dir='filings')

# Example 5: Download every form 3 for a specific date
downloader.download(form ='3', date='2024-05-21', output_dir='filings')

# Example 6: Download every 10K for a year
downloader.download(form='10-K', date=('2024-01-01', '2024-12-31'), output_dir='filings')

# Example 7: Download every form 4 for a list of dates
downloader.download(form = '4',date=['2024-01-01', '2024-12-31'], output_dir='filings')

downloader.download_using_api() downloads filings using the datamule API instead. For more information look at SEC Router.

It uses the datamule sec router endpoint.

downloader.download_using_api(form='10-K',ticker='AAPL')

Both functions operate mostly the same. If return_urls is set to True, returns filing primary document urls instead of downloading them. If human_readable = True, it will download human readable versions of the filings. For more information look at the Human Readable Jupyter Notebook

download_datasets

downloader.download_dataset('10K')
downloader.download_dataset('MDA')

Need a better way to store datasets, as I'm running out of storage. Currently stored on Dropbox 2gb free tier.

parsing

Uses endpoint: https://jgfriedman99.pythonanywhere.com/parse_url with params url and return_type. Current endpoint can be slow. If it's too slow for your use-case, please contact me.

simplified html

simplified_html = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm',return_type='simplify')

Download Example

interactive html

interactive_html = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm',return_type='interactive')

Download Example

json

d = dm.parse_textual_filing(url='https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/tsla-20211231.htm',return_type='json')

Download Example

TODO

• standardize accession number to not include '-'. Currently db does not have '-' but submissions_index.csv does.
• add code to convert parsed json to interactive html
• add mulebot

Update Log

9/16/24 v0.26

• added indexer.watch(interval,cik,form) to monitor when EDGAR updates. v0.25
• added human_readable option to download, and download_using_api.

9/15/24

• fixed downloading filings overwriting each other due to same name.

9/14/24

• added support for parser API

9/13/24

• added download_datasets
• added option to download indices
• added support for jupyter notebooks

9/9/24

• added download_using_api(self, output_dir, **kwargs). No indices required.

9/8/24

• Added integration with datamule's SEC Router API

9/7/24

• Simplified indices approach
• Switched from pandas to polar. Loading indices now takes under 500 milliseconds.