The Federal Energy Regulatory Commission (FERC) has moved to collecting and distributing data using XBRL. XBRL is primarily designed for financial reporting, and has been adopted by regulators in the US and other countries. Much of the tooling in the XBRL ecosystem is targeted towards filers, and rendering individual filings in a human readable way, but there is very little targeted towards accessing and analyzing large collections of filings. This tool is designed to provide that functionality for FERC XBRL data. Specifically, it can extract data from a set of XBRL filings, and write that data to a SQLite database whose structure is generated from an XBRL Taxonomy. While each XBRL instance contains a reference to a taxonomy, this tool requires a path to a single taxonomy that will be used to interpret all instances being processed. This means even if instances were created from different versions of a Taxonomy, the provided taxonomy will be used when processing all of these instances, so the output database will have a consistent structure. For more information on the technical details of the XBRL extraction, see the docs.
As of this point in time this tool is only tested with FERC Form 1, but is intended to extend support to other FERC forms. It is possible it could be used with non-FER taxonomies with some tweaking, but we do not provide any official support for non-FERC data.
Usage
Installation
To install using conda, run the following command, and activate the environment.
$ conda env create -f environment.ymlActivate:
$ conda activate ferc-xbrl-extractCLI
This tool can be used as a library, as it is in PUDL, or there is a CLI provided for interacting with XBRL data. The only required options for the CLI are a path a single XBRL filing, or directory of XBRL filings, and a path to the SQLite database.
$ xbrl_extract {path_to_filings} {path_to_database}By default, the CLI will use the 2022 version of the FERC Form 1 Taxonomy to create the structure of the output database. To specify a different taxonomy use the --taxonomy option.
$ xbrl_extract {path_to_filings} {path_to_database} --taxonomy {url_of_taxonomy}Parsing XBRL filings can be a time consuming and CPU heavy task, so this tool implements some basic multiprocessing to speed this up. It uses a process pool to do this. There are two options for configuring the process pool, --batch-size and --workers. The batch size configures how many filings will be processed by each child process at a time, and workers specifies how many child processes to create in the pool. It may take some experimentation to get these options optimally configured.
$ xbrl_extract {path_to_filings} {path_to_database} --workers {number_of_processes} --batch-size {filings_per_batch}3 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file catalystcoop.ferc_xbrl_extractor-0.4.0.tar.gz.
File metadata
- Download URL: catalystcoop.ferc_xbrl_extractor-0.4.0.tar.gz
- Upload date:
- Size: 28.1 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.1 CPython/3.10.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e4dda23ae483b6421784f409215e77d8eef7d31137f0d38d0cdb2c7f2749b059
|
|
| MD5 |
dd2bc9c5199409b44863911dc674d388
|
|
| BLAKE2b-256 |
52899f3d49e7c480ed66ee436c617e3baefd50028ca5842ab8df91b247d1afbb
|
File details
Details for the file catalystcoop.ferc_xbrl_extractor-0.4.0-py3-none-any.whl.
File metadata
- Download URL: catalystcoop.ferc_xbrl_extractor-0.4.0-py3-none-any.whl
- Upload date:
- Size: 22.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.1 CPython/3.10.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fd2b51f54e1ecf31dc3b022ab77bf8b7203c482280ec686711e6b83769a1b46a
|
|
| MD5 |
3953b170fae347f030ba5483facb309f
|
|
| BLAKE2b-256 |
f0ffa3440d4e4d7a83bf66ee62136656b6f22d79d1e07446e1ed5fe0b88e1563
|
