Skip to main content

Tools for scraping personal financial data.

Project description

Python package for scraping personal financial data from financial institutions.

License: GPL v2 PyPI Build

This package may be useful on its own, but is specifically designed to be used with beancount-import.

Supported data sources


To install the most recent published package from PyPi, simply type:

pip install finance-dl

To install from a clone of the repository, type:

pip install .

or for development:

pip install -e .


Create a Python file like

Refer to the documentation of the individual scraper modules for details.

Basic Usage

You can run a scraping configuration named myconfig as follows:

python -m finance_dl.cli --config-module example_finance_dl_config --config myconfig

The configuration myconfig refers to a function named CONFIG_myconfig in the configuration module.

Make sure that your configuration module is accessible in your Python sys.path. Since sys.path includes the current directory by default, you can simply run this command from the directory that contains your configuration module.

By default, the scrapers run fully automatically, and the ones based on selenium and chromedriver run in headless mode. If the initial attempt for a selenium-based scraper fails, it is automatically retried again with the browser window visible. This allows you to manually complete the login process and enter any multi-factor authentication code that is required.

To debug a scraper, you can run it in interactive mode by specifying the -i command-line argument. This runs an interactive IPython shell that lets you manually invoke parts of the scraping process.

Automatic Usage

To run multiple configurations at once, and keep track of when each configuration was last updated, you can use the finance_dl.update tool.

To display the update status, first create a logs directory and run:

python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs status

Initially, this will indicate that none of the configurations have been updated. To update a single configuration myconfig, run:

python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig

With a single configuration specified, this does the same thing as the finance_dl.cli tool, except that the log messages are written to logs/myconfig.txt and a logs/myconfig.lastupdate file is created if it is successful.

If multiple configurations are specified, as in:

python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig1 myconfig2

then all specified configurations are run in parallel.

To update all configurations, run:

python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update --all


Copyright (C) 2014-2018 Jeremy Maitin-Shepard.

Distributed under the GNU General Public License, Version 2.0 only. See LICENSE file for details.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

finance_dl-1.3.3-py3-none-any.whl (78.6 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page