Yahoo-ticker-downloader 3.0.1

A web scraper for ticker symbols from yahoo finance

Produces .csv, .xlsx, .json and .yaml files (All files contain same data but in a different format) for stocks, ETF, futures, indexes, mutual funds, currency, warrants and bonds. The ticker symbol, company name and exchange are saved for all symbols.

It gets its data from https://finance.yahoo.com/lookup/. Please note: it is not possible to get all the symbols due to limitations set by Yahoo.

Requirements

Python 2.7 or Python 3.5+

Install

From python package manager (preferred):

pip install Yahoo-ticker-downloader

From source:

python setup.py install

Example Usage

usage: YahooTickerDownloader.py [-h] [-i] [-e] [-E EXCHANGE] [-s SLEEP] [-p]
                                [type]

positional arguments:
  type                  The type to download, this can be: generic

optional arguments:
  -h, --help            show this help message and exit
  -i, --insecure        use HTTP instead of HTTPS
  -e, --export          export immediately without downloading (Only useful if
                        you already downloaded something to the .pickle file)
  -E EXCHANGE, --Exchange EXCHANGE
                        Only export ticker symbols from this exchange (the
                        filtering is done during the export phase)
  -s SLEEP, --sleep SLEEP
                        The time to sleep in seconds between requests
  -p, --pandantic       Stop and warn the user if some rare assertion fails

For example to download all stock symbols you run it like:

YahooTickerDownloader.py

The program takes a few weeks before it is finished. The program supports suspending and resuming a download. Press CTRL+C to suspend download. Restart the program in the same working directory to resume downloading. It is possible to export partially downloaded results using the -e flag.

Example of CSV output:

Ticker      Name    Exchange        exchangeDisplay Type    TypeDisplay
JNUG        Direxion Daily Jr Gld Mnrs Bull 3X ETF  ASE     NYSE MKT        E       ETF
DWDP        DowDuPont Inc.  NYQ     NYSE    S       Equity
E   Eni S.p.A.      NYQ     NYSE    S       Equity
EQH AXA Equitable Holdings, Inc.    NYS     NYSE    S       Equity
XOM Exxon Mobil Corporation NYQ     NYSE    S       Equity
ETP Energy Transfer Partners, L.P.  NYQ     NYSE    S       Equity
ES=F        E-mini S&P 500 Index Futures,Jufuture   CME     Chicago Mercantile Exchange     F       Futures
NQ=F        E-mini Nasdaq 100 Index Futuresfuture   CME     Chicago Mercantile Exchange     F       Futures
GE=F        Eurodollar Futures,Sep-2018future       CME     Chicago Mercantile Exchange     F       Futures
...ect

Depending on the type you are downloading, you will get between 3.000 and 100.000+ entries.

Further resources

Download history for symbols: ystockquote

Changelog

• Version 3.0.1 (2018-12-01)

  • Removed reppy dependency

• Version 3.0.0 (2018-05-27)

  • Switched over to different JSON api (searchassist)

• Version 2.2.0 (2018-01-31)

  • Continue exporting to different formats if one export fails ( #41 )

  • Check robots.txt ( anti-feature )

• Version 2.1.1 (2017-08-02)

  • A too old requests dependency was listed ( #35 )

• Version 2.1.0 (2017-05-10)

  • Added market parameter ( pull request #33 )

• Version 2.0.1 (2017-05-07)

  • Fixed issue where all downloads except stock and currency stopped working.

• Version 2.0.0 (2017-05-05)

  • Switched over to JSON api

• Version 1.0.0 (2017-04-04)

  • Reverted some changes from 0.10.0. Bond is back. Reverted back to English site instead of German.

  • Resolved CSV issue again. Closes #23 and #16.

  • Merged #26 Workaround Y! b>2000 limit

  • Scraper now scrapes a lot more at the expense of runtime.

  • Support for python2 is back. Latest python 2 & 3 are supported.

  • Removed xls support

  • Added xlsx support (#29)

• Version 0.10.1 (2017-02-04)

  • More descriptive help message

• Version 0.10.0 (2017-02-02)

  • Removed bond downloading option.

  • Uses different yahoo source. Fixes #18

  • Removed python2 from classifiers. Related to #16

• Version 0.9.0 (unreleased)

  • Added a flag to restrict output to specific stock exchanges.

• Version 0.8.1 (2016-08-17)

  • Workaround for #7 : downloading interruption

  • Solution for #9 : UnicodeEncodeError

• Version 0.7.0 (2016-03-20)

  • Added –export option. It will transcode the .pickle file immediately to the desired output formats.

• Version 0.6.0 (unreleased)

  • Add 3 retries with an exponential back-off if HTTPError or ChunkedEncodingError is raised when processing _fetchHtml.

• Version 0.5.0 (2015-08-16)

  • Allows downloading using a insecure connection.

  • The temporarily download file-names now include the ticker type.

• Version 0.4.0 (2014-10-28)

  • Warrant symbols can now be downloaded.

  • Bond symbols can now be downloaded.

• Version 0.3.0 (2014-08-14)

  • Use HTTPS instead of HTTP

  • Retry to fetch a page if it contains no symbols (A “fix” for issue #4)

  • Renamed all ‘Curreny’ to ‘Currency’

  • Relative imports are used

  • Fix: .csv file it outputs is encoded in UTF-8 when using python2

  • Performance: Considerable reduced memory consumption

  • It now outputs .json, .yaml and .xls files in addition to .csv