A web scraper for ticker symbols from yahoo finance
Produces .csv, .xlsx, .json and .yaml files (All files contain same data but in a different format) for stocks, ETF, futures, indexes, mutual funds, currency, warrants and bonds. The ticker symbol, company name and exchange are saved for all symbols. The stock symbols also have a category name.
It gets its data from https://finance.yahoo.com/lookup/. Please note: it is not possible to get all the symbols due to limitations set by Yahoo. About 75%-90% of all symbols are gathered using this script depending on type.
Python 2.7 or Python 3.5+
From python package manager (preferred):
pip install Yahoo-ticker-downloader
python setup.py install
usage: YahooTickerDownloader.py [-h] [-i] [-e] [-E EXCHANGE] [-s SLEEP] [-p] [-m MARKET] type positional arguments: type The type to download, this can be: future mutualfund etf stocks index currency optional arguments: -h, --help show this help message and exit -i, --insecure use HTTP instead of HTTPS -e, --export export immediately without downloading (Only useful if you already downloaded something to the .pickle file) -E EXCHANGE, --Exchange EXCHANGE Only export ticker symbols from this exchange (the filtering is done during the export phase) -s SLEEP, --sleep SLEEP The time to sleep in seconds between requests -p, --pandantic Stop and warn the user if some rare assertion fails -m MARKET, --market MARKET Specify the Region of queried exchanges (us = USA+Canada, dr=Germany, fr=France, hk=Hongkong, gb=United Kingdom, default= all)
The first positional argument must be one of the following: stocks etf future index mutualfund currency warrant bond
For example to download all stock symbols you run it like:
The program takes several hours before it is finished so if you don’t need all symbols it might be a good idea to specify the Region of the exchanges you’re interested in. The program supports suspending and resuming a download. Press CTRL+C to suspend download. Restart the program in the same working directory to resume downloading. It is possible to export a partially downloaded results using the -e flag.
Example of CSV output:
Ticker,Name,Exchange,categoryName,categoryNr ENZ,"Enzo Biochem Inc.",NYQ,"Medical Laboratories & Research",525 ENZN,"Enzon Pharmaceuticals Inc.",NMS,Biotechnology,515 ENZR,"Energizer Resources Inc.",PNK,,0 EOAA.DE,"E.ON AG",GER,"Diversified Utilities",913 KMX,"CarMax Inc.",NYQ,"Auto Dealerships",744 KMY.MU,KIMBERLY-CLARK,MUN,"Personal Products",323 KN1.DU,ECOUNION,DUS,"Business Software & Services",826 KNCAY,"Konica Minolta Holdings Inc.",PNK,,0 KND,"Kindred Healthcare Inc.",NYQ,"Long-Term Care Facilities",523 KNDI,"Kandi Technologies, Corp",NGM,"Auto Manufacturers - Major",330 ...ect
Depending on the type you are downloading, you will get between 3.000 and 100.000+ entries.
Download history for symbols: ystockquote
- Version 2.1.1 (2017-08-02)
- A too old requests dependency was listed ( #35 )
- Version 2.1.0 (2017-05-10)
- Added market parameter ( pull request #33 )
- Version 2.0.1 (2017-05-07)
- Fixed issue where all downloads except stock and currency stopped working.
- Version 2.0.0 (2017-05-05)
- Switched over to JSON api
- Version 1.0.0 (2017-04-04)
- Reverted some changes from 0.10.0. Bond is back. Reverted back to English site instead of German.
- Resolved CSV issue again. Closes #23 and #16.
- Merged #26 Workaround Y! b>2000 limit
- Scraper now scrapes a lot more at the expense of runtime.
- Support for python2 is back. Latest python 2 & 3 are supported.
- Removed xls support
- Added xlsx support (#29)
- Version 0.10.1 (2017-02-04)
- More descriptive help message
- Version 0.10.0 (2017-02-02)
- Removed bond downloading option.
- Uses different yahoo source. Fixes #18
- Removed python2 from classifiers. Related to #16
- Version 0.9.0 (unreleased)
- Added a flag to restrict output to specific stock exchanges.
- Version 0.8.1 (2016-08-17)
- Workaround for #7 : downloading interruption
- Solution for #9 : UnicodeEncodeError
- Version 0.7.0 (2016-03-20)
- Added –export option. It will transcode the .pickle file immediately to the desired output formats.
- Version 0.6.0 (unreleased)
- Add 3 retries with an exponential back-off if HTTPError or ChunkedEncodingError is raised when processing _fetchHtml.
- Version 0.5.0 (2015-08-16)
- Allows downloading using a insecure connection.
- The temporarily download file-names now include the ticker type.
- Version 0.4.0 (2014-10-28)
- Warrant symbols can now be downloaded.
- Bond symbols can now be downloaded.
- Version 0.3.0 (2014-08-14)
- Use HTTPS instead of HTTP
- Retry to fetch a page if it contains no symbols (A “fix” for issue #4)
- Renamed all ‘Curreny’ to ‘Currency’
- Relative imports are used
- Fix: .csv file it outputs is encoded in UTF-8 when using python2
- Performance: Considerable reduced memory consumption
- It now outputs .json, .yaml and .xls files in addition to .csv