Scrape Charts

Follow the steps completed in test_scraper.py (located in the tests folder or below). The virtual environment is not necessary, pip will download all required packagesdoes not need to be activited, but the import statements probably appear slightly differently, however comments will rectify your confusion on the correct string to paste.

Main feature: scrape multiple charts from one website or multiple websites, and then turn it into one large list

Features include:

• Scraping a chart from a website or multiple websites
• Choosing what charts you want from a website along with combining multiple websites' charts into one large chart
• Processing that larger chart into a list
• Cleaning that list
• Cleaning the text with regex functions
• No errors in the python package, eg. works out of the box
• Converting that cleaned list into a panda dictionary
• Converting that cleaned list into a dictionary
• Saving that cleaned list as a json file, etc
• A very friendly to use class that does everything for you
• Return statements at every stage so that if one part doesn't work, you can see the data and process it yourself (a contingency, because if you contacted me, I would fix the issue)
• A maintainer
• No bugs or issues at the time of writing, (unit testing exists partially)
• Regex functions that are explained below

from src.chart_scraper.ChartScraper import Scraper
# Importing from pip can be done without the src

# This is for educational purposes only, this code is example code, but not for usage
chartScraper = Scraper("https://www.learnthat.org/pages/view/roots.html", chartNumber=[2])
# At this stage. chartScraper.combinedCharts hold this one mega list, so you can manually change one or two things, however this isn't necessary, the following code below will still work as if nothing happened
chartScraper.cleanList(whichToKeep="[a-zA-Z0-9 ]+", whereToSplit="\(|,", whereToCombine="/", whereToClean=[[" -", ":"],[";", ","], ["[^a-zA-Z ,:]+", ""], [" +", " "]])
chartScraper.listToDict(includePrintStatement=False)
chartScraper.getDictKeys(includePrintStatement=False)
# All lowercase
chartScraper.findWordComponents("philology")
chartScraper.createDataFrame()
chartScraper.saveFiles(fileType=2)

chartScraper.cleanList(whichToKeep="[a-zA-Z0-9]+", whereToSplit="\(|,", whereToCombine="/", whereToClean=[[" -", ":"],[";", ","], ["[^a-zA-Z ,:]+", ""], [" +", " "]])
# whichToKeep="[a-zA-Z0-9 ]+" removes strings that don't contain either letters (a-z, A-Z) or numbers (0-9, eg. the larger number 123 works)
# whereToSplit="\(|," splits the string whenever there is a parantehsis or comma, the paranthesis is backslashed because regex requires it
# whereToCombine="/" combines a/b/c into ["a", "ab", "ac"], my niche case required it when I built this package
# whereToClean=[[" -", ":"],[";", ","], ["[^a-zA-Z ,:]+", ""], [" +", " "]]
# [" -", ":"] turns any " -" into ":"
# Likewise, [";", ","] converts ";" into ","
# ["[^a-zA-Z ,:]+", ""] removes characters that aren't letters (a-z, A-z), spaces (" "), commas (,), or colons (:)
# [" +", " "] rectifies the issue of multiple spaces into one space, eg "            " into " "