Dawson College PyScrapper v1.1.1
A Python module which contains useful functions to help scrape data from Dawson College which is a CEGEP in Montreal Quebec Canada.
Features
- Get information on all the programs offered by Dawson College (ex: Computer Science, Computer Engineering, etc.)
- Get an estimate of the total number of students enrolled
- Get the total number of faculty members
- Get the general metrics of Dawson College (ex: total number of programs offered, number of programs, number of profiles, number of disciplines, number of special studies, number of general studies, etc.)
Usage
Installation
pip install git+ssh://git@github.com/jdboisvert/dawson-college-pyscrapper
Using the core functionality
Getting program details for a specific program
from dawson_college_pyscrapper.scrapper import get_program_details
program_url = "https://www.dawsoncollege.qc.ca/programs/program-name"
# Get the BeautifulSoup Tag object of the program that is listed on the programs page.
listed_program = BeautifulSoup(requests.get(PROGRAMS_LISTING_URL).text.strip(), "html.parser").find("tr")
# Get the details of the program at the given URL.
program_details = get_program_details(program_url=program_url, listed_program=listed_program)
print(program_details)
Get details of all programs
from dawson_college_pyscrapper.scrapper import get_programs
programs = get_programs()
for program in programs:
print(f"Program Name: {program.name}")
print(f"Modified Date: {program.modified_date}")
print(f"Program Type: {program.program_type}")
print(f"Program URL: {program.url}")
print("\n")
Get the total number of students enrolled
from dawson_college_pyscrapper.scrapper import get_total_number_of_students
total_number_of_students = get_total_number_of_students()
print(f"Total number of students: {total_number_of_students}")
Get the total number of faculty members
from dawson_college_pyscrapper.scrapper import get_total_number_of_faculty_members
total_number_of_faculty_members = get_total_number_of_faculty_members()
print(f"Total number of faculty members: {total_number_of_faculty_members}")
Get the general metrics of Dawson College
from dawson_college_pyscrapper.scrapper import scrape
general_metrics = scrape()
print(f"Total programs offered: {general_metrics.total_programs_offered}")
print(f"Number of programs: {general_metrics.number_of_programs}")
print(f"Number of profiles: {general_metrics.number_of_profiles}")
print(f"Number of disciplines: {general_metrics.number_of_disciplines}")
print(f"Number of special studies: {general_metrics.number_of_special_studies}")
print(f"Number of general studies: {general_metrics.number_of_General_studies}")
print("\n")
print("Year count:")
for year, count in general_metrics.total_year_counts.items():
print(f"{year}: {count}")
print("\n")
print("Programs:")
for program in general_metrics.programs:
print(f"Program Name: {program.name}")
print(f"Modified Date: {program.modified_date}")
print(f"Program Type: {program.program_type}")
print(f"Program URL: {program.url}")
print("\n")
More examples
Check out the examples in the tests directory.
Development
Getting started
# install pyenv (if necessary)
brew install pyenv pyenv-virtualenv
echo """
export PYENV_VIRTUALENV_DISABLE_PROMPT=1
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
""" > ~/.zshrc
source ~/.zshrc
# create a virtualenv
pyenv install 3.11.1
pyenv virtualenv 3.11.1 dawson_college_pyscrapper
pyenv activate dawson_college_pyscrapper
# install dependencies
pip install -U pip
pip install -e ".[dev]"
Pre-commit
A number of pre-commit hooks are set up to ensure all commits meet basic code quality standards.
If one of the hooks changes a file, you will need to git add that file and re-run git commit before being able to continue.
To Install: pre-commit install
Testing
pytest and tox are used for testing. Tox is configured to try testing against both Python 3.8 and Python 3.9 if you have them available. If one is missing, Tox will skip it rather than fail out.
# just the unit tests against your current python version
pytest
# just the unit tests with a matching prefix
pytest -k test_some_function
# full test suite and code coverage reporting
tox
Credits
- Jeffrey Boisvert (jdboisvert) info.jeffreyboisvert@gmail.com
Metadata
Release files for dawson-college-pyscrapper 1.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dawson_college_pyscrapper-1.1.1.tar.gz | 18.5 kB | Without provenance | Details for dawson_college_pyscrapper-1.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dawson_college_pyscrapper-1.1.1-py3-none-any.whl | Python 3 | none | any | Details for dawson_college_pyscrapper-1.1.1-py3-none-any.whl |
Total release size: 29.2 kB
Release files / dawson_college_pyscrapper-1.1.1.tar.gz
| Download URL | dawson_college_pyscrapper-1.1.1.tar.gz |
|---|---|
| Size | 18.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5ad1d3eb809f206da6a668afad83f433b0589a4fbe29b0b546ab8e564f32a937
|
|
BLAKE2b-256 checksum How to use checksums |
6885fdc3e61b50f9c38bbb9114bec76197b69c73105ced7c2ed1f45a1c120bff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/6.0.0 pkginfo/1.9.6 requests/2.28.2 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.11.1
|
Release files / dawson_college_pyscrapper-1.1.1-py3-none-any.whl
| Download URL | dawson_college_pyscrapper-1.1.1-py3-none-any.whl |
|---|---|
| Size | 10.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
df5b6c4981c4085c7e09443ad1a4d41deebee375a10f0895e07bb2dda37e2d86
|
|
BLAKE2b-256 checksum How to use checksums |
2511a2d11b404e2b791f718f6b42daf6533230f420555baa967a6f565528dec8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/6.0.0 pkginfo/1.9.6 requests/2.28.2 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.11.1
|
Security for dawson-college-pyscrapper 1.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.