A simple web scraping/crawler tool for university sites.
pip install university-scraper
then:
from university_scraper import available, init
# Get list of available universities
print(available())
# Give the abbreviation as a string, from the list of available universities
scraper = init('USYD')
scraper.programs
scraper.units
# Details can be retrieved for a certain program or unit using the respective kwargs
scraper.program_detail(...)
scraper.unit_detail(...)
Scrapers available for:
Contribute
Part of the reason I want this open sourced is because if a university makes a design change, the scraper for it should be modified.
If you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue ASAP.
If you are a programmer, PRs with fixes are warmly welcomed and acknowledged with a virtual 🍺
If you want a scraper for a new university added
-
Open an Issue providing us the university name, as well as the direction on how to get the neccessary details
- Unit details
- Program details
-
You are a developer and want to code the scraper on your own feel free to make a PR for us to review :)
For Devs / Contribute
Assuming you have python3 installed, navigate to the directory where you want this project to live in and drop these lines
git clone https://github.com/giulianocelani/university-scraper.git &&
cd university-scraper &&
pip install pipenv &&
pipenv shell &&
pipenv install &&
python -m unittest -v
Acknowledgement
Project was built with reference to https://github.com/hhursev/recipe-scrapers
Metadata
Release files for university-scraper 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| university_scraper-0.0.2.tar.gz | 6.9 kB | Without provenance | Details for university_scraper-0.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| university_scraper-0.0.2-py3-none-any.whl | Python 3 | none | any | Details for university_scraper-0.0.2-py3-none-any.whl |
Total release size: 16.5 kB
Release files / university_scraper-0.0.2.tar.gz
| Download URL | university_scraper-0.0.2.tar.gz |
|---|---|
| Size | 6.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3adbd5b2d9b5d004e4a7887a6c2e9b3a0f9a950554596c81602e7df47c48770c
|
|
BLAKE2b-256 checksum How to use checksums |
620e390c86defb903975c443c692f407b89c359c3019e7509d3869962180fb3f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.8.5
|
Release files / university_scraper-0.0.2-py3-none-any.whl
| Download URL | university_scraper-0.0.2-py3-none-any.whl |
|---|---|
| Size | 9.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
26b246486ffa170c0eb825f4b5414513d9eb3c8e182691620458fa7bf7cc2e35
|
|
BLAKE2b-256 checksum How to use checksums |
f309dacdae8613734d8a65af11bc1f7da9073e48893bb0f3312c474620b1df98
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.8.5
|
Security for university-scraper 0.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.