No project description provided
Project description
Final Data Crawler Project
Description
Final Data Crawler Project is a Python package designed for crawling specific real estate listing page. This project is done to test the knowledge of python and to learn data collection.
Installation
Using a package manager
You can install the crawler as a package: Using pip
:
pip install final_data_crawler_project
Or using poetry
:
poetry add final_data_crawler_project
Cloning the repository
You can also clone the repository and install the dependencies. Using poetry
:
git clone https://github.com/kmaciekus/final_data_crawler_project.git
cd final_data_crawler_project
poetry install
Afterwards you can checkout and run some example scripts, e.g.:
poetry run python example/apartment.py
Usage
As a module
from final_data_crawler_project import crawl_real_estate
print(crawl_real_estate(object="house", query="Kauno raj", time_limit=60, return_format="records"))
For more examples look in the examples directory.
Structure
The project is structured as follows:
final_data_crawler_project/
: Main package directory.__init__.py
: Package initialization file.crawler_objects/
: Directory containing individual crawler scripts.__init__.py
: Initialization file for crawlers module.apartment_crawler.py
: Crawler for apartments listings.house_crawler.py
: Crawler for house listings.plot_crawler.py
: Crawler for plot listings.
utils/
: Directory containing help functions.converters.py
: Holds function for handling convertion to int or float.error_handler.py
: Holds function for handling text value errors.
definitions.py
: Definitions.main.py
: Main script for the crawler package.
example/
: Directory containing example scripts.apartment.py/
: Examples getting apartment listings.house.py/
: Examples getting house listings.plot.py/
: Examples getting plot listings.
tests/
: Test scripts for the package.mock/
: Directory containing mock data- ``
__init__.py
: Initialization file for tests.converters_test.py
: Testing file for converter functions.text_error_handler_test.py
: Testing file for error handling function.
License
This project is licensed under the MIT license.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for final_data_crawler_project-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 999a89e93f22cfa2309691216438229dcce8513260022f2019e0713ff6c34ed3 |
|
MD5 | e8a54b8333dab69b29561371228be13b |
|
BLAKE2b-256 | 9661f9fbf9c11c9b87c5c69f5ca07ce93636c4223b66939ca7bb0eec67166ed0 |
Close
Hashes for final_data_crawler_project-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c5630634dff1d5a1619a22d73d704e39feeca5a8349659f996b5f6b26fb6d07 |
|
MD5 | f128eec4954dcbccd729dcd873cd78c8 |
|
BLAKE2b-256 | a90fdf554dbe207b83621b02b8faab646e51ce615e74dc5a5d352fe48e912f7f |