A python based scraper for any data source.
Project description
# pyscrape
### Setup Local Dev Install python, virtualenv and deps. To get started go [here](https://realpython.com/blog/python/flask-by-example-part-1-project-setup).
Once
Then: pip install -r requirements.txt
### Pushing to Production/Staging on Heroku (Don’t do this, ci should do this) This is an example of how a sample app would deploy. This shouldn’t be here. git remote add heroku-staging git@heroku.com:pyscrape-staging.git git remote add heroku-production git@heroku.com:pyscrape-production.git Or make deploy
### Release First, create a new pip package. This will bump the patch version and write it to VERSION. make package
Then, to push to the package to the repository: make release
### Usage Pyscraper is meant as a framework to help with the extraction, transformation and loading of data between sources.
To get started, create a new Python project and then pip install pypscraper-framework.
To run the app flask app frontend: pyscraper_flask To run the worker process: pyscraper_worker
The following two environment vars are required: ` export APP_SETTINGS='DevelopmentConfig' # name of the corresponding config class for this env. export APP_BASEDIR=$(pwd) # must point to directory containing your config file. `
A config file is also required. See config.py.example.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for pyscraper_framework-0.0.22.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6dcf01917ef31ef4516135ac968e78dbd0b6ec45e9c4ab552c01c2dae83aa20 |
|
MD5 | 601606c9f502b753ae33b339a0aa5e56 |
|
BLAKE2b-256 | 37066d6fd9f4b2b2282fcd785aca2b2c28c7fceb96285ee32b54844da4a7cf1e |