A python based scraper for any data source.
Project description
# pyscrape
### Setup Local Dev
Install python, virtualenv and deps. To get started go [here](https://realpython.com/blog/python/flask-by-example-part-1-project-setup).
Once
Then: `pip install -r requirements.txt`
### Pushing to Production/Staging on Heroku (Don't do this, ci should do this)
This is an example of how a sample app would deploy. This shouldn't be here.
git remote add heroku-staging git@heroku.com:pyscrape-staging.git
git remote add heroku-production git@heroku.com:pyscrape-production.git
Or
`make deploy`
### Release
First, create a new pip package. This will bump the patch version and write it to `VERSION`.
`make package`
Then, to push to the package to the repository:
`make release`
### Usage
Pyscraper is meant as a framework to help with the extraction, transformation and loading of data between sources.
To get started, create a new Python project and then `pip install pypscraper-framework`.
To run the app flask app frontend: `pyscraper_flask`
To run the worker process: `pyscraper_worker`
The following two environment vars are required:
```
export APP_SETTINGS='DevelopmentConfig' # name of the corresponding config class for this env.
export APP_BASEDIR=$(pwd) # must point to directory containing your config file.
```
A config file is also required. See `config.py.example`.
### Concepts
The framework relies heavily on naming conventions and magically imports.
#### Pipe
This represents the flow of data from a source to a target. `pipe.start()` should do whatever is necessary determine and enqueue any and all `ETLJob`s that must execute in order for a run to be considered succesfull.
#### ETLJob
A base class defined in the framework. It has three methods: extract, transform, load.
#### Transformer/Extractor/Loader
Extend this class with a class that has the same name as your pipe's class. The ETLJob will run `{Transformer|Extractor|Loader}.execute()` when executing.
### Setup Local Dev
Install python, virtualenv and deps. To get started go [here](https://realpython.com/blog/python/flask-by-example-part-1-project-setup).
Once
Then: `pip install -r requirements.txt`
### Pushing to Production/Staging on Heroku (Don't do this, ci should do this)
This is an example of how a sample app would deploy. This shouldn't be here.
git remote add heroku-staging git@heroku.com:pyscrape-staging.git
git remote add heroku-production git@heroku.com:pyscrape-production.git
Or
`make deploy`
### Release
First, create a new pip package. This will bump the patch version and write it to `VERSION`.
`make package`
Then, to push to the package to the repository:
`make release`
### Usage
Pyscraper is meant as a framework to help with the extraction, transformation and loading of data between sources.
To get started, create a new Python project and then `pip install pypscraper-framework`.
To run the app flask app frontend: `pyscraper_flask`
To run the worker process: `pyscraper_worker`
The following two environment vars are required:
```
export APP_SETTINGS='DevelopmentConfig' # name of the corresponding config class for this env.
export APP_BASEDIR=$(pwd) # must point to directory containing your config file.
```
A config file is also required. See `config.py.example`.
### Concepts
The framework relies heavily on naming conventions and magically imports.
#### Pipe
This represents the flow of data from a source to a target. `pipe.start()` should do whatever is necessary determine and enqueue any and all `ETLJob`s that must execute in order for a run to be considered succesfull.
#### ETLJob
A base class defined in the framework. It has three methods: extract, transform, load.
#### Transformer/Extractor/Loader
Extend this class with a class that has the same name as your pipe's class. The ETLJob will run `{Transformer|Extractor|Loader}.execute()` when executing.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file pyscraper_framework-0.0.42.tar.gz
.
File metadata
- Download URL: pyscraper_framework-0.0.42.tar.gz
- Upload date:
- Size: 8.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa1a8d507007695611eb44aa73f19bf94c0479c63b79ab4a6112003856aba1ca |
|
MD5 | 9b92ee7543e374ca95e80770ac230f70 |
|
BLAKE2b-256 | c0701e0c77ffa8983f00a9e0641d8b7a0ceb4528751b4c9b8c81d7eaea54312c |