ruia-peewee-async

A Ruia plugin that uses peewee-async to store data to MySQL or PostgreSQL or both of them.

Installation

Using pip or pipenv or poetry to install.

pip install ruia-peewee-async[aiomysql]
pipenv install ruia-peewee-async[aiomysql]
poetry add ruia-peewee-async[aiomysql]

or

pip install ruia-peewee-async[aiopg]
pipenv install ruia-peewee-async[aiopg]
poetry add ruia-peewee-async[aiopg]

or

pip install ruia-peewee-async[all]
pipenv install ruia-peewee-async[all]
poetry install ruia-peewee-async[all]

ruia-peewee-async[all] means to install both aiomysql and aiopg.

Usage

A complete example is in the example directory.

There's a create_model method to create the Peewee model based on database configuration. You can use the create_model method to manipulate tables before starting the spider.

from ruia_peewee_async import create_model

mysql_model, mysql_manager, postgres_model, postgres_manager = create_model(mysql=mysql) # or postgres=postgres or both
# create the table at the same time
mysql_model, mysql_manager, postgres_model, postgres_manager = create_model(mysql=mysql, create_table=True) # or postgres=postgres or both
rows = mysql_model.select().count()
print(rows)

And class Spider from ruia_peewee_async has attributes below related to database you can use.

from peewee import Model
from typing import Callable, Dict
from typing import Optional as TOptional
from peewee_async import (
    AsyncQueryWrapper,
    Manager,
    MySQLDatabase,
    PooledMySQLDatabase,
    PooledPostgresqlDatabase,
    PostgresqlDatabase,
)
from ruia import Spider as RuiaSpider

class Spider(RuiaSpider):
    mysql_model: Union[Model, Dict] # It will be a Model instance after spider started.
    mysql_manager: Manager
    postgres_model: Union[Model, Dict] # same above
    postgres_manager: Manager
    mysql_db: MySQLDatabase
    postgres_db: PostgresqlDatabase
    mysql_filters: TOptional[AsyncQueryWrapper]
    postgres_filters: TOptional[AsyncQueryWrapper]

For more information, check out peewee's documentation and peewee-async's documentation.

Development

Using pyenv to install the version of python that you need. For example

pyenv install 3.7.9

Then go to the root of the project and run:

poetry install && poetry install -E aiomysql -E aiopg

to install all dependencies.

MacOS users have to run brew install postgresql to install postgresql and export the pg_config to the PATH, so that the psycorg2 dependency can be installed successfully with pip.

• Using poetry shell to enter the virtual environment. Or open your favorite editor and select the virtual environment to start coding.
• Using pytest to run unit tests under tests folder.
• Using pytest --cov . to run all tests and generate coverage report in terminal.

Thanks

• ruia
• peewew
• peewee-async
• aiomysql
• aiopg
• schema
• pytest and its awesome plugins