gn2pg_client 0.1.1.dev0

Import tool from GeoNature to a PostgreSQL database through Export module API (client side)

This project provides an import tool between GeoNature instances (client side). Widely inspired from ClientApiVN

Topics

• Project Setup

• Issues

• HowTo

  • Help

  • Init config file

  • InitDB Schema and tables

  • Full download

  • Incremental download

  • Debug mode

  • Logs

  • Import datas into GeoNature datas

• Contributing

  • Renew requirements file for non poetry developers

• Licence

• Team

Warning

Actually in development.

Project Setup

GN2PG Client can be installed by running pip. It requires Python 3.7.4# to run.

pip install gn2pg-client

Issues

Please report any bugs or requests that you have using the GitHub issue tracker!

HowTo

Help

gn2pg_cli --help

Init config file

This command init a TOML config file within ~/.gn2pg hidden directory (in user HOME directory), named as you want. PLEASE DO NOT SPECIFY PATH!

gn2pg_cli --init <myconfigfile>

Config file is structured as this. [[source]] block can be duplicate as many as needed (one block for each source).

# GN2PG configuration file

# Local db configuration
[db]
db_host = "localhost"
db_port = 5432
db_user = "<dbUser>"
db_password = "<dbPassword>"
db_name = "<dbName>"
db_schema_import = "schema"
    # Additional connection options (optional)
    [db.db_querystring]
    sslmode = "prefer"


# Source configuration,
# Ducplicate this block for each source (1 source = 1 export)
[[source]]
# Source name, will be use to tag stored data in import table
name = "Source1"
# GeoNature source login
user_name = "<monuser>"
# GeoNature source password
user_password = "<monPwd>"
# GeoNature source URL
url = "<http://geonature1/>"
# GeoNature source Export id
export_id = 1
# Data type is facultative. By default the value is 'synthese'. Therefore, triggers from to_gnsynthese.sql are not activated.
# If you want to insert your date into a GeoNature database please choose either 'synthese_with_cd_nomenclature' or 'synthese_with_label'.
# If not, delete the line.
data_type = "synthese_with_cd_nomenclature"


[[source]]
# Source configuration
name = "Source2"
user_name = "<monuser>"
user_password = "<monPwd>"
url = "<http://geonature2/>"
export_id = 1
data_type = "synthese_with_cd_nomenclature"

Tip

You can add variable in source block enable = false to disable a source

InitDB Schema and tables

To create json tables where datas will be downloaded, run :

gn2pg_cli --json-tables-create <myconfigfile>

Full download

To download all datas from API, run :

gn2pg_cli --full <myconfigfile>

Incremental download

To update data since last download, run :

gn2pg_cli --update <myconfigfile>

To automate the launching of updates, you can write the cron task using the following command, for example every 30 minutes.

*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli --update <myconfigfile>" > /dev/null 2>&1

Debug mode

Debug mode can be activated using --verbose CLI argument

Logs

Log files are stored in $HOME/.gn2pg/log directory.

Import datas into GeoNature datas

Default script to auto populate GeoNature is called “to_gnsynthese”.

gn2pg_cli --custom-script to_gnsynthese <myconfigfile>

Tip

You can also replacing synthese script by your own scripts, using file path instead of to_gnsynthese.

Contributing

All devs must be done in forks.

Pull requests must be pulled to dev branch. For example with this command:

gh repo fork --clone lpoaura/gn2pg_client

Install project and development requirements (require poetry):

poetry install

Make your devs and pull requests.

Run gn2pg_cli command in dev mode

poetry run gn2pg_cli <options>

Renew requirements file for non poetry developers

poetry export -f requirements.txt > requirements.txt

Licence

GNU AGPLv3

Team

• @lpofredc (LPO Auvergne-Rhône-Alpes), main developper

---

With the financial support of the DREAL Auvergne-Rhône-Alpes.