pygazpar 0.2.0

Retrieve gas consumption from GrDF web site (French Gas Company)

PyGazpar

PyGazpar is a Python library for getting natural gas consumption from GrDF French provider.

Their natural gas meter is called Gazpar. It is wireless and transmit the gas consumption once per day.

All consumption data is available on the client account at GrDF Web Site (https://monespace.grdf.fr).

PyGazpar automatically go through the Web Site and download the consumption data Excel file, and make it available in a Python structure (list of dictionaries).

Installation

Requirements

PyGazpar is working with Selenium Python library to automate navigation through GrDF Web site. Selenium requires a WebDriver that acts as gateway between automatic actions from PyGazpar and a native browser already installed on the system.

PyGazpar has been developped and tested with Firefox browser (version 68.8) and its corresponding Web Driver geckodriver (version 0.24).

Firefox browser installation

Follow instructions here

Firefox Web Driver (geckodriver) installation

Follow instructions here

Create your virtual environment

$ cd /path/to/my_project_folder/

$ python3 -m venv .venv

PyGazpar installation

Activate your virtual environment.

source .venv/bin/activate

Use the package manager pip to install PyGazpar.

pip install pygazpar

You can also download the source code and install it manually.

cd /path/to/pygazpar/
python setup.py install

Usage

Command line:

$ pygazpar -u 'your login' -p 'your password' -w 'path/to/Selenium Web Driver' -s 30 -t 'temporary directory where to store XSLX file (ex: /tmp)'

Library:

import pygazpar

client = pygazpar.Client('your login',
                         'your password',
                         'path/to/Selenium Web Driver',
                         30,
                         'temporary directory where to store XSLX file (ex: /tmp)')

client.update()

data = client.data()

Output:

data =>
[
  {
    "time_period": "16/04/2021",
    "start_index_m3": 13685.0,
    "end_index_m3": 13695.0,
    "volume_m3": 9.9,
    "energy_kwh": 111.0,
    "converter_factor_kwh/m3": 11.268,
    "temperature_degC": 8.0,
    "type": "MES",
    "timestamp": "2021-04-20T10:21:46.265119"
  },
  {
    "time_period": "17/04/2021",
    "start_index_m3": 13695.0,
    "end_index_m3": 13702.0,
    "volume_m3": 7.7,
    "energy_kwh": 86.0,
    "converter_factor_kwh/m3": 11.268,
    "temperature_degC": 9.0,
    "type": "MES",
    "timestamp": "2021-04-20T10:21:46.265119"
  },
  {
    "time_period": "18/04/2021",
    "start_index_m3": 13702.0,
    "end_index_m3": 13708.0,
    "volume_m3": 6.2,
    "energy_kwh": 69.0,
    "converter_factor_kwh/m3": 11.268,
    "temperature_degC": 10.0,
    "type": "MES",
    "timestamp": "2021-04-20T10:21:46.265119"
  }
]

More features

By default, PyGazpar queries for daily consumption data.

However, it is also possible to get weekly or monthly consumption data :

Weekly

Command line:

$ pygazpar -u 'your login' -p 'your password' -w 'path/to/Selenium Web Driver' -s 30 -t 'temporary directory where to store XSLX file (ex: /tmp)' -f WEEKLY

Library:

import pygazpar

client = pygazpar.Client('your login',
                         'your password',
                         'path/to/Selenium Web Driver',
                         30,
                         'temporary directory where to store XSLX file (ex: /tmp)',
                         meterReadingFrequency = Frequency.WEEKLY)

client.update()

data = client.data()

Output:

data =>
[
  {
    "time_period": "Du 29/03/2021 au 04/04/2021",
    "volume_m3": 21.4,
    "energy_kwh": 240.0,
    "timestamp": "2021-04-20T09:22:12.166500"
  },
  {
    "time_period": "Du 05/04/2021 au 11/04/2021",
    "volume_m3": 58.9,
    "energy_kwh": 663.0,
    "timestamp": "2021-04-20T09:22:12.166500"
  },
  {
    "time_period": "Du 12/04/2021 au 18/04/2021",
    "volume_m3": 57.1,
    "energy_kwh": 643.0,
    "timestamp": "2021-04-20T09:22:12.166500"
  }
]

Monthly

Command line:

$ pygazpar -u 'your login' -p 'your password' -w 'path/to/Selenium Web Driver' -s 30 -t 'temporary directory where to store XSLX file (ex: /tmp)' -f MONTHLY

Library:

import pygazpar

client = pygazpar.Client('your login',
                         'your password',
                         'path/to/Selenium Web Driver',
                         30,
                         'temporary directory where to store XSLX file (ex: /tmp)',
                         meterReadingFrequency = Frequency.MONTHLY)

client.update()

data = client.data()

Output:

data =>
[
  {
    "time_period": "F\u00e9vrier 2021",
    "volume_m3": 317.6,
    "energy_kwh": 3547.0,
    "timestamp": "2021-04-20T09:34:31.728125"
  },
  {
    "time_period": "Mars 2021",
    "volume_m3": 261.1,
    "energy_kwh": 2937.0,
    "timestamp": "2021-04-20T09:34:31.728125"
  },
  {
    "time_period": "Avril 2021",
    "volume_m3": 130.7,
    "energy_kwh": 1472.0,
    "timestamp": "2021-04-20T09:34:31.728125"
  }
]

Test mode

In testing environment (unit test or debugging), it may be cumbersome to work with live data: slowness, values are always changing.

There is a test mode that permits to work with static data. Those data are taken from internal resource files.

Command line:

$ pygazpar -u 'your login' -p 'your password' -w 'path/to/Selenium Web Driver' -s 30 -t 'temporary directory where to store XSLX file (ex: /tmp)' --testMode

Library:

import pygazpar

client = pygazpar.Client('your login',
                         'your password',
                         'path/to/Selenium Web Driver',
                         30,
                         'temporary directory where to store XSLX file (ex: /tmp)',
                         testMode = True)

client.update()

data = client.data()

Limitation

PyGazpar relies on how GrDF Web Site is built. It goes through each Web pages and automatically fill forms, click buttons using their internal identifiers.

Any change in the Web site structure or identifier naming may break this library.

We expect in close Future that GrDF makes available a standard API from which we can get safely their data.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

MIT

Project status

PyGazpar has been initiated for integration with Home Assistant.

Corresponding Home Assistant integration custom component is available here.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

0.2.0 - 2021-04-21

Added

• #12: Be able to retrieve consumption not only on a daily basis, but weekly and monthly:
  • API : Add a new parameter 'meterReadingFrequency' to Client that accepts enumeration: Frequency.DAILY, Frequency.WEEKLY or Frequency.MONTHLY.
  • Command line : Add a new argument '--frequency' that accepts values : DAILY, WEEKLY or MONTHLY.
• Be able to test using offline data :
  • API : Add a new parameter 'testMode' to Clients used to specify whether we want to get some live data (testMode=False) and static testing data (testMode=True).
  • Command line : Add a new argument '--testMode' (True if specified and False by default).

Changed

• Some ouput energy property names have changed:
  • 'date' => 'time_period'.
  • 'converter_factor' => 'converter_factor_kwh/m3'.
  • 'local_temperature' => 'temperature_degC'.

0.1.27 - 2021-04-20

Fixed

• #10 : Does not download data file in tmpdir as expected (instead it is downloaded in the default user Download directory). The previous attempt in version 0.1.26 is a failure. The origin of the bug is that we send a relative tmp directory path to Webdriver. Additionally, this path has to be normalized to the runtime OS (using slash or backslash).

0.1.26 - 2021-04-18

Fixed

• #10 : Does not download data file in tmpdir as expected (instead it is downloaded in the default user Download directory).

Added

• A new parameter to drive whether we want Selenium in headless mode or not (mainly for troubleshooting purpose).

0.1.25 - 2021-04-15

Fixed

• Remove useless warning log messages (log message level has been decreased to debug).

0.1.24 - 2021-04-14

Added

• README.md amendment (thanks to pbranly).

0.1.23 - 2021-04-09

Changed

• Final release with CI/CD workflow improvement.

0.1.22 - 2021-04-09

Changed

• Improve CI/CD workflow.

0.1.21 - 2021-04-07

Changed

• Cleanup some codes managing Privacy Conditions popup.

0.1.20 - 2021-04-07

Changed

• Close an eventual Privacy Conditions popup just before clicking the daily button (instead of Cookie banner).

0.1.19 - 2021-04-06

Added

• Close an eventual Cookie popup just before clicking the daily button.
• Add log messages and screenshot capture around click() and send_keys() methods.

0.1.18 - 2021-04-05

Fixed

• Typo in logger usage.

0.1.17 - 2021-04-05

Fixed

• Logger name must be different in each instance. Using name is a good practice, we can get the module hierarchy.

0.1.16 - 2021-04-05

Fixed

• Logger 'pygazpar' initialization must be done inside the main program and not inside the library.

0.1.15 - 2021-04-05

Added

• Many log message to help debugging if GrDF site changes something : pygazpar.log
• Take a screenshot of the corresponding page where a selenium command fails : error_screenshot.png.

0.1.14 - 2020-09-23

Fixed

• GrDF survey popup has to be closed at the home page.
• GrDF Assistant popup may hide the Download button. We have to close it.
• GrDF bottom banner that invite to accept cookies may also hide the Download button. We have to accept it.

Added

• A new parameter 'lastNRows' to get only the last N most recent records.

0.1.13 - 2020-06-30

Fixed

• GrDF data retrieval from Excel file is not limited to the 1000 first rows any more.

0.1.12 - 2020-06-30

Fixed

• The previous 0.1.11 is not sufficient. Hence, 2 new changes : First, make configurable the waiting time so the user can adapt to its context usage. Second, the condition on clicking on Download button is now based on the corresponding event and not anymore on its identifier.

0.1.11 - 2020-06-29

Fixed

• When GrDF Web site is slower than usual, the client may not wait enough time for pages to load and miss to reach the data file. It occurs with the page containing 'Jour' button which is very long to load (I increase the wait to load time from 5s to 30s).

0.1.10 - 2020-06-03

Fixed

• Extract rows from Excel until line 1000 (instead of 365 as before).

0.1.9 - 2019-08-31

Fixed

• WebDriver window size must be large enough to display all clickable components.

0.1.8 - 2019-08-31

Changed

• Use PropertyNameEnum type to store all property names.

0.1.7 - 2019-08-29

Added

• Add wait_time option to control how much time the library has to wait for Web page element to load (see https://selenium-python.readthedocs.io/waits.html for details).
• Add LoginError exception raised when PyGazpar is unable to sign in the GrDF Web site with the given username/password.
• Refactor all data property names.

0.1.6 - 2019-08-26

Added

• Add README.md and CHANGELOG.md.
• Add Client.data() method to get the updated data.

Removed

• Remove Client.data property to get the updated data. Replaced with Client.__data private property.

MIT License

Copyright (c) 2019 Stéphane Senart

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.