co2mpas 1.0.3

A vehicle simulator predicting CO2 emissions for NEDC using WLTP time-series

##################################################################
CO2MPAS: Vehicle simulator predicting NEDC CO2 emissions from WLTP
##################################################################

:Release: 1.0.3
:Home: http://co2mpas.io/
:Releases: http://files.co2mpas.io/
:Sources: https://github.com/JRCSTU/co2mpas
:pypi-repo: http://pypi.co2mpas.io/ (will move to standard Python-repo in the future)
:Keywords: CO2, fuel-consumption, WLTP, NEDC, vehicle, automotive,
EU, JRC, IET, STU, back-translation, policy,
simulator, engineering, scientific
:Developers: .. include:: ../AUTHORS.rst
:Copyright: 2015 European Commission (`JRC-IET
<https://ec.europa.eu/jrc/en/institutes/iet>`_)
:License: `EUPL 1.1+ <https://joinup.ec.europa.eu/software/page/eupl>`_


**CO2MPAS** is backward-looking longitudinal-dynamics CO\ sub(`2`) and
fuel-consumption simulator for light-duty vehicles (cars and vans),
specially crafted to back-translate consumption figures from WLTP cycles
into NEDC ones.

It is an open-source project developed with Python-3.4,
using Anaconda & WinPython under Windows 7, Anaconda under MacOS, and
Linux's standard python environment.
The program runs as a *console command*.

History
=======
The *European Commission* is supporting the introduction of the *WLTP cycle*
for Light-duty vehicles developed at the *United Nations (UNECE)*
level, in the shortest possible time-frame. Its introduction requires
the adaptation of CO\ sub(`2`) certification and monitoring procedures set
by European regulations. European Commission's *Joint Research Centre* has been
assigned the development of this vehicle simulator to facilitate this
adaptation.



Quickstart
==========
.. Tip::
**About console-commands:**

- Console-commands beginning with ``$`` symbol are for the ``bash`` shell
(UNIX).

- Console-commands beginning with ``>`` symbol are for *Windows* ``cmd.exe``
command-prompt.

- You can adapt commands between the two shells with minor modifications
(i.e. ``mkdir <--> md``, ``rm <--> del``).

- You may download and install the *all-in-one* archive which contains
both shells configured in a console supporting decent copy-paste and
resizing capabilities (see ref:`all-in-one`_).


IF you have familiarity with v1 release AND IF you already have a full-blown
*python-3 environment* (i.e. *Linux* or the *all-in-one* archive) you can
immediately start working with the following *bash* commands; otherwise
follow the detailed instructions under sections ref:`install`_ and
ref:`usage`_.

.. code-block:: console

## Install co2mpas.
## NOTE: If behind proxy, specify additionally this option:
## --proxy http://user:password@yourProxyUrl:yourProxyPort
##
$ pip install co2mpas --extra-index http://pypi.co2mpas.io/simple/ --trusted-host pypi.co2mpas.io

## Where to store input and output files.
## In *Windows* cmd-prompt use `md` command instead.
$ mkdir input output

## Create a template excel-file for inputs.
$ co2mpas template input/vehicle_1.xlsx

###################################################
## Edit generated `./input/vehicle_1.xlsx` file. ##
###################################################

## Run simulator.
$ co2mpas -I input -O output

###################################################
## Inspect generated results inside `./output/`. ##
###################################################


.. _end-opening:
.. contents:: Table of Contents
:backlinks: top
:depth: 4



.. _install:

Install
=======
The installation procedure has 2-stages:

1. Install (or Upgrade) Python (2 choices under *Windows*).
2. Install CO2MPAS:
a. Install (or Upgrade) executable.
b. (optional) Install documents.
c. (optional) Install sources.

On *Windows* you may alternatively install the *all-In-One* archive
instead of performing the above 2 steps separately.


.. _all-in-one:

*All-In-One* Installation under Windows
---------------------------------------
- Download **all-in-one archive** from
http://files.co2mpas.io/.
Ensure that you download the correct 32/64 architecture for your PC
(the 64bit archive CANNOT run on 32bit PCs, but the opposite is possible).

- Use the original `"7z" extraxtor <http://portableapps.com/apps/utilities/7-zip_portable>`_,
since "plain-zip" produces out-of-memory errors when expanding long
directories.
Prefer an **installation-folder without any spaces leading to it**.

- If you have already downloaded a previous version of the *all-in-one*
archive, you may prefer to just upgrade CO2MPAS contained within.
Follow the instructions in the "Upgrade" section, below.

- After installation, check that the version of CO2MPAS contained in the
archive corresponds to the latest/the one described in these instructions.
If not, follow the instructions in the "Upgrade" section, below.

- Visit the guidelines for its usage: doc(`allinone`)
(also contained within the archive).


Python Installation
-------------------
If you already have a suitable python-3 installation with all scientific
packages updated to their latest versions, you may skip this 1st stage.

.. Note::
**Installing Python under Windows:**

The program requires CPython-3, and depends on *numpy*, *scipy*, *pandas*,
*sklearn* and *matplotlib* packages, which depend on C-native backends
and need a C-compiler to install from sources.

In *Windows* it is strongly suggested **NOT to install the standard CPython
distribution that comes up first(!) when you google for "python windows"**,
unless you are an experienced python-developer, and you know how to
hunt down pre-compiled dependencies from the *PyPi* repository and/or
from the `Unofficial Windows Binaries for Python Extension Packages
<http://www.lfd.uci.edu/~gohlke/pythonlibs/>`_.

Therefore we suggest that you download one of the following two
*scientific-python* distributions:

#. `WinPython <https://winpython.github.io/>`_ **python-3** (prefer 64 bit)
#. `Anaconda <http://continuum.io/downloads>`_ **python-3** (prefer 64 bit)



Install WinPython
~~~~~~~~~~~~~~~~~
The *WinPython* distribution is just a collection of the standard pre-compiled
binaries for *Windows* containing all the scientific packages, and much more.
It is not update-able, and has a quasi-regular release-cycle of 3 months.


1. Install the latest python-3 (preferably 64 bit) from https://winpython.github.io/.
Prefer an **installation-folder without any spaces leading to it**.

2. Open the WinPython's command-prompt console, by locating the folder where
you just installed it and run (double-click) the following file::

<winpython-folder>\"WinPython Command Prompt.exe"


3. In the console-window check that you have the correct version of
WinPython installed, and expect a similar response:

.. code-block:: console

> python --version
Python 3.4.3

REM Check your python is indeed where you installed it.
> where python
....


4. Use this console and follow ref:`co2mpas-install`_ instructions, below.



Install Anaconda
~~~~~~~~~~~~~~~~
The *Anaconda* distribution is a non-standard Python environment that
for *Windows* containing all the scientific packages we need, and much more.
It is not update-able, and has a semi-regular release-cycle of 3 months.

1. Install Anaconda python 3.4 (preferably 64 bit) from http://continuum.io/downloads.
Prefer an **installation-folder without any spaces leading to it**.

.. Note::
When asked by the installation wizard, ensure that *Anaconda* gets to be
registered as the default python-environment for the user's account.

2. Open a *Windows* command-prompt console::

"windows start button" --> `cmd.exe`

3. In the console-window check that you have the correct version of
Anaconda-python installed, by typing:

.. code-block:: console

> python --version
Python 3.4.3 :: Anaconda 2.3.0 (64-bit)

REM Check your python is indeed where you installed it.
> where python
....

4. Use this console and follow ref:`co2mpas-install`_ instructions, below.


.. _co2mpas-install:

CO2MPAS installation
--------------------
1. Install CO2MPAS executable internally into your python-environment with
the following console-command:

.. code-block:: console

> pip install co2mpas --extra-index http://pypi.co2mpas.io/simple/ --trusted-host pypi.co2mpas.io
Collecting co2mpas
Downloading http://pypi.co2mpas.io/packages/co2mpas-...
...
Installing collected packages: co2mpas
Successfully installed co2mpas-1.0.3

.. Note::
**Installing Behind Firewall:**

This previous step requires http-connectivity to Python's
"standard" repository (https://pypi.python.org/) and to co2mpas-site.
In case you are behind a **corporate proxy**, you may either:

a) Append the following option to all ``pip`` commands, appropriatly
adapted: ``--proxy http://user:password@yourProxyUrl:yourProxyPort``.

For averting any security deliberations, JRC cryptographically signes
all *final releases*, so that you or your IT staff may
`validate their authenticity
<https://www.davidfischer.name/2012/05/signing-and-verifying-python-packages-with-pgp/>`_
and detect *man-in-the-middle* attacks, however impossible.

b) Download all *wheel* packages from `co2mpas-site
<http://files.co2mpas.io>`_ for the specific version you are
interested in , and install them one by one:

.. code-block:: console

REM Download MANUALLY all `*.whl` files contained in release folder
REM from co2mpas-site in some folder.
> cd <folder-where-wheels_downloaded>
> pip install *.whl


.. Warning::
If you cannot install CO2MPAS, re-run the command adding 2 *verbose*
flags ``-vv``, copy-paste the console-output, and send it to JRC.


2. Check that when you run ``co2mpas``, the version executed is indeed the one
installed above:

.. code-block:: console

> python -v --version
co2mpas_version: 1.0.3
co2mpas_path: d:\co2mpas_ALLINONE-XXbit-v1.0.3\Apps\WinPython\python-3.4.3\lib\site-packages\co2mpas
python_path: D:\co2mpas_ALLINONE-XXbit-v1.0.3\WinPython\python-3.4.3
python_version: 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 XXX]


.. Warning::
The above procedure installs the *latest* CO2MPAS, which
*might be more up-to-date than the version described here!*

In that case you can either:

a) Visit the documents for version actually installed.
b) "Pin" the exact version you wish to install with a ``pip`` command
like this:

.. code-block:: console

> pip install co2mpas==1.0.3 ... # Other options, like above.



Install extras
~~~~~~~~~~~~~~
Internally CO2MPAS uses an algorithmic scheduler to execute model functions.
In order to visualize the *design-time models* and *run-time workflows*
you need to install the **Graphviz** visualization library from:
http://www.graphviz.org/.

If you skip this step, the ``graphplot`` sub-command and the ``--plot-workflow``
option would both fail to run (see ref:`debug`_).



Upgrade CO2MPAS
~~~~~~~~~~~~~~~

1. Uninstall (see below) and re-install it.


Uninstall CO2MPAS
~~~~~~~~~~~~~~~~~
To uninstall CO2MPAS type the following command, and confirm it with ``y``:

.. code-block:: console

> pip uninstall co2mpas
Uninstalling co2mpas-<installed-version>
...
Proceed (y/n)?


Re-run the command *again*, to make sure that no dangling installations are left
over; disregard any errors this time.



.. _usage:

Usage
=====
.. Note::
The following commands are for the **bash console**, specifically tailored
for the **all-in-one** archive.


First ensure that the latest version of CO2MPAS is properly installed, and that
its version match the version declared on this file.

The main entry for the simulator is the ``co2mpas`` console-command,
which **is not visible, but it is installed in your PATH.**
To get the syntax of the ``co2mpas`` console-command, open a console where
you have installed CO2MPAS (see ref:`install`_ above) and type:

.. code-block:: console

$ co2mpas --help
Predict NEDC CO2 emissions from WLTP cycles.

Usage:
co2mpas [simulate] [-v] [--predict-wltp] [--more-output] [--no-warn-gui] [--plot-workflow]
[-I <fpath>] [-O <fpath>]
co2mpas demo [-v] [-f] <folder>
co2mpas template [-v] [-f] <excel-file-path> ...
co2mpas ipynb [-v] [-f] <folder>
co2mpas modelgraph [-v] [-l | <models> ...]
co2mpas [-v] --version
co2mpas --help

-I <fpath> Input folder or file, prompted with GUI if missing [default: ./input]
-O <fpath> Input folder or file, prompted with GUI if missing [default: ./output]
-l, --list List available models.
dict-wltp Whether to predict also WLTP values.
--more-output Output also per-vehicle output-files.
--no-warn-gui Does not pause batch-run to report inconsistencies.
--plot-workflow Open workflow-plot in browser, after run finished.
-f, --force Overwrite template/sample excel-file(s).
-v, --verbose Print more verbosely messages.

* Items enclosed in `[]` are optional.


Sub-commands:
simulate [default] Run simulation for all excel-files in input-folder (-I).
demo Generate demo input-files inside <folder>.
template Generate "empty" input-file at <excel-file-path>.
ipynb Generate IPython notebooks inside <folder>; view them with cmd:
ipython --notebook-dir=<folder>
modelgraph List all or plot available models.

Examples:

# Create sample-vehicles inside the `input` folder.
# (the `input` folder must exist)
co2mpas demo input

# Run the sample-vehicles just created.
# (the `output` folder must exist)
co2mpas -I input -O output

# Create an empty vehicle-file inside `input` folder.
co2mpas template input/vehicle_1.xlsx

# View a specific submodel on your browser.
co2mpas modelgraph gear_box_calibration

The default sub-command (``simulate``) accepts either a single **input-excel-file**
or a folder with multiple input-files for each vehicle, and generates a
**summary-excel-file** aggregating the major result-values from these vehicles,
and (optionally) multiple **output-excel-files** for each vehicle run.


Running Samples
---------------
The simulator contains input-files for demo-vehicles that are a nice
starting point to try out.

1. Choose a folder where you will store the *input* and *output* files:

.. code-block:: console

## Skip this if ``tutorial`` folder already exists.
$ mkdir tutorial
$ cd tutorial

## Skip also this if folders exist.
$ mkdir input output

.. Note::
The input & output folders do not have to reside in the same parent,
neither to have these names.
It is only for demonstration purposes that we decided to group them both
under a hypothetical ``some-folder``.

3. Create the demo vehicles inside the *input-folder* with the ``demo``
sub-command:


.. code-block:: console

$ co2mpas demo input
Creating DEMO INPUT file 'input\co2mpas_demo_1_full_data.xlsx'...
Creating DEMO INPUT file 'input\co2mpas_demo_2_wltp_high_only.xlsx'...
Creating DEMO INPUT file 'input\co2mpas_demo_3_wltp_low_only.xlsx'...
Creating DEMO INPUT file 'input\co2mpas_demo_4_baseline_no_battery_currents - Copy.xlsx'...
Creating DEMO INPUT file 'input\co2mpas_demo_5_baseline_no_gears.xlsx'...
You may run DEMOS with:
co2mpas simulate -I input

4. Run the simulator:

.. code-block:: console

$ co2mpas -I input -O output
Processing 'input' --> 'output'...
Processing: co2mpas_demo_1_full_data
...
...
Done! [90.765501 sec]


6. Inspect the results:

.. code-block:: console

$ start output/*summary.xlsx ## More summaries might exist in the folder from previous runs.
$ start output ## View the folder with all files generated.


Output files
~~~~~~~~~~~~
Below is the structure of the output-files produced for each vehicle::

## Input and calibrated values for electrics.
<timestamp>_precondition_WLTP_<inp-fname>.xls

## Input and calibrated values.
<timestamp>_calibration_WLTP-H_<inp-fname>.xls

## Input and calibrated values.
<timestamp>_calibration_WLTP-L_<inp-fname>.xls

## Input and predicted values.
<timestamp>_prediction_NEDC_<inp-fname>.xls

## Major CO2 values from all vehicles in the batch-run.
<timestamp>_summary.xls


Entering new vehicles
---------------------
You may modify the samples vehicles and run again the model.
But to be sure that your vehicle does not contain by accident any of
the sample-data, use the ``template`` sub-command to make an *empty* input
excel-file:


1. Decide the *input/output* folders. Assuming we are still in the ``tutorial``
folder and we wish to re-use the ``input/output`` folders from the example
above, we may clear all their contents with this:

.. code-block:: console

$ rm -r ./input/* ./output/* Replace `rm` with `del` in *Windows* (`cmd.exe`)


2. Create an empty vehicle template-file (eg. ``vehicle_1.xlsx``) inside
the *input-folder* with the ``template`` sub-command:

.. code-block:: console

$ co2mpas template input/vehicle_1.xlsx ## Note that here we specify the filename, not the folder!
Creating TEMPLATE INPUT file 'input/vehicle_1.xlsx'...


3. Open the template excel-file to fill-in your vehicle data
(and save it afterwards):

.. code-block:: console

$ start input/vehicle_1.xlsx ## Opens the excel-file. Use `start` in *cmd.exe*.

.. Tip::
The generated file contains help descriptions to help you populate it
with vehicle data. For items where an array of values is required
(i.e. gear-box ratios) you may reference different parts of
the spreadsheet following the syntax of the `"xlref" mini-language
<https://pandalone.readthedocs.org/en/latest/reference.html#module-pandalone.xleash>`_.

You may repeat these last 2 steps if you want to add more vehicles in
the *batch-run*.

4. Run the simulator. Specify the single excel-file as input:

.. code-block:: console

$ co2mpas -I ./input/vehicle_1.xlsx -O output
Processing './input/vehicle_1.xlsx' --> 'output'...
Processing: vehicle_1
...
Done! [12.938986 sec]

5. Assuming you do receive any error, you may now inspect the results:

.. code-block:: console

$ start output/*summary.xlsx ## More summaries might open from previous runs.
$ start output ## View all files generated (see below).


6. In the case of errors, or if the results are not satisfactory, repeat the
above procedure from step 3 to modify the vehicle and re-run the model.
See also ref:`debug`_, below.


Using IPython
-------------
You may enter the data for a single vehicle and run its simulation, plot its
results and experiment in your browser using `IPython <http://ipython.org/>`_.

The usage pattern is similar to "demos" but requires to have **ipython**
installed:

1. Ensure *ipython* with *notebook* "extra" is installed:

.. Warning::
This step requires too many libraries to provide as standalone files,
so unless you have it already installed, you will need a proper
*http-connectivity* to the standard python-repo.

.. code-block:: console

$ pip install ipython[notebook]
Installing collected packages: ipython[notebook]
...
Successfully installed ipython-x.x.x notebook-x.x.x


2. Then create the demo ipython-notebook(s) into some folder
(i.e. assuming the same setup from above, ``tutorial/input``):

.. code-block:: console

$ pwd ## Check our current folder (``cd`` alone for Windows).
.../tutorial

$ co2mpas ipynb ./input

3. Start-up the server and open a browser page to run the vehicle-simulation:

.. code-block:: console

$ ipython notebook ./input

4. A new window should open to your default browser (AVOID IEXPLORER) listing
the ``simVehicle.ipynb`` notebook (and all the demo xls-files).
Click on the ``*.ippynb`` file to "load" the notebook in a new tab.

The results are of a simulation run already pre-generated for this notebook
but you may run it yourself again, by clicking the menu::

"menu" --> `Cell` --> `Run All`

And watch it as it re-calculates *cell* by cell.

5. You may edit the python code on the cells by selecting them and clicking
``Enter`` (the frame should become green), and then re-run them,
with ``Ctrl + Enter``.

Navigate your self around by taking the tutorial at::

"menu" --> `Help` --> `User Interface Tour`

And study the example code and diagrams.

6. When you have finished, return to the console and issue twice ``Ctrl + C``
to shutdown the *ipython-server*.


.. _debug:

Debugging and investigating results
-----------------------------------

- Make sure that you have installed `graphviz`, and when running the simulation,
append also the ``--plot-workflow`` option.

- Use the ``modelgraph`` sub-command to plot the offending model (or just
out of curiosity). For instance:

.. code-block:: console

$ co2mpas modelgraph gear_box_calibration

.. _static/GearModel.png
:alt: Flow-diagram of the Gear-calibration model.
:height: 240
:width: 320

- Inspect the functions mentioned in the workflow and models and search them
in `CO2MPAS documentation <http://files.co2mpas.io/>`_ ensuring you are
visiting the documents for the actual version you are using.