Statistical estimation of a clear sky signal from PV system power data
Project description
StatisticalClearSky
Statistical estimation of a clear sky signal from PV system power data
Getting Started
You can install pip package or Anaconda package for this project.
Installation
If you are using pip:
$ pip install statisticalclearsky
As of February 11, 2019, it fails because scs package installed as a dependency of cxvpy expects numpy to be already installed. scs issue 85 says, it is fixed. However, it doesn't seem to be reflected in its pip package. Also, cvxpy doesn't work with numpy version less than 1.16. As a work around, install numpy separatly first and then install this package. i.e.
$ pip install 'numpy>=1.16'
$ pip install statisticalclearsky
If you are using Anaconda, the problem described above doesn't occur since numpy is already installed. And during statisticalclearsky installation, numpy is upgraded above 1.16:
$ conda install c slacgismo statisticalclearsky
Solvers
The default convex solver included with cvxpy is ECOS, which is open source. However this solver tends to fail on problems with >1000 variables, as it does not work for this algorithm.
So, the default behavior of the code is to use the commercial Mosek solver. Thus, we encourage you to install it separately as below and obtain the license on your own.

mosek  For using MOSEK solver.
If you are using pip:
$ pip install f https://download.mosek.com/stable/wheel/index.html Mosek
If you are using Anaconda:
$ conda install c mosek mosek==8.1.43
Academic licenses are available for free here: https://www.mosek.com/products/academiclicenses/
Usage
As a part of Python code or inside Jupyter notebook
Example 1: Simplest example with the fewest number of input parameters.
Using default solver (Open Source solver: ECOS)
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d) iterative_fitting.execute() clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate()
Example 2: Estimating clear sky signals without degradation.
You can estimate clear sky signals based on the assumption that there is no yeartoyear degradation. In this case, you can set is_degradation_calculated keyword argument to False in execute method. By default, it's set to True.
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d) iterative_fitting.execute(is_degradation_calculated=False) clear_sky_signals = iterative_fitting.clear_sky_signals()
Example 3: Using a different solver.
The default solver ECOS is not stable with large set of input data. The following example shows how to specify to use Mosek solver by passing solver_type keyword argument (to the constructor).
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d, solver_type='MOSEK') iterative_fitting.execute() clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate()
Example 4: Setting rank for Generalized Low Rank Modeling.
By default, rank of low rank matrices is specified to be 6. You can change it by specifying rank_k keyword argument (in the constructor).
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d, rank_k=6) iterative_fitting.execute() # Get the resulting left low rank matrix and right low rank matrix for evaluation. left_low_rank_matrix = iterative_fitting.left_low_rank_matrix() # The above can be also obtained as l_cs_value: l_cs_value = iterative_fitting.l_cs_value # Get the resulting right low rank matrix for evaluation. right_low_rank_matrix = iterative_fitting.right_low_rank_matrix() # The above can be also obtained as r_cs_value: r_cs_value = iterative_fitting.r_cs_value clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate() # The above can be also obtained as beta_value: beta_value = iterative_fitting.beta_value
Example 5: Setting different hyperparameters for minimization of objective function of Generalized Low Rank Modeling.
There are three hyperparameters in the objective function of Generalized Low Rank Modeling, i.e. mu_l, mu_r, and tau. By default, mu_l is set to 1.0, mu_r is set to 20.0, and tau is set to 0.8. You can change it by specifying mu_l, mu_r, and tau keyword arguments in execute method.
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d) iterative_fitting.execute(mu_l=5e2, mu_r=1e3, tau=0.9) clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate()
Example 6: Setting different control parameters for minimization of objective function of Generalized Low Rank Modeling.
There are three control parameters in the objective function of Generalized Low Rank Modeling, i.e. exit criteria  exit_criterion_epsilon, and maximum number of iteration  max_iteration. By default, exit_criterion_epsilon is set to 1e3, max_iteration is set to 100. You can change it by specifying eps and max_iteration keyword arguments in execute method.
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d) iterative_fitting.execute(exit_criterion_epsilon=1e6, max_iteration=10) clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate()
Example 7: Setting limit on degradation rate.
You can specify the maximum degradation and minimum degradation by setting max_degradation and min_degradation keyword arguments in execute method. By default, they are set not to be used.
import numpy as np from statistical_clear_sky.algorithm.iterative_fitting import IterativeFitting # Usually read from a CSV file or a database with more data, # covering 1 day (column) and a few years (row): power_signals_d = np.array([[0.0, 0.0, 0.0, 0.0], [1.33389997, 1.40310001, 0.67150003, 0.77249998], [1.42349994, 1.51800001, 1.43809998, 1.20449996], [1.52020001, 1.45150006, 1.84809995, 0.99949998]]) iterative_fitting = IterativeFitting(power_signals_d) iterative_fitting.execute(max_degradation=0.0, min_degradation=0.5) clear_sky_signals = iterative_fitting.clear_sky_signals() degradation_rate = iterative_fitting.degradation_rate()
Jupyter notebook examples
Alternatively, you can clone this repository (GIT) and execute the example codes under notebooks folder.
Simplest way to install dependencies if you are using pip is by
$ pip install r requirements.txt
As mentioned in the section, "Getting Started" above, as of February 11, 2019, it fails because scs package installed as a dependency of cxvpy expects numpy to be already installed. scs issue 85 says, it is fixed. However, it doesn't seem to be reflected in its pip package. Also, cvxpy doesn't work with numpy version less than 1.16. As a work around, install numpy separatly first and install the other packages using requirements.txt. i.e.
$ pip install 'numpy>=1.16'
$ pip install r requirements.txt
Running the tests
Unit tests (developer tests)

GIT clone this project.

In the project directory in terminal,
$ python m unittest
This runs all the tests under tests folder.
All the tests are placed under "tests" directory directly under the project directory. It is using "unittest" that is a part of Python Standard Library by default. There may be a better unit testing framework. But the reason is to invite as many contributors as possible with variety of background.
Coding style tests
pylint is used to check if coding style is conforming to "PEP 8  Style Guide for Python Code"
Note: We are open to use LGTM. However, since we decided to use another code coverage tool codecov based on a comment by project's Technical Advisory Council, we decided not to use another tool that does code coverage. We are also open to use other coding style tools.
Example of using pylint:
In the project directory in terminal,
$ pylint statistical_clear_sky
Contributing
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
Versioning
We use Semantic Versioning for versioning. For the versions available, see the tags on this repository.
Authors

Bennet Meyers  Initial work and Main research work  Bennet Meyers GitHub

Tadatoshi Takahashi  Refactoring and Packaging work and Research support work  Tadatoshi Takahashi GitHub
See also the list of contributors who participated in this project.
License
This project is licensed under the BSD 2Clause License  see the LICENSE file for details
References
[1] B. Meyers, M. Tabone, and E. C. Kara, "Statistical Clear Sky Fitting Algorithm," IEEE Photovoltaic Specialists Conference, 2018.
Acknowledgments
 The authors would like to thank Professor Stephen Boyd from Stanford University for his input and guidance and Chris Deline, Mike Deceglie, and Dirk Jordan from NREL for collaboration.
Project details
Release history Release notifications
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size  File type  Python version  Upload date  Hashes 

Filename, size statistical_clear_sky0.2.1py3noneany.whl (44.1 kB)  File type Wheel  Python version py3  Upload date  Hashes View hashes 
Filename, size statisticalclearsky0.2.1.tar.gz (34.1 kB)  File type Source  Python version None  Upload date  Hashes View hashes 
Hashes for statistical_clear_sky0.2.1py3noneany.whl
Algorithm  Hash digest  

SHA256  4bfa5c33e86164b2fc19290bd2c8617877a0f88972b84e9330e7296845a1ce69 

MD5  d0004660b53b3179f037a6d698233366 

BLAKE2256  58dba2880bb63e44cdb88212f323f605b355739b04ef97a7a76f77317dfba9e0 
Hashes for statisticalclearsky0.2.1.tar.gz
Algorithm  Hash digest  

SHA256  ca9ad405f285f7a02a0e5144c9cbb549f4a4187a75e5c290819ca7740ff64df1 

MD5  9e6953a3d44604b3be686292799072ea 

BLAKE2256  99161aa852a6890870ef9b60ed744a87de231d4243822af46fcbdbb9e467c2a3 