Formulate human-readable queries and retrieve data from ENTSO-E into pandas.DataFrame format.
Project description
Disclaimer: This library was written to experiment with various concepts. The submodules entsoe_client.ParameterTypes and entsoe_client.Queries are straight forward. To refactor entsoe_client.Parsers, the next steps would be to establish fixtures, standardize flattening of the XML tree and decouple the records from the DataFrame library.
ENTSO-E Client
Formulate readable queries and handle data in Pandas, including an exhaustive set of pre-defined queries.
>>> import requests
>>> from lxml import objectify
>>> from lxml.etree import dump
>>> url = 'https://transparency.entsoe.eu/api?' \
... 'documentType=A81&businessType=A95&psrType=A04&type_MarketAgreement.Type=A01&controlArea_Domain=10YNL----------L' \
... f'&periodStart=202101010000&periodEnd=202104010000&securityToken={api_key}'
>>> response = requests.Session().get(url=url)
>>> element = objectify.fromstring(response.content)
>>> dump(element)
<Balancing_MarketDocument xmlns="urn:iec62325.351:tc57wg16:451-6:balancingdocument:3:0">
<mRID>051b91beed574b48b4548214e9001afc</mRID>
<revisionNumber>1</revisionNumber>
<type>A81</type>
<process.processType>A34</process.processType>
<sender_MarketParticipant.mRID codingScheme="A01">10X1001A1001A450</sender_MarketParticipant.mRID>
<sender_MarketParticipant.marketRole.type>A32</sender_MarketParticipant.marketRole.type>
<receiver_MarketParticipant.mRID codingScheme="A01">10X1001A1001A450</receiver_MarketParticipant.mRID>
<receiver_MarketParticipant.marketRole.type>A33</receiver_MarketParticipant.marketRole.type>
<createdDateTime>2021-10-04T18:12:43Z</createdDateTime>
<controlArea_Domain.mRID codingScheme="A01">10YNL----------L</controlArea_Domain.mRID>
<period.timeInterval>
<start>2020-12-31T23:00Z</start>
<end>2021-03-31T22:00Z</end>
</period.timeInterval>
<TimeSeries>
<mRID>1</mRID>
<businessType>A95</businessType>
<type_MarketAgreement.type>A01</type_MarketAgreement.type>
<mktPSRType.psrType>A04</mktPSRType.psrType>
<flowDirection.direction>A03</flowDirection.direction>
<quantity_Measure_Unit.name>MAW</quantity_Measure_Unit.name>
<curveType>A01</curveType>
<Period>
<timeInterval>
<start>2020-12-31T23:00Z</start>
<end>2021-01-01T23:00Z</end>
</timeInterval>
<resolution>PT60M</resolution>
<Point>
<position>1</position>
<quantity>44</quantity>
</Point>
<Point>
<position>2</position>
<quantity>44</quantity>
[...]
becomes
>>> import entsoe_client as ec
>>> from entsoe_client.ParameterTypes import *
>>> client = ec.Client(api_key)
>>> parser = ec.Parser
>>> query = ec.Query(
... documentType=DocumentType("Contracted reserves"),
... psrType=PsrType("Generation"),
... businessType=BusinessType("Frequency containment reserve"),
... controlArea_Domain=Area("NL"),
... type_MarketAgreementType=MarketAgreementType("Daily"),
... periodStart="2021-01-01T00:00",
... periodEnd="2021-04-01T00:00"
... )
>>> response = client(query)
>>> df = parser.parse(response)
>>> df.iloc[:,:3].head()
position quantity Period.timeInterval.start...
2020-12-31 23:00:00+00:00 1 44 2020-12-31T23:00Z
2021-01-01 00:00:00+00:00 2 44 2020-12-31T23:00Z
2021-01-01 01:00:00+00:00 3 44 2020-12-31T23:00Z
2021-01-01 02:00:00+00:00 4 44 2020-12-31T23:00Z
2021-01-01 03:00:00+00:00 5 44 2020-12-31T23:00Z
...
predefined queries are subset of the generic Query class, covering all examples of the ENTSO-E API guide.
>>> predefined_query = ec.Queries.Balancing.AmountOfBalancingReservesUnderContract(
... controlArea_Domain=Area("NL"),
... type_MarketAgreementType=MarketAgreementType("Daily"),
... psrType=PsrType("Generation"),
... periodStart="2021-01-01T00:00",
... periodEnd="2021-04-01T00:00"
... )
...
>>> predefined_query() == query()
True
Query templates abstract the API specifics through Enumerated types.
Parse responses into Pandas DataFrames without loss of any information.
Main contributions
- Exhaustive List of ParameterTypes.
These allow mapping between natural language and the codes required for GET requests, e.g.
DocumentType.A85 == DocumentType("Imbalance price")
. This feature allows keeping track of queries without jumping between documents or adding comments.
- Exhaustive List of Pre-defined Queries from ENTSO-E API Guide.
ENTSO-E API Guide is a minial set for any API connector to implement and reflects all dashboards on ENTSO-E Transparency Platform.
- Parsers
Response Documents come in XML schema which can be parsed into pandas DataFrames.
Implemented: GL_MarketDocuments, TransmissionNetwork_MarketDocuments, Publication_MarketDocuments and Balancing_MarketDocuments.
Missing: Outages, Congestion Management and System Operations.
Nevertheless, ENTSO-E Client seeks to be minimal to go from Query to DataFrame and requires domain- knowledge on how to formulate queries and interpret various columns of a parsed response.
ENTSO-E relies on many codes (Type) to map to desired queries. Types are encoded in Enum classes with a .help() function to list the all. They can be addressed through Type[code] or Type(string), making interaction easy. HTTP requests and responses usually require the code, whereas we want to formulate the query as a human-readable string.
from entsoe_client import Queries from entsoe_client.ParameterTypes import * Queries.Transmission.CapacityAllocatedOutsideEU( out_Domain=Area('SK'), in_Domain=Area('UA_BEI'), marketAgreementType=MarketAgreementType('Daily'), # Original code: A01 auctionType=AuctionType('Explicit'), # Original code: A02 auctionCategory=AuctionCategory('Hourly'), # Original code: A04 classificationSequence_AttributeInstanceComponent_Position=1, periodStart=201601012300, periodEnd=201601022300)
>>> ParameterTypes.DocumentType['A25'] == ParameterTypes.DocumentType('Allocation result document') True >>> ec.ParameterTypes.DocumentType.help() --- DocumentType --- API_PARAMETER: DESCRIPTION [...] A25: Allocation result document A71: Generation forecast A72: Reservoir filling information A73: Actual generation A85: Imbalance prices A86: Imbalance volume [...] API_PARAMETER: DESCRIPTION --- DocumentType --- >>> ec.ParameterTypes.BusinessType.help() --- BusinessType --- API_PARAMETER: DESCRIPTION [...] A25: General Capacity Information A29: Already allocated capacity(AAC) A97: Manual frequency restoration reserve B08: Total nominated capacity C22: Shared Balancing Reserve Capacity C24: Actual reserve capacity [...] API_PARAMETER: DESCRIPTION --- BusinessType ---
#shortened from sample_plot.py import entsoe_client as ec from settings import api_key # Instantiate Client, Parser and Query. client = ec.Client(api_key) parser = ec.Parser() query = ec.Queries.Generation.AggregatedGenerationPerType( in_Domain=ec.ParameterTypes.Area('DE_LU'), periodStart=202109050200, periodEnd=202109070200) # Extract data. response = client(query) df = parser(response) [...] # Transform data. production = df[~consumption_mask][['quantity', 'TimeSeries.MktPSRType.psrType']] ## PsrType, e.g. `B01` := `Biomass`. production['GenerationType'] = production['TimeSeries.MktPSRType.psrType']. \ apply(lambda x: ParameterTypes.PsrType[x].value) # Map ENTSO-E PsrTypes into human-readable string. production_by_type = pd.pivot_table(production, index=production.index, columns='GenerationType', values='quantity') [...] # Plot. production_by_type.plot.bar(title="Production by Generation Type in DE-LU", xlabel="UTC", ylabel='MWh', ax=ax, **plot_params) [...]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for entsoe_client-0.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0dfd8f9999927695adfcbfe1fdf5ae60f880879834d2a6f38f3764571d2a9519 |
|
MD5 | ea3db2e05490dee6365ad4dc8c126e4d |
|
BLAKE2b-256 | f8cefd7e1872febbf62f0873df74db3a9add52a00f6103ec9c6a937e2c4e4f96 |