pyngsi

What is it ?

pyngsi is a Python framework that helps you write a pipeline for your Fiware dataflow.

Writing a NGSI agent that relies on pyngsi avoids all the plumbing so you can focus on writing your own logic to build NGSI entities.

Key Features

• NGSI v2 support
• Map Python-native data to NGSI entities
• Write NGSI entities to Fiware Orion
• Handle incoming data through a common interface
• Compute statistics
• Allow visualization/debugging facilities

Where to get it

The source code is currently hosted on GitHub at : https://github.com/Orange-OpenSource/pyngsi

Binary installer for the latest released version is available at the Python package index.

pip install pyngsi

Getting started

Build your first NGSI entity

from pyngsi.ngsi import DataModel

m = DataModel(id="Room1", type="Room")
m.add_url("dataProvider", "https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html#entity-creation")
m.add("pressure", 720)
m.add("temperature", 23.0)

m.pprint()

The resulting JSON looks like this :

{
  "id": "Room1",
  "type": "Room",
  "dataProvider": {
    "value": "https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html#entity-creation",
    "type": "URL"
  },
  "pressure": {
    "value": 720,
    "type": "Integer"
  },
  "temperature": {
    "value": 23.0,
    "type": "Float"
  }
}

Send the NGSI entity to the Orion broker

from pyngsi.sink import SinkOrion

sink = SinkOrion()
sink.write(m.json())

Develop your own NGSI Agent

Let's quickly create a CSV file to store values from our room sensors

echo -e "Room1;23;720\nRoom2;21;711" > room.csv

Let's code a function that converts incoming rows to NGSI entities

def build_entity(row: Row) -> DataModel:
    id, temperature, pressure = row.record.split(';')
    m = DataModel(id=id, type="Room")
    m.add_url("dataProvider", row.provider)
    m.add("temperature", float(temperature))
    m.add("pressure", int(pressure))
    return m

Let's use it in in our new NGSI Agent

from pyngsi.sources.source import Source, Row
from pyngsi.sink import SinkOrion
from pyngsi.agent import NgsiAgent

src = Source.from_file("room.csv")
sink = SinkOrion()
agent = NgsiAgent.create_agent(src, sink, process=build_entity)
agent.run()

This basic example shows how the pyngsi framework is used to build a NGSI Agent.
Here data are stored on the local filesystem.
By changing just one line you could retrieve incoming data from a FTP server or HTTP server.

from pyngsi.sources.source import Source, Row
from pyngsi.sources.server import ServerHttpUpload
from pyngsi.sink import SinkOrion
from pyngsi.agent import NgsiAgent

src = ServerHttpUpload() # line updated !
sink = SinkOrion()
agent = NgsiAgent.create_agent(src, sink, process=build_entity)
agent.run()

The HTTP server is running. Now you can send the file to the endpoint.

curl -F file=@room.csv http://127.0.0.1:8880/upload

JSON and text formats are natively supported.
Many sources and sinks are provided, i.e. SinkStdout to just displays entities, eliminating the need of having an Orion server running.
One could create a custom Source to handle custom data. The MicrosoftExcelSource is given as exemple.
One could extend the framework according to his needs.

Dependencies

• loguru
• requests
• requests-toolbelt
• shortuuid
• more_itertools
• geojson
• flask
• cherrypy
• schedule
• openpyxl

License

Apache 2.0

Documentation

The official documentation is hosted at https://pixel-ports.github.io/pyngsi-tutorial.html

Known Issues

SourceMicrosoftExcel may fail to open some odd Excel files due to an openpyxl bug (i.e. sometimes cannot read graphs).
In this case try to remove the offending sheet or prefer working with CSV files.

Background

Work on pyngsi started at Orange in 2019 for the needs of the PIXEL european project.

Funding

pyngsi has been developed as part of the PIXEL project, H2020, funded by the EC under Grant Agreement number 769355.