cubicweb-semnews 0.2.0

Cube for news storage and analysis

Summary

Cube for news storage and analysis

This cube provides an implementation of Semnews:

• store news articles and tweets.

• extract and synthetize information.

• provide semantic useful and original visualisation.

• analytics tools and datamining/machine learning processings.

Installation

Creation of the instance:

• Create an instance using: cubicweb-ctl create semnews <name-of-instance>

• Create the instance’s database using: cubicweb-ctl db-create <name-of-instance>

Add articles sources

Source of articles could be created using:

• Blogs/RSS feeds:

  session.create_entity('CWSource', name=<name of the source>, type=u'datafeed',
                        parser=u'rss-parser', lang=<lang of the source>,
                        url=<url of the blog/rss feed>,
                        config=u'synchronization-interval=120min')

• Tweet:

  session.create_entity('CWSource', name=<name of the source>, type=u'datafeed',
                        parser=u'tweet-parser', lang=<lang of the source>,
                        url=<url of the blog/rss feed>,
                        config=u'synchronization-interval=120min')

The synchronization interval could be setted to a more specific value, or setted to “no” for manual synchronization only.

Semnews comes with some predifined blogs/tweets/rss feeds:

• Some french political blogs. You can add them using:

  cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_blogs_fr.py

• Some international english newspapers. You can add them using:

  cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_newspapers.py

• Some french newspapers. You can add them using:

  cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_newspapers_fr.py

• Some french politician tweets. You can add them using:

  cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_twitters_fr.py

Add Named Entities sources

Semnews is based on a named entities process, that you have to define:

session.create_entity('NerProcess', name=<name of process>, host=<appid or sparql endpoint url>,
                      type=<rql or sparql>, lang=<optional lang of the ner source>,
                      request=<request to be performed>)

See the document of the NER cube for more details. Example of source:

session.create_entity('NerProcess', name=u'dbpedia38-en', host=u'ner',
                      type=u'rql', lang=u'en',
                      request=u'Any U WHERE X label %(token)s, X cwuri U, '
                               'X ner_source NS, NS name "dbpedia38-en"')

Commands

Semnews provide to commands:

• A command to extract named entities from articles:

  cubicweb-ctl process-ner <name-of-instance>

• A command to cleanup recognized entities according to some Dbpedia categories (see entities/external_resources.py):

  cubicweb-ctl cleanup-ner <name-of-instance>