conceptnet-lite 0.2.0

Python library to work with ConceptNet offline without the need of PostgreSQL

conceptnet-lite [maintainance mode]

Conceptnet-lite is a Python library for working with ConceptNet offline without the need for PostgreSQL.

The library comes with Apache License 2.0, and is separate from ConceptNet itself. The ConceptNet is available under CC-BY-SA-4.0 license, which also applies to the formatted database file that we provide. See here for the list of conditions for using ConceptNet data.

This is the official citation for ConceptNet if you use it in research:

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. "ConceptNet 5.5: An Open Multilingual Graph of General Knowledge." In proceedings of AAAI 31.

Status

ConceptNet was not updated since September 2021, so Conceptnet-lite won't be updated either. Please do not use it in any new projects.

Installation

To install conceptnet-lite use pip:

$ pip install conceptnet-lite

Connecting to the database

Before you can use conceptnet-lite, you will need to obtain ConceptNet dabase file. You have two options: download pre-made one or build it yourself from the raw ConceptNet assertions file.

Downloading the ConceptNet database

ConceptNet releases happen once a year. You can use conceptnet-lite to build your own database from the raw assertions file (see below), but if there is a pre-built file it will be faster to just get that one. conceptnet-lite can download and unpack it to the specified folder automatically.

Here is a link to a compressed database for ConceptNet 5.7. This link is used automatically if you do not supply the alternative.

import conceptnet_lite

conceptnet_lite.connect("/path/to/conceptnet.db")

This command both downloads the resource (our build for ConceptNet 5.7) and connects to the database. If path specified as the first argument does not exist, it will be created (unless there is a permissions problem). Note that the database file is quite large (over 9 Gb).

If your internet connection is intermittent, the built-in download function may give you errors. If so, just download the file separately, unpack it to the directory of your choice and provide the path to the .connect() method as described below.

Building the database for a new release.

If a database file is not found in the folder specified in the db_path argument, conceptnet-lite will attempt to automatically download the raw assertions file from here and build the database. This takes a couple of hours, so we recommend getting the pre-built file.

If you provide a path, this is where the database will be built. Note that the database file is quite large (over 9 Gb). Note that you need to pass db_download_url=None to force the library build the database from dump.

import conceptnet_lite

conceptnet_lite.connect("/path/to/conceptnet.db", db_download_url=None)

If the specified does not exist, it will be created (unless there is a permissions problem). If no path is specified, and no database file is not found in the current working directory, conceptnet-lite will attempt to build one in the current working directory.

Once the database is built, conceptnet-lite will connect to it automatically.

Loading the ConceptNet database

Once you have the database file, all you need to do is to pass the path to it to the .connect() method.

import conceptnet_lite

conceptnet_lite.connect("/path/to/conceptnet.db")

If no path is specified, conceptnet-lite will check if a database file exists in the current working directory. If it is not found, it will trigger the process of downloading the pre-built database (see above).

Accessing concepts

Concepts objects are created by looking for every entry that matches the input string exactly. If none is found, the peewee.DoesNotExist exception will be raised.

from conceptnet_lite import Label

cat_concepts = Label.get(text='cat').concepts  
for c in cat_concepts:
    print("    Concept URI:", c.uri)
    print("    Concept text:", c.text)

Concept URI: /c/en/cat
Concept text: cat
Concept URI: /c/en/cat/n
Concept text: cat
Concept URI: /c/en/cat/n/wn/animal
Concept text: cat
Concept URI: /c/en/cat/n/wn/person
...

concept.uri provides access to ConceptNet URIs, as described here. You can also retrieve only the text of the entry by concept.text.

Working with languages

You can limit the languages to search for matches. Label.get() takes an optional language attribute that is expected to be an instance Language, which in turn is created by calling Language.get() with name argument. List of available languages and their codes are described here.

from conceptnet_lite import Label

cat_concepts = Label.get(text='cat', language='en').concepts  
for c in cat_concepts:
    print("    Concept URI:", c.uri)
    print("    Concept text:", c.text)
    print("    Concept language:", c.language.name)

    Concept URI: /c/en/cat
    Concept text: cat
    Concept language: en
    Concept URI: /c/en/cat/n
    Concept text: cat
    Concept language: en
    Concept URI: /c/en/cat/n/wn/animal
    Concept text: cat
    Concept language: en
    Concept URI: /c/en/cat/n/wn/person
    Concept text: cat
    Concept language: en
...

Querying edges between concepts

To retrieve the set of relations between two concepts, you need to create the concept objects (optionally specifying the language as described above). cn.edges_between() method retrieves all edges between the specified concepts. You can access its URI and a number of attributes, as shown below.

Some ConceptNet relations are symmetrical: for example, the antonymy between white and black works both ways. Some relations are asymmetrical: e.g. the relation between cat and mammal is either hyponymy or hyperonymy, depending on the direction. The two_way argument lets you choose whether the query should be symmetrical or not.

from conceptnet_lite import Label, edges_between

introvert_concepts = Label.get(text='introvert', language='en').concepts
extrovert_concepts = Label.get(text='extrovert', language='en').concepts
for e in edges_between(introvert_concepts, extrovert_concepts, two_way=False):
    print("  Edge URI:", e.uri)
    print("  Edge name:", e.relation.name)
    print("  Edge start node:", e.start.text)
    print("  Edge end node:", e.end.text)
    print("  Edge metadata:", e.etc)

  Edge URI: /a/[/r/antonym/,/c/en/introvert/n/,/c/en/extrovert/]
  Edge name: antonym
  Edge start node: introvert
  Edge end node: extrovert
  Edge metadata: {'dataset': '/d/wiktionary/en', 'license': 'cc:by-sa/4.0', 'sources': [{'contributor': '/s/resource/wiktionary/en', 'process': '/s/process/wikiparsec/2'}, {'contributor': '/s/resource/wiktionary/fr', 'process': '/s/process/wikiparsec/2'}], 'weight': 2.0}

• e.relation.name: the name of ConceptNet relation. Full list here.

• e.start.text, e.end.text: the source and the target concepts in the edge

• e.etc: the ConceptNet metadata dictionary contains the source dataset, sources, weight, and license. For example, the introvert:extrovert edge for English contains the following metadata:

{
	"dataset": "/d/wiktionary/en",
	"license": "cc:by-sa/4.0",
	"sources": [{
		"contributor": "/s/resource/wiktionary/en",
		"process": "/s/process/wikiparsec/2"
	}, {
		"contributor": "/s/resource/wiktionary/fr",
		"process": "/s/process/wikiparsec/2"
	}],
	"weight": 2.0
}

Accessing all relations for a given concepts

You can also retrieve all relations between a given concepts and all other concepts, with the same options as above:

from conceptnet_lite import Label, edges_for

for e in edges_for(Label.get(text='introvert', language='en').concepts, same_language=True):
    print(e.start.text, "::", e.end.text, "|", e.relation.name)

extrovert :: introvert | antonym
introvert :: extrovert | antonym
outrovert :: introvert | antonym
reflection :: introvert | at_location
introverse :: introvert | derived_from
introversible :: introvert | derived_from
introversion :: introvert | derived_from
introversion :: introvert | derived_from
introversive :: introvert | derived_from
introverted :: introvert | derived_from
...

The same set of edge attributes are available for edges_between and edges_for (e.uri, e.relation.name, e.start.text, e.end.text, e.etc).

Note that we have used optional argument same_language=True. By supplying this argument we make edges_for return relations, both ends of which are in the same language. If this argument is skipped it is possible to get edges to concepts in languages other than the source concepts language. For example, the same command as above with same_language=False will include the following in the output:

kääntyä_sisäänpäin :: introvert | synonym
sulkeutua :: introvert | synonym
sulkeutunut :: introvert | synonym
introverti :: introvert | synonym
asociale :: introvert | synonym
introverso :: introvert | synonym
introvertito :: introvert | synonym
内向 :: introvert | synonym

Accessing concept edges with a given relation direction

You can also query the relations that have a specific concept as target or source. This is achieved with concept.edges_out and concept.edges_in, as follows:

from conceptnet_lite import Label

concepts = Label.get(text='introvert', language='en').concepts  
for c in concepts:
    print("    Concept text:", c.text)
    if c.edges_out:
        print("      Edges out:")
        for e in c.edges_out:
            print("        Edge URI:", e.uri)
            print("        Relation:", e.relation.name)
            print("        End:", e.end.text)
    if c.edges_in:
        print("      Edges in:")
        for e in c.edges_in:
            print("        Edge URI:", e.uri)
            print("        Relation:", e.relation.name)
            print("        End:", e.end.text)

    Concept text: introvert
      Edges out:
        Edge URI: /a/[/r/etymologically_derived_from/,/c/en/introvert/,/c/la/introvertere/]
        Relation: etymologically_derived_from
        End: introvertere
...
      Edges in:
        Edge URI: /a/[/r/antonym/,/c/cs/extrovert/n/,/c/en/introvert/]
        Relation: antonym
        End: introvert
...

Traversing all the data for a language

You can go over all concepts for a given language. For illustration, let us try Old Norse, a "small" language with the code "non" and vocab size of 7868, according to the ConceptNet language statistics.

from conceptnet_lite import Language

mylanguage = Language.get(name='non')
for l in mylanguage.labels:
    print("  Label:", l.text)
    for c in l.concepts:
        print("    Concept URI:", c.uri)
        if c.edges_out:
            print("      Edges out:")
            for e in c.edges_out:
                print("        Edge URI:", e.uri)
        if c.edges_in:
            print("      Edges in:")
            for e in c.edges_in:
                print("        Edge URI:", e.uri)

  Label: andsœlis
    Concept URI: /c/non/andsœlis/r
      Edges out:
        Edge URI: /a/[/r/antonym/,/c/non/andsœlis/r/,/c/non/réttsœlis/]
        Edge URI: /a/[/r/related_to/,/c/non/andsœlis/r/,/c/en/against/]
        Edge URI: /a/[/r/related_to/,/c/non/andsœlis/r/,/c/en/course/]
        Edge URI: /a/[/r/related_to/,/c/non/andsœlis/r/,/c/en/sun/]
        Edge URI: /a/[/r/related_to/,/c/non/andsœlis/r/,/c/en/widdershins/]
        Edge URI: /a/[/r/synonym/,/c/non/andsœlis/r/,/c/non/rangsœlis/]
    Concept URI: /c/non/andsœlis
      Edges out:
        Edge URI: /a/[/r/external_url/,/c/non/andsœlis/,/c/en.wiktionary.org/wiki/andsœlis/]
  Label: réttsœlis
    Concept URI: /c/non/réttsœlis
      Edges in:
        Edge URI: /a/[/r/antonym/,/c/non/andsœlis/r/,/c/non/réttsœlis/]
...

Accessing Concepts by URI

You can access concept ORM objects directly by providing a desired ConceptNet URI. This is done as follows:

from conceptnet_lite import Concept

edge_object = Edge.get(start='/c/en/example')
concept_object = Concept.get(uri='/c/en/example')