conceptnet-local 1.0.2

ConceptNet and embeddings with a local DB and API.

Local ConceptNet

ConceptNet and fastText embeddings with a local DB and API. English only.

Setup

Make sure the path to the DB is included in your environment variables:

CN_DB_PATH=<path>/cn.db

The DB file can be downloaded from here (~2.2GB): https://drive.google.com/file/d/1ykKOa8oUzdiI5OShaQgyP8AZBH2iHDot/view?usp=sharing

Alternatively, the DB can be set up manually by navigating to the db_setup directory in this repository and following the instructions in the notebook there.

Example Usage

Import:

import conceptnet_local as cnl

Concept Details

List of all concept IDs:

all_concept_ids = cnl.get_all_concept_ids()

All edges connecting to a concept:

edges = cnl.get_all_edges(cn_id="/c/en/example")

Relatedess between two concepts according to fastText embeddings:

relatedness = cnl.get_relatedness("/c/en/example", "/c/en/test")

Lowest-Cost Paths

Get a variant of A* by specifying cost and heuristic weights:

from conceptnet_local import get_a_star_variant, CostFunction, HeuristicFunction

CustomAStar = get_a_star_variant(
    cost_weights={
        CostFunction.EDGE_COUNT: 2.,
        CostFunction.SIMILARITY_DIFFERENCE: 1.,
    },
    heuristic_weights={
        HeuristicFunction.SIMILARITY_TO_GOAL: 1.,
    }
)

or create a fully custom variant of A*:

from conceptnet_local import AStar, Concept, Relation

class CustomAStar(AStar):
    def get_cost(self, source: Concept, target: Concept, relation: Relation, goal: Concept) -> float:
        ...

    def get_heuristic(self, current: Concept, goal: Concept) -> float:
        ...

Get the lowest-cost path:

custom_a_star = CustomAStar()
path = custom_a_star.compute_path(input_concept="/c/en/example", output_concept="/c/en/test", print_time=True)

print(cnl.format_path(path=path))

Concept Extraction

Extract the ConceptNet concepts from a given text, along with their locations in the text:

concepts_in_text = cnl.get_concepts_in_text(text="This is an example.")

Custom Queries

Get a reference to the DB:

connection, cursor = cnl.setup_sqlite_db()

Execute queries, e.g.:

partial_id = "examp"
statement = cursor.execute("SELECT embedding FROM embeddings WHERE concept_id LIKE '%?%'", (partial_id,))
result = statement.fetchall()

Close DB connection:

cnl.close_sqlite_db(db_connection=connection)

Version History

1.0

• release 🎉

0.9

• concept extraction from text
• concept degree retrieval

0.8

• concepts table
• retrieval of similar concepts
• concept existence check

0.7

• separate utils file
• link label formatting
• natural-language path formatting

0.6

• no more FastText model built in
• embedding computation method optional in relatedness method
• optimized embedding retrieval from DB

0.5

• custom initialization method in A*

0.4

• FastText embeddings for arbitrary text

0.3

• retrieval of all concept IDs

0.2

• FastText embeddings

0.1

• reading links and embeddings from DB
• custom A* with configurable variants
• greedy version of Yen's algorithm