skipchunk 2.0.1

Solr

Start Solr first! Doesn’t work with Solr cloud yet, but we’re working on it.

You’ll need to start solr using skipchunk’s solr_home directory for now.

Then run this: python solr-blog-example.py

Elasticsearch

Start Elasticsearch first!

Then run this: python elasticsearch-blog-example.py

Engine configuration

You need an engine_config, as a dict, to create skipchunk.

The dict must contain the following entries

• host (the fully qualified URL of the engine web API endpoint)

• name (the name of the graph)

• path (the on-disk location of stateful data that will be kept)

• engine_name (either “solr” or “elasticsearch”)

engine_config_solr = {

    "host":"http://localhost:8983/solr/",

    "name":"osc-blog",

    "path":"./skipchunk_data",

    "engine_name":"solr"

}

engine_config_elasticsearch = {

    "host":"http://localhost:9200/",

    "name":"osc-blog",

    "path":"./skipchunk_data",

    "engine_name":"elasticsearch"

}

Skipchunk Initialization

When initializing Skipchunk, you will need to provide the constructor with the following parameters

• engine_config (the dict containing search engine connection details)

• spacy_model=“en_core_web_lg” (the spacy model to use to parse text)

• minconceptlength=1 (the minimum number of words that can appear in a noun phrase)

• maxconceptlength=3 (the maximum number of words that can appear in a noun phrase)

• minpredicatelength=1 (the minimum number of words that can appear in a verb phrase)

• maxpredicatelength=3 (the maximum number of words that can appear in a verb phrase)

• minlabels=1 (the number of times a concept/predicate must appear before it is recognized and kept. The lower this number, the more concepts will be kept - so be careful with large content sets!)

• cache_documents=False

• cache_pickle=False

Skipchunk Methods

• tuplize(filename=source,fields=['title','content',...]) (Produces a list of (text,document) tuples ready for processing by the enrichment.)

• enrich(tuples) (Enriching can take a long time if you provide lots of text. Consider batching at 10k docs at a time.)

• save (Saves to pickle)

• load (Loads from pickle)

Graph API

After enrichment, you can then index the graph into the engine

• index(skipchunk:Skipchunk) (Updates the knowledge graph in the search engine)

• delete (Deletes a knowledge graph - be careful!)

After indexing, you can call these methods to get autocompleted concepts or walk the knowledge graph

• conceptVerbConcepts(concept:str,verb:str,mincount=1,limit=100) -> list ( Accepts a verb to find the concepts appearing in the same context)

• conceptsNearVerb(verb:str,mincount=1,limit=100) -> list ( Accepts a verb to find the concepts appearing in the same context)

• verbsNearConcept(concept:str,mincount=1,limit=100) -> list ( Accepts a concept to find the verbs appearing in the same context)

• suggestConcepts(prefix:str,build=False) -> list ( Suggests a list of concepts given a prefix)

• suggestPredicates(prefix:str,build=False) -> list ( Suggests a list of predicates given a prefix)

• summarize(mincount=1,limit=100) -> list ( Summarizes a core)

• graph(subject:str,objects=5,branches=10) -> list ( Gets the subject-predicate-object neighborhood graph for a subject)