graphvision 0.1

Create topology map for image segments

Create topology map for image segments

GraphVision

A library fascilitating generation of topological representation fpr the image segments. The topological graph is created based on the segments and visual embeddings as nodes. The segment topology, doesn't only keep the spatial but semantic infromation as well, this is very helpful for various tasks in visual robotics[1] and localization.

GraphVision provides such graphical representations that let us perform visual query, and communicate with our segmentation graph, this would generally require a lot of preprocessing hassle. VisonGraph does it all itself, and also provides functionalities to visualize the segment topology and perfom visual query on the graph leveraging dijkstras` algorithm for localization.

Usage

The library requirements are flexible but still its suggested to use a virtual environment.

pip install graphvision

Read an image as rgb, import the Generator and SegmentGraph. Initialize the generator, call the segmentation model, generate the masks and that is it, you are ready create the topological graph for your segments.

from graphvision import Generator, SegmentGraph
gen = Generator()
mask_gen = gen.SAM()
segments = gen.generate_masks(mask_gen, image_rgb)

You can use visualize or plot it using gen.plot_segments(segments)

Create the topological graph

sg = SegmentGraph(segments, image_rgb)
G, centroids = sg.get_topology(dist_thres=150) # Other Optional Parameters: area_percent, add_to_bbox

Two graph G can be returned as a 'networkx' (default) or as a PyTorch geormetric object. This graphical representation can now be leveraged in various vision tasks such as object localization, environment mapping in robotics based on both spatial and semantic features.

Query the segment graph

You can also perform visual queries on the graph, to locate objects dependent on other nodes (neighbouring objects), This a naive implementation, please kindly go through the colab notebook to understand to understand it in greateer detail.

result = sg.query_segment_graph(G, query_pair=("Girl", "plant"), show_legend=True)

Reference

1. RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation