lens-vpr 0.1.0

LENS: Locational Encoding with Neuromorphic Systems

:eye: LENS - Locational Encoding with Neuromorphic Systems

This repository contains code for LENS - Locational Encoding with Neuromorphic Systems. LENS combines neuromorphic algoriths, sensors, and hardware to perform accurate, real-time robotic localization using visual place recognition (VPR). LENS can be used with the SynSense Speck2fDevKit board which houses a SPECK^TM dynamic vision sensor and neuromorphic processor for online VPR.

License & Citation

This repository is licensed under the MIT License

If you use our code, please cite our paper:

@arxiv{hines202xlens,
      title={Robotic localization and navigation using a biologically inspired compact neuromorphic ecosystem}, 
      author={Adam D. Hines and Michael Milford and Tobias Fischer},
      year={2024},
      booktitle={}
      
}

Installation and setup

To run LENS, please download this repository and install the required dependencies.

Get the code

Get the code by cloning the repository.

git clone [e-mail]:AdamDHines/LENS.git
cd ~/LENS

Install dependencies

All dependencies can be instlled from our PyPi package, local requirements.txt, or from conda-forge (recommended to use micromamba). Please ensure your Python version is <= 3.11.

pip install

# Install from our PyPi package
pip install lens

# Install from local requirements.txt
pip install -r requirements.txt

conda install

# Create new environment and install dependencies from conda-forge
conda create -n lens lens-vpr python=3.11 -c conda-forge

Quick start

Get started using our pretrained models and datasets to evaluate the system. For a full guide on training and evaluating your own datasets, please visit our Wiki.

Run the inferencing model

To run a simulated event stream, you can try our pre-trained model and datasets. Using the --sim_mat and --matching flag will display a similarity matrix and perform Recall@N matching based on a ground truth matrix.

python main.py --sim_mat --matching

Train a new model

New models can be trained by parsing the --train_model flag. Try training a new model with our provided reference dataset.

# Train a new model
python main.py --train_model

Optimize network hyperparameters

For new models on custom datasets, you can optimize your network hyperparameters using Weights & Biases through our convenient optimizer.py script.

# Optimize network hyperparameters
python optimizer.py

For more details, please visit the Wiki.

Deployment on neuromoprhic hardware

If you have a SynSense Speck2fDevKit, you can try out LENS using our pre-trained model and datasets by deploying simulated event streams on-chip.

# Generate a timebased simulation of event streams with pre-recorded data
python main.py --simulated_speck --sim_mat --matching

Additionally, models can be deployed onto the Speck2fDevKit for low-latency and energy efficient VPR with sequence matching in real-time.

First, install samna.

# Install samna from pip
pip install samna
# Check that the package imports properly
python
> import samna

Then, use the --event_driven flag to start the online inferencing system.

# Run the online inferencing model
python main.py --event_driven

For more details on deployment to the Speck2fDevKit, please visit the Wiki.

Issues, bugs, and feature requests

If you encounter problems whilst running the code or if you have a suggestion for a feature or improvement, please report it as an issue.