LENS: Locational Encoding with Neuromorphic Systems
Project description
:eye: LENS - Locational Encoding with Neuromorphic Systems
This repository contains code for LENS - Locational Encoding with Neuromorphic Systems. LENS combines neuromorphic algoriths, sensors, and hardware to perform accurate, real-time robotic localization using visual place recognition (VPR). LENS can be used with the SynSense Speck2fDevKit board which houses a SPECKTM dynamic vision sensor and neuromorphic processor for online VPR.
License and citation
This repository is licensed under the MIT License. If you use our code, please cite our arXiv paper:
@misc{hines2024lens,
title={A compact neuromorphic system for ultra energy-efficient, on-device robot localization},
author={Adam D. Hines and Michael Milford and Tobias Fischer},
year={2024},
eprint={2408.16754},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2408.16754},
}
Installation and setup
To run LENS, please download this repository and install the required dependencies.
Get the code
Get the code by cloning the repository.
git clone git@github.com:AdamDHines/LENS.git
cd ~/LENS
Install dependencies
All dependencies can be instlled from our conda-forge package, PyPi package, or local requirements.txt
. For the conda-forge package, we recommend using micromamba or miniforge. Please ensure your Python version is <= 3.11.
conda package
# Create a new environment and install packages
micromamba create -n lens-vpr -c conda-forge lens-vpr
# samna package is not available on conda-forge, so pip install it
micromamba activate lens-vpr
pip install samna
pip
# Install from our PyPi package
pip install lens-vpr
# Install from local requirements.txt
pip install -r requirements.txt
Quick start
Get started using our pretrained models and datasets to evaluate the system. For a full guide on training and evaluating your own datasets, please visit our Wiki.
Run the inferencing model
To run a simulated event stream, you can try our pre-trained model and datasets. Using the --sim_mat
and --matching
flag will display a similarity matrix and perform Recall@N matching based on a ground truth matrix.
python main.py --sim_mat --matching
Train a new model
New models can be trained by parsing the --train_model
flag. Try training a new model with our provided reference dataset.
# Train a new model
python main.py --train_model
Optimize network hyperparameters
For new models on custom datasets, you can optimize your network hyperparameters using Weights & Biases through our convenient optimizer.py
script.
# Optimize network hyperparameters
python optimizer.py
For more details, please visit the Wiki.
Deployment on neuromoprhic hardware
If you have a SynSense Speck2fDevKit, you can try out LENS using our pre-trained model and datasets by deploying simulated event streams on-chip.
# Generate a timebased simulation of event streams with pre-recorded data
python main.py --simulated_speck --sim_mat --matching
Additionally, models can be deployed onto the Speck2fDevKit for low-latency and energy efficient VPR with sequence matching in real-time. Use the --event_driven
flag to start the online inferencing system.
# Run the online inferencing model
python main.py --event_driven
For more details on deployment to the Speck2fDevKit, please visit the Wiki.
Issues, bugs, and feature requests
If you encounter problems whilst running the code or if you have a suggestion for a feature or improvement, please report it as an issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.