LENS: Locational Encoding with Neuromorphic Systems
Project description
:eye: LENS - Locational Encoding with Neuromorphic Systems
This repository contains code for LENS - Locational Encoding with Neuromorphic Systems. LENS combines neuromorphic algoriths, sensors, and hardware to perform accurate, real-time robotic localization using visual place recognition (VPR). LENS can be used with the SynSense Speck2fDevKit board which houses a SPECKTM dynamic vision sensor and neuromorphic processor for online VPR.
License & Citation
This repository is licensed under the MIT License
If you use our code, please cite our paper:
@arxiv{hines202xlens,
title={Robotic localization and navigation using a biologically inspired compact neuromorphic ecosystem},
author={Adam D. Hines and Michael Milford and Tobias Fischer},
year={2024},
booktitle={}
}
Installation and setup
To run LENS, please download this repository and install the required dependencies.
Get the code
Get the code by cloning the repository.
git clone git@github.com:AdamDHines/LENS.git
cd ~/LENS
Install dependencies
All dependencies can be instlled from our PyPi package, local requirements.txt
, or from conda-forge (recommended to use micromamba). Please ensure your Python version is <= 3.11.
pip install
# Install from our PyPi package
pip install lens
# Install from local requirements.txt
pip install -r requirements.txt
conda install
# Create new environment and install dependencies from conda-forge
conda create -n lens lens-vpr python=3.11 -c conda-forge
Quick start
Get started using our pretrained models and datasets to evaluate the system. For a full guide on training and evaluating your own datasets, please visit our Wiki.
Run the inferencing model
To run a simulated event stream, you can try our pre-trained model and datasets. Using the --sim_mat
and --matching
flag will display a similarity matrix and perform Recall@N matching based on a ground truth matrix.
python main.py --sim_mat --matching
Train a new model
New models can be trained by parsing the --train_model
flag. Try training a new model with our provided reference dataset.
# Train a new model
python main.py --train_model
Optimize network hyperparameters
For new models on custom datasets, you can optimize your network hyperparameters using Weights & Biases through our convenient optimizer.py
script.
# Optimize network hyperparameters
python optimizer.py
For more details, please visit the Wiki.
Deployment on neuromoprhic hardware
If you have a SynSense Speck2fDevKit, you can try out LENS using our pre-trained model and datasets by deploying simulated event streams on-chip.
# Generate a timebased simulation of event streams with pre-recorded data
python main.py --simulated_speck --sim_mat --matching
Additionally, models can be deployed onto the Speck2fDevKit for low-latency and energy efficient VPR with sequence matching in real-time.
First, install samna
.
# Install samna from pip
pip install samna
# Check that the package imports properly
python
> import samna
Then, use the --event_driven
flag to start the online inferencing system.
# Run the online inferencing model
python main.py --event_driven
For more details on deployment to the Speck2fDevKit, please visit the Wiki.
Issues, bugs, and feature requests
If you encounter problems whilst running the code or if you have a suggestion for a feature or improvement, please report it as an issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.