Skip to main content

LENS: Locational Encoding with Neuromorphic Systems

Project description

:eye: LENS - Locational Encoding with Neuromorphic Systems

PyTorch License: MIT QUT Centre for Robotics stars Downloads Conda Version PyPI - Version GitHub repo size

This repository contains code for LENS - Locational Encoding with Neuromorphic Systems. LENS combines neuromorphic algoriths, sensors, and hardware to perform accurate, real-time robotic localization using visual place recognition (VPR). LENS can be used with the SynSense Speck2fDevKit board which houses a SPECKTM dynamic vision sensor and neuromorphic processor for online VPR.

License and citation

This repository is licensed under the MIT License. If you use our code, please cite our arXiv paper:

@misc{hines2024lens,
      title={A compact neuromorphic system for ultra energy-efficient, on-device robot localization}, 
      author={Adam D. Hines and Michael Milford and Tobias Fischer},
      year={2024},
      eprint={2408.16754},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2408.16754}, 
}

Installation and setup

To run LENS, please download this repository and install the required dependencies.

Get the code

Get the code by cloning the repository.

git clone git@github.com:AdamDHines/LENS.git
cd ~/LENS

Install dependencies

All dependencies can be instlled from our conda-forge package, PyPi package, or local requirements.txt. For the conda-forge package, we recommend using micromamba or miniforge. Please ensure your Python version is <= 3.11.

conda package

# Create a new environment and install packages
micromamba create -n lens-vpr -c conda-forge lens-vpr

# samna package is not available on conda-forge, so pip install it
micromamba activate lens-vpr
pip install samna

pip

# Install from our PyPi package
pip install lens-vpr

# Install from local requirements.txt
pip install -r requirements.txt

Quick start

Get started using our pretrained models and datasets to evaluate the system. For a full guide on training and evaluating your own datasets, please visit our Wiki.

Run the inferencing model

To run a simulated event stream, you can try our pre-trained model and datasets. Using the --sim_mat and --matching flag will display a similarity matrix and perform Recall@N matching based on a ground truth matrix.

python main.py --sim_mat --matching

Train a new model

New models can be trained by parsing the --train_model flag. Try training a new model with our provided reference dataset.

# Train a new model
python main.py --train_model

Optimize network hyperparameters

For new models on custom datasets, you can optimize your network hyperparameters using Weights & Biases through our convenient optimizer.py script.

# Optimize network hyperparameters
python optimizer.py

For more details, please visit the Wiki.

Deployment on neuromoprhic hardware

If you have a SynSense Speck2fDevKit, you can try out LENS using our pre-trained model and datasets by deploying simulated event streams on-chip.

# Generate a timebased simulation of event streams with pre-recorded data
python main.py --simulated_speck --sim_mat --matching

Additionally, models can be deployed onto the Speck2fDevKit for low-latency and energy efficient VPR with sequence matching in real-time. Use the --event_driven flag to start the online inferencing system.

# Run the online inferencing model
python main.py --event_driven

For more details on deployment to the Speck2fDevKit, please visit the Wiki.

Issues, bugs, and feature requests

If you encounter problems whilst running the code or if you have a suggestion for a feature or improvement, please report it as an issue.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lens-vpr-0.1.1.tar.gz (39.6 kB view details)

Uploaded Source

Built Distribution

lens_vpr-0.1.1-py3-none-any.whl (49.8 kB view details)

Uploaded Python 3

File details

Details for the file lens-vpr-0.1.1.tar.gz.

File metadata

  • Download URL: lens-vpr-0.1.1.tar.gz
  • Upload date:
  • Size: 39.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.10

File hashes

Hashes for lens-vpr-0.1.1.tar.gz
Algorithm Hash digest
SHA256 aac0707a5bd2e3c9f6dbbd155461906b84dd5be30b0b324655ed9ae29a22ecb0
MD5 ed44295398ea295d7ddb45e7604f7108
BLAKE2b-256 d4a0241babcaa8af22fb343729bab6bc0e9680e26d2407ed814f9d0915f7442d

See more details on using hashes here.

File details

Details for the file lens_vpr-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: lens_vpr-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 49.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.10

File hashes

Hashes for lens_vpr-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 082f0fa1da2155fdd052149edef603e93333628642fa3c4ab89bfbb786ee6843
MD5 628494c296ffb82ae6e6c84015f8dbdb
BLAKE2b-256 b916091155e9b7720fee53e0cb6da3509875c8ab1ed4bdea95aed2275cbcb408

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page