deeplens-core 1.5.0

DeepLens: differentiable optical lens simulator for end-to-end cameras.

Docs • Tutorials • Community • PyPI

DeepLens is a differentiable optical lens simulator. It is developed for (1) differentiable optical design, (2) end-to-end optics-vision co-design, and (3) photorealistic image simulation. DeepLens helps researchers build custom differentiable optical systems and computational imaging pipelines with minimal effort.

Mission

1. Next-generation optical design software enhanced by differentiable optimization.
2. Next-generation computational cameras integrating optical encoding with deep learning decoding.

Features

1. Differentiable Optics. DeepLens leverages gradient backpropagation and differentiable optimization, demonstrating outstanding optimization power compared to classical optical design.
2. Automated Lens Design. Enables automated lens design using curriculum learning, optical regularization losses, and GPU acceleration.
3. Hybrid Refractive-Diffractive Optics. Supports accurate simulation and optimization of hybrid refractive-diffractive lenses (e.g., DOEs, metasurfaces).
4. Accurate Image Simulation. Delivers photorealistic, spatially-varying image simulations, verified against commercial software and real-world experiments.
5. Optics-Vision Co-Design. Supports end-to-end differentiability from optics, sensor, and ISP to vision algorithms, enabling comprehensive optics-vision co-design.

Additional features (available via collaboration):

1. Polarization Ray Tracing. Provides polarization ray tracing and differentiable optimization of coating films.
2. Non-Sequential Ray Tracing. Includes a differentiable non-sequential ray tracing model for stray light analysis and optimization.
3. Kernel Acceleration. Achieves >10x speedup and >90% GPU memory reduction with custom GPU kernels across NVIDIA and AMD platforms.
4. Distributed Optimization. Supports distributed simulation and optimization for billions of rays and high-resolution (>100k x 100k) diffractive computations.

Applications

1. Automated lens design

Fully automated lens design from scratch. Try it with AutoLens!

2. End-to-End lens design

Lens-network co-design from scratch using final images (or classification/detection/segmentation) as objective.

3. Implicit Lens Representation

A surrogate network for fast (aberration + defocus) image simulation.

4. Hybrid Refractive-Difractive Lens Model

Design hybrid refractive-diffractive lenses with a new ray-wave model.

Installation

Clone this repo:

git clone https://github.com/singer-yang/DeepLens
cd DeepLens

Create a conda environment:

conda create -n deeplens_env python=3.12
conda activate deeplens_env

# Linux and Mac
pip install torch torchvision
# Windows
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

pip install -r requirements.txt

or

conda env create -f environment.yml -n deeplens_env

Run the demo code:

python 0_hello_deeplens.py

DeepLens repo is structured around three decoupled modules (optics, sensor, and network):

DeepLens/
│
├── deeplens/
│   ├── optics/ (optics simulation and lens models)
│   ├── sensor/ (sensor + ISP simulation)
│   ├── network/ (surrogate and reconstruction networks)
│   ├── camera.py (composes lens and sensor into a camera simulator)
│   └── ...
│
├── 0_hello_deeplens.py (code tutorials)
├── ...
└── write_your_own_code.py

Community

Join our Slack workspace and WeChat Group (singeryang1999) to connect with our core contributors, receive the latest industry updates, and be part of our community. For any inquiries, contact Xinge Yang ([e-mail]).

Contribution

We welcome all contributions. To get started, please read our Contributing Guide or check out open questions. All project participants are expected to adhere to our Code of Conduct. A list of contributors can be viewed in Contributors and below:

Citation

If you use DeepLens in your research, please cite the paper. See more in History of DeepLens.

@article{yang2024curriculum,
  title={Curriculum learning for ab initio deep learned refractive optics},
  author={Yang, Xinge and Fu, Qiang and Heidrich, Wolfgang},
  journal={Nature communications},
  volume={15},
  number={1},
  pages={6572},
  year={2024},
  publisher={Nature Publishing Group UK London}
}