torchidl 0.0.4

Official package of Implicit Deep Learning and State-driven Implicit Models

Implicit Deep Learning Package

Documentation • Quick Tutorial • Installation • Examples • Citation

Authors: Hoang Phan, Bao Tran, Chi Nguyen, Bao Truong, Thanh Tran, Khai Nguyen, Alicia Y. Tsai, Hong Chu, Laurent El Ghaoui

Introduction

Traditional feedforward neural networks compute hidden states by passing inputs sequentially through layers. In contrast, Implicit Deep Learning determines these hidden states by solving fixed-point equations. This approach offers theoretical advantages in stability, memory efficiency, and expressivity.

Formally, given input $u \in \mathbb{R}^p$ and output $y \in \mathbb{R}^q$, the implicit model (Figure b) computes the equilibrium hidden state $x \in \mathbb{R^n}$ by solving:

$$ \begin{cases} \begin{aligned} x &= \phi(A x + B u), \quad &&\text{(Equilibrium equation)} \ \hat{y} &= C x + D u, \quad &&\text{(Prediction equation)} \end{aligned} \end{cases} $$

where $\phi$ is a non-linear activation function that satisfies the well-posedness condition (e.g., ReLU), and $A \in \mathbb{R}^{n \times n}, B \in \mathbb{R}^{n \times p}, C \in \mathbb{R}^{q \times n}, D \in \mathbb{R}^{q \times p}$ are learnable parameters.

This formulation can represent any feedforward network. For example, a simple MLP with 2 hidden layers (Figure a):

$$ x_1 = \phi(W_0u) \longrightarrow x_2=\phi(W_1x_1) \longrightarrow \hat{y}(u) = W_2 x_2. $$

can be rewritten in matrix form as:

$$x = \phi(A x + B u) = \phi \left( \left[\begin{array}{cc} 0 & W_1 \ 0 & 0 \end{array}\right] \left[\begin{array}{c} x_2 \ x_1 \end{array}\right] + \left[\begin{array}{c} 0 \ W_0 \end{array}\right] u \right) = \phi \left[\begin{array}{c} W_1 x_1 \ W_0 u \end{array}\right] = \left[\begin{array}{c} x_2 \ x_1 \end{array}\right] $$

$$\hat{y} = C x + D u = \left[\begin{array}{cc} W_2 & 0 \end{array}\right] \left[\begin{array}{c} x_2 \ x_1 \end{array}\right] + \left[\begin{array}{c} 0 \end{array}\right] u = W_2 x_2$$

For a conceptual introduction to Implicit Models, see this article on Medium. For mathematical foundations and technical details, refer to the SIAM journal paper.

Installation

• Via pip:

  pip install torchidl
  

• From source:

  git clone https://github.com/HoangP8/Implicit-Deep-Learning && cd Implicit-Deep-Learning
  pip install -e .
  

Quick Tour

The torchidl package provides a comprehensive framework for implementing and experimenting with implicit models. It includes the foundational ImplicitModel, the recurrent variant ImplicitRNN, and the state-driven training approach SIM. Below are simple examples of how to use each model. More details of the package including the theory and functions can be found in the documentation.

Example: ImplicitModel

from torchidl import ImplicitModel

# Prepare your dataset
train_loader, test_loader = ...

# Initialize the model
model = ImplicitModel(
    hidden_dim=100,  # Hidden state dimension
    input_dim=3072,  # Input dimension
    output_dim=10,   # Output dimension
).to(device)

# Standard PyTorch training loop
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = torch.nn.CrossEntropyLoss()

for epoch in range(num_epochs): 
    for inputs, targets in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets) 
        loss.backward()  
        optimizer.step()

Example: SIM (State-driven Implicit Modeling)

State-driven Implicit Modeling (SIM) is a training method for Implicit Models that uses a pretrained network to synthesize training data. SIM formulates the training process of Implicit Models as a convex optimization problem so that it can be solved stably and efficiently.

Requirements:

• A pretrained network
• A convex optimization solver

For more details, see the documentation.

import torch
from torchidl import SIM
from torchidl import CVXSolver

# Prepare your dataset
train_loader, test_loader = ...

# Load a pretrained explicit model
explicit_model = MLP(input_dim, hidden_dim, output_dim).to(device)
explicit_model.load_state_dict(torch.load('checkpoint.pt'))

# Initialize SIM model and solver
sim = SIM(activation_fn=torch.nn.functional.relu, device=device)
solver = CVXSolver()

# Train and evaluate
sim.train(solver=solver, model=explicit_model, dataloader=train_loader)
sim.evaluate(test_loader)

The package provides multiple solvers including:

• CVXSolver: Convex optimization solver
• ADMMSolver: Standard ADMM implementation
• ADMMMultiGPUSolver: Distributed training across multiple GPUs
• ProjectedGDLowRankSolver: Projected gradient descent with low-rank constraints
• LeastSquareSolver: Efficient least squares solver using numpy.linalg.lstsq (ignores well-posedness constraint)

Besides, custom solvers can also be implemented by extending the BaseSolver class.

Documentation and Examples

More details of the package are provided in the documentation. For a comprehensive introduction, see our Notebook tutorial. The examples directory contains implementation examples for each model variant.

To run an example:

bash examples/idl/idl.sh

Related Works

Implicit Deep Learning
SIAM, 2020
Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, Alicia Y. Tsai

Constrained Implicit Learning Framework for Neural Network Sparsification
ACML, 2025
Alicia Y. Tsai, Wenzhi Gao, Laurent El Ghaoui

The Extrapolation Power of Implicit Models
IJCAI, 2024
Juliette Decugis, Alicia Y Tsai, Max Emerling, Ashwin Ganesh, Laurent El Ghaoui

State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks
arxiv, 2022
Alicia Y. Tsai, Juliette Decugis, Laurent El Ghaoui, Alper Atamtürk

Implicit Learning in Deep Models: Enhancing Extrapolation Power and Sparsity
Dissertation, 2024
Alicia Y. Tsai

Citation

@misc{torchidl,
    author = {Hoang Phan and Bao Tran and Chi Nguyen and Bao Truong and Thanh Tran and Khai Nguyen and Alicia Y. Tsai and Hong Chu and Laurent El Ghaoui},
    title = {Implicit Deep Learning Pytorch Package},
    year = {2025},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/HoangP8/Implicit-Deep-Learning}},
}