rrelu 0.1.0

generate reliable relu activation function for DNNs

Reliable ReLU Toolbox (RReLU) To Enhance Resilience of DNNs

Background • Usage • Code • Citation •

Background

The Reliable ReLU Toolbox (RReLU) is a powerful reliability tool designed to enhance the resiliency of deep neural networks (DNNs) by generating reliable ReLU activation functions. It is Implemented for the popular PyTorch deep learning platform. RReLU allows users to find a clipped ReLU activation function using various methods. This tool is highly versatile for dependability and reliability research, with applications ranging from resiliency analysis of classification networks to training resilient models and improving DNN interpretability.

RReLU includes all state-of-the-art activation restriction methods. These methods offer several advantages: they do not require retraining the entire model, avoid the complexity of fault-aware training, and are non-intrusive, meaning they do not necessitate any changes to an accelerator. RReLU serves as the research code accompanying the paper (ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs), and it includes implementations of the following algorithms:

• ProAct (the proposed algorithm) (code).
• FitAct (paper and code).
• FtClipAct (paper and code).
• Ranger (paper and code).

Usage

you can download the rrelu on PyPI here.

Installing

From Pip

Install using pip install rrelu

From Source Download this repository into your project folder.

Importing

Import the entire package:

import rrelu

In order to use multiple gpu :

pip install torchpack
from torchpack import distributed as dist
dist.init()
torch.backends.cudnn.benchmark = True
torch.cuda.set_device(dist.local_rank())

create a model :

from rrelu.setup import build_model
model = build_model("model_name":string, n_classes:inetger,dropout_rate:float).cuda()

create a data loader :

from rrelu.setup import build_data_loader
data_loader_dict, n_classes = build_data_loader(
        dataset:string,
        image_size,
        batch_size,
        n_worker,
        data_path,
        dist.size(), # for multiple gpu 
        dist.rank(), # for multiple gpu
    )

load pretrained model :

checkpoint = load_state_dict_from_file(args.init_from)
model.load_state_dict(checkpoint) 

change the representatoin to fixed-point :

for name, param in model.named_parameters():
    if param!=None:
      param.copy_(torch.tensor(Fxp(param.clone().cpu().numpy(), True, n_word=args.n_word,n_frac=args.n_frac,n_int=args.n_int).get_val()))

replace activatoin function with clipped version :

from rrelu.setup import replace_act
model = replace_act(model,args.name_relu_bound,args.name_serach_bound,data_loader_dict,args.bounds_type,args.bitflip)

evaluate the clipped model on various fault rates :

from rrelu.setup import eval_fault
for fault_rate in args.fault_rates: # fault_rates = [10^-7,3 * 10^-7 ,10^-6 , 3 * 10^-6 , 10^-5 , 3 * 10^-5]
            val_results_fault = eval_fault(model,data_loader_dict,fault_rate,args.iterations,args.bitflip,args.n_word , args.n_frac, args.n_int)

Import a specific module:

from rrelu.search_bound import proact_bounds 

run search in command line

When Download this repository into your project folder.

torchpack dist-run -np 1 python search.py --dataset "dataset name (CIFAR10, CIFAR100)" --data_path "path to the dataset" --model "name of the model" --init_from "pretrained file path" \
                      --name_relu_bound "name of bounded relu" --name_serach_bound "name of the search algorithm" --bounds_type "type of thresholds" --bitflip "value representaiton"

Code

Structure

The main source code of framework is held in src/rrelu, which carries search_bounds, relu_bounds , extended pytorchfi and other implementations.

Citation

View the published paper. If you use or reference rrelu, please cite: