ISR 1.9.0

Image Super Resolution

Image Super-Resolution

The goal of this project is to upscale and improve the quality of low resolution images.

Includes the Keras implementations of:

• the super-scaling Residual Dense Network described in Residual Dense Network for Image Super-Resolution (Zhang et al. 2018);
• the super-scaling Residual in Residual Dense Network described in ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks(Wang et al. 2018)
• a multi-output version of the Keras VGG19 network for deep features extraction used in the perceptual loss;
• a custom discriminator network based on the one described in Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (SRGANS, Ledig et al. 2017)

Docker scripts are available to carry training and testing. Also, we provide scripts to facilitate training on the cloud with AWS and nvidia-docker with only a few commands.

We welcome any kind of contribution. If you wish to contribute, please see the Contribute section.

Contents

• Sample Results
• Getting Started
• Predict
• Train
• Installation
• Unit Testing
• Changelog
• Additional Information
• Contribute
• Maintainers
• License

Sample Results

The samples use an upscaling factor of two. The weights used to produced these images are available under sample_weights (see Additional Information).

The original low resolution image (left), the super scaled output of the network (center) and the result of the baseline scaling obtained with GIMP bicubic scaling (right).

Below a sample output of the RDN network re-trained on the weights provided in the repo to achieve artefact removal and detail enhancement. For the follow up training session we used a combination of artefact removing strategies and a few forms of a perceptual loss, using combinations of the deep features of the VGG19 and the GAN's discriminator network.

Bicubic up-scaling (baseline).

ISR standard.

ISR with artefact removal and VGG+GAN perceptual loss.

Getting Started

1. Install Docker

2. Build docker image for local usage docker build -t isr . -f Dockerfile.cpu

In order to train remotely on AWS EC2 with GPU

3. Install Docker Machine

4. Install AWS Command Line Interface

5. Set up an EC2 instance for training with GPU support. You can follow our nvidia-docker-keras project to get started

Predict

Place your images (png, jpg) under data/input/<data name>, the results will be saved under /data/output/<data name>/<model>/<training setting>.

NOTE: make sure that your images only have 3 layers (the png format allows for 4).

Check the configuration file config.yml for more information on parameters and default folders.

The --default flag in the run command will tell the program to load the weights specified in config.yml. It is possible though to iteratively select any option from the command line.

Predict locally

From the main folder run

docker run -v $(pwd)/data/:/home/isr/data -v $(pwd)/weights/:/home/isr/weights -it isr --predict --default --config config.yml

Predict on AWS with nvidia-docker

From the remote machine run (using our DockerHub image)

sudo nvidia-docker run -v $(pwd)/isr/data/:/home/isr/data -v $(pwd)/isr/weights/:/home/isr/weights -it idealo/image-super-resolution --predict --default --config config.yml

Train

Train either locally with (or without) Docker, or on the cloud with nvidia-docker and AWS.

Add you training set, including training and validation Low Res and High Res folders, under training_sets in config.yml.

Train on AWS with GPU support using nvidia-docker

To train with the default settings set in config.yml follow these steps:

1. From the main folder run bash scripts/setup.sh -m <name-of-ec2-instance> -b -i -u -d <data_name>.
2. ssh into the machine docker-machine ssh <name-of-ec2-instance>
3. Run training with sudo nvidia-docker run -v $(pwd)/isr/data/:/home/isr/data -v $(pwd)/isr/logs/:/home/isr/logs -v $(pwd)/isr/weights/:/home/isr/weights -it isr --training --default --config config.yml

<data_name> is the name of the folder containing your dataset. It must be under ./data/<data_name>.

Tensorboard

The log folder is mounted on the docker image. Open another EC2 terminal and run

tensorboard --logdir /home/ubuntu/isr/logs

and locally

docker-machine ssh <name-of-ec2-instance> -N -L 6006:localhost:6006

Notes

A few helpful details

• DO NOT include a Tensorflow version in requirements.txt as it would interfere with the version installed in the Tensorflow docker image
• DO NOT use Ubuntu Server 18.04 LTS AMI. Use the Ubuntu Server 16.04 LTS AMI instead

Train locally

Train locally with docker

From the main project folder run

docker run -v $(pwd)/data/:/home/isr/data -v $(pwd)/logs/:/home/isr/logs -v $(pwd)/weights/:/home/isr/weights -it isr --train --default --config config.yml

Installation

There are two ways to install the Image Super-Resolution package:

• Install ISR from PyPI (recommended):

pip install ISR

• Install ISR from the GitHub source:

git clone https://github.com/idealo/image-super-resolution
cd image-super-resolution
python setup.py install

Unit Testing

From the tests folder run

pytest -vs --cov=ISR --show-capture=no --disable-pytest-warnings

Changelog

• v2: added deep features from VGG19 and a discriminator for GAN training. Moved all non strictly architecture building operations outside of the model files. The models are combined when needed in the Trainer class. In order to allow for GAN training fit_generator function had to be replaced with the more granular train_on_batch. Now the project relies on custom data handlers and loggers instead of the custom Keras generator.

Additional Information

RDN Network architecture

The main parameters of the architecture structure are:

• D - number of Residual Dense Blocks (RDB)
• C - number of convolutional layers stacked inside a RDB
• G - number of feature maps of each convolutional layers inside the RDBs
• G0 - number of feature maps for convolutions outside of RDBs and of each RBD output

source: Residual Dense Network for Image Super-Resolution

RRDN Network architecture

The main parameters of the architecture structure are:

• T - number of Residual in Residual Dense Blocks (RRDB)
• D - number of Residual Dense Blocks (RDB) insider each RRDB
• C - number of convolutional layers stacked inside a RDB
• G - number of feature maps of each convolutional layers inside the RDBs
• G0 - number of feature maps for convolutions outside of RDBs and of each RBD output

source: ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

RDN Pre-trained weights

The weights of the RDN network trained on the DIV2K dataset are available in weights/sample_weights/rdn-C6-D20-G64-G064-x2_div2k-e086.hdf5.
The model was trained using C=6, D=20, G=64, G0=64 as parameters (see architecture for details) for 86 epochs of 1000 batches of 8 32x32 augmented patches taken from LR images.

The artefact removing and detail enhancing weights obtained with a combination of different training sessions using different datasets and perceptual loss with VGG19 and GAN can be found at weights/sample_weights/rdn-C6-D20-G64-G064-x2_enhanced-e219.hdf5

Contribute

We welcome all kinds of contributions, models trained on different datasets, new model architectures and/or hyperparameters combinations that improve the performance of the currently published model.

Will publish the performances of new models in this repository.

See CONTRIBUTING for more details.

Maintainers

• Francesco Cardinale, github: cfrancesco
• Zubin John, github: valiantone
• Dat Tran, github: datitran

Copyright

See LICENSE for details.