Create mosaics with overlapping tiles
Project description
MightyMosaic
Introduction
The goal of the library is to create a "mosaic", which is a sublass of np.ndarray, dividing images of arbitrary shape into a 4d or 5d array. The class is implemented so that we can apply functions on the tiles (usually the prediction function of a neural network, which has a fixed input shape) and fuse the result.
The MightyMosaic allow the overlapping of tiles, which is necessary to avoid discrepancies between adjacent tiles.
Requirements
As we use fstring (in the asserts), it is necessary to have, at least, python3.6.
 numpy
 tqdm
For running this jupyter notebook (which should have been converted to markdown at some point), we also need the following library:
 matplotlib
 pillow
 keras
How to use
Let's begin by importing the library that we need and define the filename for both the images and the model that we will use.
Btw, credits for itout (pixiv69581) and Akae Neo (pixiv17014316) for drawing the pictures that we used as examples.
import os import matplotlib.pyplot as plt import numpy as np import PIL.Image from keras.models import load_model from MightyMosaic import MightyMosaic
face_filename = "3578641 0.png" full_filename = "__original_drawn_by_itou_onsoku_tassha__samplef9c6557ccec993c08627d33e49cf4524.jpg" model_filename = "open_eyes_saliency_256.h5"
model = load_model(model_filename) print(f"The model input shape is {model.input_shape}")
The model input shape is (None, 256, 256, 3)
Open the images
The opening of the images can be done directly using PIL and numpy.
First, we show that our model can process a picture of shape (256, 256, 3) (more accurately, a batch of shape (?, 256, 256, 3)).
face_im = np.array(PIL.Image.open(face_filename))/255 full_im = np.array(PIL.Image.open(full_filename))/255 prediction = model.predict(np.expand_dims(face_im, axis=0))[0,:,:,0]
plt.figure(figsize=(16, 8)) plt.subplot(121) plt.imshow(face_im, interpolation='bilinear', vmin=0, vmax=1) plt.subplot(122) plt.imshow(prediction, interpolation='bilinear', vmin=0, vmax=1) plt.show()
Nice, right? The model return a value of 0 on closed eyes, and 1 on opened eyes.
Create a basic mosaic
However, if we want to run the model on picture of different shape, we need either to resize the picture, which is, obviously, nonideal, are to divide it in patch and run the network on each of them.
We will start by creating the mosaic, applying the model can be done later.
It is done by calling the method MightyMosaic.from_array. The basic usage is to provide only two parameters : the image that we want to transform, and the size of each tiles (here (256,256)). Please note that it is not mandatory to have square tiles, it's just how our network was constructed.
After constructing the mosaic, we will try to retrieve the original image. This is done with the get_fusion method.
mosaic_without_overlap = MightyMosaic.from_array(full_im, (256,256), overlap_factor=1) print(f'The mosaic shape is {mosaic_without_overlap.shape}') plt.figure(figsize=(16, 16)) plt.subplot(121) plt.title(f'Image with shape {full_im.shape}') plt.imshow(full_im, interpolation='bilinear', vmin=0, vmax=1) plt.subplot(122) plt.title('Mosaic reconstruction') plt.imshow(mosaic_without_overlap.get_fusion(), interpolation='bilinear', vmin=0, vmax=1) plt.show()
The mosaic shape is (4, 4, 256, 256, 3)
If the two images are different, that's mean that I commit something, broke another, and didn't check the results. Hopefully, that didn't happen.
Let's apply the predict method and check the result.
Please ensure that, if the predict take an input of shape (?, x, y, z), it returns an array of shape (?, x', y', z') or (?, x', y') (so, no additional dimensions). There is another condition on x', y' that we will see later.
We can have a progress_bar to see the progress of the prediction.
fused_prediction_without_overlap = mosaic_without_overlap.apply(model.predict, progress_bar=True) print(f'The prediction shape is : {fused_prediction_without_overlap.shape}') fused_prediction_without_overlap = fused_prediction_without_overlap.get_fusion() plt.figure(figsize=(16, 16)) plt.subplot(121) plt.title(f'Image with shape {full_im.shape}') plt.imshow(full_im, interpolation='bilinear', vmin=0, vmax=1) plt.subplot(122) plt.title('Prediction on the mosaic') plt.imshow(fused_prediction_without_overlap[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.show()
HBox(children=(IntProgress(value=0, max=16), HTML(value='')))
The prediction shape is : (4, 4, 256, 256, 1)
Meh. Not convinced about the results? Maybe it's because of the boundary of each tile, clearly visible in the prediction. And since the left eye is between four tiles, the prediction can't be accurate.
Create an overlapping mosaic
In fact, what we want is an overlapping mosaic (meaning that the tiles overlap). And that's what MightyMosaic is for.
Let's create a mosaic with an overlapping_factor of 2, meaning that the stride, between each tile is only tile_shape/2. Of course, an overlapping_factor of one means no overlapping, and you can increase the factor to any positive integer. However, please note that the number of tile increase with tile_shape^2, so might want to refrain to use a high overlapping_factor (it would be quite useless anyway).
Also, do not use a overlap_factor that can't divide the tile_shape (that also means that you want to avoid odd shapes). We previously talk about a condition on x' and y', it is the same as on x, y : they must be multiple of the correspondant overlap_factor.
mosaic = MightyMosaic.from_array(full_im, (256,256), overlap_factor=2) print(f'The mosaic shape is {mosaic.shape}')
The mosaic shape is (8, 8, 256, 256, 3)
The number of tiles is not of 16 anymore, but raises to 64.
Our prediction support batchs, so we will use it to speed up the processing a little: we add a parameter batch_size with 8 as a value.
prediction = mosaic.apply(model.predict, progress_bar=True, batch_size=8) prediction = prediction.get_fusion() plt.figure(figsize=(16, 16)) plt.subplot(121) plt.title(f'Image with shape {full_im.shape}') plt.imshow(full_im, interpolation='bilinear', vmin=0, vmax=1) plt.subplot(122) plt.title('Prediction on the mosaic') plt.imshow(prediction[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.show()
HBox(children=(IntProgress(value=0, max=8), HTML(value='')))
The prediction is quite better, without visible borders between tiles.
About the fill_mode parameter
Copying the keras fill_mode parameter, we enable to change the way the mosaic is filled. The possible values are constant, nearest and reflect. Please refer to the documentation of keras for more details.
Here, we create a mosaic with "reflect" as the fill_mode.
mosaic_with_reflection = MightyMosaic.from_array(full_im, (256,256), overlap_factor=4, fill_mode='reflect') print(f'The mosaic shape is {mosaic_with_reflection.shape}')
The mosaic shape is (16, 16, 256, 256, 3)
OK, now, we are ready to plot, side by side, the prediction for:
 the mosaic without overlap (I);
 the mosaic with an overlapping_factor of 2 (II);
 the mosaic with an overlapping_factor of 4 and a fill_mode at "nearest" (III).
We also plot abs(III) and abs(IIIII) to highlight the effects of the parameters.
prediction_with_reflection = mosaic_with_reflection.apply(model.predict, progress_bar=True, batch_size=8) prediction_with_reflection = prediction_with_reflection.get_fusion() plt.figure(figsize=(18, 16)) plt.subplot(234) plt.title('Input') plt.imshow(full_im, interpolation='bilinear', vmin=0, vmax=1) plt.subplot(231) plt.title(f'(I) shape={fused_prediction_without_overlap.shape}') plt.imshow(fused_prediction_without_overlap[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.subplot(232) plt.title(f'(II) shape={prediction.shape}') plt.imshow(prediction[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.subplot(233) plt.title(f'(III) shape={prediction_with_reflection.shape}') plt.imshow(prediction_with_reflection[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.subplot(235) plt.title('abs(III)') plt.imshow(abs(fused_prediction_without_overlapprediction)[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.subplot(236) plt.title('abs(IIIII)') plt.imshow(abs(predictionprediction_with_reflection)[:,:,0], interpolation='bilinear', vmin=0, vmax=1) plt.show()
HBox(children=(IntProgress(value=0, max=32), HTML(value='')))
That's it.
P.S. It's completely possible to use a categorizer for the prediction. Of course, since the original goal of the mosaic is to use segmentation, we assert that the output of the network should be of size (?, h, w, c) with:
 ? the number of batch ;
 h the height of the output ;
 w the width of the output ;
 c the number of canals.
A categorizer would have a output shape of (?, n) with n the number of classes, but you can tweak the lambda to get the correct dimensions.
mosaic = from_array(im, (tile_size,tile_size), overlap_factor=f) prediction = mosaic.apply(lambda x: np.tile(model.predict(x), f*f).reshape(1, f, f, n))
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size  File type  Python version  Upload date  Hashes 

Filename, size MightyMosaic1.2.3py3noneany.whl (20.6 kB)  File type Wheel  Python version py3  Upload date  Hashes View 
Hashes for MightyMosaic1.2.3py3noneany.whl
Algorithm  Hash digest  

SHA256  8f019514f607b287e58bbe2834910a0bdb49bb4b1bad583df4f5f6bd006c7e7f 

MD5  3be9c818da90378c9fd92e139fa8c03b 

BLAKE2256  0f0a5c247e6625bca3f6cf5a9edde71f6a984567226dd297b34f55f8717ef7eb 