StructureFlow: Image Inpainting via Structure-aware Appearance Flow
With development of deep neural networks, reconstructing of hazy,distorted images or removing unwanted objects is a demanding application. Termed as Image Inpainting this was approached by diffusion based,patch based and deep-neural network framework based methods but failed to generate realistic structures of missing areas and gave disappointing results on non-repetitive patterns as it lacked in separating structure and texture of an image.
This Paper takes care of that by introducing two stages: 1.Structure reconstruction & 2. Texture generation. this two stages first recovers the missing structures and generate the reconstructed fine tuned data in last stage. Avoiding high frequency textures first edge images are used for structural guidance. This two stage network flow use Edge-preserved smooth methods to remove high-frequency textures while retaining sharp edges and low-frequency structures.
.png)
## The Proposed Method ##
- Edge preserved smooth images are used to reconstruct missing sections.the use of appearance flow ensures sample features from regions with similar structures.
Then texture generator synthesizes high frequency details. And
Gaussian sampling is used instead of bilinear sampling to expand the receptive field of sampling operation.The model consists of two parts:
the structure reconstructor
and the Texture generator
-
Structure Reconstructor - Let Igt be the ground-truth image and Sgt be the edge-preserved smooth result of Igt .Now processing of the structure reconstructor will be
where M is the mask of the input image Iin. , a binarized matrix where 1 represents the missing region and 0 represents the background.
The reconstruction loss of
distance between the predicted structures
and the groundtruth structures
.
To achieve
where
is discriminator.Now the generator & discriminator is trained & optimized as
where
are regularized parameters.
-
Texture Generator - The texture generator takes
.
The adversial loss in texture generator will be
now to establish long term dependency & a proper relation between different reegions the apearence Flow is introduced.Gaussian sampling is proposed to expand the receptive field.with a kernel size n this can be written as
-
Sampling Correctness Loss - This sampling correctnes loss playes a important key as it constraints the appearence flow .
and
be features generated by VGG19 layer.The sampling correctness loss calculate the relative cosine similarity between the ground truth features and the sampled features as:
is the sampled feature calculated byGaussian sampling and
is a normalization term.
-
-
Experimental Results -
- Several resudial blocks with autoencoders were used to process the features.This model was tested Places2, CelebA,and Paris StreetView datasets. the structure reconstructor
. and the texture generator
. are trained separately using the edge-preserved image
.
.png)
- This model was compared with CA,Edge Connect and it outperformed both of them.the edge-preserved smooth images had the structures of images retrived well so,the model well balanced structure reconstruction and texture generation. Photo-realistic results are obtained hence. As these remove high frequency textures the proper tradeoff between structure reconstructor and the texture generator was acheived .
.png)
- Flow Ablation Study revealed Gaussian sampling & sampling correction loss effected real good as Bilinear sampling failed to capture long term correlations.
.png)
Conclusion -
Structure preservation and texture generator stages handled the inpaintaing challenge well.introducing appearance flow to to sample features from relative regions yield realistic image details,which improved the outcome in a geat manner. Experiments on multiple datasets shows the superior performance of the proposed network.
For code, visit this link.