GAN Inversion: A brief walkthrough — Part IV

  • Part I — Introduction to GAN Inversion and its approaches.
  • Part II, III & IV — GAN Inversion extensions, applications, and future work.

1. Applications of GAN inversion

1.1 Image interpolation

Eq. 1: Image interpolation

1.2 Image manipulation

Eq. 2: Image manipulation

1.3 Semantic diffusion

Fig. 1: Semantic diffusion results by [4]
Fig. 2: Semantic diffusion results by [2]

1.4 GAN dissection

  • GAN dissection: This suggests that for each object class c, the aim is to identify whether the uᵗʰ unit of feature map r_u,P extracted at a particular layer encodes some internal representation of the class at location P. This is done by upsampling the feature map to the image size and thresholding to mask out the non-relevant pixels. It then performs segmentation for class c on the image generated by the generator and performs an IOU check to measure the agreement between the segmented image and the thresholded feature maps.
Fig. 3: GAN dissection by [6]
  • GAN intervention: It is important to note that all units highly correlated to an object class are not necessarily responsible for rendering the object into the image. Hence, it is essential to find the set of units that cause an object to occur in the image. They decompose feature maps into unforced and causal (causing object rendering) units and force insertion and ablation of these units to identify their effect. They also incorporate a learnable continuous per-channel factor that implies the effect each unit of feature map has on the rendering.
Fig. 4: GAN intervention by [6]
Fig. 5: Layer based comparison by [6]

1.5 Interactive generation

1.6 Seeing what a GAN does not learn

Fig. 6: Image reconstruction using Progressive GAN church model
Fig. 7: Image segmentation statistics on StyleGAN, WGAN-GP, and Progressive GAN.
Eq. 3: Decomposing the generator
Eq. 4: Optimization objective by [7]
Fig. 8: Layer inversion by [7]
Fig. 9: Layer-wise inversion results for model trained on LSUN church

2. Further improvements

Evaluation metrics

Precise Control

Domain generalization

3. Summary

References

--

--

--

Leading big data and AI-powered solution company https://www.sertiscorp.com/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Using Stocktwits to Create an Investor Sentiment Metric

Supervised and Unsupervised Machine Learning Algorithms

Tutorial: OCR with PaddleOCR (PP-OCR)

Visualizing Layer Representations in Neural Networks

Announcing fast.ai part 1 now available as Kaggle Kernels

5 Things I’ve Learned as an Open Source Machine Learning Framework Creator

Artificial Neural Network using R Studio

Using Elyra to create Machine Learning pipelines on Kubeflow

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sertis

Sertis

Leading big data and AI-powered solution company https://www.sertiscorp.com/

More from Medium

Sharpness-Aware Minimization for Efficiently Improving Generalization

Review — Radiance Fields without Neural Networks

Paper Review: ConvNext or Convnets for 2020s

Inverse Problems with Unrolled Adversarial Regularization