Masked GANs for Face Completion: A Novel Deep Learning Approach

Anshuman Sharma; Biswaroop Nath; Tejaswini Kar; D Khasim

doi:10.4108/eetpht.9.4850

Authors

Anshuman Sharma KIIT University
Biswaroop Nath KIIT University
Tejaswini Kar KIIT University
D Khasim Vellore Institute of Technology University

DOI:

https://doi.org/10.4108/eetpht.9.4850

Keywords:

Generative Adversarial Networks, Masked data, face images

Abstract

INTRODUCTION: Recent deep learning based image editing methods have achieved promising results for removing object in an image but fail to generate appreciable performance for removing large objects of complex nature, especially mask from facial images. Towards this goal the objective of this work is to remove mask objects in facial images. In this study, authors propose a novel approach for face completion using Generative Adversarial Networks (GANs) that utilize masked data. This technology can help in image restoration and preservation, thus enabling us to cherish those memories that are held dear to our hearts.

OBJECTIVES: Train a GAN to learn the mapping from incomplete to complete face images by utilizing a masked input image.

METHODS: The discriminator is trained to distinguish between face images and full ground truth images. Our results indicate that our technique generates high-quality, realistic facial images that are visually comparable to the ground truth and that it can generalise to new faces that were not encountered during training.

RESULTS: Our findings indicate that GANs with masked inputs are a good approach for generating whole face images from partial or masked data.

CONCLUSION: Our experimental findings show that our method produces facial images of great quality and realism that are visually equivalent to the actual thing. Our proposed approach can also be applied to fresh faces that weren’t seen. The performance can still be improved further using larger dataset. Also, further investigation into adversial attacks may help in improving performance. This technology can be further utilized for developing realtime mask removal software as well.

Downloads

Download data is not yet available.

References

M. Ciotti, M. Ciccozzi, A. Terrinoni, W.-C. Jiang, C.-B. Wang, S. Bernardini,The covid-19 pandemic, Critical Reviews in Clinical Laboratory Sciences 57 (6)(2020)365–388,pMID:32645276.arXiv:https://doi.org/10.1080/10408363.2020.1783198, doi:10.1080/10408363.2020.1783198. DOI: https://doi.org/10.1080/10408363.2020.1783198

R. Salakhutdinov, Learning deep generative models, Annual Review of Statistics and ItsApplication 2 (1) (2015)361–385 arXiv:https://doi.org/10.1146/annurev-statistics-010814-020120, doi:10.1146/annurev-statistics-010814-020120 DOI: https://doi.org/10.1146/annurev-statistics-010814-020120

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks (2014). arXiv:1406.2661.

N. Ud Din, K. Javed, S. Bae, J. Yi, A novel gan-based network for unmasking of masked face, IEEE Access 8 (2020) 44276–44287. doi:10.1109/ACCESS.2020.2977386. DOI: https://doi.org/10.1109/ACCESS.2020.2977386

M. Mirza, S. Osindero, Conditional generative adversarial nets, arXiv preprint arXiv:1411.1784 (2014).

L. Yu, W. Zhang, J. Wang, Y. Yu, Seqgan: Sequence generative adversarial nets with policy gradient 31 (Feb. 2017).doi:10.1609/aaai.v31i1.10804.URL:https://ojs.aaai.org/index.php/AAAI/article/view/10804 DOI: https://doi.org/10.1609/aaai.v31i1.10804

T. D. Nguyen, T. Le, H. Vu, D. Phung, Dual discriminator generative adversarial nets, in: Advances in Neural Information Processing Systems, 2017, pp. 2670–2680.

Q. Hoang, T. Nguyen, T. Le, D. Phung, Multi-generator gernerative adversarial nets (CoRR ,08-2017).

Y.-M. Cheung, M. Li, R. Zou, Facial structure guided gan for identity-preserved face image de-occlusion, in: Proceedings of the 2021 International Conference on Multimedia Retrieval, ICMR ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 46–54.doi:10.1145/3460426.3463642.URL:https://doi.org/10.1145/3460426.3463642 DOI: https://doi.org/10.1145/3460426.3463642

A. Kammoun, R. Slama, H. Tabia, T. Ouni, M. Abid, Generative adversarial networks for face generation: A survey,ACMComput.Surv.55(5)(dec2022).doi:10.1145/3527850.URL:https://doi.org/10.1145/3527850 DOI: https://doi.org/10.1145/3527850

J. Huang, K. Cui, D. Guan, A. Xiao, F. Zhan, S. Lu, S. Liao, E. Xing, Masked generative adversarial networks are data-efficient generation learners, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, Vol. 35, Curran Associates, Inc., 2022, pp. 2154–2167. URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/0efcb1885b8534109f95ca82a5319d25-Paper-Conference.pdf

T. Karras, M. Aittala, S. Laine, E. H¨ark¨onen, J. Hellsten, J. Lehtinen, T. Aila, Alias-free generative adversarial networks, in: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Vol. 34, Curran Associates,Inc.,2021,pp.852–863.URL: https://proceedings.neurips.cc/paper_files/paper/2021/ file/076ccd93ad68be51f23707988e934906-Paper.pdf

D. Volkhonskiy, I. Nazarov, E. Burnaev, Steganographic generative adversarial networks, in: W. Osten, D. P. Nikolaev (Eds.), Twelfth International Conference on Machine Vision (ICMV 2019), Vol. 11433, International Society for Optics and Photonics, SPIE, 2020, p. 114333M. doi:10.1117/12.2559429.URL:https://doi.org/10.1117/12.2559429 DOI: https://doi.org/10.1117/12.2559429

L. Hou, Z. Yuan, L. Huang, H. Shen, X. Cheng, C. Wang, Slimmable generative adversarial networks, Proceedings of the AAAI Conference on Artificial Intelligence 35 (9) (2021) 7746–7753. doi:10.1609/aaai.v35i9.16946. URL https://ojs.aaai.org/index.php/AAAI/article/view/16946 DOI: https://doi.org/10.1609/aaai.v35i9.16946

E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, T. Karras, G. Wetzstein, Efficient geometry-aware 3d generative adversarial networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 16123–16133. DOI: https://doi.org/10.1109/CVPR52688.2022.01565