Colorizing Multi-Modal Medical Data: An Autoencoder-based Approach for Enhanced Anatomical Information in X-ray Images

Bunny Saini; Divya Venkatesh; Avinaash Ganesh; Amar Parameswaran; Shruti Patil; Pooja Kamat; Tanupriya Choudhury

doi:10.4108/eetpht.10.5540

Authors

Bunny Saini Symbiosis International University
Divya Venkatesh Symbiosis International University
Avinaash Ganesh Symbiosis International University
Amar Parameswaran Symbiosis International University
Shruti Patil Symbiosis International University
Pooja Kamat Symbiosis International University
Tanupriya Choudhury Graphic Era University

DOI:

https://doi.org/10.4108/eetpht.10.5540

Keywords:

Gray Image Colorization, Unsupervised Learning, Auto Encoders, Transfer Learning, Medical Imaging

Abstract

Colourisation is the process of synthesising colours in black and white images without altering the image’s structural content and semantics. The authors explore the concept of colourisation, aiming to colourise the multi-modal medical data through X-rays. Colourized X-ray images have a better potential to portray anatomical information than their conventional monochromatic counterparts. These images contain precious anatomical information that, when colourised, will become very valuable and potentially display more information for clinical diagnosis. This will help improve understanding of these X-rays and significantly contribute to the arena of medical image analysis. The authors have implemented three models, a basic auto-encoder architecture, and two combined learnings of the autoencoder module with transfer learning of pre-trained neural networks. The unique feature of this proposed framework is that it can colourise any medical modality in the medical imaging domain. The framework’s performance is evaluated on a chest x-ray image dataset, and it has produced benchmark results enabling high-quality colourisation. The biggest challenge is the need for a correct solution for the mapping between intensity and colour. This makes human interaction and external information from medical professionals crucial for interpreting the results.

Downloads

Download data is not yet available.

References

B, S., & B, R. (2022, November 9). International Journal of Recent Technology and Engineering (IJRTE). Retrieved November 15, 2022, from https://www.ijrte.org/

Farella, E. M., Malek, S., & Remondino, F. (2022). Colorizing the Past: Deep Learning for the Automatic Colorization of Historical Aerial Images. Journal of Imaging, 8(10), 269..

Garcea, F., Serra, A., Lamberti, F., & Morra, L. (2022). Data augmentation for medical imaging: A systematic literature review. Computers in Biology and Medicine, 106391. DOI: https://doi.org/10.1016/j.compbiomed.2022.106391

Khan, M. U. G., Gotoh, Y., & Nida, N. (2017, June 22). Medical image colorization for better visualization and segmentation. White Rose Research Online. Retrieved November 15, 2022, from https://eprints.whiterose.ac.uk/119356/

Tariq, A., Gill, A. Y., & Hussain, H. K. (2023). Evaluating the Potential of Artificial Intelligence in Orthopedic Surgery for Value-based Healthcare. International Journal of Multidisciplinary Sciences and Arts, 2(1), 27-35. DOI: https://doi.org/10.47709/ijmdsa.v2i1.2394

Liang, Y., Lee , D., Li1 , Y., & Shin, B.-S. (2021, January 18). Unpaired medical image colorization using generative ... - springer. Unpaired medical image colorization using generative adversarial network. Retrieved November 15, 2022, from https://link.springer.com/content/pdf/10.1007/s11042-020-10468-6.pdf

R. Ribani and M. Marengoni, "A Survey of Transfer Learning for Convolutional Neural Networks," 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), 2019, pp. 47-57, doi: 10.1109/SIBGRAPI-T.2019.00010. DOI: https://doi.org/10.1109/SIBGRAPI-T.2019.00010

D’Souza, G., Reddy, N. S., & Manjunath, K. N. (2023). Localization of lung abnormalities on chest X-rays using self-supervised equivariant attention. Biomedical Engineering Letters, 13(1), 21-30. DOI: https://doi.org/10.1007/s13534-022-00249-5

Mohan, R., Elsken, T., Zela, A., Metzen, J. H., Staffler, B., Brox, T., ... & Hutter, F. (2023). Neural Architecture Search for Dense Prediction Tasks in Computer Vision. International Journal of Computer Vision, 1-24. DOI: https://doi.org/10.1007/s11263-023-01785-y

Morra, L., Piano, L., Lamberti, F., & Tommasi, T. (2020, October 19). Bridging the gap between natural and medical images through ... - arxiv. Bridging the gap between Natural and Medical Images through Deep Colorization. Retrieved November 15, 2022, from https://arxiv.org/pdf/2005.10589.pdf DOI: https://doi.org/10.1109/ICPR48806.2021.9412444

Huang, S., Jin, X., Jiang, Q., & Liu, L. (2022). Deep learning for image colorization: Current and future prospects. Engineering Applications of Artificial Intelligence, 114, 105006.

Abbadi, N. K. E., & Razaq, E. S. (2020). Automatic gray images colorization based on lab color space. Indonesian Journal of Electrical Engineering and Computer Science, 18(3), 1501-1509. DOI: https://doi.org/10.11591/ijeecs.v18.i3.pp1501-1509

Wu, M., Jin, X., Jiang, Q., Lee, S. J., Liang, W., Lin, G., & Yao, S. (2021). Remote sensing image colorization using symmetrical multi-scale DCGAN in YUV color space. The Visual Computer, 37(7), 1707-1729. DOI: https://doi.org/10.1007/s00371-020-01933-2

Li, B., Lu, Y., Pang, W., & Xu, H. (2023). Image Colorization using CycleGAN with semantic and spatial rationality. Multimedia Tools and Applications, 1-15. DOI: https://doi.org/10.1007/s11042-023-14675-9

Cevallos, S., Pérez, N., Riofrío, D., Benítez, D., Moyano, R. F., & Baldeon-Calisto, M. (2022, July). A Deep Convolutional Autoencoder Architecture for Automatic Image Colorization. In 2022 IEEE Colombian Conference on Applications of Computational Intelligence (ColCACI) (pp. 1-6). IEEE. DOI: https://doi.org/10.1109/ColCACI56938.2022.9905295

Dong, X., Liu, C., Li, W., Hu, X., Wang, X., & Wang, Y. (2021). Self-supervised colorization towards monochrome-color camera systems using cycle CNN. IEEE Transactions on Image Processing, 30, 6609-6622. DOI: https://doi.org/10.1109/TIP.2021.3096385

Kim, S., Jang, Y., & Kim, S. E. (2021). Image-Based TF Colorization With CNN for Direct Volume Rendering. IEEE Access, 9, 124281-124294. DOI: https://doi.org/10.1109/ACCESS.2021.3100429

An, J., Kpeyiton, K. G., & Shi, Q. (2020). Grayscale images colorization with convolutional neural networks. Soft Computing, 24(7), 4751-4758. DOI: https://doi.org/10.1007/s00500-020-04711-3

Joshi, M. R., Nkenyereye, L., Joshi, G. P., Islam, S. R., Abdullah-Al-Wadud, M., & Shrestha, S. (2020). Auto-colorization of historical images using deep convolutional neural networks. Mathematics, 8(12), 2258 DOI: https://doi.org/10.3390/math8122258

Mouzon, T., Pierre, F., & Berger, M. O. (2019, June). Joint cnn and variational model for fully-automatic image colorization. In International Conference on Scale Space and Variational Methods in Computer Vision (pp. 535-546). Springer, Cham. DOI: https://doi.org/10.1007/978-3-030-22368-7_42

Nguyen, T., Mori, K., & Thawonmas, R. (2016). Image colorization using a deep convolutional neural network. arXiv preprint arXiv:1604.07904.

Hensman, P., & Aizawa, K. (2017, November). cGAN-based manga colorization using a single training image. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) (Vol. 3, pp. 72-77). IEEE. DOI: https://doi.org/10.1109/ICDAR.2017.295

Mourchid, Y., Donias, M., & Berthoumieu, Y. (2021, January). Automatic Image Colorization based on Multi-Discriminators Generative Adversarial Networks. In 2020 28th European Signal Processing Conference (EUSIPCO) (pp. 1532-1536). IEEE. DOI: https://doi.org/10.23919/Eusipco47968.2020.9287792

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29.

Jin, X., Li, Z., Liu, K., Zou, D., Li, X., Zhu, X., ... & Liu, Q. (2021, October). Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 1176-1184). DOI: https://doi.org/10.1145/3474085.3481544

Farella, E. M., Malek, S., & Remondino, F. (2022). Colorizing the Past: Deep Learning for the Automatic Colorization of Historical Aerial Images. Journal of Imaging, 8(10), 269. DOI: https://doi.org/10.3390/jimaging8100269

Dias, M., Monteiro, J., Estima, J., Silva, J., & Martins, B. (2020). Semantic segmentation and colorization of grayscale aerial imagery with W‐Net models. Expert systems, 37(6), e12622. DOI: https://doi.org/10.1111/exsy.12622

Chen, S. Y., Zhang, J. Q., Zhao, Y. Y., Rosin, P. L., Lai, Y. K., & Gao, L. (2022). A review of image and video colorization: From analogies to deep learning. Visual Informatics. DOI: https://doi.org/10.1016/j.visinf.2022.05.003

Huang, S., Jin, X., Jiang, Q., & Liu, L. (2022). Deep learning for image colorization: Current and future prospects. Engineering Applications of Artificial Intelligence, 114, 105006. DOI: https://doi.org/10.1016/j.engappai.2022.105006

Žeger, I., Grgic, S., Vuković, J., & Šišul, G. (2021). Grayscale image colorization methods: Overview and evaluation. IEEE Access. DOI: https://doi.org/10.1109/ELMAR49956.2020.9219019

Nida, N., Sharif, M., Khan, M. U. G., Yasmin, M., & Fernandes, S. L. (2016). A framework for automatic colorization of medical imaging. IIOAB J, 7, 202-209.

Mathur, A. N., Khattar, A., & Sharma, O. (2021). 2D to 3D Medical Image Colorization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2847-2856). DOI: https://doi.org/10.1109/WACV48630.2021.00289

Selvapriya B., Raghu B. (2019) Colorization using Desired Color for Medical Images. International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-7 Issue-6S3

Mooney, P. (2018, March 24). Chest X-ray images (pneumonia). Kaggle. Retrieved September 17, 2022, from https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia

Liang, C. M., Li, Y. W., Liu, Y. H., Wen, P. F., & Yang, H. (2022). Segmentation and weight prediction of grape ear based on SFNet-ResNet18. Systems Science & Control Engineering, 10(1), 722-732. DOI: https://doi.org/10.1080/21642583.2022.2110541

Tschannen, M., Bachem, O., & Lucic, M. (2018, December 12). Recent advances in autoencoder-based representation learning. arXiv.org. Retrieved November 17, 2022, from https://arxiv.org/abs/1812.05069

Bank, D., Koenigstein, N., & Giryes, R. (2021, April 3). Autoencoders. arXiv.org. Retrieved November 17, 2022, from https://arxiv.org/abs/2003.05991

Kamat, P., Sugandhi, R., & Kumar, S. (2021). Data-driven bearing fault detection using hybrid autoencoder-LSTM deep learning approach. International Journal of Modelling, Identification and Control, 38(1), 88-103. DOI: https://doi.org/10.1504/IJMIC.2021.122471

Jordan, J. (2018, March 19). Chapter 14: Autoencoders. deeplearningbook-notes. Retrieved November 17, 2022, from https://ucla-labx.github.io/deeplearningbook-notes/Ch14-Autoencoders.html

Autoencoders in Deep learning: Tutorial & use cases [2022]. V7. (2022, October 21). Retrieved November 17, 2022, from https://www.v7labs.com/blog/autoencoders-guide

Ramzan, F., Khan, M. U. G., Rehmat, A., Iqbal, S., Saba, T., Rehman, A., & Mehmood, Z. (2019, December 18). A deep learning approach for automated diagnosis and Multi-class classification of alzheimer's disease stages using resting-state fmri and residual neural networks - Journal of Medical Systems. SpringerLink. Retrieved November 18, 2022, from https://link.springer.com/article/10.1007/s10916-019-1475-2 DOI: https://doi.org/10.1007/s10916-019-1475-2

Sanakkayala, D. C., Varadarajan, V., Kumar, N., Karan, Soni, G., Kamat, P., ... & Kotecha, K. (2022). Explainable AI for bearing fault prognosis using deep learning techniques. Micromachines, 13(9), 1471. DOI: https://doi.org/10.3390/mi13091471

Jin, X., Di, Y., Jiang, Q., Chu, X., Duan, Q., Yao, S., & Zhou, W. (2023). Image colorization using deep convolutional auto-encoder with multi-skip connections. Soft Computing, 27(6), 3037-3052. DOI: https://doi.org/10.1007/s00500-022-07483-0

Hu, M., Bai, L., Fan, J., Zhao, S., & Chen, E. (2023). Vehicle color recognition based on smooth modulation neural network with multi-scale feature fusion. Frontiers of Computer Science, 17(3), 173321. DOI: https://doi.org/10.1007/s11704-022-1389-x

Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2016). Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG), 35(4), 1-11. DOI: https://doi.org/10.1145/2897824.2925974

Mouzai, M., Mustapha, A., Bousmina, Z., Keskas, I., & Farhi, F. (2023). Xray-Net: Self-supervised pixel stretching approach to improve low-contrast medical imaging. Computers and Electrical Engineering, 110, 108859.. DOI: https://doi.org/10.1016/j.compeleceng.2023.108859

Tiwari, S., Jain, A., Sapra, V., Koundal, D., Alenezi, F., Polat, K., ... & Nour, M. (2023). A smart decision support system to diagnose arrhythymia using ensembled ConvNet and ConvNet-LSTM model. Expert Systems with Applications, 213, 118933. DOI: https://doi.org/10.1016/j.eswa.2022.118933

Treneska, S., Zdravevski, E., Pires, I. M., Lameski, P., & Gievska, S. (2022). GAN-Based Image Colorization for Self-Supervised Visual Feature Learning. Sensors, 22(4), 1599. MDPI AG. Retrieved from http://dx.doi.org/10.3390/s22041599 DOI: https://doi.org/10.3390/s22041599