Colorizing Multi-Modal Medical Data: An Autoencoder-based Approach for Enhanced Anatomical Information in X-ray Images




Gray Image Colorization, Unsupervised Learning, Auto Encoders, Transfer Learning, Medical Imaging


Colourisation is the process of synthesising colours in black and white images without altering the image’s structural content and semantics. The authors explore the concept of colourisation, aiming to colourise the multi-modal medical data through X-rays. Colourized X-ray images have a better potential to portray anatomical information than their conventional monochromatic counterparts. These images contain precious anatomical information that, when colourised, will become very valuable and potentially display more information for clinical diagnosis. This will help improve understanding of these X-rays and significantly contribute to the arena of medical image analysis. The authors have implemented three models, a basic auto-encoder architecture, and two combined learnings of the autoencoder module with transfer learning of pre-trained neural networks. The unique feature of this proposed framework is that it can colourise any medical modality in the medical imaging domain. The framework’s performance is evaluated on a chest x-ray image dataset, and it has produced benchmark results enabling high-quality colourisation. The biggest challenge is the need for a correct solution for the mapping between intensity and colour. This makes human interaction and external information from medical professionals crucial for interpreting the results.


Download data is not yet available.


B, S., & B, R. (2022, November 9). International Journal of Recent Technology and Engineering (IJRTE). Retrieved November 15, 2022, from

Farella, E. M., Malek, S., & Remondino, F. (2022). Colorizing the Past: Deep Learning for the Automatic Colorization of Historical Aerial Images. Journal of Imaging, 8(10), 269..

Garcea, F., Serra, A., Lamberti, F., & Morra, L. (2022). Data augmentation for medical imaging: A systematic literature review. Computers in Biology and Medicine, 106391. DOI:

Khan, M. U. G., Gotoh, Y., & Nida, N. (2017, June 22). Medical image colorization for better visualization and segmentation. White Rose Research Online. Retrieved November 15, 2022, from

Tariq, A., Gill, A. Y., & Hussain, H. K. (2023). Evaluating the Potential of Artificial Intelligence in Orthopedic Surgery for Value-based Healthcare. International Journal of Multidisciplinary Sciences and Arts, 2(1), 27-35. DOI:

Liang, Y., Lee , D., Li1 , Y., & Shin, B.-S. (2021, January 18). Unpaired medical image colorization using generative ... - springer. Unpaired medical image colorization using generative adversarial network. Retrieved November 15, 2022, from

R. Ribani and M. Marengoni, "A Survey of Transfer Learning for Convolutional Neural Networks," 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), 2019, pp. 47-57, doi: 10.1109/SIBGRAPI-T.2019.00010. DOI:

D’Souza, G., Reddy, N. S., & Manjunath, K. N. (2023). Localization of lung abnormalities on chest X-rays using self-supervised equivariant attention. Biomedical Engineering Letters, 13(1), 21-30. DOI:

Mohan, R., Elsken, T., Zela, A., Metzen, J. H., Staffler, B., Brox, T., ... & Hutter, F. (2023). Neural Architecture Search for Dense Prediction Tasks in Computer Vision. International Journal of Computer Vision, 1-24. DOI:

Morra, L., Piano, L., Lamberti, F., & Tommasi, T. (2020, October 19). Bridging the gap between natural and medical images through ... - arxiv. Bridging the gap between Natural and Medical Images through Deep Colorization. Retrieved November 15, 2022, from DOI:

Huang, S., Jin, X., Jiang, Q., & Liu, L. (2022). Deep learning for image colorization: Current and future prospects. Engineering Applications of Artificial Intelligence, 114, 105006.

Abbadi, N. K. E., & Razaq, E. S. (2020). Automatic gray images colorization based on lab color space. Indonesian Journal of Electrical Engineering and Computer Science, 18(3), 1501-1509. DOI:

Wu, M., Jin, X., Jiang, Q., Lee, S. J., Liang, W., Lin, G., & Yao, S. (2021). Remote sensing image colorization using symmetrical multi-scale DCGAN in YUV color space. The Visual Computer, 37(7), 1707-1729. DOI:

Li, B., Lu, Y., Pang, W., & Xu, H. (2023). Image Colorization using CycleGAN with semantic and spatial rationality. Multimedia Tools and Applications, 1-15. DOI:

Cevallos, S., Pérez, N., Riofrío, D., Benítez, D., Moyano, R. F., & Baldeon-Calisto, M. (2022, July). A Deep Convolutional Autoencoder Architecture for Automatic Image Colorization. In 2022 IEEE Colombian Conference on Applications of Computational Intelligence (ColCACI) (pp. 1-6). IEEE. DOI:

Dong, X., Liu, C., Li, W., Hu, X., Wang, X., & Wang, Y. (2021). Self-supervised colorization towards monochrome-color camera systems using cycle CNN. IEEE Transactions on Image Processing, 30, 6609-6622. DOI:

Kim, S., Jang, Y., & Kim, S. E. (2021). Image-Based TF Colorization With CNN for Direct Volume Rendering. IEEE Access, 9, 124281-124294. DOI:

An, J., Kpeyiton, K. G., & Shi, Q. (2020). Grayscale images colorization with convolutional neural networks. Soft Computing, 24(7), 4751-4758. DOI:

Joshi, M. R., Nkenyereye, L., Joshi, G. P., Islam, S. R., Abdullah-Al-Wadud, M., & Shrestha, S. (2020). Auto-colorization of historical images using deep convolutional neural networks. Mathematics, 8(12), 2258 DOI:

Mouzon, T., Pierre, F., & Berger, M. O. (2019, June). Joint cnn and variational model for fully-automatic image colorization. In International Conference on Scale Space and Variational Methods in Computer Vision (pp. 535-546). Springer, Cham. DOI:

Nguyen, T., Mori, K., & Thawonmas, R. (2016). Image colorization using a deep convolutional neural network. arXiv preprint arXiv:1604.07904.

Hensman, P., & Aizawa, K. (2017, November). cGAN-based manga colorization using a single training image. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) (Vol. 3, pp. 72-77). IEEE. DOI:

Mourchid, Y., Donias, M., & Berthoumieu, Y. (2021, January). Automatic Image Colorization based on Multi-Discriminators Generative Adversarial Networks. In 2020 28th European Signal Processing Conference (EUSIPCO) (pp. 1532-1536). IEEE. DOI:

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29.

Jin, X., Li, Z., Liu, K., Zou, D., Li, X., Zhu, X., ... & Liu, Q. (2021, October). Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 1176-1184). DOI:

Farella, E. M., Malek, S., & Remondino, F. (2022). Colorizing the Past: Deep Learning for the Automatic Colorization of Historical Aerial Images. Journal of Imaging, 8(10), 269. DOI:

Dias, M., Monteiro, J., Estima, J., Silva, J., & Martins, B. (2020). Semantic segmentation and colorization of grayscale aerial imagery with W‐Net models. Expert systems, 37(6), e12622. DOI:

Chen, S. Y., Zhang, J. Q., Zhao, Y. Y., Rosin, P. L., Lai, Y. K., & Gao, L. (2022). A review of image and video colorization: From analogies to deep learning. Visual Informatics. DOI:

Huang, S., Jin, X., Jiang, Q., & Liu, L. (2022). Deep learning for image colorization: Current and future prospects. Engineering Applications of Artificial Intelligence, 114, 105006. DOI:

Žeger, I., Grgic, S., Vuković, J., & Šišul, G. (2021). Grayscale image colorization methods: Overview and evaluation. IEEE Access. DOI:

Nida, N., Sharif, M., Khan, M. U. G., Yasmin, M., & Fernandes, S. L. (2016). A framework for automatic colorization of medical imaging. IIOAB J, 7, 202-209.

Mathur, A. N., Khattar, A., & Sharma, O. (2021). 2D to 3D Medical Image Colorization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2847-2856). DOI:

Selvapriya B., Raghu B. (2019) Colorization using Desired Color for Medical Images. International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-7 Issue-6S3

Mooney, P. (2018, March 24). Chest X-ray images (pneumonia). Kaggle. Retrieved September 17, 2022, from

Liang, C. M., Li, Y. W., Liu, Y. H., Wen, P. F., & Yang, H. (2022). Segmentation and weight prediction of grape ear based on SFNet-ResNet18. Systems Science & Control Engineering, 10(1), 722-732. DOI:

Tschannen, M., Bachem, O., & Lucic, M. (2018, December 12). Recent advances in autoencoder-based representation learning. Retrieved November 17, 2022, from

Bank, D., Koenigstein, N., & Giryes, R. (2021, April 3). Autoencoders. Retrieved November 17, 2022, from

Kamat, P., Sugandhi, R., & Kumar, S. (2021). Data-driven bearing fault detection using hybrid autoencoder-LSTM deep learning approach. International Journal of Modelling, Identification and Control, 38(1), 88-103. DOI:

Jordan, J. (2018, March 19). Chapter 14: Autoencoders. deeplearningbook-notes. Retrieved November 17, 2022, from

Autoencoders in Deep learning: Tutorial & use cases [2022]. V7. (2022, October 21). Retrieved November 17, 2022, from

Ramzan, F., Khan, M. U. G., Rehmat, A., Iqbal, S., Saba, T., Rehman, A., & Mehmood, Z. (2019, December 18). A deep learning approach for automated diagnosis and Multi-class classification of alzheimer's disease stages using resting-state fmri and residual neural networks - Journal of Medical Systems. SpringerLink. Retrieved November 18, 2022, from DOI:

Sanakkayala, D. C., Varadarajan, V., Kumar, N., Karan, Soni, G., Kamat, P., ... & Kotecha, K. (2022). Explainable AI for bearing fault prognosis using deep learning techniques. Micromachines, 13(9), 1471. DOI:

Jin, X., Di, Y., Jiang, Q., Chu, X., Duan, Q., Yao, S., & Zhou, W. (2023). Image colorization using deep convolutional auto-encoder with multi-skip connections. Soft Computing, 27(6), 3037-3052. DOI:

Hu, M., Bai, L., Fan, J., Zhao, S., & Chen, E. (2023). Vehicle color recognition based on smooth modulation neural network with multi-scale feature fusion. Frontiers of Computer Science, 17(3), 173321. DOI:

Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2016). Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG), 35(4), 1-11. DOI:

Mouzai, M., Mustapha, A., Bousmina, Z., Keskas, I., & Farhi, F. (2023). Xray-Net: Self-supervised pixel stretching approach to improve low-contrast medical imaging. Computers and Electrical Engineering, 110, 108859.. DOI:

Tiwari, S., Jain, A., Sapra, V., Koundal, D., Alenezi, F., Polat, K., ... & Nour, M. (2023). A smart decision support system to diagnose arrhythymia using ensembled ConvNet and ConvNet-LSTM model. Expert Systems with Applications, 213, 118933. DOI:

Treneska, S., Zdravevski, E., Pires, I. M., Lameski, P., & Gievska, S. (2022). GAN-Based Image Colorization for Self-Supervised Visual Feature Learning. Sensors, 22(4), 1599. MDPI AG. Retrieved from DOI:




How to Cite

Saini B, Venkatesh D, Ganesh A, Parameswaran A, Patil S, Kamat P, Choudhury T. Colorizing Multi-Modal Medical Data: An Autoencoder-based Approach for Enhanced Anatomical Information in X-ray Images. EAI Endorsed Trans Perv Health Tech [Internet]. 2024 Mar. 25 [cited 2024 Apr. 21];10. Available from:

Most read articles by the same author(s)