Deep spectral network for time series clustering

Duc-Trung Hoang; Mahdi Achache; Vinay Kumar Jain

doi:10.4108/eetsis.3907

Authors

Duc-Trung Hoang University of Clermont Auvergne
Mahdi Achache Aix-Marseille University
Vinay Kumar Jain IFP Énergies nouvelles

DOI:

https://doi.org/10.4108/eetsis.3907

Keywords:

Deep clustering, neural network, time series, spectral clustering, mutual information, multi label classification

Abstract

Deep clustering is an approach that uses deep learning to cluster data, since it involves training a neural network model to become familiar with a data representation that is suitable for clustering. Deep clustering has been applied to a wide range of data types, including images, texts, time series and has the advantage of being able to automatically learn features from the data, which can be more effective than using hand-crafted features. It is also able to handle high-dimensional data, such as time series with many variables, which can be challenging for traditional clustering techniques. In this paper, we introduce a novel deep neural network type to improve the performance of the auto-encoder part by ignoring the unnecessary extra-noises and labelling the input data. Our approach is helpful when just a limited amount of labelled data is available, but labelling a big amount of data would be costly or time-consuming. It also applies for the data in high-dimensional and difficult to define a good set of features for clustering.

References

A. Alqahtani, M. Ali, X. Xie, and M. Jones "Deep TimeSeries Clustering: A Review: a review", Electronics 2, vol. 10, no.23, 2021.

S. Aghabozorgi, A. Shirkhorshidi, and T. Wah, "Timeseries clustering – A decade review", Information Systems, vol. 53, pp. 16-38, Nov. 2015.

A. Ng, M. Jordan, and Y. Weiss "On spectral clustering: analysis and an algorithm", Proceedings of the 14th International Conference on Neural Information Processing Systems:, pp. 849-856, Jan. 2001.

Y. Asano, C. Rupprecht, and A. Vedaldi, "Self-labelling via simultaneous clustering and representation learning", ICLR Conference Paper, 2020.

F. Bach and M. Jordan "Learning Spectral Clustering", Proceedings of the 16th International Conference on Neural Information Processing Systems:, 2003.

D. Bo, X. Wang, C. Shi, M. Zhu, E. Lu, and P. Cui, "Structural deep clustering network", Proceedings of The Web Conference, pp 1400–1410, 2020.

S. Chang, Y. Zhang, W. Han, M. Yu, X. Guo, W. Tan, X. Cui, M. Witbrock, M. Johnson, and T. Huang, "Dilated recurrent neural networks", Advances in Neural Information Processing Systems, pp. 77–87, 2017. [8] J. Chang, L. Wang, G. Meng, S. Xiang, and C. Pan "Deep Adaptive Image Clustering", ICCV, 2017.

J. Chien, K. Kuo, "Variational Recurrent Neural Networks for Speech Separation", Interspeech, 2017.

M. Corduas and D. Piccolo, "Time series clustering and classification by the autoregressive metric", Computational Statistics Data Analysis, vol. 52, no.4, pp. 1860-1872, Jan. 2008.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradientbased learning applied to document recognition", IEEE, 1998.

H. Dau, A. Bagnall, K. Kamgar, C. Yeh, S. Gharghabi,

C. Ratanamahatana, and E. Keogh "The UCR Time Series Archive", IEEE/CAA J. Autom. Sinica, vol. 6, no. 6, pp. 1293-1305, Nov. 2019.

K. Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang H, "Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization", Proceedings of the IEEE international conference on computer vision, pp 5736–5745, 2017.

H. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P. Muller, "Deep learning for time series classification: a review", Data Mining and Knowledge Discovery volume, vol. 33, pp. 917-963, 2019.

N.El Malki, "New partition-based and density-based approaches for improving clustering", thèse de doctorat, https://theses.hal.science/tel-03716266/document, 2021.

J. Franceschi, A. Dieuleveut, and M. Jaggi, "Unsupervised scalable representation learning for multivariate time series", Advances in Neural Information Processing Systems, pp 4652–4663, 2019.

X. Guo, X. Liu, E. Zhu, and J. Yin, "Deep Clustering with Convolutional Autoencoders" In Proceedings of the International Conference on Neural Information Processing, pp. 373–382, 2017.

X. Guo, L. Gao, X. Liu and J. Yin, "Improved deep embedded clustering with local structure preservation", IJCAI, pp 1753–1759, 2017.

R. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, "Learning deep representations by mutual information estimation and maximization", ICLR, 2019.

P. Huang, Y. Huang, W. Wang, L. Wang, "Deep embedding network for clustering", In Proceedings of the International Conference on Pattern Recognition, pp. 1532–1537, 2014.

A. Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks", Advances in Neural Information Processing Systems 25 (NIPS), 2012.

B. Lafabregue, J. Weber, P. Gancarski, and G. Forestier, "End-to-end deep representation learning for time series clustering: a comparative study", Data Mining and Knowledge Discovery volume, vol. 36, pp. 29-81, 2022.

Q. Ma and J. Zheng and S. Li and G. Cottrel, "Learning representations for time series clustering", Advances in neural information processing systems, page 3781-3791, 2019.

Q. Ma, C. Chen, S. Li, and G. Cottrell, "Learning Representations for Incomplete Time Series Clustering" AAAI Technical Track on Machine Learning, Vol. 35, No. 10, 2021.

M. Law, R. Urtasun and R. Zemel, "Deep spectral clustering learning," Proceedings of the 34th International Conference on Machine Learning, vol 70, pp. 1985–1994, 2017.

N. Madiraju, S. Sadat, D. Fisher, and J. Karimabadi, "Deep Temporal Clustering: Fully Unsupervised Learning of Time-Domain Features", arXiv 2018, arXiv:1802.01059, 2018.

E. Min, X. Guo, Q. Liu, G. Zhang, J. Cui and J. Long, "A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture," IEEE Access, vol. 6,

pp. 39501-39514, 2018

A. Sammani, A. Bagheri, and W. Asselbergs, "Automatic multilabel detection of ICD10 codes in Dutch cardiology discharge letters using neural networks," nature portfolio, 2021.

K. Simonyan, A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," ICLR, 2015.

P. Tzirakis, M. Nicolaou, B. Schuller, and S. Zafeiriou, "Time-series Clustering with Jointly Learning Deep

Representations, Clusters and Temporal Boundaries", 14th IEEE International Conference on Automatic Face Gesture Recognition , 2019.

U. Shaham, K. Stanton, H. Li, B. Nadler, R. Basri and Y. Kluger, "SpectralNet: Spectral clustering using deep neural networks," 6th International Conference on Learning Representations, 2018.

J. Xie, R. Girshick and A. Farhadi, "Unsupervised deep embedding for clustering analysis", International conference on machine learning, pp. 478–487, 2016.

H. Zha, X. He, C. Ding, H. Simon, and M. Gu "Spectral relaxation for K-means clustering", Proceedings of the 14th International Conference on Neural Information Processing Systems:, pp. 1057-1064, Jan. 2001.

Y. Zhao, Y. Yuan, F. Nie, and Q. Wang, "Spectral clustering based on iterative optimization for large-scale and high-dimensional data", Neurocomputing, vol. 318, pp. 227-235, Jan. 2018.

F. Wang and C. Zhang, "Spectral Clustering for Time Series", Pattern Recognition and Data Mining, pp. 345-354, 2005.

Z. Wang Z, W. Yan, and T. Oates, "Time series classification from scratch with deep neural networks: A strong baseline", International joint conference on neural networks (IJCNN), IEEE, pp 1578–1585, 2017.

X. Yang, C. Deng, F. Zheng, J. Yan, and W. Liu, "Deep Spectral Clustering Using Dual Autoencoder Network", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

B. Yang, X. Fu, N. Sidiropoulos, M. Hong "Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering", Proceedings of the 34th International Conference on Machine Learning PMLR, vol 70 pp. 38613870, 2017.

N. Yulita, M. Ivan Fanany, A. Murni Arymuthya, "Bidirectional Long Short-Term Memory using Quantized data of Deep Belief Networks for Sleep Stage Classification", Procedia Computer Science, vol 116 pp. 530-538, 2017.

S. Zhang, H. Tong, J. Xu, and R. Maciejewski, "A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions", Computational social network, pp 06–11, 2019.

S. Zhou, H. Xu, and Z. Zheng, "Graph convolutional networks: a comprehensive review", ArXiv, https://arxiv.org/abs/2206.07579, 2022.

Y. Ren, J. Pu, Z. Zhang, J. Xu, X. Pu, S. Yu, and L. He, "Deep Clustering: A Comprehensive Survey", ArXiv, https://arxiv.org/abs/2210.04142, 2022.