Research on the Deployment Strategy of Big Data Visualization Platform by the Internet of Things Technology
DOI:
https://doi.org/10.4108/eetsis.v10i3.3051Keywords:
CPU, Field Programmable Gate Array, genetic algorithm, IOT, ant colony scheduling, big dataAbstract
INTRODUCTION: To improve the big data visualization platform's performance and task scheduling capability, a big data visualization platform is constructed based on Field Programmable Gate Array (FPGA) chip application power equipment.
OBJECTIVES: This study proposes to combine a genetic algorithm and an ant colony scheduling (ACOS) algorithm to design a big data visualization platform deployment strategy based on an improved ACOS algorithm.
METHODS: Firstly, big data technology is analyzed. Then, the basic theory of the ant colony algorithm is studied. According to the basic theory of ACOS and genetic algorithm, an improved ACOS algorithm model is constructed. The improved ACOS algorithm scheduler is compared with the other three schedulers. Under the same environment, the completion time of scheduling the same job and different task amounts are analyzed. The Central Processing Unit (CPU) utilization is analyzed when different schedulers have entirely different workloads.
RESULTS: The results show that the constructed big data visualization platform based on the improved ACOS algorithm model has higher task scheduling efficiency than other schedulers and can greatly shorten the data processing time. The experimental results show that under the homogeneous cluster, the completion time of the improved ACOS algorithm generally lags the capacity scheduler and the fair scheduler. Under the heterogeneous cluster, the improved ACOS algorithm scheduler can reasonably allocate tasks to nodes with different performances, reducing the task completion time. When the number of completed tasks increases from 50 to 200, the time increases by 45s, and the completion time is shorter than other schedulers. The CPU utilization of different task volumes is the highest, and the utilization rate increases from 81% to 95%.
CONCLUSION: The improved ACOS algorithm scheduler has the shortest data processing time and the highest efficiency. This work provides a specific reference value for optimizing the big data visualization platform's deployment strategy and improving the platform's performance.
References
Arif C, Setiawan BI, Saptomo SK, Taufik M, Wiranto, Mizoguchi M. Developing it infrastructure of evaporative irrigation by adopting iot technology. IOP Conference Series: Earth and Environmental Science. 2021, 622(1).
Xu Y, Zhang Z, Liu M. Design of cancer classification and visualization platform based on Internet big data. Journal of Physics: Conference Series.2020,1650(3).
Song Z, Yang Y, Guo H. Analysis of data crawling and visualization methods for recruitment industry information. Journal of Physics: Conference Series. 2021, 1971(1).
Davis R, Vochozka M, Vrbka J, Octav N. Industrial Artificial Intelligence, Smart Connected Sensors, and Big Data-driven Decision-Making Processes in Internet of Things-based Real-Time Production Logistics. Economics, Management, and Financial Markets. 2020, 15(3): 9-16.
Cocoros NM, Kirby C, Zambarano B, Ochoa A, Eberhardt K, Rocchio Sb C, Ursprung WS, Nielsen VM, Durham NN, Menchaca JT, Josephson M, Erani D, Hafer E, Weiss M, Herrick B, Callahan M, Isaac T, Klompas M. RiskScape: A Data Visualization and Aggregation Platform for Public Health Surveillance Using Routine Electronic Health Record Data. Am J Public Health. 2021,111(2):269-276.
Ming C, Bo L, Zuo H. College Entrance Examination Voluntary Filing System Based on Big Data. International Journal of Advanced Research in Big Data Management System. 2018, 2(21): 23-36.
Chen J, Tian J, Jiang S, Zhou Y, Li H, Xu J. The Allocation of Base Stations with Region Clustering and Single-Objective Nonlinear Optimization. Mathematics. 2022, 10(13): 2257.
Harb H, Mroue H, Mansour A, Nasser A, Cruz Em. A hadoop-based platform for patient classification and disease diagnosis in healthcare applications. Sensors. 2020, 20(7):1931.
Atat R, Liu L, Wu J, Li G, Ye C, Yang Y. Big data meet cyber-physical systems: A panoramic survey. IEEE Access. 2018, 6: 73603-73636.
Zhao Y, Ye P, Yang K, Meng J, Lei M. A field programmable gate array based synchronization mechanism of analog and digital local oscillators in bandwidth-interleaved data acquisition systems. Review of Scientific Instruments. 2021, 92(3).
Protopsaltis A, Sarigiannidis P, Margounakis D, Lytos A.. Data Visualization in Internet of Things: Tools, Methodologies, and Challenges. ARES '20: Proceedings of the 15th International Conference on Availability, Reliability and Security, 2020, 1-11.
Kumar S, Tiwari P, Zymbler M. Internet of Things is a revolutionary approach for future technology enhancement: a review. J Big Data. 2019, 6(111).
Bashir MR, Gill AQ. Towards an IoT Big Data Analytics Framework: Smart Buildings Systems. 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Sydney, NSW, Australia, 2016, pp. 1325-1332.
Khare S, Totaro M. Big Data in IoT. 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kanpur, India, 2019, pp. 1-7.
Preeti G, Ayushi C. Big data analytics for IOT. International Journal of Advanced Research in Engineering and Technology (IJARET). 2020, 11(6): pp. 593-603.
Sudhir A. Exploratory study for big data visualization in the Internet of things. International Journal of Creative Research Thoughts (IJCRT). 2017,5(3):805-809.
Mrs Poonam and Mrs Aditi Mittal. Eminent Data Visualization Tools for Integration of Big Data with IoT. International Journal of Advanced Research in Science, Communication and Technology (IJARSCT). 2021, 5(1).
Yuya S. A Survey on IoT Big Data Analytic Systems: Current and Future. IEEE Internet of Things Journal. 2022, 9(2).
Pandey V, Saini P. Constraint programming versus heuristic approach to MapReduce scheduling problem in Hadoop YARN for energy minimization. J Supercomput . 2021,77: 6788-6816.
Bawankule KL, Dewang RK, Singh A K. Performance Analysis of Hadoop YARN Job Schedulers in a Multi-Tenant Environment on HiBench Benchmark Suite. International Journal of Distributed Systems and Technologies (IJDST). 2021,12( )3: 64-82.
Saraswat H, Sharma N. Enhancing the Traditional File System to HDFS: A Big Data Solution. International Journal of Computer Applications. 2017, 167(9):975-8887.
Ergüzen, ünver M. Developing a File System Structure to Solve Healthy Big Data Storage and Archiving Problems Using a Distributed File System. Applied Sciences.2018, 8(6):913.
Pachghare A, Jadhav A, Panigrahi S, Smitha D. Implementation of MapReduce Using Pig for Election Analysis. International Conference on Innovative Computing and Communications. 2019, 56: 231.
Kim YP, Hong CH, Yoo C. Performance impact of JobTracker failure in Hadoop. International journal of communication systems. 2015, 28(7):1265-1281.
Subbulakshmi T, Manjaly JS. TaskTracker Aware Scheduler with Resource Availability Control for Hadoop MapReduce. International Journal of Advanced Intelligence Paradigms. 2018,1(1):1.
Huang W, Meng L, Zhang D, Zhang W. In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing. 2017, 10(1): 3-19.
Jarrah M, Al-Quraan M, Jararweh Y, Al-Ayyoub M. MedGraph: a graph-based representation and computation to handle large sets of images. Multimedia Tools & Applications. 2017, 76(2):2769-2785.
Chen J, Li K, Zhuo T, Bilal K, Yu S, Weng C, Li K. A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment. IEEE Transactions on Parallel & Distributed Systems. 2017, 28(4): 919-933.
Guo Y, Zhang Z, Jiang J, Wu W, Zhange C, Cui B, Li J. Model averaging in distributed machine learning: a case study with Apache Spark. The VLDB Journal. 2021, 30(4):693-712.
Fu SY. Design of high speed data acquisition system for linear array CCD based on FPGA. Procedia Computer Science. 2020, 166: 414-418, 2020.
Mendrofa H, Muis A. Serial Manipulator Control Optimization Using Ant Colony Algorithm. Journal of Physics: Conference Series. 2021, 1993(1).
Wan L, Du C. An approach to evaluation of environmental benefits for ecological mining areas based on ant Colony algorithm. Earth Science Informatics. 2021,14(2):797-808.
Yi G, He Y, Gao L, He W. Propagation Path Optimization of Product Attribute Design Changes Based on Petri Net fusion Ant Colony Algorithm. Expert Systems with Applications. 2021, 173.
Shi, Zhang Y. A Novel Algorithm to Optimize the Energy Consumption Using IoT and Based on Ant Colony Algorithm. Energies. 2021, 14(6):1-17.
Wu F. Contactless Distribution Path Optimization Based on Improved Ant Colony Algorithm. Mathematical Problems in Engineering. 2021,7: 1-11.
Lv G, Chen S. Routing optimizationin wireless sensor network based on improved ant colony algorithm. International Core Journal of Engineering. 2020, 6(2):1-11.
Ghosh M, Dey N, Mitra D, Chakrabarthi A. A novel quantum algorithm for ant colony optimization. IET Quantum Communication. 2021, 3(1):13-29.
Sekiner SU, Shumye A, Geer S. Minimizing Solid Waste Collection Routes Using Ant Colony Algorithm: A Case Study in Gaziantep District. Journal of Transportation and Logistics. 2021, 6(1): 29-47.
Euchi J, Sadok A. Optimising the travel of home health carers using a hybrid ant colony algorithm. Transport. 2021, 3:1-22.
Chen Y, Zhou X. Path Planning of Robot Based on Improved Ant Colony Algorithm in Computer Technology. Journal of Physics: Conference Series. 2021, 1744(4).
Zhou Y, Fu X. Research on the combination of improved Sobel operator and ant colony algorithm for defect detection. MATEC Web of Conferences. 2021, 336(11).
Wang Y, Yang R R, Xu YX, Li X, Shi JL. Research on Multi-Agent Task Optimization and Scheduling Based on Improved Ant Colony Algorithm. IOP Conference Series: Materials Science and Engineering. 2021, 1043(3).
Luan J, Yao Z, Zhao F, Song X. A novel method to solve supplier selection problem: Hybrid algorithm of genetic algorithm and ant colony optimization. Mathematics and Computers in Simulation.2019,156:294-309.
Cai L, Qi Y, Wei W, Wu J, Li J. mrMoulder: a recommendation-based adaptive parameter tuning approach for big data processing platform. Future Generation Computer Systems. 2019, 93: 570-582, 2019.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Guangtao Zhang
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.
