Scalable Big Data Analytics Pipeline for Real Time Network Traffic Prediction Using Advanced Machine Learning

Abstract

The paper presents a scalable big data analytics pipeline to predict network traffic in real-time, which meets the increasing need of effective traffic management in high velocity network systems. It works on the principle of distributed stream processing and sophisticated machine learning models, such as the Random Forest model, XGBoost model, and Long Short-Term Memory (LSTM) network to achieve high accuracy of 98.9% and low latency of 19 ms. The hybrid model improves the performance of predictions using non-linear interaction of features, optimization of gradient boosting, and temporal trends. The pipeline incurs a Root Mean Square error of 0.17 and a throughput rate of 20,800 records per second, which proves that it can process large volumes of network data at real time effectively. The offered system is able to balance accurately, scale well, and is computationally efficient enough to implement it on enterprise network, cloud environments and smart cities communication networks. The paper brings out the strengths of the pipeline to identify anomalies, optimize flow of traffic and resource assignment. The most important input is the combination of various machine learning algorithms that is used to develop a strong prediction system in real-time. The next phase of research will involve federated learning to train in privacy-preserving mode, integration of edge computing in 6G networks, and it will describe explainable AI to increase the transparency and efficiency of the model. The progress will also streamline the system to the next-generation network infrastructures.