Real-Time Bot Detection on XCorp: Enhanced Feature Engineering with Apache Spark

The rapid growth of misinformation, spam, and automated bot activity on social media platforms such as X (formerly Twitter, operated by XCorp) has created a pressing need for accurate and real-time detection systems. Bots are one of the most important challenges at XCorp. A bot on XCorp is software that sends tweets to users automatically. XCorp bots may be used for several advantageous purposes or may be created for malicious purposes such as spreading fake news campaigns, spamming, and violating the privacy of others. This study presents a scalable framework for real-time tweets analysis using Apache Spark, focusing on the integration of user-based features and context-based features to enhance bot detection performance. Three distinct datasets were utilized to build and train the proposed model, enabling robust feature learning and generalization across different data sources. A large stream of real-time XCorp data was then processed using Spark Streaming, where newly engineered features captured user and content features within tweets. The model was constructed using five PySpark machine learning techniques: Gradient Boosted Trees Classification, Linear Support Vector Machine Classification, Decision Tree Classification, Logistic Regression, and Random Forest Classification. Experimental results show that incorporating the proposed features significantly improves accuracy, precision, and recall compared with baseline models and existing studies, while maintaining a relatively stable computation time. Among the evaluated models, the Random Forest classifier achieved the best performance, reaching 99.1% precision, 99.7% recall, 99.40% F1-Score and 99.6% accuracy.