Ransomware Detection: The Efficacy of Behavior-Based and Machine Learning Techniques

Abstract

Ransomware remains one of the most pervasive cybersecurity threats, exploiting both technological and human vulnerabilities to inflict severe economic and operational damage. This thesis investigates the efficacy of hybrid detection methodologies that integrate behavior-based analysis with machine learning (ML) and deep learning Long Short-Term Memory (LSTM) approaches to improve detection accuracy and generalization across diverse ransomware variants. The proposed framework unifies three behavioral dimensions—File System Monitoring (FSM), Process Behavior Analysis (PBA), and Network Behavior Analysis (NBA)—into a comprehensive dataset of 15,411 instances and 224 features, aligned through a Timestamp-Based Integration process. Multiple classifiers, including Random Forest, Naïve Bayes, Support Vector Machine (SVM), Gradient Boosting, and LSTM, were trained and evaluated. Two integration strategies—decision-level fusion (voting) and model-level stacking- were compared empirically to identify the most robust hybrid configuration. Experimental results demonstrated that the stacking ensemble [BB, XGB, NB] achieved superior macro-average performance (F1 ≈ 0.93, AUPRC ≈ 0.91), validating the advantage of multi-model learning for ransomware detection. Additionally, Synthetic Minority Over-sampling Technique (SMOTE) balancing and probability calibration improved fairness and stability across minority ransomware families such as Ryuk, Sodinokibi, and LockBit. The study also incorporated statistical validation (McNemar’s test) and sensitivity analysis to ensure the reliability of results under variable conditions. Finally, ethical and policy considerations were highlighted to guide the responsible deployment of AI-driven cybersecurity systems. This research bridges a major gap in ransomware detection studies by operationalizing a cross-domain hybrid framework that synchronizes host and network behavioral data, providing a replicable and scalable foundation for intelligent, interpretable, and ethically aligned ransomware defense systems.