Evaluating Anomaly Detection Algorithms: A Multi-Metric Analysis Across Variable Class Imbalances

Anomaly detection is critical for ensuring integrity and performance in numerous high-stakes domains, ranging from financial systems to network security. The effectiveness of anomaly detection algorithms is significantly affected by the class imbalances that characterize real-world data. This research conducts an in-depth analysis of five anomaly detection algorithms—Angle-based Outlier Detector (ABOD), K-Nearest Neighbors (KNN), Local Outlier Factor (LOF), Isolation Forest (IsoForest), and One-Class SVM (OCSVM). Our evaluation spans datasets with a spectrum of feature complexities and observation volumes, alongside a targeted resampling of anomaly percentages, shifting from a balanced 50/50 distribution to imbalances ranging from 10% to 40%. A multi-metric evaluation framework is deployed, encompassing F1, ROC AUC, PR AUC, MCC, and Kappa, to deliver a layered assessment of algorithmic performance. Our findings reveal distinct stability in the correlations of F1 with Kappa and MCC, and between MCC and Kappa, signifying their potential as consistent performance indicators across various datasets and models. In contrast, F1’s correlation with ROC and PR AUC, and to a lesser degree PR AUC’s correlation with ROC, displayed notable fluctuations, indicating a differential impact of class distribution on these metrics. The study underscores the imperative of utilizing a multi-metric approach for a comprehensive evaluation of anomaly detection algorithms, ensuring adaptability to the diverse and skewed distributions encountered in practice. The insights from this analysis provide a pathway for practitioners to make informed decisions in selecting and deploying anomaly detection models that can withstand the challenges posed by varying class imbalances.