Learn why imbalanced datasets rarely need rebalancing. Compare SMOTE, sampling and class weights with strong classifiers, probability scoring and threshold tuning.

In Machine Learning, classification problem refers to predictive modeling where a class label needs to be predicted for a given...

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.

Classification tasks in machine learning involving more than two classes are known by the name of "multi-class classification". Performance indicators are very useful when the aim is to evaluate and compare different classification models or machine learning techniques. Many metrics come in handy to test the ability of a multi-class classifier. Those metrics turn out to be useful at different stage of the development process, e.g. comparing the performance of two different models or analysing the behaviour of the same model by tuning different parameters. In this white paper we review a list of the most promising multi-class metrics, we highlight their advantages and disadvantages and show their possible usages during the development of a classification model.

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.

Learn what Precision-Recall curves are, how they compare to ROC curves, and how to plot, interpret and evaluate them in Python with imbalanced data examples.

Read Sole's discussion of three recent articles that shift the conversation around handling imbalanced datasets. These articles explore whether generating synthetic data is effective with current state-of-the-art algorithms, and examine the implications of using synthetic data on model performance and deployment.

Learn how to implement SMOTE in Python and whether you should still use it to work with imbalanced datasets in 2026.

Background Predictive models employing machine learning algorithms are increasingly being used in clinical decision making, and improperly calibrated models can result in systematic harm. We sought to investigate the impact of class imbalance correction, a commonly applied preprocessing step in machine learning model development, on calibration and modelled clinical decision making in a large real-world context. Methods A histogram boosted gradient classifier was trained on a highly imbalanced national dataset of > 1.8 million patients undergoing surgery, to predict the risk of 90-day mortality and complications after surgery. Class imbalance correction strategies including random oversampling, synthetic minority oversampling technique, random under-sampling, and cost-sensitive learning were compared to the natural distribution (‘natural’). Models were tested and compared with classification metrics, calibration plots, decision curve analysis, and simulated clinical impact analysis. Results The natural model demonstrated high performance (AUROC 0.94, 95% CI 0.94–0.95 for mortality; 0.84, 95% CI 0.84–0.85 for complications) and calibration (log loss 0.05, 95% CI 0.04–0.05 for mortality; 0.23, 95% CI 0.23–0.24 for complications). Class imbalance mitigation (CSL, ROS, RUS, and SMOTE) did not improve AUROC or AUPRC but increased recall and F1 scores at the expense of precision and accuracy. However, these methods severely compromised model calibration, leading to significant over-prediction of risks (up to a 62.8 % increase) as further evidenced by increased log loss across all mitigation techniques. Decision curve analysis and clinical scenario testing confirmed that the natural model provided the highest net benefit. Conclusion Class imbalance correction methods result in significant miscalibration, leading to possible harm when used for clinical decision making. ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement This work was funded by the Health Research Council of New Zealand 2025 Artificial Intelligence (AI) in Healthcare Project Grant. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethical approval was obtained from the Auckland Health Research Ethics Committee (AH29056). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data can be obtained from the New Zealand Ministry of Health National Collections team. Requests can be made by contacting data-enquiries{at}health.govt.nz. A Python package was created to create training and evaluation pipelines and is freely available on Github at <<https://github.com/Surgical-Recovery-and-Safety-Lab/medpipe>>. The code used to train the models and create the figures is freely available on GitHub at <https://github.com/Surgical-Recovery-and-Safety-Lab/AI-risk-score>.

Learn to explain interpretable and black box machine learning models with LIME, Shap, partial dependence plots, ALE plots, permutation feature importance and more, utilizing Python open source libraries..

Learn why imbalanced datasets rarely need rebalancing. Compare SMOTE, sampling and class weights with strong classifiers, probability scoring and threshold tuning.

This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian noise upsampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate to extreme (15% to 1% churn rate). Employing metrics such as F1 score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, this research provides a comprehensive evaluation of classifier performance under different imbalance scenarios, focusing on applications in the telecommunications domain. The findings highlight that tuned XGBoost paired with SMOTE (Tuned_XGB_SMOTE) consistently achieves the highest F1 score and robust performance across all imbalance levels. SMOTE emerged as the most effective upsampling method, particularly when used with XGBoost, whereas Random Forest performed poorly under severe imbalance. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, and GNUS produced inconsistent results. This study underscores the impact of data imbalance, with MCC, Kappa, and F1 scores fluctuating significantly, whereas ROC AUC and PR AUC remained relatively stable. Moreover, rigorous statistical analyses employing the Friedman test and Nemenyi post hoc comparisons confirmed that the observed improvements in F1 score, PR-AUC, Kappa, and MCC were statistically significant (p < 0.05), with Tuned_XGB_SMOTE significantly outperforming Tuned_RF_GNUS. While differences in ROC-AUC were not significant, the consistency of these results across multiple performance metrics underscores the reliability of our framework, offering a statistically validated and attractive solution for model selection in imbalanced classification scenarios.

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.

In Machine Learning, classification problem refers to predictive modeling where a class label needs to be predicted for a given...

Learn to explain interpretable and black box machine learning models with LIME, Shap, partial dependence plots, ALE plots, permutation feature importance and more, utilizing Python open source libraries..

Learn what Precision-Recall curves are, how they compare to ROC curves, and how to plot, interpret and evaluate them in Python with imbalanced data examples.

Classification tasks in machine learning involving more than two classes are known by the name of "multi-class classification". Performance indicators are very useful when the aim is to evaluate and compare different classification models or machine learning techniques. Many metrics come in handy to test the ability of a multi-class classifier. Those metrics turn out to be useful at different stage of the development process, e.g. comparing the performance of two different models or analysing the behaviour of the same model by tuning different parameters. In this white paper we review a list of the most promising multi-class metrics, we highlight their advantages and disadvantages and show their possible usages during the development of a classification model.

Class Central ratings are the most trusted measurement of quality for online courses. Now you can easily decide which Train in Data course to take next.

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.

Master imbalanced data with practical, modern machine learning techniques. Learn why class weights and resampling don’t improve performance, and how threshold tuning leads to better cost-sensitive decisions.