3.990
Winning Strategies for Data Science Competitions

Introduction

The rise of data science competitions has profoundly shaped modern machine learning, both in practice and pedagogy. Platforms such as Kaggle, DrivenData, Zindi, KDD Cup, CrowdAI, and Topcoder have created ecosystems where individuals and teams compete to build the most accurate predictive models on shared datasets. These competitions are not just gamified analytics challenges—they are often tied to real-world problems in healthcare, finance, climate science, public policy, and e-commerce. In many of these competitions, ensemble methods consistently form the backbone of the winning solutions.

Among the most well-known platforms, Kaggle—acquired by Google in 2017—stands as the largest and most influential. It hosts hundreds of competitions annually, ranging from open research challenges to private industry-sponsored problems offering significant financial rewards. Platforms like DrivenData and Zindi focus on social impact problems and emerging markets, respectively, while KDD Cup remains a respected academic competition tied to the annual ACM SIGKDD conference.

Participation is open to anyone with a free account. Competitors are typically provided with a labeled training set, an unlabeled test set, and a problem description. Submissions are evaluated automatically against a hidden holdout test set using a predefined metric (e.g., root mean squared error, log-loss, AUC). A real-time leaderboard displays public scores, though final rankings depend on performance on a private test partition to prevent leaderboard overfitting.

Historical Context: Ensembles Dominante

The dominance of ensemble methods in competitions can be traced to the famous Netflix Prize (2006–2009), where teams competed to improve Netflix’s movie recommendation system. The $1 million prize was awarded to a stacked ensemble of hundreds of models, many of them independently trained and blended using ridge regression and neural nets. This established a pattern: the best results often arise not from a single model, but from clever combinations of many diverse models.

Competitions like the Kaggle Heritage Health Prize, BNP Paribas Cardif Claim Management, and Home Credit Default Risk further reinforced this pattern. In nearly all top-tier solutions, ensemble learning—especially stacking, blending, bagged boosting models, and model averaging—played a central role.

Common Winning Ensemble Architectures

Modern winning ensembles are rarely trivial. They often follow hierarchical, multi-layered architectures such as:

• Stacked generalization with multiple first-layer models (e.g., LightGBM, CatBoost, neural nets, logistic regression) and a second-level meta-learner (e.g., XGBoost or ridge regression).
• Blended models, which average predictions from different models trained with different seeds, features, or folds.
• Cross-validated folds ensembles, where base models are trained on different folds and their predictions are averaged or passed to a meta-model.
• Hybrid ensembles, combining tree-based models (for structured data) with deep learning models (for embeddings or time series features).

These architectures exploit both model diversity and error decorrelation, and often include hundreds of individual models, trained with subtle variations in hyperparameters, feature selection, and data transformations (Breiman, 1996; Wolpert, 1992; Freund & Schapire, 1997).

Competition-Style Ensemble in R: A Practical Illustration

To illustrate how an ensemble might be constructed in a competition, consider the House Prices competition (a popular Kaggle example). We’ll use a simplified version in R that includes:

1. A base layer of diverse models (lm, rpart, xgboost)
2. A meta-model trained on their predictions

# Load required packages
library(caret)
library(caretEnsemble)
library(xgboost)
library(rpart)
library(MASS)

# Load the dataset
data(Boston)
housing <- Boston
names(housing)[which(names(housing) == "medv")] <- "price"

# Train-test split
set.seed(123)
train.index <- createDataPartition(housing$price, p = 0.8, list = FALSE)
train.data <- housing[train.index, ]
test.data <- housing[-train.index, ]

# Train base models
control <- trainControl(method = "cv", number = 5,
                        savePredictions = "final",
                        allowParallel = TRUE)

base.models <- caretList(
  price ~ ., data = train.data,
  trControl = control,
  tuneList = list(
    lm = caretModelSpec(method = "lm"),
    tree = caretModelSpec(method = "rpart"),
    xgb = caretModelSpec(method = "xgbTree", 
                         tuneGrid = expand.grid(nrounds = 100,
                                                max_depth = 3,
                                                eta = 0.1,
                                                gamma = 0,
                                                colsample_bytree = 1,
                                                min_child_weight = 1,
                                                subsample = 1))
  )
)

# Stacking meta-model
stack.model <- caretStack(base.models, method = "glm", trControl = control)

# Predict and evaluate
preds <- predict(stack.model, newdata = test.data)
rmse <- sqrt(mean((preds - test.data$price)^2))
cat("Stacked RMSE on test set:", round(rmse, 2), "\n")

This code demonstrates the essence of competition-style ensembling using stacking. In practice, competitors would iterate over dozens of such base models, engineer features extensively, and optimize hyperparameters with advanced tools like mlr3 or BayesianOptimization.

Ethical Considerations and Critiques

While competitions foster innovation and collaboration, they are not without criticism. Key ethical and practical concerns include:

• Leaderboard overfitting: Repeated submissions encourage tuning to the public leaderboard, which may harm generalization. Kaggle addresses this with private leaderboards, but the risk remains.
• Computational privilege: Top-performing solutions often require substantial computing power, favoring competitors with access to GPUs and clusters.
• Black-box modeling: Competitions reward predictive accuracy, not interpretability. This can lead to deployment of opaque models in sensitive domains like health or finance without adequate scrutiny.
• Reproducibility issues: Many winning solutions are too complex or poorly documented to replicate, which undermines transparency and knowledge transfer.
• Cultural bias: Platforms like Kaggle are English-centric and dominated by teams from high-income countries, limiting global inclusion.

Despite these concerns, competitions have played a transformative role in democratizing access to real-world data problems and popularizing best practices in ensemble modeling (Dodge et al., 2019; Lipton, 2018).

Practical Tips for Students and Practitioners

For those looking to compete or to simulate real-world model evaluation, the following practices are common among top competitors:

• Start simple: Begin with a strong single model (e.g., xgboost, ranger) and develop a solid cross-validation scheme.
• Log everything: Track performance across folds, seeds, and parameter settings. Reproducibility is key.
• Stack wisely: Use out-of-fold predictions for stacking to avoid overfitting. Keep the meta-model simple.
• Feature engineering wins: Clean data and insightful features often matter more than model complexity.
• Blend diverse models: Combine models with different assumptions (trees vs. linear vs. neighbors). Diversity matters.
• Respect leakage: Avoid using test data or derived variables that “peek” into the future or outcome.

Finally, always be cautious when adapting competition-winning models for deployment in real-world settings. Competitions optimize for score, not always for fairness, explainability, or long-term reliability.

Competing with Model Stacking

---

Files & Resources

All Files for Lesson 3.990

---

References

Foundational Works on Ensembles and Stacking:

Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. https://doi.org/10.1007/BF00058655

Freund, Y., & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119–139. https://doi.org/10.1006/jcss.1997.1504

Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259. https://doi.org/10.1016/S0893-6080(05)80023-1

Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble selection from libraries of models. In Proceedings of the Twenty-First International Conference on Machine Learning (pp. 18). https://doi.org/10.1145/1015330.1015432

Sill, J., Tikk, D., Zhang, Y., & Kadlec, R. (2009). Feature-weighted linear stacking. arXiv preprint arXiv:0911.0460. https://arxiv.org/abs/0911.0460

Historical Context and Netflix Prize:

Bell, R. M., Koren, Y., & Volinsky, C. (2007). The BellKor solution to the Netflix Prize. Netflix Prize documentation. Retrieved from https://www.netflixprize.com/assets/ProgressPrize2007_BPC_BellKor.pdf

Koren, Y. (2009). The BellKor solution to the Netflix Grand Prize. Netflix Prize documentation. Retrieved from https://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf

Platform and Competition Descriptions:

Kaggle. (2023). Kaggle competitions overview. Retrieved from https://www.kaggle.com/competitions

DrivenData. (2023). DrivenData: Data science competitions for social good. Retrieved from https://www.drivendata.org/

Zindi. (2023). Zindi: The data science competition platform for Africa. Retrieved from https://zindi.africa/

Ethics, Reproducibility, and Competition Culture:

Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2185–2194). https://doi.org/10.18653/v1/D19-1224

Lipton, Z. C. (2018). The mythos of model interpretability. Communications of the ACM, 61(10), 36–43. https://doi.org/10.1145/3233231

Errata

Let us know.