Detecting fraudulent meter manipulation for Tunisian utility STEG. Given 15 years of client billing history, predict which clients are committing fraud at a 5.6% positive rate. Key innovations: stochastic feature selection across 100 LightGBM iterations and a 14-model diversity ensemble with an elastic net meta-learner.
STEG, the Tunisian electricity and gas utility, lost 200 million Tunisian Dinars to fraudulent meter manipulation. Given a client's complete billing history — invoices, counter readings, consumption levels, over 15 years (2005–2019) — the task is to predict which clients are involved in fraud.
Three core challenges made this hard: invoice-level data had to be aggregated into client-level features before any modeling could begin, the fraud rate was only 5.6% (heavily imbalanced), and categorical variables had inconsistent levels between train and test sets that would silently break encoders if not handled explicitly.
Constants, library loading, reproducibility seeds.
Load client + invoice CSVs, join, and harmonize factor levels between train and test. Rare or test-only levels are merged into meaningful groups based on fraud rate rather than dropped.
Aggregate invoice-level rows into client-level features: consumption totals, invoice frequency, counter change patterns, billing gap statistics, seasonal splits.
Five encoding strategies for high-cardinality categoricals. 100 random feature subset iterations to identify the most impactful feature combinations.
3 LightGBM variants · 4 XGBoost variants · 3 CatBoost · H2O AutoML · Random Forest. Diversity across algorithm, hyperparameters, and random seed.
Elastic net trained on the 14 base model probability outputs. Learns the optimal blend weight for each base model.
Instead of greedy forward/backward selection, run 100 iterations where each randomly samples 12–35 features, trains a LightGBM, and records AUC. The features appearing in the top-5 performing subsets form the final set. This explores the feature interaction space stochastically — a single greedy search can get trapped in local optima where a jointly useful combination is never found by looking at each feature individually.
for (i in 1:100) {
set.seed(i)
m <- sample(num_features, sample(12:35, 1))
bst <- lgb.train(data = lgb.Dataset(x[, m], label = y), ...)
higher_auc_list[[i]] <- AUC(predict(bst, xt[, m]), ytest)
parameters_list[[i]] <- m
}
best_features <- Reduce(union, parameters_list[top_5_indices])
For the region variable (high cardinality), five encoding strategies were applied and kept as separate features: target mean P(fraud | region), WOE (Weight of Evidence), James-Stein shrinkage, M-estimator, and leave-one-out. The model then learns which encoding captures the most signal for each tree split — rather than committing to one encoding before the model has seen any data.
Train and test had mismatched factor levels — some values of counter_statue appeared in train but not test. Rather than dropping, rare levels were merged into meaningful groups based on their fraud rate. This preserves signal while ensuring predictions work on unseen category values.
Diversity was created across three axes: algorithm (LightGBM, XGBoost, CatBoost, H2O, RF), hyperparameters (deep/shallow, high/low learning rate, dart/gbdt booster), and random seed. The elastic net meta-learner on 14 probability outputs assigns higher weight to models that are both accurate and contribute unique signal.
# Meta-learner: elastic net on 14 base model predictions
meta_features <- cbind(lgbm1, lgbm2, lgbm3,
xgb1, xgb2, xgb3, xgb4,
cat1, cat2, cat3,
h2o_automl, rf1, rf2, rf3)
enet_meta <- train(meta_features, y_target, method = "glmnet")
At 5.6% fraud, a naive model predicts all zeros. Multiple strategies were tested: stratified undersampling (fraud proportion fixed while reducing majority), SMOTE, 50/50 undersampling for fast feature selection iteration, and full data with class weighting for final models. No single strategy was universally best — it depended on the base algorithm.
In a 20-hour hackathon, the two highest-leverage decisions were feature selection strategy and ensemble diversity. Standard greedy selection would have missed interaction-dependent features. Standard single-model optimization would have capped the ceiling. The stochastic subset search and 14-model blend are what pushed the result to Rank 6.
Design insight: Feature interaction spaces are too large for greedy search. Stochastic sampling across many random subsets is not just a computational shortcut — it explores combinations that sequential selection structurally cannot find.