Kaggle Wins

In the summer of 2022 I spent my evenings on two very different Kaggle competitions: one about predicting credit default from tabular time series, the other about judging the quality of argumentative essays. Both ended with medals, and both were team efforts. Here's what each solution did.

Competition	Finish	Medal	Team
American Express – Default Prediction	16th	Gold	Shivam Gupta, Priyanshu Chaudhary, me
Feedback Prize – Predicting Effective Arguments	14th	Silver	Abhishek Thakur, Bhavya Dhingra, Shivam Gupta, Rai, me

American Express: diversity over a single best model

The task was to predict whether a customer would default, from monthly statement profiles. Our 16th-place solution leaned on two ideas: varied feature sets, and a diverse ensemble.

A blank payment card standing at the front of a row of glass panes, one per monthly statement, each marked with a data point, and a smooth blue rod threaded through them that rises toward the card — Each customer is a card and the statements behind it, one per month. The features summarise that history: means, spreads, last values, differences, and Hull moving averages like the smooth line threaded through the monthly points.

Feature engineering

Mean, standard deviation and last-value features were common to every model. On top of that we trained in three ways:

only Hull moving average (HMA) features,
only difference features,
HMA and difference features together, which only worked with CatBoost.

Using every difference feature turned out to be the wrong call for some models: for neural networks and XGBoost they introduced leakage, and cross-validation stopped correlating with the leaderboard. HMA features proved much better than differences.

Models

Model	CV	Private LB	Public LB	Features
LightGBM	0.7973	0.80687	0.79906	Diff + last
XGBoost	0.7972	0.80639	0.79718	HMA + last
CatBoost	0.7952	0.80468	0.79614	HMA + diff + last
NN 1	0.7923	0.80190	0.79240	HMA + last
NN 2	0.7921	0.80186	0.79188	Diff + last
TabNet	0.7933	–	–	HMA + last

Ensemble

Cross-validation and the leaderboard tracked each other well, so we used Optuna to pick ensemble weights and submitted a rank ensemble. A few strong public models had no out-of-fold predictions, so their weights were set by hand.

In hindsight, our best submission (0.80839) wasn't one we picked, and giving more weight to TabNet and LightGBM would have landed us 13th. Things we didn't get to try: predicting the missing values of B_29 and using them as features, LightGBM on HMA features, and pseudo-labelling on the private test data.

Thanks to the community notebooks from Chris Deotte, Raddar, Robert Hatch, Jiwei Liu and Ragnar that shaped the whole competition. Original write-up ↗

Feedback Prize: one model, one clever embedding

Here the task was to rate each argumentative element in a student essay (claim, evidence, rebuttal and so on) as ineffective, adequate or effective. Our 14th-place result came from a single strong model, run with a few variations and averaged.

Seven paper strips laid side by side, each mounted on a card of a different colour, with a small white token at the end of each showing one, two or three blue dots — One essay, cut into its argument. Each element (lead, position, claim, evidence, counterclaim, rebuttal, conclusion) gets its own colour, and the model rates how well each one works: one dot for ineffective, two for adequate, three for effective.

Domain pre-training

First, DeBERTa-v3-large was further pre-trained as a masked language model on all the essays from the previous Feedback competition, reaching a perplexity of about 4.5.

Framing it as token classification

Inspired by a public kernel from Nicholas Broad, each essay became one sequence, with special tokens marking where each discourse element starts and ends:

[CLS] some text [CLS_CLAIM] the claim text [END_CLAIM] ... [SEP]

Every token gets a discourse-type id. Only the start and end tokens of each element carry an effectiveness label; everything else is ignored in the loss.

The lift: a discourse-type embedding

The change that helped most was adding a second embedding table for discourse types inside DeBERTa's embedding layer and summing it with the word embeddings, so every token knows what kind of argument it belongs to:

self.disc_type_embeddings = nn.Embedding(9, self.embedding_size)
...
embeddings += self.disc_type_embeddings(disc_type_ids)

Training used AdamW with a polynomial learning-rate schedule over five folds. At inference, the probabilities at each element's start and end tokens were averaged to get its label.

The write-up and full code were published by Abhishek Thakur, using his training library tez. Original write-up ↗