Ten weeks ahead, one horizon at a time: 1st place at the Entel 2022 forecasting datathon

Retail demand for telecom devices is a difficult signal to forecast. Most store-and-model series are short and intermittent: zero for weeks, then a burst of units when a model launches or a promotion lands. Any single series carries little information. The structure shared across thousands of series carries much more, and so does knowing which features will still mean the same thing once you move into the test window.

This post describes the solution that took first place in the second challenge of the Entel 2022 datathon. I worked on it with Renato Castro Cruz as team Insight ML. The method has three parts. Each forecast horizon gets its own gradient-boosted model, built only from information that exists at forecast time. Adversarial validation is used to audit which features behave differently in the test period. The final assembly step is conservative and anchors new predictions to weekly totals and to an earlier strong submission. Everything below comes from the public repository, and every number is one recorded there. The repository has cleared notebook outputs and does not include the proprietary data, so I quote only scores that appear in the code or its comments.

The problem

The raw training table is wide. Each row is a combination of five anonymized descriptors (brand, product range, model, department, point of sale), followed by weekly unit columns SEMANA_01 … SEMANA_50. The test table has the same descriptors and ten placeholder week columns, SEMANA_51 … SEMANA_60, filled with the value 99. Preprocessing replaces the hashed descriptor codes with frequency-ranked labels such as MOD_1 and PVENT_1, and keeps a reverse mapping so submissions can be translated back. It then melts both tables into long format and assigns calendar dates: training weeks start on Monday 2021-05-17, and test weeks start on Monday 2022-05-02, immediately after the last training week.

A series is identified by the triple $i = (m, p, g)$: model $m$, point of sale $p$ and product range $g$. Rows that share this key are summed, and brand and department are kept as attributes of the series. Let $y_{i,t} \ge 0$ be the units of series $i$ in week $t$, and let $T = 50$ be the last observed week. The task is to predict

for every series in the test file. Submissions are scored by the root mean squared error over the set $\mathcal{T}$ of all test pairs $(i, t)$:

The code computes this with mean_squared_error(..., squared=False). RMSE is dominated by large errors, so the series that sell tens or hundreds of units in a week matter far more than the long tail of near-zero series. The per-week diagnostic plots in the notebooks reflect this: next to every scatter of predicted against actual values, they show separate box plots for weeks where actual demand is above 10 and above 100.

Evaluation and validation

A horizon-aware feature set

The most important design decision affects validation as much as modeling. The feature grid is built by stacking training and test weeks, with test demand set to zero. A lag feature $y_{i,t-\ell}$ evaluated at a test week $T+h$ therefore holds a real value only when $\ell \ge h$. For $\ell < h$ it reads one of the zero placeholders. A model trained with lag 1 and then asked to predict week $T+3$ sees inputs that never occurred during training.

The solution uses a direct multi-horizon strategy. For each horizon $h$ (the SHIFT parameter), the function get_feature_names_shift keeps a time-dependent feature only if its shift $\ell$ satisfies

and keeps every feature that does not depend on time. The model for horizon $h$ is

where the three lags are $y_{i,T+h-\ell}$ for $\ell \in \{h, h+1, h+2\}$, $\mathbf{r}_{i,h}$ are rolling statistics whose shifts lie in the same range, and $\mathbf{x}_i$ are static attributes and encodings. Every input to $f_h$ is observed when the forecast is made.

Time-based holdout plus in-window cross-validation

The main training notebook (02.01-Training-3models) combines two schemes:

• Temporal holdout. Training rows come from $[\text{2021-08-01}, \text{2022-03-01})$. The holdout is $[\text{2022-03-01}, \text{2022-05-01})$, the last eight weeks of history. This mimics the real setup: fit on the past and score on later weeks.
• K-fold inside the training window. Within the training window, KFold(n_splits=3, shuffle=True, random_state=100) produces out-of-fold predictions and a CV RMSE. Each fold model also predicts the holdout and the test set, and these predictions are averaged over folds.

If $f^{(k)}_h$ is the model trained without fold $k$, then the holdout and test predictions are $\hat y = \frac{1}{K} \sum_{k=1}^{K} f^{(k)}_h(\mathbf{x})$ with $K = 3$. The CV score uses the out-of-fold predictions, and the holdout score uses the fold-averaged predictions. The per-library notebooks also contain purely temporal train/validation/test splits; one XGBoost run, for example, trains up to 2022-02-28, validates on March 2022 and tests on April 2022.

Adversarial validation

Random K-fold cannot detect a feature whose distribution moves between the training period and the test period. Adversarial validation checks for that shift directly. Label every training row $z = 0$ and every test row $z = 1$, train a classifier $g(\mathbf{x}) \approx P(z = 1 \mid \mathbf{x})$, and measure its ROC AUC on held-out rows:

If $\mathrm{AUC} \approx 0.5$, the classifier cannot tell the two periods apart, and so the features carry no period signature. An AUC well above 0.5 means some features encode when a row comes from rather than what it is, and the gain importances of $g$ show which ones.

In 02.04-Adversarial, $g$ is a LightGBM binary classifier (num_leaves=32, learning_rate=0.2, lambda_l1=1, scale_pos_weight=8 for the class imbalance, up to 1000 rounds with early stopping after 100) trained on a shuffled 80/20 split of the stacked train and test rows. The notebook keeps the 30 most discriminative features of one iteration in a comment, and they include item_id, brand, point of sale, range, calendar flags such as is_month_start, tm_wm and tm_m_end, several rolling means, and long lags. Most of these columns then appear in the notebook's drop list. Src/utils/adversarial_validation.py adds two more tools: a per-feature audit that trains a shallow classifier (max_depth=2, num_leaves=4, learning_rate=0.02) on each column alone under 5-fold CV and reports the mean AUC for that column, and an out-of-fold adversarial scorer.

The final feature set follows this audit only in part. All calendar features (year, month, day, day_of_week, day_of_year, the month-start, month-end and quarter-start flags, and the derived week-of-month indicators) are excluded from training, with one exception: week_of_month, which is kept as a categorical. The series identifiers are kept, because the holdout preferred them. A comment in the training notebook records holdout RMSE 2.766 without identifiers and 2.731 with them. I treated adversarial validation as a source of hypotheses to test against the time holdout, not as an automatic filter.

Feature engineering

The features are built in 01.02-Preprocessing on the stacked train and test grid, sorted by series and date, and are merged in 01.03-MakeDataset.

Lags. $y_{i,t-\ell}$ for $\ell = 1, \dots, 15$ (Demanda_lag_shift_ℓ), stored as float16. The horizon filter above keeps three of them for any given model.

Rolling statistics. For shift $d \in \{1, \dots, 16\}$ and window $w \in \{2, 4\}$:

These are computed with groupby(keys)[y].transform(lambda x: x.shift(d).fillna(0).rolling(w, min_periods=1)), using population standard deviation (ddof=0). There are 16 grouping keys: the series id, each of the five descriptors alone, and all ten descriptor pairs. That gives $16 \times 2 \times 16 \times 2 = 1024$ columns, computed in parallel with joblib (8 workers). Because the grid is sorted by series and then by date, a shift inside a coarser group mostly reads earlier weeks of the same series. These columns should therefore be read as close relatives of item-level windows, not as true group aggregates.

Target encodings. The mean, standard deviation and maximum of demand per group, for the 15 descriptor keys above plus the full (model, point of sale, range) triple, computed after masking every week beyond the training range: 48 columns. They describe a series' typical level and volatility over the whole history.

Categoricals and calendar. Model, point of sale, range, brand and department are passed as categorical features. The calendar features described in the previous section are computed but not used for training, apart from week_of_month. A "release week" feature, the first week with positive demand, is also computed and then excluded.

Feature selection. Each horizon is trained twice. Version c_model_v2 uses the full horizon-filtered feature set. Version c_model_v3 uses only the top N_FEATURE_IMPORTANCE = 15 features, ranked by importance averaged over the folds of the first run.

Models and ensemble

All three libraries go through a common interface (Src/utils/LGBM.py, XGBOOST.py, CATBOOST.py) and a shared training loop (Src/training.py). That loop handles folds, holdout scoring, feature-importance plots and the submission files. Every model clips its predictions at zero, $\hat y \leftarrow \max(\hat y, 0)$.

The Tweedie objective suits zero-inflated, non-negative counts. With variance power $1 < p < 2$, it models $\operatorname{Var}(y) \propto \mu^{p}$, a compound Poisson-gamma distribution that puts positive mass at exactly zero. The per-library notebooks explore it extensively. The LightGBM notebook records holdout RMSE 2.509 with $p = 1.03$ and 2.456 with $p = 1.07$. The LightGBM wrapper also contains a Bayesian hyperparameter search (scikit-optimize BayesSearchCV with a Gaussian-process surrogate, 60 candidates, Tweedie DART regressor), but it is disabled by default.

Blending. The assembly notebook (02.02-Ensemble-Models) builds the model forecasts for horizons $h \in \{1, 2\}$. For each horizon it gathers the fold-averaged submissions of every library and both feature-set versions that exist on disk. That is a set $\mathcal{M}_h$ of up to six runs, combined with equal weights:

Assembly with a reference submission. The same notebook loads one of the team's earlier submission files (Submission_73.csv) as a reference $y^{\text{ref}}$. The code calls it Demanda_real, but it is a submission, not ground truth. A sibling notebook loads another one under the name df_demanda_best_submission. Weeks not covered by the direct models are taken from this reference. For the covered weeks, three steps follow. Predictions below 0.05 are set to zero. Predictions are rescaled so that each week's total matches the reference total:

and the result is blended with the reference:

with $\alpha = 0.35$ for week 51 and $\alpha = 0.2$ for week 52 (the week-52 blend is applied before rescaling). The new models therefore act as a correction to an existing forecast: they redistribute demand across series within the near-term weeks while the weekly totals stay fixed. The notebook writes several candidate files (with and without rescaling, with different weights), and the repository does not record which one was submitted last.

Architecture

flowchart LR
  A["Wide CSV<br/>SEMANA_01..60"] --> B["Melt + encode IDs<br/>week dates"]
  B --> C["Sum per model,<br/>POS, range"]
  C --> D["Train+test grid"]
  D --> E["Lags 1-15"]
  D --> F["Rolling mean/std<br/>shift x window x group"]
  D --> G["Target encodings"]
  E --> H["Merge features"]
  F --> H
  G --> H
  H --> I["Horizon filter<br/>h to h+2"]
  J["Adversarial<br/>validation"] -.-> I
  I --> K["LightGBM"]
  I --> L["XGBoost"]
  I --> M["CatBoost"]
  K --> N["3-fold CV<br/>+ time holdout"]
  L --> N
  M --> N
  N --> O["Equal-weight<br/>average per week"]
  O --> P["Zero small, rescale,<br/>blend with reference"]
  P --> Q["Submission"]

Results

The solution placed 1st on the final leaderboard. The repository does not contain the public or private leaderboard scores, so I do not quote any.

What the repository does contain are local results left in notebook comments. They are single runs on different splits and configurations, so each row is only meaningful against the row it was compared with:

Two patterns appear. First, the identifiers help on the time holdout even though adversarial validation flags them as period-discriminative. Second, the target encodings lower XGBoost's training error (1.763 against 2.129) but raise its April error. CatBoost, which handles categoricals with its own ordered statistics, gains slightly from the same encodings.

What did not work

Only experiments the repository records as unsuccessful are listed here.

• Share-of-week target. One variant (01.01-Data-Creation-ratios) models each series' percentage of that week's total, $s_{i,t} = 100\, y_{i,t} / \sum_j y_{j,t}$, and converts back to units by renormalizing the predicted shares and multiplying by a weekly total taken from an earlier submission. A comment in the training notebooks labels a ratio run as the "bad version" (version mala de ratios).
• Target encodings for XGBoost. The XGBoost notebook concludes in a comment that the encodings add noise (ENCODINGS GENERAN RUIDO), supported by the train/test split shown in the table above.

Lessons

1. Build horizon-aware inputs. With a stacked grid, any lag shorter than the horizon reads a placeholder at test time. Restricting model $f_h$ to shifts $\ell \in [h, h+2]$ removes this mismatch by construction, at the cost of training one model per horizon.
2. Treat adversarial validation as hypothesis generation. The adversarial classifier pointed at calendar flags, identifiers and some rolling windows. Removing the calendar features followed naturally, since in a 50-week history they mostly encode position in time. Removing the identifiers would have cost accuracy on the time holdout. The holdout settles what the AUC can only suggest.
3. Validate leakage-prone features per learner. Target encodings computed over the full training history improved the training fit of XGBoost while worsening its April holdout, yet helped CatBoost slightly. The same feature can behave differently depending on how a learner already handles categoricals.
4. Match the loss to the data. Intermittent non-negative counts favour Tweedie objectives with variance power just above 1. The documented LightGBM comparison moved in that direction ($p = 1.07$ beat $1.03$), and every model clips its output at zero.
5. Name references honestly. The assembly code compares predictions against a column called Demanda_real that is in fact an earlier submission. A "referential RMSE" against such a file measures agreement with that file, not accuracy. Shrinking toward a strong earlier forecast is a valid way to reduce variance, but the metric has to be read for what it is.

Reproducibility

The code is in CristianLazoQuispe/Datathon-Entel-2022. The notebooks in Codes/ run in numeric order: 00.00 (melt and encode) → 01.01 (aggregate per series; a ratio variant also exists) → 01.02 (lags, rolling windows, encodings) → 01.03 (merge into per-shift training tables) → 02.01 (per-library or combined training for a given SHIFT) → 02.02-Ensemble-Models (assembly) → 02.04 (adversarial validation). Shared code lives in Src/ (training.py, the three model wrappers, feature engineering, preprocessing and submission analysis).

Some practical notes:

• Data. The competition data belongs to Entel and is not redistributed. The notebooks expect Data/train/train.csv and Data/test/test.csv, and they write intermediate pickles under Results/.
• Seeds. SEED = 42 for NumPy, Python and the model libraries, and random_state=100 for the cross-validation folds.
• Library versions. The code uses fit-time early_stopping_rounds and XGBoost's ntree_limit, both removed in later major releases of LightGBM and XGBoost. Pin pre-2023 versions or port those calls. The XGBoost and CatBoost wrappers are configured for GPU.
• Paths. The reverse ID mapping is written to Results/reverse_dict_mapping_list.txt but read from utils/reverse_dict_mapping_list.txt in the training notebooks, so one of the two paths has to be aligned.
• Notebook state. Some cells are stored as raw (disabled), and the training loop was last left at a single SHIFT. To reproduce the assembly, train horizons 1 and 2 for all three libraries and both feature-set versions.