Step 6 - End-to-end workflow (capstone)¶

Goal: tie steps 1 through 5 together on the packaged dummy_data.csv.

Workflow:

1. Split the series around the intervention (step 1).
2. Sanity-check the CV fold layout (step 2), making sure no fold touches the evaluation window.
3. Tune prophet_xgb via tune_model() on a small budget (step 3).
4. Fit the final model via run_single_its() using the tuned best_params (step 4).
5. Inspect the MBB CI fan (step 5) alongside the ATE summary.
6. Compare against ARIMA with compare_models().

Every stochastic step carries an explicit seed (LHS, MBB, model RNG) so the run is reproducible.

In [1]:

%matplotlib inline

import logging
import warnings
from pathlib import Path

import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

warnings.filterwarnings('ignore')
logging.basicConfig(level=logging.WARNING)

from its2s import run_single_its, tune_model, compare_models, time_series_cv

OUT_DIR = Path.cwd() / 'figures'
OUT_DIR.mkdir(exist_ok=True)

DATA = Path('data/dummy_data.csv')
INTERVENTION = pd.Timestamp('2022-03-15')
HOLDOUT_DAYS = 42

df = pd.read_csv(DATA, parse_dates=['ds']).sort_values('ds').reset_index(drop=True)
print(f'n={len(df)} rows, {df.ds.min().date()} -> {df.ds.max().date()}')
print(f'intervention={INTERVENTION.date()}')

%matplotlib inline import logging import warnings from pathlib import Path import matplotlib.dates as mdates import matplotlib.pyplot as plt import numpy as np import pandas as pd warnings.filterwarnings('ignore') logging.basicConfig(level=logging.WARNING) from its2s import run_single_its, tune_model, compare_models, time_series_cv OUT_DIR = Path.cwd() / 'figures' OUT_DIR.mkdir(exist_ok=True) DATA = Path('data/dummy_data.csv') INTERVENTION = pd.Timestamp('2022-03-15') HOLDOUT_DAYS = 42 df = pd.read_csv(DATA, parse_dates=['ds']).sort_values('ds').reset_index(drop=True) print(f'n={len(df)} rows, {df.ds.min().date()} -> {df.ds.max().date()}') print(f'intervention={INTERVENTION.date()}')

n=1576 rows, 2018-01-01 -> 2022-04-25
intervention=2022-03-15

6a. Split + CV sanity check¶

Confirm that no CV fold test window touches the run_single_its evaluation window [intervention - test_days, intervention).

In [2]:

# cv_end_date defaults to the start of the run's held-out test window,
# derived from the config's periods split (GH #40) -- the same split
# run_single_its uses below, so the cap matches the run's actual window.
cv_result = time_series_cv(
    df=df,
    intervention_date=INTERVENTION,
    model_name='arima',
    n_folds=3,
    test_obs=90,
    min_train_obs=365,
    skip_obs=0,
)
last_fold_end = max(f.test_end for f in cv_result.folds)
assert last_fold_end < cv_result.cv_end_date, 'CV fold leaked into evaluation window!'
print(cv_result.summary())
print(f'\nlast CV fold ends:   {last_fold_end.date()}')
print(f'derived cv_end_date: {cv_result.cv_end_date.date()}')
print(f'intervention_date:   {INTERVENTION.date()}')
print('-> No leakage.')

# cv_end_date defaults to the start of the run's held-out test window, # derived from the config's periods split (GH #40) -- the same split # run_single_its uses below, so the cap matches the run's actual window. cv_result = time_series_cv( df=df, intervention_date=INTERVENTION, model_name='arima', n_folds=3, test_obs=90, min_train_obs=365, skip_obs=0, ) last_fold_end = max(f.test_end for f in cv_result.folds) assert last_fold_end < cv_result.cv_end_date, 'CV fold leaked into evaluation window!' print(cv_result.summary()) print(f'\nlast CV fold ends: {last_fold_end.date()}') print(f'derived cv_end_date: {cv_result.cv_end_date.date()}') print(f'intervention_date: {INTERVENTION.date()}') print('-> No leakage.')

Cross-validation: arima (3 folds)
  RMSE: 6.7608 +/- 0.7267
  MAE:  5.8262 +/- 0.7137
  MAPE: 11.10%

last CV fold ends:   2019-09-27
derived cv_end_date: 2021-05-12
intervention_date:   2022-03-15
-> No leakage.

6b. Tuning prophet_xgb¶

Small n_trials + n_folds keeps this notebook fast. For publication-grade tuning use n_trials=100, n_folds=5, test_obs=365, min_train_obs=730, skip_obs=365.

In [3]:

tuning_result = tune_model(
    df=df,
    intervention_date=INTERVENTION,
    model_name='prophet_xgb',
    n_trials=8,
    n_folds=3,
    split_method='observations',
    test_obs=90,
    min_train_obs=365,
    skip_obs=0,
    metric='rmse',
    n_jobs=1,
    seed=42,
)
print(f'best CV RMSE: {tuning_result.best_rmse:.3f} +/- {tuning_result.best_std_rmse:.3f}')
print('best_params:'); display(tuning_result.best_params)

tuning_result = tune_model( df=df, intervention_date=INTERVENTION, model_name='prophet_xgb', n_trials=8, n_folds=3, split_method='observations', test_obs=90, min_train_obs=365, skip_obs=0, metric='rmse', n_jobs=1, seed=42, ) print(f'best CV RMSE: {tuning_result.best_rmse:.3f} +/- {tuning_result.best_std_rmse:.3f}') print('best_params:'); display(tuning_result.best_params)

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

INFO:prophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.

best CV RMSE: 7.172 +/- 5.423
best_params:

{'prophet': {'changepoint_prior_scale': 0.2557713605607101,
  'seasonality_prior_scale': 1.0430587071386823,
  'changepoint_range': 0.8597584299949586},
 'xgb': {'n_estimators': 60,
  'max_depth': 9,
  'learning_rate': 0.007220349783043519,
  'min_child_weight': 3,
  'subsample': 0.9968091371947913,
  'colsample_bytree': 0.6357730517504636,
  'gamma': 2.552152057841753}}

6c. Fit the final model with tuned params¶

In [4]:

from its2s.metrics.excess import calc_ate_summary

pr = run_single_its(
    df=df,
    intervention_date=INTERVENTION,
    model_name='prophet_xgb',
    config_overrides={
        'models': {'prophet_xgb': tuning_result.best_params},
        'bootstrap': {'n_sim': 200, 'block_length': 14},  # small n_sim for notebook speed
    },
)
ate = calc_ate_summary(pr.excess_table)
print('ATE summary:'); display(ate)
print('\ntest metrics:'); print(pr.metrics_test)
print('\nexcess (period):'); display(pr.excess_table.period_excess)

from its2s.metrics.excess import calc_ate_summary pr = run_single_its( df=df, intervention_date=INTERVENTION, model_name='prophet_xgb', config_overrides={ 'models': {'prophet_xgb': tuning_result.best_params}, 'bootstrap': {'n_sim': 200, 'block_length': 14}, # small n_sim for notebook speed }, ) ate = calc_ate_summary(pr.excess_table) print('ATE summary:'); display(ate) print('\ntest metrics:'); print(pr.metrics_test) print('\nexcess (period):'); display(pr.excess_table.period_excess)

WARNING:its2s.diagnostics:Ljung-Box test (p=0.0278) suggests significant residual autocorrelation for model 'prophet_xgb'. Bootstrap CIs may undercover.

ATE summary:

	metric	estimate	ci_lo	ci_hi	n_obs
0	Total ATE	347.279542	297.197239	398.350799	42
1	Mean ATE per obs	8.268561	7.076125	9.484543	42

test metrics:
MetricsResult(rmse=2.179646266316057, mae=1.7562491787486079, mape=3.383103044838158, mase=0.7604734811835387, mase_m=7, mase_denominator=2.3094154131651314)

excess (period):

	period	start_date	end_date	n_obs	total_observed	total_expected	total_excess	excess_ci_lo	excess_ci_hi	excess_pct
0	Full holdout	2022-03-15	2022-04-25	42	3029.800778	2682.521236	347.279542	297.197239	398.350799	12.946013

6d. Plot the tuned counterfactual with MBB CI fan¶

In [5]:

boot = pr.bootstrap_result

fig, ax = plt.subplots(figsize=(12, 4.5))
ax.plot(boot.dates, boot.actual, color='black', lw=1, label='observed')
ax.plot(boot.dates, boot.predicted, color='tab:blue', lw=1.5, label='counterfactual')
ax.fill_between(boot.dates, boot.conf_lo, boot.conf_hi, color='tab:blue', alpha=0.2, label='95% CI (MBB)')
ax.axvline(INTERVENTION, color='red', linestyle='--', label='intervention')
ax.set_title('Tuned prophet_xgb counterfactual with MBB CI')
ax.legend(loc='upper left')
ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m'))
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig(OUT_DIR / 'step6_tuned_counterfactual.png', dpi=150)
display(fig)

boot = pr.bootstrap_result fig, ax = plt.subplots(figsize=(12, 4.5)) ax.plot(boot.dates, boot.actual, color='black', lw=1, label='observed') ax.plot(boot.dates, boot.predicted, color='tab:blue', lw=1.5, label='counterfactual') ax.fill_between(boot.dates, boot.conf_lo, boot.conf_hi, color='tab:blue', alpha=0.2, label='95% CI (MBB)') ax.axvline(INTERVENTION, color='red', linestyle='--', label='intervention') ax.set_title('Tuned prophet_xgb counterfactual with MBB CI') ax.legend(loc='upper left') ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m')) ax.grid(True, alpha=0.3) plt.tight_layout() plt.savefig(OUT_DIR / 'step6_tuned_counterfactual.png', dpi=150) display(fig)

Key takeaways¶

1. The pieces compose cleanly. prepare_splits -> time_series_cv -> tune_model -> run_single_its -> MBB CI -> compare_models. No manual remapping between stages.
2. Leakage-free tuning is the default. cv_end_date is derived from the run's split config, keeping step-3 tuning honest; pass cv_end_date=intervention_date only when you deliberately want all pre-intervention data.
3. Seeds are load-bearing. LHS seed, bootstrap random_state, and any model-internal RNG must be set for results to be reproducible between sessions.
4. The notebook uses small budgets for speed. Production settings (n_trials=100, n_sim=1000, n_folds=5, skip_obs=365) will give tighter and more trustworthy CIs at the cost of runtime.