ML Pipeline

Unified skill for the complete ML pipeline within a quant trading research system.
Consolidates eight prior skills into a single authoritative reference covering
the full lifecycle: data validation, feature creation, selection,
transformation, anti-leakage checks, pipeline automation, deep learning optimization, and deployment.

---

1. When to Use

Activate this skill when the task involves any of the following:

• - Creating, selecting, or transforming features for an ML-driven strategy.
• Auditing an existing feature pipeline for data leakage or overfitting risk.
• Automating an end-to-end ML pipeline (data prep through model export).
• Evaluating feature importance, scaling, encoding, or interaction effects.
• Integrating features with a feature store (Feast, Tecton, custom Parquet store).
• Explaining core ML concepts (bias-variance, cross-validation, regularisation)

in the context of feature engineering decisions.

---

2. Inputs to Gather

Before starting work, collect or confirm:

Input	Details
Objective	Target metric (Sharpe, accuracy, RMSE ...), constraints, time horizon.
Data

Symbols / instruments, timeframe, bar type, sampling frequency, data sources. |
| Leakage risks | Point-in-time concerns, survivorship bias, look-ahead in labels or features. |
| Compute budget | CPU/GPU limits, wall-clock budget for AutoML search. |
| Latency | Online vs. offline inference, acceptable prediction latency. |
| Interpretability | Regulatory or research need for explainable features / models. |
| Deployment target | Where the model will run (notebook, backtest harness, live engine). |

---

3. Feature Creation Patterns

3.1 Numerical Features

• - Interaction terms: price * volume, high / low, close - open.
• Rolling statistics: mean, std, skew, kurtosis over configurable windows.
• Polynomial / log transforms: log(volume + 1), spread^2.
• Binning / discretisation: equal-width, quantile-based, or domain-driven bins.

3.2 Categorical Features

• - One-hot encoding: for low-cardinality categoricals (sector, exchange).
• Target encoding: mean-target per category with smoothing (careful of leakage -- use only in-fold means).
• Ordinal encoding: when categories have a natural order (credit rating).

3.3 Time-Series Specific

• - Lag features: return_{t-1}, return_{t-5}, etc.
• Calendar features: day-of-week, month, quarter, options-expiry flag.
• Rolling z-score: (x - rolling_mean) / rolling_std for stationarity.
• Fractional differentiation: preserve memory while achieving stationarity (Lopez de Prado).

3.4 Feature Selection Techniques

• - Filter methods: mutual information, variance threshold, correlation pruning.
• Wrapper methods: recursive feature elimination (RFE), forward/backward selection.
• Embedded methods: L1 regularisation, tree-based importance, SHAP values.
• Permutation importance: model-agnostic; run on out-of-fold predictions.

---

4. Anti-Leakage Checks

Data leakage is the single most common cause of inflated backtest results.
Apply these checks at every pipeline stage:

4.1 Label Leakage

• - Labels must be computed from future returns relative to the feature

timestamp. Verify that the label window does not overlap the feature window.

• - Use purging and embargo when labels span multiple bars.

4.2 Feature Leakage

• - No feature may use information from time t+1 or later at prediction time t.
• Rolling statistics must use a closed left window: df['feat'].rolling(20).mean().shift(1).
• Target-encoded categoricals must be computed on the training fold only.

4.3 Cross-Validation Leakage

• - Use purged k-fold or walk-forward CV for time-series. Never use random

k-fold on ordered data.

• - Insert an embargo gap between train and test folds to prevent bleed-through

from autocorrelation.

4.4 Survivorship & Selection Bias

• - Ensure the universe of instruments at time t reflects what was actually

tradable at that time (delisted stocks, halted symbols removed later).

• - Backfill from point-in-time databases where available.

4.5 Validation Checklist

Run before every backtest:

CODEBLOCK0

---

5. Pipeline Automation (AutoML)

5.1 Prerequisites

• - Python environment with one or more AutoML libraries:

Auto-sklearn, TPOT, H2O AutoML, PyCaret, Optuna, or custom Optuna pipelines.

• - Training data in CSV / Parquet / database.
• Problem type identified: classification, regression, or time-series forecasting.

5.2 Pipeline Steps

Step	Action
1. Define requirements	Problem type, evaluation metric, time/resource budget, interpretability needs.
2. Data infrastructure

Load data, quality assessment, train/val/test split strategy, define feature transforms. | | 3. Configure AutoML | Select framework, define algorithm search space, set preprocessing steps, choose tuning strategy (Bayesian, random, Hyperband). | | 4. Execute training | Run automated feature engineering, model selection, hyperparameter optimisation, cross-validation. | | 5. Analyse & export | Compare models, extract best config, feature importance, visualisations, export for deployment. |

5.3 Pipeline Configuration Template

CODEBLOCK1

5.4 Output Artifacts

• - automl_config.py -- pipeline configuration.
• INLINECODE13 / .joblib / .onnx -- serialised model.
• INLINECODE16 -- fitted preprocessing + feature transforms.
• INLINECODE17 -- metrics, confusion matrix / residuals, feature rankings.
• INLINECODE18 -- prediction API code, input validation, requirements.txt.

---

6. Core ML Fundamentals (Feature-Engineering Context)

6.1 Bias-Variance Trade-off

• - More features increase model capacity (lower bias) but risk overfitting (higher variance).
• Use regularisation (L1/L2), feature selection, or dimensionality reduction to manage.

6.2 Evaluation Strategy

• - Walk-forward validation: the gold standard for time-series strategies.

Roll a fixed-width training window forward; test on the next out-of-sample period.

• - Monte Carlo permutation tests: shuffle labels and re-evaluate to estimate

the probability that observed performance is due to chance.

• - Combinatorial purged CV (CPCV): generate many train/test combinations with

purging for more robust performance estimates.

6.3 Feature Scaling

• - Fit scalers (StandardScaler, MinMaxScaler, RobustScaler) on the training set only.
• Apply the same fitted scaler to validation and test sets.
• RobustScaler is often preferred for financial data due to heavy tails.

6.4 Handling Missing Data

• - Forward-fill then backward-fill for price data (be aware of leakage on backfill).
• Indicator column for missingness can itself be informative.
• Tree-based models can handle NaN natively; linear models cannot.

---

7. Workflow

For any feature engineering task, follow this sequence:

1. 1. Restate the task in measurable terms (metric, constraints, deadline).
2. Enumerate required artifacts: datasets, feature definitions, configs, scripts, reports.
3. Propose a default approach and 1-2 alternatives with trade-offs.
4. Implement feature pipeline with anti-leakage checks built in.
5. Validate with walk-forward CV, Monte Carlo, and the leakage checklist above.
6. Deliver repo-ready code, documentation, and a run command.

---

8. Deep Learning Optimization

8.1 Optimizer Selection

Optimizer	Best For	Learning Rate
Adam	Most cases, adaptive	1e-3 to 1e-4
AdamW

Transformers, weight decay | 1e-4 to 1e-5 | | SGD + Momentum | Large batches, fine-tuning | 1e-2 to 1e-3 | | RAdam | Stability without warmup | 1e-3 |

8.2 Learning Rate Scheduling

• - OneCycleLR: Best for short training, fast convergence
• CosineAnnealing: Smooth decay, good generalization
• ReduceOnPlateau: Adaptive when validation loss plateaus
• Warmup + Decay: Standard for transformers

8.3 Regularization Techniques

• - Dropout: 0.1-0.5 for fully connected layers
• L2 (Weight Decay): 1e-4 to 1e-2
• Batch Normalization: Stabilizes training
• Early Stopping: Monitor validation loss, patience 5-10 epochs

8.4 PyTorch Lightning Integration

CODEBLOCK2

8.5 Financial Reinforcement Learning

• - State: Market features, portfolio state, position
• Action: Buy/Sell/Hold, position sizing
• Reward: Risk-adjusted returns (Sharpe, Sortino)
• Frameworks: Stable-Baselines3, RLlib, FinRL

---

9. Error Handling

Problem	Cause	Fix
AutoML search finds no good model	Insufficient time budget or poor features	Increase budget, engineer better features, expand algorithm search space.
Out of memory during training

Dataset too large for available RAM | Downsample, use incremental learning, simplify feature engineering. | | Model accuracy below threshold | Weak signal or overfitting | Collect more data, add domain-driven features, regularise, adjust metric. | | Feature transforms produce NaN/Inf | Division by zero, log of negative | Add guards: np.where(denom != 0, ...), np.log1p(np.abs(x)). | | Optimiser fails to converge | Bad hyperparameter ranges | Tighten search bounds, increase iterations, exclude unstable algorithms. |

---

10. Bundled Scripts

All scripts live in scripts/ within this skill directory.

Script	Purpose
INLINECODE22	Validate input data quality before pipeline execution.
INLINECODE23

Evaluate trained model performance and generate reports. |
| pipeline_deployment.py | Deploy a trained pipeline to a target environment with rollback support. |
| feature_engineering_pipeline.py | End-to-end feature engineering: load, clean, transform, select, train. |
| feature_importance_analyzer.py | Analyse feature importance (permutation, SHAP, tree-based). |
| data_visualizer.py | Visualise feature distributions and relationships to target. |
| feature_store_integration.py | Integrate with feature stores (Feast, Tecton) for online/offline serving. |

---

11. Resources

Frameworks

• - scikit-learn -- preprocessing, feature selection, pipelines.
• Auto-sklearn / TPOT / H2O AutoML / PyCaret -- automated pipeline search.
• Optuna -- flexible hyperparameter optimisation.
• SHAP -- model-agnostic feature importance.
• Feast / Tecton -- feature store management.
• PyTorch Lightning -- https://lightning.ai/docs/pytorch/stable/
• Stable-Baselines3 -- https://stable-baselines3.readthedocs.io/
• FinRL -- https://github.com/AI4Finance-Foundation/FinRL

Key References

• - Lopez de Prado, Advances in Financial Machine Learning (2018) -- purged CV, fractional differentiation, meta-labelling.
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning -- bias-variance, regularisation, model selection.
• scikit-learn user guide: feature extraction, preprocessing, model selection.

Best Practices

• - Always start with a simple baseline before running AutoML.
• Balance automation with domain knowledge -- blind search rarely beats informed priors.
• Monitor resource consumption; set hard timeouts.
• Validate on true out-of-sample holdout data, not just cross-validation.
• Document every pipeline decision for reproducibility.