Why Your Backtest Is Lying to You
The first time I ran a serious backtest, the numbers looked great. Dual moving average crossover on AAPL, 2020 to 2023, fast period 20, slow period 60. Ten trades, +67% return. I thought I had found something. I had not. I had found a curve-fitted pattern in one ticker over one window that told me almost nothing about the future.
The three things that made my results fake
The first was parameter overfitting. I had grid-searched over every combination of fast and slow period and picked the best one on the full dataset. Of course it looked good. I had essentially asked "which parameters would have been best in hindsight?" and used the answer to evaluate the strategy.
The second was no transaction costs. Real trades have commission and slippage. My framework defaults to 0.1% commission and 0.05% slippage per trade. Small numbers, but they compound. A strategy that looks like +67% gross can come in significantly lower after realistic costs are applied.
The third was in-sample evaluation. I optimized parameters on the full history, then measured performance on the same history. That is not validation. That is memorization.
Walk-forward: what actually works
The framework I built has a walk-forward splitter built in. You define a training window and a test window. The optimizer runs a grid search over the training window to find the best parameters, then evaluates those exact parameters on the held-out test window. Then it slides forward and repeats.
Out-of-sample performance is always lower than in-sample. If it is not, you are probably doing something wrong.
At the end of a walk-forward run you have out-of-sample results across multiple non-overlapping periods, not one big in-sample number. The Sharpe on my live framework sits at 0.74. That is the out-of-sample number, computed from the test windows, with full transaction costs. It is lower than the in-sample number, and that is the point.
What the system actually does
The data layer fetches OHLCV from Yahoo Finance with an Alpha Vantage fallback, caches everything in SQLite with TTL expiry so you are not hitting the API on every run, and validates the schema with Pydantic before anything touches the strategy engine. The strategies run on Backtrader. The analytics layer computes Sharpe, Sortino, max drawdown, win rate, and profit factor from the equity curve. QuantStats handles the tearsheet export.
I also built a Plotly Dash dashboard so you can change the ticker, date range, strategy, and parameters interactively and see the equity curve update live. The whole thing has 53 tests. I wanted to be able to trust the numbers.
What I took away from it
Backtesting is easy to do badly and hard to do well. The difference between a backtest that means something and one that does not comes down to a few decisions: are your parameters chosen in-sample or out-of-sample, are your costs realistic, is your evaluation set one you have never touched during development. Most published retail backtests fail at least one of those.