{ "site": "quants.wiki", "section": "Backtest validity", "url": "https://quants.wiki/backtest/", "reviewed": "2026-08-27", "license": "CC BY 4.0", "changes": "https://quants.wiki/changes.json", "tables": [ { "title": "Expected maximum of N independent standard normals", "intro": "The Bailey and Lopez de Prado approximation uses the Euler-Mascheroni constant gamma = 0.5772156649: E[max] = (1-gamma)*Z^-1(1 - 1/N) + gamma*Z^-1(1 - 1/(N*e)). This is the expected best t-statistic from N independent trials of a strategy with no edge at all. The last column shows the cruder sqrt(2*ln(N)) approximation for comparison.", "columns": [ "N trials", "Z^-1(1 - 1/N)", "Z^-1(1 - 1/(N e))", "E[max Z]", "sqrt(2 ln N)" ], "rows": [ [ "2", "0.000000", "0.900453", "0.519755", "1.177410" ], [ "5", "0.841621", "1.449666", "1.192594", "1.794123" ], [ "10", "1.281552", "1.789242", "1.574598", "2.145966" ], [ "20", "1.644854", "2.088110", "1.900708", "2.447747" ], [ "45", "2.009875", "2.401003", "2.235640", "2.759225" ], [ "100", "2.326348", "2.680210", "2.530603", "3.034854" ], [ "500", "2.878162", "3.180243", "3.052528", "3.525509" ], [ "1000", "3.090232", "3.375895", "3.255122", "3.716922" ], [ "10000", "3.719016", "3.964416", "3.860665", "4.291932" ], [ "100000", "4.264891", "4.483073", "4.390829", "4.798526" ], [ "1000000", "4.753424", "4.951679", "4.867860", "5.256522" ] ] }, { "title": "Expected maximum spurious annualised Sharpe ratio, and trials needed for a Sharpe of 2", "intro": "For a true Sharpe ratio of zero with iid normal monthly returns, the estimated annualised Sharpe ratio has a standard deviation of sqrt(12/T). The expected maximum over N independent trials is that standard deviation times E[max Z]. The right-hand column inverts the relationship: how many independent trials are needed before the expected best result is an annualised Sharpe ratio of 2.", "columns": [ "Sample length T, months", "SD of estimated annualised SR", "Best of 10 trials", "Best of 100", "Best of 1,000", "Best of 10,000", "Trials for an expected best SR of 2" ], "rows": [ [ "24 (2 years)", "0.707107", "1.1134", "1.7894", "2.3017", "2.7299", "approximately 243" ], [ "60 (5 years)", "0.447214", "0.7042", "1.1317", "1.4557", "1.7265", "approximately 1.46 x 10^5" ], [ "120 (10 years)", "0.316228", "0.4979", "0.8002", "1.0294", "1.2208", "approximately 4.4 x 10^9" ], [ "240 (20 years)", "0.223607", "0.3521", "0.5659", "0.7279", "0.8633", "approximately 3 x 10^18" ] ] }, { "title": "Minimum backtest length in years", "intro": "The number of years of data required before the observed Sharpe ratio exceeds what N independent trials would produce by chance. Exact column uses (E[max Z_N]/SR_ann)^2; the approximation column uses 2*ln(N)/SR_ann^2 as given in the source. Below the minimum backtest length, an in-sample Sharpe ratio at that level is not evidence of anything.", "columns": [ "N trials", "Target annualised Sharpe", "Minimum years, exact", "Minimum years, 2 ln N / SR^2" ], "rows": [ [ "10", "1.0", "2.4794", "4.6052" ], [ "45", "1.0", "4.9981", "7.6133" ], [ "100", "1.0", "6.4040", "9.2103" ], [ "1000", "1.0", "10.5958", "13.8155" ], [ "10000", "1.0", "14.9047", "18.4207" ], [ "45", "0.5", "19.9923", "30.4533" ], [ "45", "1.5", "2.2214", "3.3837" ], [ "45", "2.0", "1.2495", "1.9033" ] ] }, { "title": "Deflated Sharpe ratio of the base series against trial count", "intro": "The base series has a monthly Sharpe ratio of 0.272400 over T = 24 observations, with skewness -0.508966 and kurtosis 2.368633. The deflated Sharpe ratio is the probabilistic Sharpe ratio evaluated against a threshold SR0 = sd(trial Sharpes) * E[max Z_N]. The table assumes a standard deviation of 0.15 in the monthly Sharpe ratios across trials, an input that must be measured from the actual trial set rather than assumed.", "columns": [ "N trials", "E[max Z]", "Threshold SR0, monthly", "Deflated Sharpe ratio" ], "rows": [ [ "1 (no selection, PSR against zero)", "0.000000", "0.000000", "0.887023" ], [ "2", "0.519755", "0.077963", "0.806286" ], [ "4", "1.052123", "0.157818", "0.694739" ], [ "6", "1.300141", "0.195021", "0.634561" ], [ "10", "1.574598", "0.236190", "0.563936" ], [ "20", "1.900708", "0.285106", "0.477480" ], [ "30", "2.073392", "0.311009", "0.431868" ], [ "100", "2.530603", "0.379590", "0.316870" ], [ "1000", "3.255122", "0.488268", "0.168640" ] ] }, { "title": "Combinatorial purged cross-validation: splits and backtest paths", "intro": "With the sample divided into N contiguous groups and k of them held out in each test set, there are C(N,k) train-test splits and phi = C(N,k)*k/N distinct out-of-sample paths that can be assembled from them. Walk-forward is the single-path special case.", "columns": [ "Groups N", "Test groups k", "Splits C(N,k)", "Backtest paths", "Train groups per split" ], "rows": [ [ "6", "2", "15", "5", "4" ], [ "8", "2", "28", "7", "6" ], [ "10", "2", "45", "9", "8" ], [ "12", "3", "220", "55", "9" ], [ "16", "2", "120", "15", "14" ], [ "20", "2", "190", "19", "18" ], [ "10", "5", "252", "126", "5" ], [ "any N", "k = N (all held out)", "1", "1", "0, no training data" ], [ "walk-forward", "-", "N-1 sequential fits", "1", "expanding or rolling" ] ] }, { "title": "Purge, embargo, and the cost in training observations", "intro": "When a label at bar t depends on prices up to bar t+h, any training observation whose label window overlaps the test set leaks information. Purging removes those observations; an embargo removes a further block after the test set to break serial dependence in the features. The cost is stated as a fraction of the training set.", "columns": [ "Total bars T", "Folds", "Test bars per fold", "Label span h", "Embargo", "Bars removed from training", "Percent of training set" ], "rows": [ [ "1000", "5", "200", "20", "20", "up to 40", "5.0000" ], [ "1000", "5", "200", "50", "50", "up to 100", "12.5000" ], [ "2520", "10", "252", "20", "25", "up to 45", "1.9841" ], [ "1000", "5", "200", "0", "0", "0", "0.0000" ], [ "252", "5", "50", "20", "20", "up to 40", "19.8020" ] ] } ], "entries": [ { "id": "multiple-testing-selection-bias", "term": "Why a backtest result is not an estimate", "definition": "The Sharpe ratio of a chosen strategy is the maximum over the configurations that were tried, and the maximum of N draws has a higher expectation and a smaller variance than a single draw. Reporting it as if it were a single estimate is a category error, not a small bias.", "formula": "If N independent configurations of a true-zero strategy are tested and each produces an estimated Sharpe ratio with standard deviation s, the expected best result is s * E[max Z_N], with E[max Z_N] = (1-gamma)*Z^-1(1-1/N) + gamma*Z^-1(1-1/(N e)) and gamma = 0.5772156649", "fields": [ { "label": "Worked", "value": "With T = 60 monthly observations, the estimated annualised Sharpe ratio of a true-zero strategy has a standard deviation of sqrt(12/60) = 0.447214. Over 100 independent trials the expected best is 0.447214 * 2.530603 = 1.1317", "mono": true }, { "label": "Growth in N", "value": "E[max Z] grows roughly as sqrt(2 ln N), so the selected result improves slowly and relentlessly: 1.574598 at N = 10, 2.530603 at 100, 3.255122 at 1,000, 3.860665 at 10,000", "mono": true }, { "label": "What counts as a trial", "value": "Every parameter value examined, every universe filter, every rebalancing frequency, every stop level, and every variant discarded on the way. The count is the size of the search, not the number of strategies presented", "mono": true }, { "label": "The uncomfortable case", "value": "Over 5 years of monthly data, approximately 1.46 x 10^5 independent trials give an expected best annualised Sharpe ratio of 2. Over only 2 years, approximately 243 trials do", "mono": true } ], "notes": [ "N is almost never known, and it is always larger than the number the researcher reports. A grid search over three parameters with ten values each is a thousand trials whether or not the researcher thinks of it that way, and every abandoned data-cleaning decision adds more.", "The independence assumption cuts in the researcher's favour. Correlated trials - neighbouring parameter values on a smooth surface - have a lower expected maximum than independent ones, so treating a grid search as N independent trials overstates the correction. The right adjustment is to an effective number of independent trials, which is smaller and hard to estimate.", "The approximation is asymptotic in N and poor at small N. At N = 2 it gives 0.519755 against the exact value of 1/sqrt(pi) = 0.564190, an 8 percent understatement. It is accurate enough for the N values that matter.", "No amount of out-of-sample testing fixes this if the out-of-sample period was consulted during development. The correction applies to the total number of times the data influenced a decision." ], "source": "Bailey, Borwein, Lopez de Prado and Zhu 2014" }, { "id": "expected-maximum-sharpe", "term": "Expected maximum Sharpe ratio from N trials", "definition": "The specific number a backtest has to beat. It is the standard deviation of the Sharpe estimator under the null multiplied by the expected maximum of N standard normals, and it is computable from three inputs: the sample length, the periodicity and the trial count.", "formula": "E[max SR_ann] = sqrt(P/T) * E[max Z_N] under a true Sharpe ratio of zero and iid normal returns, with P periods per year and T observations", "fields": [ { "label": "Worked, base series length", "value": "T = 24 monthly, so sqrt(12/24) = 0.707107. Best of 10 trials: 0.707107 * 1.574598 = 1.1134. Best of 100: 1.7894. Best of 1,000: 2.3017", "mono": true }, { "label": "Worked, five years", "value": "T = 60, sqrt(12/60) = 0.447214. Best of 10: 0.7042. Best of 100: 1.1317. Best of 1,000: 1.4557. Best of 10,000: 1.7265", "mono": true }, { "label": "The base series in context", "value": "The base series has an observed annualised Sharpe ratio of 0.943622 over 24 months. That is below the expected best of just 10 independent trials on 24 months of no-edge data, which is 1.1134", "mono": true }, { "label": "Length dominates", "value": "Doubling T divides the null standard deviation by sqrt(2). Multiplying N by 100 multiplies E[max Z] by roughly 1.5 at these magnitudes. Sample length is the more powerful lever by a wide margin", "mono": true } ], "notes": [ "The comparison for the base series is the whole point of publishing it. A 0.94 annualised Sharpe ratio over two years does not clear ten trials, let alone the search that actually produced it. Nothing about the strategy is being criticised; the sample is simply too short to say anything.", "The null assumes iid normal returns, which makes the standard deviation sqrt(P/T) rather than the more general sqrt((1 + SR^2/2)*P/T). At SR = 0 those coincide exactly, which is convenient: the null is the case where the simple form is correct.", "Autocorrelated returns inflate the null standard deviation, so a strategy on smoothed data faces a higher threshold than this, not a lower one." ] }, { "id": "probabilistic-sharpe-ratio", "term": "Probabilistic Sharpe ratio", "definition": "The probability that the true Sharpe ratio exceeds a stated threshold, given the observed Sharpe ratio, the sample length, and the skewness and kurtosis of the returns. It converts a point estimate into a probability statement and it is the machinery the deflated Sharpe ratio is built on.", "formula": "PSR(SR*) = Z[ (SR_hat - SR*) * sqrt(T - 1) / sqrt(1 - gamma3*SR_hat + ((gamma4 - 1)/4)*SR_hat^2) ], with SR_hat and SR* per period, gamma3 skewness and gamma4 kurtosis (3 for a normal), and Z the standard normal CDF", "fields": [ { "label": "Worked, threshold zero", "value": "Base series: SR_hat = 0.272400 monthly, T = 24, gamma3 = -0.508966, gamma4 = 2.368633. Denominator = sqrt(1 + 0.138642 + 0.342158*0.074202) = 1.078903. z = 0.272400*4.795832/1.078903 = 1.210846. PSR(0) = 0.887023", "mono": true }, { "label": "Normal-moment comparison", "value": "With gamma3 = 0 and gamma4 = 3 the same data gives z = 1.282805 and PSR(0) = 0.900220. The observed higher moments cost 1.3 percentage points of confidence", "mono": true }, { "label": "Consistency with the standard error", "value": "At gamma3 = 0 and gamma4 = 3 the denominator becomes sqrt(1 + SR^2/2), so PSR is the normal-theory t-test on the Sharpe ratio with T-1 rather than T. The two are the same statement", "mono": true }, { "label": "Interpretation", "value": "PSR(0) = 0.887 means an 11.3 percent probability that the true Sharpe ratio is at or below zero, for a single untried strategy. It says nothing about a selected one", "mono": true } ], "notes": [ "Note the kurtosis term uses (gamma4 - 1)/4, with gamma4 the full kurtosis, not excess. Substituting excess kurtosis is a common implementation error and it inflates PSR: on the base series it would give a denominator of sqrt(1 + 0.138642 - 0.030263) = 1.052557 rather than sqrt(1 + 0.138642 + 0.025391) = 1.078903, raising PSR(0) from 0.887023 to 0.895998.", "Negative skewness raises the denominator here, because the -gamma3*SR_hat term is positive when gamma3 is negative. It therefore lowers PSR, which is the correct direction: negative skewness makes a given Sharpe ratio less trustworthy.", "PSR against a zero threshold is not a defence against overfitting. It corrects for sample length and higher moments only. The threshold has to be raised to account for selection, which is what the deflated Sharpe ratio does." ], "source": "Bailey and Lopez de Prado 2012, The Sharpe Ratio Efficient Frontier" }, { "id": "deflated-sharpe-ratio", "term": "Deflated Sharpe ratio", "definition": "The probabilistic Sharpe ratio evaluated against a threshold set by the number of trials rather than against zero. It answers the question a backtest actually poses: given that this was the best of N attempts, what is the probability the true Sharpe ratio is positive.", "formula": "DSR = PSR(SR0) where SR0 = sd(SR across the N trials) * [ (1-gamma)*Z^-1(1 - 1/N) + gamma*Z^-1(1 - 1/(N e)) ]. All Sharpe ratios per period, not annualised", "fields": [ { "label": "Worked, N = 10", "value": "Base series SR_hat = 0.272400 monthly. Assume the 10 trial Sharpe ratios have a standard deviation of 0.15 monthly. E[max Z_10] = 1.574598, so SR0 = 0.15*1.574598 = 0.236190. z = (0.272400 - 0.236190)*4.795832/1.078903 = 0.160957. DSR = 0.563936", "mono": true }, { "label": "Worked, N = 100", "value": "SR0 = 0.15*2.530603 = 0.379590, which exceeds the observed 0.272400. z = -0.476470, DSR = 0.316870", "mono": true }, { "label": "Worked, N = 1000", "value": "SR0 = 0.488268, z = -0.959555, DSR = 0.168640", "mono": true }, { "label": "Sensitivity to the trial dispersion", "value": "At N = 100, a trial standard deviation of 0.10 gives DSR = 0.534254; 0.15 gives 0.316870; 0.20 gives 0.149422. The single most influential input is the one least often measured", "mono": true } ], "notes": [ "The trial standard deviation is not a nuisance parameter to be assumed. It is measured from the Sharpe ratios of all the configurations tested, and it is precisely the quantity a researcher who only kept the winner cannot supply. A deflated Sharpe ratio computed with a guessed dispersion is a guess.", "The base series drops from a PSR of 0.887 against zero to a deflated Sharpe ratio of 0.564 at ten trials and 0.317 at a hundred. Nothing about the data changed; only the honest accounting of how it was found.", "DSR is a probability, not a Sharpe ratio, despite the name. It ranges over [0,1] and is not comparable with an ordinary Sharpe ratio in any way.", "The framework treats trials as independent draws from a common distribution. A grid search over a smooth parameter surface violates that badly, and the correct effective N is smaller than the raw count. Both the raw count and any effective count should be reported, because the gap between them is the researcher's judgement." ], "source": "Bailey and Lopez de Prado 2014, The Deflated Sharpe Ratio" }, { "id": "minimum-backtest-length", "term": "Minimum backtest length", "definition": "The sample length below which a claimed Sharpe ratio cannot be distinguished from the best of N trials on no-edge data. It is a function of the trial count and the target Sharpe ratio only, and it is a necessary condition rather than a sufficient one.", "formula": "MinBTL in years = (E[max Z_N] / SR_ann)^2, with the source giving the approximation MinBTL is approximately 2*ln(N)/SR_ann^2", "fields": [ { "label": "Worked, the canonical case", "value": "N = 45 trials targeting an annualised Sharpe ratio of 1: E[max Z_45] = 2.235640, so MinBTL = (2.235640/1)^2 = 4.9981 years. The 2*ln(N) approximation gives 2*ln(45) = 7.6133 years", "mono": true }, { "label": "By trial count at SR = 1", "value": "N = 10 needs 2.4794 years; N = 100 needs 6.4040; N = 1,000 needs 10.5958; N = 10,000 needs 14.9047", "mono": true }, { "label": "By target Sharpe at N = 45", "value": "SR 0.5 needs 19.9923 years; SR 1.0 needs 4.9981; SR 1.5 needs 2.2214; SR 2.0 needs 1.2495", "mono": true }, { "label": "Approximation gap", "value": "The 2*ln(N) form is consistently more conservative, by 52 percent at N = 45 and 24 percent at N = 10,000, because sqrt(2 ln N) exceeds E[max Z_N] at every N", "mono": true } ], "notes": [ "The relationship is inverse-square in the target Sharpe ratio, which is why low-Sharpe strategies are effectively unverifiable. Demonstrating a true annualised Sharpe ratio of 0.5 against even 45 trials needs 20 years of data, which for most strategies exceeds the period over which the market structure was stable.", "This is a necessary condition and not a sufficient one. A backtest longer than the minimum length has not been shown to be valid; it has merely not been ruled out on multiple-testing grounds alone. Look-ahead bias, survivorship bias and regime change are separate failures with separate tests.", "The formula is stated in years and uses an annualised Sharpe ratio, so the periodicity of the data does not enter. A daily backtest and a monthly backtest of the same length in years face the same minimum, which is correct: more frequent sampling of the same period does not add independent information about the mean." ], "source": "Bailey, Borwein, Lopez de Prado and Zhu 2014, Pseudo-Mathematics and Financial Charlatanism" }, { "id": "trials-for-a-spurious-sharpe-of-two", "term": "How many trials it takes to find a spurious Sharpe of 2", "definition": "The inverse of the expected-maximum calculation. Given a sample length, solve for the number of independent trials at which the expected best result from a strategy with no edge is an annualised Sharpe ratio of 2. The answer is a strong function of sample length and a weak function of the target.", "formula": "Solve E[max Z_N] = SR_target * sqrt(T/P) for N, where the left side is the Bailey and Lopez de Prado approximation and the right side is the target annualised Sharpe divided by the null standard deviation sqrt(P/T)", "fields": [ { "label": "Worked, 2 years of monthly data", "value": "T = 24, null SD = sqrt(12/24) = 0.707107. Required E[max Z] = 2/0.707107 = 2.828427. Solving gives N of approximately 243 trials", "mono": true }, { "label": "Worked, 5 years", "value": "T = 60, null SD = 0.447214, required E[max Z] = 4.472136, N approximately 1.46 x 10^5", "mono": true }, { "label": "Worked, 10 years", "value": "T = 120, null SD = 0.316228, required E[max Z] = 6.324555, N approximately 4.43 x 10^9", "mono": true }, { "label": "Worked, 20 years", "value": "T = 240, required E[max Z] = 8.944272, N approximately 3 x 10^18 - beyond any feasible search", "mono": true } ], "notes": [ "The jump from about 243 trials at two years to about 146,000 at five years is the argument for length over cleverness, in one line. Because E[max Z] grows as sqrt(2 ln N), buying an extra unit of spurious Sharpe ratio costs exponentially more trials, while buying it with sample length costs only linearly more data.", "243 trials on two years of data is well within a single afternoon of parameter exploration. A two-year backtest showing a Sharpe ratio of 2 is not surprising under the null and should not be treated as informative without the trial count.", "These figures assume independent trials, which no real search satisfies. The effective number of independent trials in a correlated grid search is smaller than the raw count, so the real numbers of parameter combinations required are larger than shown - which makes long samples more important, not less.", "The same arithmetic run backwards is the useful diagnostic: given how many configurations were actually tried and how long the sample is, what Sharpe ratio should have been expected under the null. If the observed result is below it, there is nothing to explain." ] }, { "id": "probability-of-backtest-overfitting", "term": "Probability of backtest overfitting", "definition": "The probability that the configuration selected as best in sample performs below median out of sample. It is estimated by combinatorially symmetric cross-validation: split the sample into groups, form every balanced train-test partition, pick the in-sample winner in each, and record its out-of-sample rank.", "formula": "For each of the C(S, S/2) symmetric splits, let omega_c be the relative out-of-sample rank of the in-sample-best configuration, in (0,1). Define the logit lambda_c = ln(omega_c/(1 - omega_c)). PBO = P[lambda_c <= 0], the fraction of splits in which the in-sample winner ranks at or below median out of sample", "fields": [ { "label": "Worked, split count", "value": "With S = 16 groups and half held out, C(16,8) = 12,870 symmetric splits. With S = 10, C(10,5) = 252, and with S = 8, C(8,4) = 70. The count grows fast enough that S is usually capped at 16", "mono": true }, { "label": "Reading the statistic", "value": "PBO near 0.5 means the in-sample ranking carries no out-of-sample information at all. PBO above 0.5 means the in-sample winner is worse than a randomly chosen configuration out of sample, which is the signature of an overfit selection procedure", "mono": true }, { "label": "Required input", "value": "The full matrix of performance for every configuration over every group. It cannot be computed from the winner alone, which is why it is rarely reported", "mono": true }, { "label": "Related output", "value": "The same procedure yields the out-of-sample performance degradation - the regression of out-of-sample on in-sample performance across splits. A slope at or below zero is the same finding as PBO above 0.5", "mono": true } ], "notes": [ "PBO measures the selection procedure, not the selected strategy. A low PBO says the researcher's method of choosing between candidates generalises; it does not say the winner has an edge. The two questions are separate and need separate tests.", "The symmetric splits are contiguous groups, not random samples, precisely so that the temporal structure survives. Randomly permuting observations into folds destroys the serial dependence and produces a meaninglessly low PBO.", "The procedure inherits the purging requirement: if labels overlap in time, the train and test groups must be purged and embargoed or the out-of-sample ranks are contaminated and PBO is understated.", "PBO is not a probability that a strategy will lose money. It is a probability about the ranking of one configuration relative to its siblings, and a strategy can have a low PBO and no edge, or a high PBO and a real one." ], "source": "Bailey, Borwein, Lopez de Prado and Zhu 2017, The Probability of Backtest Overfitting" }, { "id": "look-ahead-bias", "term": "Look-ahead bias, and the forms that survive review", "definition": "Using information at time t that was not available at time t. The obvious forms are caught immediately; the forms that survive are the ones where the leakage is in a preprocessing step rather than in the signal.", "formula": "A feature x_t is admissible only if x_t is a function of information observable strictly at or before t. Any statistic computed over the full sample and then applied at every t violates this", "fields": [ { "label": "Worked, full-sample standardisation", "value": "Scaling a feature by its full-sample mean and standard deviation leaks the future into every observation. On the base series the full-sample mean is 0.766667 percent, but at month 6 only the first six returns were available and their mean is 2.016667 percent - a gap of 125 basis points injected into every early observation. The correct version uses an expanding or rolling window computed only from data up to t", "mono": true }, { "label": "Restatement", "value": "Fundamental data as currently reported is not what was known at the time. A point-in-time database records the as-first-reported value and its publication date; a current-value database silently substitutes revisions", "mono": true }, { "label": "Index membership", "value": "Constructing a universe from today's index constituents and running it back through history uses membership decisions that had not been made. Membership must be as-of", "mono": true }, { "label": "Timestamps", "value": "A daily bar labelled with a date does not say when within that day the data became usable. Trading on a close-based signal at that same close is a one-bar look-ahead, and it is the single most common form", "mono": true } ], "notes": [ "The magnitude of look-ahead bias is unbounded and unpredictable. Unlike multiple testing, it has no correcting formula: the only remedy is to rebuild the pipeline so the leak cannot occur, which is why point-in-time data infrastructure is not an optional refinement.", "Full-sample feature normalisation is the leak most likely to survive a code review, because it looks like a preprocessing detail rather than a signal. It is also one of the most damaging, since it injects future information into every single observation rather than a few.", "A leak often shows up as a backtest whose performance is implausibly stable across regimes. Real edges vary with the environment; a leak does not, because it has access to the whole sample everywhere." ] }, { "id": "survivorship-bias", "term": "Survivorship bias", "definition": "Restricting a universe to entities that exist at the end of the sample. It removes the worst outcomes from the historical record by construction, and its direction is always favourable to the strategy.", "formula": "The bias is the difference between the return of the full historical universe including delistings and the return of the surviving subset. It is measurable only with a database that retains dead entities and their terminal values", "fields": [ { "label": "Worked, mechanism", "value": "A universe of 100 equal-weighted assets over one year in which 5 go to zero and the other 95 return 10 percent has a true return of 0.95*1.10 - 1 = 4.5000 percent. The survivor-only universe reports 10.0000 percent. The 5.50 percentage point gap is the 5 percent delisted weight multiplied by 1.10, the growth factor the survivors reported, and it is invisible in the survivor data", "mono": true }, { "label": "Delisting returns", "value": "A dataset that contains dead entities but records no terminal return still has the bias, because the position silently disappears at its last observed price rather than at its recovery value", "mono": true }, { "label": "Fund databases", "value": "The same problem in a second form: a manager who closes stops reporting, so a database of live reporters is a survivor universe. Backfill, where a manager joins a database and supplies past returns, compounds it in the same direction", "mono": true }, { "label": "Direction", "value": "Always favourable to the backtest. Unlike multiple testing, whose sign is also known, the magnitude has no formula and depends entirely on the delisting rate of the specific universe", "mono": true } ], "notes": [ "The test is structural, not statistical: count the entities in the universe at the start of the sample and at the end. If the entry and exit counts are zero, the universe is a survivor set regardless of what the vendor claims.", "Because the magnitude is universe-specific, no published figure for the size of survivorship bias transfers to a different universe or period. Any correction has to be measured on the actual data, not imported.", "Survivorship and look-ahead bias interact. A universe defined by today's membership is both survivor-biased and look-ahead-biased, and the two effects are not separable after the fact." ] }, { "id": "overlapping-labels-effective-sample", "term": "Overlapping labels and the effective sample size", "definition": "When the label at bar t depends on prices over the following h bars, consecutive labels share most of their information. The raw observation count then overstates the independent information in the sample, and every standard error computed from it is too small.", "formula": "With one label per bar and each label spanning h bars, the approximate number of independent observations is T/h. Standard errors computed on T rather than T/h are understated by a factor of approximately sqrt(h)", "fields": [ { "label": "Worked", "value": "T = 1000 bars with labels spanning h = 20 bars: approximately 50 independent observations, and standard errors are understated by sqrt(20) = 4.472136. A t-statistic of 4 becomes 0.89", "mono": true }, { "label": "Scaling", "value": "h = 1 gives 1000 effective observations and no understatement. h = 5 gives 200 and a factor of 2.236068. h = 50 gives 20 and a factor of 7.071068", "mono": true }, { "label": "Average uniqueness", "value": "The proportionate measure is the average uniqueness of a label: the mean, over bars, of the reciprocal of the number of labels that overlap that bar. It reduces to 1/h in the simple case above and is computable exactly for irregular label spans", "mono": true }, { "label": "Consequence for sampling", "value": "Sampling one observation per h bars, or sampling by average uniqueness, restores approximate independence at the cost of discarding data. Both are preferable to reporting a t-statistic inflated by sqrt(h)", "mono": true } ], "notes": [ "This is the most quantitatively severe error in the section because it is multiplicative on the t-statistic and because the fix looks like throwing away data. A model with a 20-bar holding period trained on daily bars has 20 times fewer independent observations than its row count suggests.", "The problem is not solved by using non-overlapping labels only if the resulting sample is too small to train on - which it usually is. The honest approach is to train on the overlapping sample and evaluate significance on the effective count.", "Sample weights proportional to average uniqueness are the standard partial remedy: they let the model use every row while down-weighting redundant information. They correct the fit, not the standard error, which still has to be computed on the effective count." ], "source": "Lopez de Prado 2018, Advances in Financial Machine Learning" }, { "id": "purged-k-fold", "term": "Purged k-fold cross-validation", "definition": "Ordinary k-fold cross-validation leaks when labels overlap the test set in time, because a training label whose window extends into the test period was formed partly from test-period prices. Purging removes exactly those training observations.", "formula": "For a test set spanning [t1, t2] and labels at t spanning [t, t+h]: purge every training observation whose label window intersects [t1, t2], that is every t with t+h >= t1 and t <= t2. In the contiguous case this removes the h bars immediately before t1 in addition to the test set itself", "fields": [ { "label": "Worked cost", "value": "T = 1000 bars, 5 folds of 200, label span h = 20, embargo 20: up to 40 bars removed from the 800-bar training set, 5.0000 percent", "mono": true }, { "label": "Worse case", "value": "T = 252 bars, 5 folds of 50, h = 20, embargo 20: up to 40 bars removed from a 202-bar training set, 19.8020 percent. On short samples with long labels, purging consumes a large fraction of the training data", "mono": true }, { "label": "Larger sample", "value": "T = 2520 bars, 10 folds of 252, h = 20, embargo 25: 45 bars from 2268, 1.9841 percent. The cost falls quickly with sample length", "mono": true }, { "label": "Interior folds", "value": "An interior test fold requires purging on both sides, before and after; the first and last folds require it on one side only. The stated cost is the worst case across folds", "mono": true } ], "notes": [ "Without purging, cross-validated performance on overlapping labels is optimistic by an amount that grows with h. The failure is silent: nothing in the fit or the score indicates that the folds shared information.", "Purging is necessary and not sufficient. It removes label-window overlap; it does not remove serial dependence in the features themselves, which is what the embargo addresses.", "The purge width is determined by the label definition, so it is fixed once the prediction horizon is fixed. A researcher who lengthens the holding period must widen the purge, and the training set shrinks accordingly - a real cost of longer-horizon models that is usually overlooked." ], "source": "Lopez de Prado 2018, Advances in Financial Machine Learning" }, { "id": "embargo", "term": "Embargo", "definition": "A block of observations after the test set that is excluded from training, on top of purging. It exists because features are serially correlated: a training observation immediately after the test period can carry information about it even when its label window does not overlap.", "formula": "After a test set ending at t2, exclude training observations in (t2, t2 + he] where he is the embargo length. Conventionally he is a small fraction of T, chosen to exceed the serial dependence length of the features", "fields": [ { "label": "Worked", "value": "T = 2520 bars with a 1 percent embargo gives he = 25 bars. Combined with a label span of h = 20 and 10 folds of 252, the training set loses up to 45 bars of 2268, or 1.9841 percent", "mono": true }, { "label": "Asymmetry", "value": "Purging is required on both sides of an interior test fold. The embargo is required only after, because the concern is feature information flowing forward from the test period into the training period", "mono": true }, { "label": "Choosing he", "value": "It should exceed the autocorrelation length of the features. There is no formula; the defensible procedure is to measure the feature autocorrelation and set he beyond where it becomes negligible", "mono": true }, { "label": "When it is unnecessary", "value": "If the features are genuinely non-overlapping and serially uncorrelated - a rare case - purging alone suffices and the embargo costs data for nothing", "mono": true } ], "notes": [ "The embargo length is a judgement, not a derivation, and it should be reported. An unstated embargo makes a cross-validation result irreproducible in exactly the way an unstated purge does.", "On short samples the embargo and purge together can consume a fifth of the training set, as the 252-bar case shows. That is a real argument for longer samples rather than for skipping the correction.", "An embargo is not a substitute for purging and purging is not a substitute for an embargo. They address different leakage channels - label windows and feature serial dependence - and both are needed when both are present." ], "source": "Lopez de Prado 2018, Advances in Financial Machine Learning" }, { "id": "walk-forward-vs-cpcv", "term": "Walk-forward against combinatorial purged cross-validation", "definition": "Walk-forward testing produces exactly one out-of-sample path, so it yields one Sharpe ratio and no distribution. Combinatorial purged cross-validation forms every balanced train-test partition and assembles multiple distinct out-of-sample paths from them, giving a distribution of outcomes from the same data.", "formula": "With N contiguous groups and k held out per test set: C(N,k) splits and phi = C(N,k)*k/N distinct backtest paths. Walk-forward is the phi = 1 case", "fields": [ { "label": "Worked", "value": "N = 6, k = 2: C(6,2) = 15 splits and 15*2/6 = 5 paths. N = 10, k = 2: 45 splits and 9 paths. N = 16, k = 2: 120 splits and 15 paths. N = 10, k = 5: 252 splits and 126 paths", "mono": true }, { "label": "What a distribution buys", "value": "A single walk-forward Sharpe ratio cannot be tested. Five or fifteen paths give a dispersion, and the dispersion is what the deflated Sharpe ratio needs as its trial standard deviation", "mono": true }, { "label": "Cost", "value": "C(N,k) model fits rather than N-1. At N = 16 and k = 2 that is 120 fits against 15 for walk-forward, an eight-fold increase in compute for the same data", "mono": true }, { "label": "Walk-forward's remaining advantage", "value": "It never trains on data after the test period, so it matches deployment exactly. Combinatorial cross-validation trains on future data in most of its splits, which is acceptable for measuring generalisation and not for simulating a live process", "mono": true } ], "notes": [ "The two answer different questions. Walk-forward asks what a process deployed at the time would have produced; combinatorial purged cross-validation asks how much of the in-sample result generalises. A researcher who reports only walk-forward has one number and cannot deflate it; one who reports only cross-validation has a distribution and no simulation of deployment.", "Walk-forward has its own overfitting channel that is easy to miss: the choice of window length, refit frequency and the number of walk-forward variants examined are all trials, and they count toward N.", "Both require purging and embargoing when labels overlap. Walk-forward is often assumed safe because it is chronological, but a label at the end of the training window that spans into the test window leaks in exactly the same way." ], "source": "Lopez de Prado 2018, Advances in Financial Machine Learning" }, { "id": "t-statistic-threshold", "term": "Why a t-statistic of 2 is not enough", "definition": "The conventional 5 percent significance threshold assumes one test. In a literature or a research process with hundreds of tests, a t-statistic of 2 is the expected best of a modest search, and the threshold has to be raised to preserve the intended error rate.", "formula": "Under N independent tests, controlling the family-wise error rate at alpha requires a per-test threshold of approximately Z^-1(1 - alpha/(2N)) rather than Z^-1(1 - alpha/2). Harvey, Liu and Zhu 2016 argue for a threshold near 3.0 for newly proposed factors after multiple-testing adjustment", "fields": [ { "label": "Worked, Bonferroni arithmetic", "value": "At alpha = 0.05: one test needs Z^-1(1 - 0.025) = 1.9600. 10 tests need Z^-1(1 - 0.0025) = 2.8070. 100 tests need Z^-1(1 - 0.00025) = 3.4808. 1,000 tests need Z^-1(1 - 0.000025) = 4.0556", "mono": true }, { "label": "Under the expected-maximum view", "value": "E[max Z] over 20 independent trials is 1.900708, so a t-statistic of 2 is roughly the median-to-expected best result of a twenty-trial search on no-edge data", "mono": true }, { "label": "The haircut view", "value": "Harvey and Liu 2015 express the same correction as a haircut applied to the reported Sharpe ratio rather than as a threshold on the t-statistic. The two are algebraically the same adjustment expressed in different units", "mono": true }, { "label": "Applied to the base series", "value": "t = 1.310396 on 24 observations. It does not clear 1.96, let alone 3.0", "mono": true } ], "notes": [ "Bonferroni is conservative because it assumes independence and controls the family-wise error rate, which is a strict criterion. False-discovery-rate methods are less conservative and are the better choice when the goal is to keep the proportion of false positives among discoveries low rather than to avoid any false positive.", "The threshold depends on N, and in a published literature N is unknowable: it includes every test that was run and not published. Any threshold derived from a count of published results is a lower bound on the correct one.", "Raising the threshold does not make a result true. It reduces the rate at which noise is accepted, at the cost of rejecting real effects with modest t-statistics - which, given the minimum backtest length arithmetic, includes most real effects." ], "source": "Harvey, Liu and Zhu 2016; the haircut formulation is Harvey and Liu 2015" } ] }