Why High Model Scores Do Not Guarantee Profit
A model can predict the next price with a high statistical score and still produce poor trading decisions.
One of our earlier development updates reported high R-squared scores for a convolutional neural network trained to predict the next ETHUSD price. Those results described a prediction task. They did not establish that the model could select trades, overcome costs, or produce positive net expectancy.
Later research made this distinction harder to ignore. Models with useful-looking ranking scores still produced negative economic results on later feedback holdouts. The lesson is not that model metrics are useless. It is that each metric answers a narrower question than the word “performance” suggests.
Evidence classification: Historical development prediction metrics, train/calibration ranking diagnostics, and chronological feedback-holdout P&L. These evidence categories should not be treated as interchangeable.
R-squared measures fit, not tradeability
Our 2025 development update reported training R-squared between 98% and 99.6% and validation R-squared between 85% and 96% for next-price prediction. Those numbers were accurate descriptions of that modeling exercise, but they were not trading results.
Price levels are highly persistent. The next price is usually close to the current price, so a model can explain a large share of next-price variance without predicting the direction or magnitude of the next move well enough to trade. A simple persistence baseline—such as assuming the next price will resemble the current one—can also score well on price-level fit.
Before connecting R-squared to a strategy, we still need to ask whether the model beats a simple baseline, whether errors are directional at the intended horizon, whether the prediction is available causally, and whether the forecast magnitude exceeds expected trading costs.
AUC measures ranking, not net return
In later causal meta-ranking research, one model reached a train AUC of 0.6615 and a calibration AUC of 0.6665. An AUC above 0.5 suggests the model had some ability to rank positive labels above negative labels in those samples.
Yet its mean return on the feedback holdout was -5.88%. That is not a contradiction in the mathematics. AUC does not know the size of wins and losses, how often the model trades, where a threshold is placed, or what the strategy pays in fees, spread, slippage, and funding.
A ranking model can be directionally informative while emphasizing events whose payoff distribution is still unfavorable. It can also rank a weak candidate family without making that family economically viable.
Calibration asks a third question
A calibrated probability should mean what it says. Among events assigned a 70% probability, roughly 70% should satisfy the defined outcome over a sufficiently large and comparable sample.
Our probability-layer experiments showed calibration failures. A model can preserve decent ranking while its numeric probabilities are overconfident, underconfident, or unstable across regimes. That matters when probabilities control admission thresholds or position size.
Calibration also depends on the target definition. A probability of a labeled movement is not automatically a probability of a profitable trade. Entry delay, exit rules, costs, and overlapping positions can separate those outcomes.
Net expectancy is the economic test
A trading system ultimately needs positive expectancy after realistic implementation. A simplified expression is:
Net expectancy = average gains − average losses − trading and execution costs
A responsible strategy scorecard therefore needs more than one model metric. It should report chronological and untouched results, benchmark comparisons, net return, drawdown, exposure, turnover, trade count, cost assumptions, regime stability, and the exact timing at which each input became available.
R-squared, AUC, and calibration remain useful diagnostics:
- R-squared: How much variation in a numeric target did the model explain?
- AUC: How well did the model rank one class above another?
- Calibration: Did predicted probabilities match observed frequencies?
- Net expectancy: Did the executable decisions make money after costs?
None of the first three can answer the fourth by itself. That is why we no longer treat a high prediction score as a shortcut to a trading claim.
Follow our ongoing research notes on the Evolve Quant blog or join our newsletter.
Research status as of August 5, 2026. Later evidence may narrow or supersede this interpretation.
Important: This article is for educational and research purposes only. It is not investment advice, a trading signal, or a promise of future performance.