Glossary · Testing
Retrodiction compares a method with results already known
Retrodiction tests a method against outcomes that are already known. In materials research, it can check agreement with published properties, but its meaning depends on whether the method had access to those answers or closely related data. [1]
- Published
- Reading time
- 2 min
- Cited
- 2 sources
- DefinitionRetrodiction
- Testing a method against outcomes that are already known, ideally withholding those outcomes from the method. Agreement checks performance on the existing record but does not itself demonstrate prediction of an unmeasured result.
A known result can still be a useful test
A reference set makes it possible to compare a model’s output with measured or previously calculated properties. If the answers are genuinely withheld, the comparison can reveal systematic errors and performance within that set. Reproducing a known result is useful evidence about a method’s behavior under the stated conditions.
The interpretation changes when parameters were adjusted using the same answers. Agreement then measures fitting rather than performance on unseen outcomes. A useful report distinguishes training data, tuning data, and the final evaluation set. It also identifies whether the reference values are experimental measurements, calculations, or values compiled from mixed sources. [1]
Leakage can make agreement look stronger
Information leakage occurs when material that should be unavailable during evaluation reaches the method or its development process. Duplicate records, closely related samples, target-derived features, and preprocessing performed across the full dataset can all compromise a split. Patterns examines leakage as a source of misleading scientific performance claims. [1]
With pretrained language models, a nominally withheld paper may already have appeared in pretraining or retrieval. Removing it from a local evaluation file does not prove that its content was unavailable. Reports should state known exposure pathways and unresolved ones, rather than describing every successful answer as independent prediction of unknown physics.
The test set must resemble the intended challenge
A random split can distribute similar compositions or related records across training and evaluation. That tests a different challenge from predicting a new chemical family or a later discovery. Splitting by time, composition family, or experimental campaign can reveal weaknesses hidden by a familiar reference set, although each split answers a different question.
npj Computational Materials shows that strong database benchmark performance need not translate into strong predictions on newly added compounds. Changes in the distribution of materials can expose poor generalization. A retrodiction claim should therefore identify both the reference population and the new population to which the authors intend to apply the method. [2]
Prospective experiments provide different evidence
A prospective test fixes a prediction before the target outcome is available, then compares it with a later measurement. It can address information-exposure concerns that a retrospective benchmark cannot settle. Sample identification, measurement uncertainty, and independent reproduction still matter: predicting a transition does not establish that the intended material was synthesized.
For any reported back-test, ask what was hidden, what the method had previously seen, how settings were chosen, and which baseline it exceeded. Then ask whether the same procedure has been tested on genuinely new outcomes. These questions preserve the value of retrodiction while keeping agreement with the existing record distinct from a demonstrated discovery. [1][2]
Sources
- Leakage and the reproducibility crisis in machine-learning-based sciencePatterns, 2023
- A critical examination of robustness and generalizability of machine learning prediction of materials propertiesnpj Computational Materials, 2023
SuperMatics: AI built for superconductors.