The Oracle Problem in Software Testing¶
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE, 41(5), 507-525.
Cited by¶
3 citations across 3 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Ground Truth
- Software testing and speech processing: oracle outputs function as ground truth for a candidate implementation; human transcripts as ground truth for ASR; expert-marked event times for detection.
This sourceSurveys test oracles — including the 'ground truth oracle' that always gives the right answer — as the reference against which a candidate implementation is scored.
Supported in partVerified against the work's full text
Barr et al. define a 'ground truth oracle' as the always-correct reference for judging test oracles; it says nothing on speech recognition transcripts or expert-marked event times.
“The ground truth is another form of oracle, a conceptual oracle, that always gives the”
- Software testing and speech processing: oracle outputs function as ground truth for a candidate implementation; human transcripts as ground truth for ASR; expert-marked event times for detection.
Mechanisms¶
- Bijection Test Suite
- Its honest failure mode is the oracle problem: a test suite only checks the properties someone thought to assert, so a green suite proves the absence of the failures it looks for, never bijectivity in general.
This sourceDefines the oracle problem: tests can evaluate only supplied correctness criteria, so passing asserted checks does not establish an untested property in general.
- Its honest failure mode is the oracle problem: a test suite only checks the properties someone thought to assert, so a green suite proves the absence of the failures it looks for, never bijectivity in general.
- Endpoint Equivalence Test Suite
- Because the suite is defined by the downstream contract rather than by each path's own notion of "done," it catches the CSV path's unnormalized numbers even though that path considered them fine — the test is an oracle
This sourceDefines a test oracle as a procedure that checks observed behavior against desired behavior supplied by specifications or contracts.
- Because the suite is defined by the downstream contract rather than by each path's own notion of "done," it catches the CSV path's unnormalized numbers even though that path considered them fine — the test is an oracle
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 3 citations: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:b093bcc5be59 · see in the full table