In rare circumstances, a timeout of a multi-node cluster test (which should result in TESTERROR) can lead to a situation where

  1. No abstract machine reports this TESTERROR.
  2. The verdicts of test cases (and the test procedure) remains PASS,
    despite the fact that this result is not reliable.
2024-04-29

Symptoms:

  • testdata/SUM.tags contains a line:
    ANONYMOUS/<TP-NAME>,TESTERROR,TM 00000000000,SUM.tags#TERMINATION
  • testdata/ALL-TC-COV.csv contains lines with both verdict TESTERROR and other verdicts (like PASS)

Affected Versions:

  • RTT 7.0.0 - RTT 7.0.4
  • It might also happen in older versions, which contain fewer safeguards concerning TESTERROR.

Mitigation:

  • Check testdata/ALL-TC-COV.csv for occurrences of ,TESTERROR,
  • If TESTERROR occurs, but the test procedure / test case verdict does say PASS (or FAIL),
    then the test results are not reliable.