At the US National Institute of Standards and Technology, the Software Assurance Metrics And Tool Evaluation project, or SAMATE for short, is dedicated to improving software assurance by developing methods for evaluating software tools, measuring their effectiveness, and identifying gaps in methods and techniques.
The SAMATE project recognizes the value and importance of sound static code analyzers. During the 6th Static Analysis Tool Exposition (SATE VI), the NIST team evaluated static analyzers with respect to the SATE VI Ockham Sound Analysis Criteria.
In brief, these criteria are:
- The tool is claimed to be sound.
- For at least one weakness class and one test case, the tool produces findings for a minimum of 75% of appropriate sites.
- Even just one incorrect finding disqualifies a tool for SATE VI.
The definition of a finding includes reporting a buggy site as buggy, but also passing a correct code location without alarm. In other words, in order to satisfy the SATE VI Ockham Sound Analysis Criteria, all defects must be found, and the rate of false alarms must be low.
At least four tools were considered, only two of which satisfied the SATE VI criteria: Astrée and Frama-C.
Astrée was run on 28 sets of test cases from the Juliet 1.3 C test suite, containing a total of 18,954 buggy sites. All 18,954 were reported by Astrée.
These included test cases for buffer overflows/underflows, invalid pointer dereferences, integer overflows/underflows, divisions by zero, use of uninitialized variables, dead code, infinite loops, double free and use after free.
Additionally, Astrée discovered thousands of unintended defects in the Juliet 1.3 benchmark set.
The SAMATE report emphasizes Astrée’s outstanding precision, both with respect to the analysis results as well as the analysis model itself.
“Alarms from Astrée led us to find and fix thousands of mistakes in what was intended as the Juliet known-bug list, manifest.xml.
Because Astrée analyzes code very precisely and we checked meticulously, details of modeling that otherwise would be inconsequential showed up and had to be resolved.”
The full report was published in May 2020 and is available for free as PDF from
nvlpubs.nist.gov/nistpubs/ir/2020/NIST.IR.8304.pdf
Try now
Try Astrée on your own code, free of charge, for a period of 30 days.