summaryrefslogtreecommitdiffstats
path: root/evaluation.tex
diff options
context:
space:
mode:
authorYann Herklotz <git@yannherklotz.com>2021-09-10 18:38:04 +0100
committerYann Herklotz <git@yannherklotz.com>2021-09-10 18:38:04 +0100
commit4d019a44c0fc9d78dd498d4775d904f6a846d29f (patch)
treefed749efbc0fc9416bdd23cd2cae04e9cfe40e74 /evaluation.tex
parenta84ed32608424bfc9d941caf49c45b9f643120b4 (diff)
downloadoopsla21_fvhls-4d019a44c0fc9d78dd498d4775d904f6a846d29f.tar.gz
oopsla21_fvhls-4d019a44c0fc9d78dd498d4775d904f6a846d29f.zip
Some final changes
Diffstat (limited to 'evaluation.tex')
-rw-r--r--evaluation.tex2
1 files changed, 1 insertions, 1 deletions
diff --git a/evaluation.tex b/evaluation.tex
index a80ee79..0147bdc 100644
--- a/evaluation.tex
+++ b/evaluation.tex
@@ -208,7 +208,7 @@ By looking at the median, when division/modulo operations are enabled, we see th
To gain further confidence that the Verilog designs generated by \vericert{} are actually correct, and that the correctness theorem is indeed effective, we fuzzed \vericert{} using Csmith~\cite{yang11_findin_under_bugs_c_compil}. \citeauthor{yang11_findin_under_bugs_c_compil} previously used Csmith in an extensive fuzzing campaign on CompCert and found a handful of bugs in the unverified parts of that compiler, so it is natural to explore whether it can find bugs in \vericert{} too. \citet{herklotz21_empir_study_reliab_high_level_synth_tools} have recently used Csmith to fuzz other HLS tools including \legup{}, so we configured Csmith in a similar way. In addition to the features turned off by \citeauthor{herklotz21_empir_study_reliab_high_level_synth_tools}, we turned off the generation of global variables and non-32-bit operations. The generated designs were tested by simulating them and comparing the output value to the results of compiling the test-cases with GCC 10.3.0.
The results of the fuzzing run are shown in Fig.~\ref{tab:fuzzing}. Out of 155267 test-cases generated by Csmith, 26\% of them passed, meaning they compiled without error and resulted in the same final value as GCC. Most of the test-cases, 73.97\%, failed at compile time. The most common reasons for this were unsigned comparisons between integers (\vericert{} requires them to be signed), and the presence of 8-bit operations (which \vericert{} does not support, and which we could not turn off due to a limitation in Csmith). % The latter programs are present because of a slight limitation with how Csmith can be configured, as it does not allow 8-bit operations to be turned off completely, and will get stuck when it generates the C code.
-Because the test-cases generated by Csmith could not be tailored exactly to the C fragment that \vericert{} supports, such a high compile-time failure rate is not unexpected. Finally, and most interestingly, there were a total of 39 run-time failures, which the correctness theorem should be proving impossible. However, all 39 of these failures are due to a bug in the pretty-printing of the final Verilog code, where a logical negation (\texttt{!}) was accidentally used instead of a bitwise negation (\verb|~|). Once this bug was fixed, all test-cases passed.
+Because the test-cases generated by Csmith could not be tailored exactly to the C fragment that \vericert{} supports, such a high compile-time failure rate is expected. Finally, and most interestingly, there were a total of 39 run-time failures, which the correctness theorem should be proving impossible. However, all 39 of these failures are due to a bug in the pretty-printing of the final Verilog code, where a logical negation (\texttt{!}) was accidentally used instead of a bitwise negation (\verb|~|). Once this bug was fixed, all test-cases passed.
%%% Local Variables:
%%% mode: latex