Reduction in sections

1 files changed, 8 insertions, 8 deletions

diff --git a/eval.tex b/eval.tex
index 661ae74..46b58a0 100644
--- a/eval.tex
+++ b/eval.tex
@@ -1,9 +1,9 @@
 \section{Evaluation}\label{sec:evaluation}
 
-We generate \totaltestcases{} test-cases and provide them to three HLS tools: Vivado HLS, LegUp HLS and Intel i++. 
+We generate \totaltestcases{} test-cases and provide them to four HLS tools: Vivado HLS, LegUp HLS, Intel i++ and Bambu.
 We use the same test-cases across all tools for fair comparison (except the HLS directives, which have tool-specific syntax).
 We were able to test three different versions of Vivado HLS (v2018.3, v2019.1 and v2019.2). 
-We tested one version of Intel i++ (version 18.1), and one version of LegUp (4.0). 
+We tested one version of Intel i++ (version 18.1), LegUp (4.0) and Bambu (v0.9.7).
 LegUp 7.5 is GUI-based and therefore we could not script our tests. 
 However, we were able to manually reproduce all the bugs found in LegUp 4.0 in LegUp 7.5. 
 
@@ -49,7 +49,7 @@ However, we were able to manually reproduce all the bugs found in LegUp 4.0 in L
 \end{figure}
 
 Figure~\ref{fig:existing_tools} shows a Venn diagram of our results. 
-We see that 167 (2.5\%), 83 (1.2\%) and 26 (0.4\%) test-cases fail in LegUp, Vivado HLS and Intel i++ respectively. 
+We see that 918 (13.7\%), 167 (2.5\%), 83 (1.2\%) and 26 (0.4\%) test-cases fail in Bambu, LegUp, Vivado HLS and Intel i++ respectively.
 Despite i++ having the lowest failure rate, it has the highest time-out rate (540 test-cases), because of its remarkably long compilation time.
 % We remark that although the Intel HLS Compiler had the smallest number of confirmed test-case failures, it had the most time-outs (which could be masking additional failures)  
 Note that the absolute numbers here do not necessarily correspond to the number of bugs in the tools, because a single bug in a language feature that appears frequently in our test suite could cause many programs to crash or fail.
@@ -129,7 +129,7 @@ As in our Venn diagram, the absolute numbers here do not necessary correspond to
 
 \subsection{Some specific bugs found}
 
-We now describe two more of the bugs we found: one crash bug in LegUp and one miscompilation bug in Vivado HLS. As in Example~\ref{ex:vivado_miscomp}, each bug was first reduced automatically using \creduce{}, and then reduced further manually to achieve the minimal test-case. Although we did find test-case failures in Intel i++, the long compilation times for that tool meant that we did not have time to reduce any of the failures down to an example that is minimal enough to present here.
+We now describe three more of the bugs we found: one crash bug in LegUp, and a miscompilation in Intel and Bambu respectively. As in Example~\ref{ex:vivado_miscomp}, each bug was first reduced automatically using \creduce{}, and then reduced further manually to achieve the minimal test-case.
 
 \begin{example}[A crash bug in LegUp]
 The program shown below leads to an internal compiler error (an unhandled assertion in this case) in LegUp 4.0 and 7.5. 
@@ -176,13 +176,13 @@ int main() {
 static int b = 0x10000;
 static volatile short a = 0;
 
-int result() {
+int main() {
   a++;
   b = (b >> 8) & 0x100;
   return b;
 }
 \end{minted}
-\caption{Miscompilation bug in Bambu HLS.  As the value of \texttt{b} is shifted to the right by 8, the output should be \texttt{0x100}.  However, the actual output is 0 in Bambu.}\label{fig:eval:bambu:mismatch}
+\caption{Miscompilation bug in Bambu.  As the value of \texttt{b} is shifted to the right by 8, the output should be \texttt{0x100}.  However, Bambu outputs 0.}\label{fig:eval:bambu:mismatch}
 \end{figure}
 
 %\begin{example}[A miscompilation bug in Vivado HLS]
@@ -200,8 +200,8 @@ int result() {
 Figure~\ref{fig:eval:intel:mismatch} shows a miscompilation bug that was found in Intel i++.  Intel i++ does not seem to notice the assignment to 3 in the previous for loop, or tries to perform some optimisations that seem to analyse the array incorrectly and therefore results in a wrong value being returned.
 \end{example}
 
-\begin{example}[A miscompilation bug in Bambu HLS]
-Figure~\ref{fig:eval:bambu:mismatch} shows the bug 
+\begin{example}[A miscompilation bug in Bambu]
+Figure~\ref{fig:eval:bambu:mismatch} shows a miscompilation bug in Bambu, where the result of the value in \texttt{b} is affected by the increment operation on \texttt{a}.
 \end{example}
 
 %%% Local Variables: