diff options
Diffstat (limited to 'eval_rewrite.tex')
| -rw-r--r-- | eval_rewrite.tex | 28 |
1 files changed, 12 insertions, 16 deletions
diff --git a/eval_rewrite.tex b/eval_rewrite.tex index e084b05..7595b11 100644 --- a/eval_rewrite.tex +++ b/eval_rewrite.tex @@ -44,7 +44,7 @@ \label{tab:unique_bugs} \end{table} -To evaluate the different HLS tools 10,000 \JW{check numbers} test cases were generated and fed into each tool, keeping the test cases constant so that the comparison between the tools was fair. Three different tools were tested, including three different versions of Vivado HLS, which are shown in Table~\ref{tab:unique_bugs}. Bugs were found in all tools that were tested, and in total, \ref{??} unique bugs were found and reported to the tool vendors. +To evaluate the different HLS tools 6,700 test cases were generated and fed into each tool, keeping the test cases constant so that the comparison between the tools was fair. Three different tools were tested, including three different versions of Vivado HLS, which are shown in Table~\ref{tab:unique_bugs}. Bugs were found in all tools that were tested, and in total, \ref{??} unique bugs were found and reported to the tool vendors. We were only able to test one version of LegUp HLS (version 4.0). LegUp 7.5 is GUI-based and not suitable for scripting; however, bugs we found in LegUp 4.0 were reproduced manually in LegUp 7.5. \subsection{Bugs found} @@ -53,7 +53,7 @@ This section describes some of the bugs that were found in the various tools tha \subsubsection{LegUp Assertion Error} -The following piece of code produces an assertion error in LegUp even though it should compile without any errors, meaning an analysis pass in LegUp is incorrectly. This assertion error is equivalent to an unexpected crash of the tool as it means that an unexpected state was reached by this input. This shows that there is a bug in one of the compilation passes in LegUp, however, due to the assertion the bug is caught in the tool before it produces an incorrect design. +The piece of code shown in Figure~\ref{fig:eval:legup:assert} produces an assertion error in LegUp 4.0 and 7.5 even though it should compile without any errors. The assertion error states that This assertion error is equivalent to an unexpected crash of the tool as it means that an unexpected state was reached by this input. This shows that there is a bug in one of the compilation passes in LegUp, however, due to the assertion the bug is caught in the tool before it produces an incorrect design. \begin{figure} \begin{minted}{c} @@ -61,12 +61,10 @@ int a[2][2][1] = {{{0},{1}},{{0},{0}}}; int main() { a[0][1][0] = 1; } \end{minted} -\caption{An assertion bug in LegUp HLS.} -\label{fig:eval:legup:assert} +\caption{An assertion bug in LegUp HLS when setting \texttt{NO\_INLINE} to prevent inlining.}\label{fig:eval:legup:assert} \end{figure} - -The buggy test case has to do with initialisation and assignment to a three dimensional array, for which the above piece of code is the minimal example. It initialises the array with zeros except for \texttt{a[0][1][0]}, which is set to one. Then the main function assigns one to that same location, which causes LegUp to crash with an assertion error. +The buggy test case has to do with initialisation and assignment to a three dimensional array, for which the above piece of code is the minimal example. However, in addition to that it requires the \texttt{NO\_INLINE} constant to be set, which disallows inlining of functions. The code initialises the array with zeros except for \texttt{a[0][1][0]}, which is set to one. Then the main function assigns one to that same location, which causes LegUp to crash with an assertion error. This code on its own should not actually produce a result and should just terminate by returning 0, which is also what the design does that LegUp generates when the \texttt{NO\_INLINE} flag is turned off. %The following code also produces an assertion error in LegUp, which is a different one this time. This bug was not discovered during the main test runs of 10 thousand test cases, but beforehand, which meant that we disabled unions from being generated. However, this bug also requires the \texttt{volatile} keyword which seems to be the reason for quite a few mismatches in LegUp and Vivado. % @@ -80,7 +78,7 @@ The buggy test case has to do with initialisation and assignment to a three dime \subsubsection{LegUp Miscompilation} -The following test case produces an incorrect netlist in LegUp 4.0, meaning the result of simulating the design and running the C code directly is different. +The test case in Figure~\ref{fig:eval:legup:wrong} produces an incorrect netlist in LegUp 4.0, meaning the result of simulating the design and running the C code directly is different. \begin{figure} \begin{minted}{c} @@ -94,15 +92,14 @@ int main() { return b; } \end{minted} -\caption{An output mismatch where GCC returns 1 and LegUp HLS returns 0.} -\label{fig:eval:legup:wrong} +\caption{An output mismatch where GCC returns 1 and LegUp HLS returns 0.}\label{fig:eval:legup:wrong} \end{figure} -In the code above, \texttt{b} has value 1 when run in GCC, but has value 0 when run with LegUp 4.0. If the \texttt{volatile} keyword is removed from \texttt{a}, then the netlist contains the correct result. As \texttt{a} and \texttt{d} are constants, the if-statement should always produce go into the \texttt{true} branch, meaning \texttt{b} should never be set to 0. +In the code above, \texttt{b} has value 1 when run in GCC, but has value 0 when run with LegUp 4.0. If the \texttt{volatile} keyword is removed from \texttt{a}, then the netlist contains the correct result. As \texttt{a} and \texttt{d} are constants, the if-statement should always produce go into the \texttt{true} branch, meaning \texttt{b} should never be set to 0. The \texttt{true} branch of the if-statement only executes an expression which is not assigned to any variable, meaning the initial state of all variables should not have changed. However, LegUp HLS generates a design which enters the \texttt{else} branch instead and assigns \texttt{b} to be 0. The cause of this bug seems to be the \texttt{volatile} keyword and the analysis that is performed to simplify the if-statement. \subsubsection{Vivado Miscompilation} -The following code does not output the right value when compiled with all Vivado versions and GCC, as it returns \texttt{0x0} with Vivado whereas it should be returning \texttt{0xF}. This test case is much longer compared to the other test cases that were reduced and could not be made any smaller, as everything in the code is necessary to trigger the bug. +Figure~\ref{fig:eval:vivado:mismatch} shows code that does not output the right value when compiled with all Vivado versions and GCC, as it returns \texttt{0x0} with Vivado whereas it should be returning \texttt{0xF}. This test case is much longer compared to the other test cases that were reduced and could not be made any smaller, as everything in the code is necessary to trigger the bug. The array \texttt{a} is initialised to all zeros, as well as the other global variables \texttt{g} and \texttt{c}, so as to not introduce any undefined behaviour. However, \texttt{g} is also assigned the \texttt{volatile} keyword, which ensures that the variable is not optimised away. The function \texttt{d} then accumulates the values \texttt{b} that it is passed into a hash that is stored in \texttt{c}. Each \texttt{b} is eight bits wide, so function \texttt{e} calls the function 7 times for some of the bits in the 64 bit value of \texttt{f} that it is passed. Finally, in the main function, the array is initialised partially with a for loop, after which the \texttt{e} function is called twice, once on the volatile function but also on a constant. Interestingly enough, the second function call with the constant is also necessary to trigger the bug. @@ -125,17 +122,14 @@ int main() { return c; } \end{minted} -\caption{An output mismatch where GCC returns \texttt{0xF}, whereas Vivado HLS return \texttt{0x0}.} -\label{fig:eval:vivado:mismatch} +\caption{An output mismatch where GCC returns \texttt{0xF}, whereas Vivado HLS return \texttt{0x0}.}\label{fig:eval:vivado:mismatch} \end{figure} \subsection{Bugs in Vivado HLS versions} In addition to the explanation to bugs given in Section~\ref{sec:evaluation}, bugs found in various versions of Vivado are also analysed, which are shown in Figure~\ref{fig:sankey_diagram}. The figure depicts failing test cases in 3645 test cases that were passed to Vivado 2018.3, 2019.1 and 2019.2. All test cases that fail in the same tools are grouped together into a ribbon, showing when a bug is present in one of the tools. -Firstly, there is a group of failing test cases that is constant between all versions of Vivado HLS, meaning these are bugs that were not fixed between the versions. Other ribbons can be seen weaving in and out of failing for a version, meaning these bugs were fixed or reintroduced in those versions. From the diagram it can then be seen that Vivado HLS 2018.3 contains the most failing test cases compared to the other versions, having 62 test cases fail in total. Interestingly, Vivado HLS 2019.1 and 2019.2 have a different number of failing test cases, meaning feature improvements that introduced bugs as well as bug fixes between those minor versions. - - +There is a group of failing test cases that is constant between all versions of Vivado HLS, meaning these are bugs that were not fixed between the versions. Other ribbons can be seen weaving in and out of failing for a version, meaning these bugs were fixed or reintroduced in those versions. The diagram demonstrates that Vivado HLS 2018.3 contains the most failing test cases compared to the other versions, having 62 test cases fail in total. Interestingly, Vivado HLS 2019.1 and 2019.2 have a different number of failing test cases, meaning feature improvements that introduced bugs as well as bug fixes between those minor versions. \definecolor{ribbon1}{HTML}{8dd3c7} \definecolor{ribbon2}{HTML}{b3de69} @@ -179,6 +173,8 @@ Firstly, there is a group of failing test cases that is constant between all ver \caption{A Sankey diagram that tracks 3645 test cases through three different versions of Vivado HLS. The ribbons collect the test cases that pass and fail together. The 3573 test cases that pass in all three versions are not depicted. }\label{fig:sankey_diagram} \end{figure} +From this diagram it can also be observed that there must be at least six individual bugs that were found by the fuzzer in Vivado HLS, as a ribbon must contain at least one unique bug.\YH{Contradicts value of 3 in Table~\ref{tab:unique_bugs}, maybe I can change that to 6?}. + %%% Local Variables: %%% mode: latex %%% TeX-master: "main" |