\section{Evaluation}\label{sec:evaluation}

\definecolor{vivado}{HTML}{7fc97f}
\definecolor{intel}{HTML}{beaed4}
\definecolor{legup}{HTML}{fdc086}
\begin{figure}
  \resizebox{0.47\textwidth}{!}{%
    \begin{tikzpicture}
      \draw (-14.5,7.65) rectangle (0,-1);
      \fill[vivado,fill opacity=0.5] (-4.4,4.4) ellipse (3.75 and 2.75);
      \fill[intel,fill opacity=0.5] (-10.2,4.4) ellipse (3.75 and 2.75);
      \fill[legup,fill opacity=0.5] (-7.3,2) ellipse (3.75 and 2.75);
      \draw[black] (-4.4,4.4) ellipse (3.75 and 2.75); % making the
      \draw[black] (-10.2,4.4) ellipse (3.75 and 2.75); % outlines
      \draw[black] (-7.3,2) ellipse (3.75 and 2.75);    % fully opaque
      \node at (-10.2,6.3) {\Large\textsf{\textbf{Vivado HLS}}};
      \node at (-4.4,6.3) {\Large\textsf{\textbf{Intel HLS}}};
      \node at (-7.3,0) {\Large\textsf{\textbf{LegUp}}};

      \node at (-5.5,3) {\Huge 1 (\textcolor{red}{1})};
      \node at (-9.1,3) {\Huge 4 (\textcolor{red}{0})};
      \node at (-3,5) {\Huge 26 (\textcolor{red}{540})};
      \node at (-11.6,5) {\Huge 79 (\textcolor{red}{20})};
      \node at (-7.3,1) {\Huge 162 (\textcolor{red}{6})};
      \node at (-7.3,5.2) {\Huge 0 (\textcolor{red}{5})};
      \node at (-7.3,3.8) {\Huge 0 (\textcolor{red}{0})};
      \node at (-13.6,-0.5) {\Huge 5856};
    \end{tikzpicture}
  }
\caption{A Venn diagram showing the number of failures in each tool out of 6700 test cases that were run.  Overlapping regions mean that the test cases failed in all those tools.  The numbers in parentheses represent the number of test cases that timed out.}\label{fig:existing_tools}
\end{figure}

\begin{table}
  \centering
  \begin{tabular}{lr}\toprule
    \textbf{Tool} & \textbf{Unique Bugs}\\
    \midrule
    Vivado HLS & $\ge 2$\\
    LegUp HLS & $\ge 3$\\
    Intel HLS & $\ge 1$\\
    \bottomrule
  \end{tabular}
  \caption{Unique bugs found in each tool.}
  \label{tab:unique_bugs}
\end{table}

To evaluate the different HLS tools 10,000 \JW{check numbers} test cases were generated and fed into each tool, keeping the test cases constant so that the comparison between the tools was fair.  Three different tools were tested, including three different versions of Vivado, which are shown in Table~\ref{tab:unique_bugs}.  Bugs were found in all tools that were tested, and in total, ?? unique bugs were found and reported to the tool vendors.

\subsection{Bugs found}

This section describes some of the bugs that were found in the various tools that were tested, going over different types of bugs in the process.  Each of the following bugs was first reduced automatically using C-Reduce, and then reduced further manually to achieve the minimal test case.

\subsubsection{LegUp Assertion Error}

The following piece of code produces an assertion error in LegUp even though it should compile without any errors, meaning an analysis pass in LegUp is incorrectly.  This assertion error is equivalent to an unexpected crash of the tool as it means that an unexpected state was reached by this input.  This shows that there is a bug in one of the compilation passes in LegUp, however, due to the assertion the bug is caught in the tool before it produces an incorrect design.

\begin{figure}
\begin{minted}{c}
int a[2][2][1] = {{{0},{1}},{{0},{0}}};

int main() { a[0][1][0] = 1; }
\end{minted}
\caption{An assertion bug in LegUp HLS.}
\label{fig:eval:legup:assert}
\end{figure}


The buggy test case has to do with initialisation and assignment to a three dimensional array, for which the above piece of code is the minimal example.  It initialises the array with zeros except for \texttt{a[0][1][0]}, which is set to one.  Then the main function assigns one to that same location, which causes LegUp to crash with an assertion error.

%The following code also produces an assertion error in LegUp, which is a different one this time.  This bug was not discovered during the main test runs of 10 thousand test cases, but beforehand, which meant that we disabled unions from being generated.  However, this bug also requires the \texttt{volatile} keyword which seems to be the reason for quite a few mismatches in LegUp and Vivado.
%
%\begin{minted}{c}
%union U1 { int a; };
%
%volatile union U1 un = {0};
%
%int main() { return un.a; }
%\end{minted}

\subsubsection{LegUp Miscompilation}

The following test case produces an incorrect netlist in LegUp 4.0, meaning the result of simulating the design and running the C code directly is different.

\begin{figure}
\begin{minted}{c}
volatile int a = 0;
int b = 1;

int main() { 
  int d = 1;
  if (d + a) b || 1;
  else b = 0;
  return b;
}
\end{minted}
\caption{An output mismatch where GCC returns 1 and LegUp HLS returns 0.}
\label{fig:eval:legup:wrong}
\end{figure}

In the code above, \texttt{b} has value 1 when run in GCC, but has value 0 when run with LegUp 4.0.  If the \texttt{volatile} keyword is removed from \texttt{a}, then the netlist contains the correct result.  As \texttt{a} and \texttt{d} are constants, the if-statement should always produce go into the \texttt{true} branch, meaning \texttt{b} should never be set to 0.

\subsubsection{Vivado Miscompilation}

The following code does not output the right value when compiled with Vivado 2019.2 and GCC.

\begin{figure}
\begin{minted}{c}
volatile unsigned int g_2 = 0;
int a[256] = {0};
int c = 0;

void d(char b) { c = (c & 4095) ^ a[(c ^ b) & 15]; }

void e(long f) {
  d(f); d(f >> 8); d(f >> 16); d(f >> 24);
  d(f >> 32); d(f >> 40); d(f >> 48);
}

int main() {
  for (int i = 0; i < 56; i++) a[i] = i;
  e(g_2); e(-2L);
  return c;
}
\end{minted}
\caption{An output mismatch where GCC returns \texttt{0xF}, whereas Vivado HLS return \texttt{0x0}.}
\label{fig:eval:vivado:mismatch}
\end{figure}

\subsection{Bugs in Vivado HLS versions}

In addition to the explanation to bugs given in Section~\ref{sec:evaluation}, bugs found in various versions of Vivado are also analysed, which are shown in Figure~\ref{fig:sankey_diagram}.  The figure depicts failing test cases in 3645 test cases that were passed to Vivado 2018.3, 2019.1 and 2019.2.  All test cases that fail in the same tools are grouped together into a ribbon, showing when a bug is present in one of the tools.

Firstly, there is a group of failing test cases that is constant between all versions of Vivado HLS, meaning these are bugs that were not fixed between the versions.  Other ribbons can be seen weaving in and out of failing for a version, meaning these bugs were fixed or reintroduced in those versions.  From the diagram it can then be seen that Vivado HLS 2018.3 contains the most failing test cases compared to the other versions, having 62 test cases fail in total.  Interestingly, Vivado HLS 2019.1 and 2019.2 have a different number of failing test cases, meaning feature improvements that introduced bugs as well as bug fixes between those minor versions.



\definecolor{ribbon1}{HTML}{8dd3c7}
\definecolor{ribbon2}{HTML}{b3de69}
\definecolor{ribbon3}{HTML}{bebada}
\definecolor{ribbon4}{HTML}{fb8072}
\definecolor{ribbon5}{HTML}{80b1d3}
\definecolor{ribbon6}{HTML}{fdb462}
\begin{figure}
  \centering
  \begin{tikzpicture}
    \draw[white, fill=ribbon1] (-1.0,4.1) -- (0.0,4.1000000000000005) to [out=0,in=180] (2.0,4.1000000000000005) to [out=0,in=180] (4.0,4.1000000000000005) -- (6.0,4.1000000000000005) -- %(7.55,3.325) -- 
    (6.0,2.5500000000000003) -- (4.0,2.5500000000000003) to [out=180,in=0] (2.0,2.5500000000000003) to [out=180,in=0] (0.0,2.5500000000000003) -- (-1.0,2.55) -- cycle;
    \draw[white, fill=ribbon2] (-1.0,2.55) -- (0.0,2.5500000000000003) to [out=0,in=180] (2.0,1.8) to [out=0,in=180] (4.0,1.55) -- (6.0,1.55) -- %(7.3,0.9) -- 
    (6.0,0.25) -- (4.0,0.25) to [out=180,in=0] (2.0,0.5) to [out=180,in=0] (0.0,1.25) -- (-1.0,1.25) -- cycle;
    \draw[white, fill=ribbon3] (-1.0,1.25) -- (0.0,1.25) to [out=0,in=180] (2.0,2.5500000000000003) to [out=0,in=180] (4.0,0.25) -- (6.0,0.25) -- %(6.05,0.225) -- 
    (6.0,0.2) -- (4.0,0.2) to [out=180,in=0] (2.0,2.5) to [out=180,in=0] (0.0,1.2000000000000002) -- (-1.0,1.2) -- cycle;
    \draw[white, fill=ribbon4] (-1.0,0.5) -- (0.0,0.5) to [out=0,in=180] (2.0,2.5) to [out=0,in=180] (4.0,0.2) -- (6.0,0.2) -- %(6.2,0.1) --
    (6.0,0.0) -- (4.0,0.0) to [out=180,in=0] (2.0,2.3000000000000003) to [out=180,in=0] (0.0,0.30000000000000004) -- (-1.0,0.3) -- cycle;
    \draw[white, fill=ribbon5] (-1.0,1.2) -- (0.0,1.2000000000000002) to [out=0,in=180] (2.0,0.5) to [out=0,in=180] (4.0,2.5500000000000003) -- (6.0,2.5500000000000003) -- %(6.2,2.45) --
    (6.0,2.35) -- (4.0,2.35) to [out=180,in=0] (2.0,0.30000000000000004) to [out=180,in=0] (0.0,1.0) -- (-1.0,1.0) -- cycle;
    \draw[white, fill=ribbon6] (-1.0,0.3) -- (0.0,0.30000000000000004) to [out=0,in=180] (2.0,0.30000000000000004) to [out=0,in=180] (4.0,2.35) -- (6.0,2.35) -- %(6.3,2.2) -- 
    (6.0,2.0500000000000003) -- (4.0,2.0500000000000003) to [out=180,in=0] (2.0,0.0) to [out=180,in=0] (0.0,0.0) -- (-1.0,0.0) -- cycle;

    \draw[white, fill=black] (-0.4,4.1) rectangle (0.0,1.0);
    \draw[white, fill=black] (1.8,4.1) rectangle (2.2,2.3);
    \draw[white, fill=black] (3.8,4.1) rectangle (4.2,2.05);

    \node at (-0.2,4.5) {2018.3};
    \node at (2,4.5) {2019.1};
    \node at (4,4.5) {2019.2};
    %\node at (2,5) {Vivado HLS};

    \node at (5.5,3.325) {31};
    \node at (5.5,0.9) {26};
    \node at (5.5,2.2) {6};

    \node[white] at (-0.2,1.2) {62};
    \node[white] at (2,2.5) {36};
    \node[white] at (4,2.25) {41};
  \end{tikzpicture}
  \caption{A Sankey diagram that tracks 3645 test cases through three different versions of Vivado HLS. The ribbons collect the test cases that pass and fail together. The 3573 test cases that pass in all three versions are not depicted. }\label{fig:sankey_diagram}
\end{figure}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:
