summaryrefslogtreecommitdiffstats
path: root/eval.tex
diff options
context:
space:
mode:
authorJohn Wickerson <j.wickerson@imperial.ac.uk>2020-09-22 10:36:40 +0000
committeroverleaf <overleaf@localhost>2020-10-24 15:09:08 +0000
commit25fc95d19a586f774a99630ea34e58fb76e4e629 (patch)
tree5809da7bc117d8bfd2049b3bca90aa610589c5d5 /eval.tex
parentdfac4f477dfa32611be640c2fef65646e717a6f0 (diff)
downloadfccm21_esrhls-25fc95d19a586f774a99630ea34e58fb76e4e629.tar.gz
fccm21_esrhls-25fc95d19a586f774a99630ea34e58fb76e4e629.zip
Update on Overleaf.
Diffstat (limited to 'eval.tex')
-rw-r--r--eval.tex18
1 files changed, 9 insertions, 9 deletions
diff --git a/eval.tex b/eval.tex
index 8d42ce3..03d1357 100644
--- a/eval.tex
+++ b/eval.tex
@@ -13,7 +13,7 @@
\draw[white] (-4.4,4.4) ellipse (3.75 and 2.75); % making the
\draw[white] (-10.2,4.4) ellipse (3.75 and 2.75); % outlines
\draw[white] (-7.3,2) ellipse (3.75 and 2.75); % fully opaque
- \node[align=center] at (-10.2,6.3) {\Large\textsf{\textbf{Xilinx Vivado HLS}} \\ \Large\textsf{\textbf{2019.1}}};
+ \node[align=center] at (-10.2,6.3) {\Large\textsf{\textbf{Xilinx Vivado HLS}} \\ \Large\textsf{\textbf{v2019.1}}};
\node at (-4.4,6.3) {\Large\textsf{\textbf{Intel i++ 18.1}}};
\node at (-7.3,0) {\Large\textsf{\textbf{LegUp 4.0}}};
@@ -40,7 +40,7 @@
Intel i++ & $\ge 1$\\
\bottomrule
\end{tabular}
- \caption{Unique bugs found in each tool. %\JW{is `all versions' correct here? and should we add version numbers like in the Venn?}\YH{Yes it is actually correct here, I don't mind adding the specific version either though}\JW{Ok let's leave it as-is.}
+ \caption{Unique bugs found in each tool. The ``$\ge$'' sign signifies a lower bound on the number of unique bugs found and correspond to unique test cases after reduction. %\JW{is `all versions' correct here? and should we add version numbers like in the Venn?}\YH{Yes it is actually correct here, I don't mind adding the specific version either though}\JW{Ok let's leave it as-is.}
}
\label{tab:unique_bugs}
\end{table}
@@ -67,10 +67,10 @@ We write `$\ge$' in the table to indicate that all the bug counts are lower boun
\subsection{Results across versions of an HLS tool}
-Besides comparing the reliability of different HLS tools, we also investigated the reliability of Vivado HLS over time. Figure~\ref{fig:sankey_diagram} shows the results of giving 3645 test-cases to Vivado HLS 2018.3, 2019.1 and 2019.2.
+Besides comparing the reliability of different HLS tools, we also investigated the reliability of Vivado HLS over time. Figure~\ref{fig:sankey_diagram} shows the results of giving 3645 test-cases to Vivado HLS v2018.3, v2019.1 and v2019.2.
Test-cases that pass and fail in the same tools are grouped together into a ribbon.
-For instance, the topmost ribbon represents the 31 test-cases that fail in all three versions of Vivado HLS. Other ribbons can be seen weaving in and out; these indicate that bugs were fixed or reintroduced in the various versions. The diagram demonstrates that Vivado HLS 2018.3 contains the most failing test-cases compared to the other versions, having 62 test-cases fail in total. %Interestingly, Vivado HLS 2019.1 and 2019.2 have a different number of failing test cases, meaning feature improvements that introduced bugs as well as bug fixes between those minor versions.
-Interestingly, as an indicator of reliability of HLS tools, the blue ribbon shows that there are test-cases that fail in v2018.3, pass in v2019.1 but then fail again in 2019.2.
+For instance, the topmost ribbon represents the 31 test-cases that fail in all three versions of Vivado HLS. Other ribbons can be seen weaving in and out; these indicate that bugs were fixed or reintroduced in the various versions. The diagram demonstrates that Vivado HLS v2018.3 contains the most failing test-cases compared to the other versions, having 62 test-cases fail in total. %Interestingly, Vivado HLS 2019.1 and 2019.2 have a different number of failing test cases, meaning feature improvements that introduced bugs as well as bug fixes between those minor versions.
+Interestingly, as an indicator of reliability of HLS tools, the blue ribbon shows that there are test-cases that fail in v2018.3, pass in v2019.1 but then fail again in v2019.2.
\definecolor{ribbon1}{HTML}{8dd3c7}
\definecolor{ribbon2}{HTML}{b3de69}
@@ -98,9 +98,9 @@ Interestingly, as an indicator of reliability of HLS tools, the blue ribbon show
\draw[white, fill=black] (1.8,4.1) rectangle (2.2,2.3);
\draw[white, fill=black] (3.8,4.1) rectangle (4.2,2.05);
- \node at (-0.2,4.5) {2018.3};
- \node at (2,4.5) {2019.1};
- \node at (4,4.5) {2019.2};
+ \node at (-0.2,4.5) {v2018.3};
+ \node at (2,4.5) {v2019.1};
+ \node at (4,4.5) {v2019.2};
%\node at (2,5) {Vivado HLS};
\node at (5.5,3.325) {31};
@@ -122,7 +122,7 @@ In addition to that, it can then be seen that Vivado HLS v2018.3 must have at le
\subsection{Some specific bugs found}
-This section describes some of the bugs that were found in the various tools that were tested. We describe two bugs in LegUp and one in Vivado HLS; in each case, the bug was first reduced automatically using \creduce{}, and then reduced further manually to achieve the minimal test-case. Although we did find test-case failures in Intel i++, the very long compilation times for that tool meant that we did not have time to reduce any of the failures down to an example that is minimal enough to present here.
+This section describes some of the bugs that were found in the various tools that were tested. We describe two bugs in LegUp and one in Vivado HLS; in each case, the bug was first reduced automatically using \creduce{}, and then reduced further manually to achieve the minimal test-case. Although we did find test-case failures in Intel i++, the long compilation times for that tool meant that we did not have time to reduce any of the failures down to an example that is minimal enough to present here.
\subsubsection{LegUp assertion error}