summaryrefslogtreecommitdiffstats
path: root/eval.tex
diff options
context:
space:
mode:
authorYann Herklotz <git@yannherklotz.com>2020-09-09 09:49:45 +0100
committerYann Herklotz <git@yannherklotz.com>2020-09-09 09:49:45 +0100
commit796e4081cfb7a17ad5c3444729989ad640b7b4a5 (patch)
tree06f205b6b0c04df09b623193c6d3b91481df99e4 /eval.tex
parent3f121b82024a84d8ffdb2be9f1cb48118b03df0d (diff)
downloadfccm21_esrhls-796e4081cfb7a17ad5c3444729989ad640b7b4a5.tar.gz
fccm21_esrhls-796e4081cfb7a17ad5c3444729989ad640b7b4a5.zip
Add venn diagram
Diffstat (limited to 'eval.tex')
-rw-r--r--eval.tex38
1 files changed, 37 insertions, 1 deletions
diff --git a/eval.tex b/eval.tex
index a18998c..29e9a33 100644
--- a/eval.tex
+++ b/eval.tex
@@ -1,8 +1,44 @@
\section{Evaluation}
+\definecolor{vivado}{HTML}{7fc97f}
+\definecolor{intel}{HTML}{beaed4}
+\definecolor{legup}{HTML}{fdc086}
+\begin{figure}
+ \resizebox{0.5\textwidth}{!}{%
+ \begin{tikzpicture}
+ \begin{scope}
+ \draw[fill=vivado,fill opacity=0.5] (-4.4,4.4) ellipse (3.75 and 2.75);
+ \draw[fill=intel,fill opacity=0.5] (-10.2,4.4) ellipse (3.75 and 2.75);
+ \draw[fill=legup,fill opacity=0.5] (-7.3,2) ellipse (3.75 and 2.75);
+ \node at (-10.2,6.3) {\Large\textsf{\textbf{Vivado}}};
+ \node at (-4.4,6.3) {\Large\textsf{\textbf{Intel HLS}}};
+ \node at (-7.3,0) {\Large\textsf{\textbf{Legup}}};
+ \end{scope}
+
+ \end{tikzpicture}
+ }
+\caption{Venn diagram showing the existing tools and their current features. \textbf{Implementation} refers to usable HLS tools, whereas \textbf{Proof} refers to papers that demonstrate proofs of an algorithm without necessarily linking that proof with the algorithm that implements it.}\label{fig:existing_tools}
+\end{figure}
+
+\begin{table}
+ \centering
+ \begin{tabular}{lr}\toprule
+ \textbf{Tool} & \textbf{Bugs found}\\
+ \midrule
+ Vivado 2018.3 & 2\\
+ Vivado 2019.1 & 2\\
+ Vivado 2019.2 & 2\\
+ Legup 4.0 & 4\\
+ Intel HLS & 1\\
+ \bottomrule
+ \end{tabular}
+ \caption{Unique bugs found in each tool.}
+ \label{tab:unique_bugs}
+\end{table}
+
During the implementation stage, the testing system successfully detected bugs among all three tools. The test cases used were not the same for each tool, because, as being mentioned, each tool has its own supported syntax. Thus, to ensure the validness of test cases during the implementation stage, test cases were all generated based on the supported syntax of the targeted tools.
-After the implementation of testing system is proven to be working, a constant set of 10,000 test cases are generated and fed into HLS tools for constructing the bigger picture regarding the quality of HLS tools. The 10,000 test cases were kept constant to ensure the comparison is fair between tools. Unfortunately, 10,000 test runs were still in progress by the time of writing this thesis, as runtime for each test case can range from 5 minutes up to several hours. Data showing in \ref{result table for 10k test cases} is collected when writing this report.
+After the implementation of testing system is proven to be working, a set of 10,000 test cases are generated and fed into HLS tools for constructing the bigger picture regarding the quality of HLS tools. The 10,000 test cases were kept constant to ensure the comparison is fair between tools. Unfortunately, 10,000 test runs were still in progress by the time of writing this thesis, as runtime for each test case can range from 5 minutes up to several hours. Data showing in \ref{result table for 10k test cases} is collected when writing this report.
Three versions of Vivado HLS, including version 2019.2, 2019.1, and 2018.3, were tested. As being mentioned before, Vivado HLS version 2019.2 and 2019.1 won’t process logical AND operator with constant, which will warn about changing to bitwise AND operator (\verb|&|) and then error out. Thus, as shown in \ref{result table}, the column named as “Invalid test cases” indicates test cases that have logical AND operator (\verb|&&|) with constants. It is also being said that version 2018.3 managed to cope with the logical AND operator, so the “invalid test cases” section for version 2018.3 only count the test cases that has GCC and Vivado HLS’s C simulation result unmatched.