Update on Overleaf.

1 files changed, 43 insertions, 0 deletions

diff --git a/eval.tex b/eval.tex
new file mode 100644
index 0000000..4d16d9d
--- /dev/null
+++ b/eval.tex
@@ -0,0 +1,43 @@
+
+\section{Result}
+During the implementation stage, the testing system successfully detected bugs among all three tools. The test cases used were not the same for each tool, because, as being mentioned, each tool has its own supported syntax. Thus, to ensure the validness of test cases during the implementation stage, test cases were all generated based on the supported syntax of the targeted tools. 
+
+After the implementation of testing system is proven to be working, a constant set of 10,000 test cases are generated and fed into HLS tools for constructing the bigger picture regarding the quality of HLS tools. The 10,000 test cases were kept constant to ensure the comparison is fair between tools. Unfortunately, 10,000 test runs were still in progress by the time of writing this thesis, as runtime for each test case can range from 5 minutes up to several hours. Data showing in \ref{result table for 10k test cases} is collected when writing this report.  
+
+Three versions of Vivado HLS, including version 2019.2, 2019.1, and 2018.3, were tested. As being mentioned before, Vivado HLS version 2019.2 and 2019.1 won’t process logical AND operator with constant, which will warn about changing to bitwise AND operator (\verb|&|) and then error out. Thus, as shown in \ref{result table}, the column named as “Invalid test cases” indicates test cases that have logical AND operator (\verb|&&|) with constants. It is also being said that version 2018.3 managed to cope with the logical AND operator, so the “invalid test cases” section for version 2018.3 only count the test cases that has GCC and Vivado HLS’s C simulation result unmatched. 
+
+Moving on to the LegUp HLS version 4.0, the C/RTL unmatched result section is composed of assertion errors and C and RTL results unmatched. The assertion error happens when translating from C to RTL, which results in no Verilog file being produced. Therefore, this condition is also considered as inconsistent results since the LegUp HLS failed to translate an accurate RTL description for the inputted valid C code. Although, proportionally, the total number of C and RTL mismatch detected in LegUp HLS is much higher compared to which of Vivado HLS, two points must be emphasized. Firstly, LegUp HLS version 4.0 was implemented long time ago in 2015 and published as open sources for academic and research uses. By the time of writing this thesis, version 8.0 was just released for commercial uses. As mentioned before, version 7.5 was installed and run as GUI. Although it was not being fuzzing-tested under 10,000 test cases due to difficulties encountered when running it through the command line, it was used as a reference for comparing differences between versions released. By trying out several failing tests on version 7.5, some of the failing tests passed successfully without causing the same problem. Then we can confirm that some of the embedded issues have been solved already. Thus, lesser discrepancy in results when running through the newer versions should be expected. Secondly, Vivado HLS version 2019.2 errors out plenty of test cases due to the pragma error. Only a subset of test cases was synthesized to RTL and being simulated. Reducing in overall valid test cases can result in a lower amount of unmatched test cases. Thus, the results for LegUp HLS should not be treated “equally” with which for Vivado HLS. Those two points needed to be taken into consideration when analyzing the results obtained from LegUp HLS. 
+
+Finally, as for Intel HLS, three points need to be taken into account when analyzing the result for Intel HLS. Firstly, the hashing function used for Intel HLS is much simpler comparing to which is used for both Vivado HLS and LegUp HLS. One possible result of using simple hashing function is that a considerable amount of bug can go undetected, which can lower the total number of cases that have C and RTL results unmatched. Secondly, as Intel HLS is running under the Windows environment, it runs relatively slower comparing to other tools that run under the Linux system. Based on the result, a considerable amount of test runs was timed out, which ultimately decreases the total number of valid test runs. Therefore, similar to what has been mentioned about the reduced overall valid test cases for Vivado HLS, when analyzing the Intel HLS, this should also be taken into consideration. Lastly, as Intel HLS uses the I++ compiler instead of GCC, differences can exist. Although theoretically, the C++ compiler does have stricter requirements, it should be compatible with compiling C programs. And as the test programs were generated through Csmith in C language, words like “bool” or “class”, which are only supported in C++ but not in C, do not exist. Also, Csimth was forbidden to generate operations like malloc, which might trigger incompatibility between C and C++. Thus, although compatibility can pose a potential problem, it should not have a huge impact. Besides those three points, similar to Vivado HLS, Intel HLS will alert about using logical AND operation with constants, but it does not error out immediately. So the “invalid test cases” section is set to not applicable. 
+
+
+\section{Evaluation}
+
+We generate $10$ thousand test programs to test on three different HLS tools, including three versions of Vivado HLS, one version of LegUp HLS and one version of Intel HLS. 
+We provide the same set of programs to all HLS tools and show that different tools give rise to unique bugs. 
+
+\NR{
+What we need for this section:
+\begin{itemize}
+    \item Venn diagrams to show the overall results. Onlyy missing information from Venn diagram is the unique bugs per tool, which we can provide with a smaller table. This table can actually lead nicely to the bug portfolio.
+    \item Sankey diagram for different versions of Vivado HLS. 
+    \item A portfolio of bugs on various tools, together with intuitive explanation for why we see these bugs. We can also mention filing of bug reports, where relevant.
+    \item Is the area and latency of buggy hardware of interest to us?
+\end{itemize}
+}
+
+\subsection{Venn diagram of results}
+
+\subsection{Bug portfolio}
+\NR{In fact, the structure of this section relates to the Venn diagram.}
+
+\subsubsection{Vivado HLS}
+
+\subsubsection{LegUp HLS}
+
+\subsubsection{Intel HLS}
+
+
+
+
+  
\ No newline at end of file