Update on Overleaf.

2 files changed, 26 insertions, 32 deletions

diff --git a/conference.bib b/conference.bib
index 6d65750..dbf355b 100644
--- a/conference.bib
+++ b/conference.bib
@@ -114,4 +114,12 @@ with {LegUp} High-Level Synthesis},
   doi = {10.1145/3373087.3375310},
   publisher = {{ACM}},
   year = {2020}
+}
+
+@inproceedings{creduce,
+  title={Test-case reduction for {C} compiler bugs},
+  author={Regehr, John and Chen, Yang and Cuoq, Pascal and Eide, Eric and Ellison, Chucky and Yang, Xuejun},
+  booktitle={Proceedings of the 33rd ACM SIGPLAN conference on Programming Language Design and Implementation},
+  pages={335--346},
+  year={2012}
 }
\ No newline at end of file
diff --git a/method-new.tex b/method-new.tex
index bbfcd2e..77f72c0 100644
--- a/method-new.tex
+++ b/method-new.tex
@@ -20,7 +20,7 @@ We present the following material in this section:
 % \item How we run each HLS tool, using timeouts as appropriate.
 % \end{itemize}
 
-\subsection{Program generation via CSmith}
+\subsection{Generating programs via CSmith}
 \label{sec:method:csmith}
 
 
@@ -101,7 +101,7 @@ We also limit the depth of statements and expressions of programs generated by C
 
 \NR{I remove Zewei's writing but please feel free to re-instate them} 
 
-\subsection{Program annotation for HLS testing}
+\subsection{Annotating programs for HLS testing}
 
 We annotate the programs generated by Csmith to prepare them for HLS testing. 
 We perform annotations in two ways: program instruction and directive injection.
@@ -139,7 +139,7 @@ This script instructs the tool to create a design project and perform the necess
 \NR{Have I missed any LegUp directives? What is these lines referring to. Supported Makefile directives include partial loop unrolling with a threshold, disable inline, and disable all optimizations. Available Config TCL directives include partial loop pipeline, all loop pipeline, disable loop pipeline, resource-sharing loop pipeline, and accelerating functions. }
 
 
-\subsection{HLS testing}
+\subsection{Testing various HLS tools}
 
 % \NR{Some notes on key points from this section: 
 % \begin{itemize}
@@ -178,35 +178,21 @@ When testing Intel HLS, we place three time-outs since its execution is generall
 
 \subsection{Reducing buggy programs}
 
-\NR{Once we discover any programs that crashes the HLS tool or whose C and RTL simulation do not match, we must further scrutinise these programs to identify the root cause(s) for these undesirable behaviours. As the programs generated by CSmith can be fairly large, we must systematically \emph{reduce} these programs to identify the source of a bug.}
+\NR{Draft paragraphs. Skeleton for Yann.}
 
-The reduction process will only be triggered if the C and RTL result does not match. Test cases that either crash or forced to terminate by the timeout do not proceed to the reduction stage. As reduction is performed iteratively, long runtime is expected. Only two tools, Vivado HLS and LegUp HLS, are equipped with the reduction method. It is ignored for Intel HLS due to excessive runtime. The reduction flow can be described as follows.
+Once we discover any programs that crashes the HLS tool or whose C and RTL simulation do not match, we must further scrutinise these programs to identify the root cause(s) for these undesirable behaviours. 
+As the programs generated by CSmith can be fairly large, we must systematically reduce these programs to identify the source of a bug.
 
-\begin{enumerate}
-  \item Reduction starts with the top-level function, which by default named func\underline{ }1. As being said, the random program generated by CSmith follows a strict order that the main always call the top-level function. So, we can trace down to the root problem by iteratively reduce the top-level function and see if any sub-function has been called. 
+Reduction is performed by removing some part of the original program and then provide this reduced program to the HLS tool for re-synthesis and co-simulation. 
+The goal is to find a program that is small enough that still crashes or does not match the output of the C execution. 
+We apply two consecutive methods of reduction in this work. 
+We first perform a custom reduction in which we iteratively remove lines in strict order from the top-level function. 
+\NR{We can add one or two more sentences summarising how we reduce the programs. Zewei is probably the best person to add these sentences.}
 
-    The program is first being processed to extract the number of removable lines inside the func\underline{ }1. Lines, include variable declarations, variable initialization, for loops, if/else statements, while loops, goto and goto labels, continue, break, and return, do not count as removable lines. The reason for not removing variable declarations and initialization is that it can cause the undeclared variable error. Other non-removable features such as if/else statements and goto are kept to ensure the original logic flow. Changing the logic flow can lead to bugs went undetected, since if changed, some path might not be taken but was taken initially. By keeping those lines, the original functionality and logic flow of the program can be maximumly preserved. 
-
-  \item By getting the total number of possible removable lines, each line will be iterative commented out. One line per time eases the aim of finding the exact line that causes the discrepancy in results. A new golden result will need to be produced since each line can have an impact on the final hash variable. The timeout command is placed to avoid non-terminating programs. 
-  
-  \item Once the new golden result is generated, the program will be fed into HLS tools and go through the same testing procedures as standard test cases. A new RTL result will be produced under the timeout constrain. 
-  
-  \item 4.	The new golden result is then compared against the new RTL result. Two possible cases can happen:
-    \begin{itemize}
-     \item If the RTL result matches with the golden result, we can confirm that the current comment-out line causes the discrepancy in results previously. Then several modifications will be made on this problematic line.
-     
-    Firstly, lines after the bug-trigger line but within the current function will be removed directly since the problem has been detected. The bug-trigger line will be the new return statement. Then, close curly brackets will be added to meet the standard C syntax if needed. For example, if the problematic line is contained inside a nested for-loop originally, removing the following lines removes the close curly brackets simultaneously, which can lead to an invalid program. So close curly brackets must be added back to match with the number of open curly brackets used.  
-    
-    Secondly, the problematic line will be checked to see if it calls other functions. If it does, the reduction focus will be moved to the functions it called. For example, if func\underline{ }1 calls the sub-function func\underline{ }2, and this calling statement is detected as problematic. Func\underline{ }2 needs to be reduced as well since the root problem can be embedded inside the func\underline{ }2. Whereas if the other functions are not being called at this line, we can conclude the reduction process and confirm that the current line is the root problem.
-
-     \item If the RTL result again does not match with the golden result, the current comment-out line is not the bug-trigger. The reduction process will move on to the next possible removable line and repeat the procedure. 
-    \end{itemize}
-    
-    \item The reduction process will terminate when the problematic line is detected, as described in step 4.1, or when there are no more possible removable lines inside the func\underline{ }1. The second case means that the discrepancy in results is not caused by a single line. It is possible that the bug is triggered by a combination of several statements or is triggered by the hashing function embedded inside the main. Bugs triggered by a combination of statements require a thorough understanding of the logic flow of the program, which is hard to reduce automatically. And as for hashing functions, the transparent\underline{ }crc hashing is complicated and involved massive bug-prone operations such as shifts and sign extensions. The imprecise automatic reduction can introduce new bugs. Thus, both cases require manual reduction work. 
-\end{enumerate}
-A short note will be generated automatically once exited from the reduction process. The memo can describe three different exit conditions including 1) the program is reduced and the exact problematic line is detected; 2) the exact problematic line cannot be confirmed, and manual work is required; 3) the reduction process does not function properly which might cause by the timeout commands. The note is useful for later checking the condition of the reduced program.
-
-By the time of writing, the reduction method showed the ability to reduce the program but unable to achieve the minimal working example sometimes due to manual work required.  
-
-\subsection{How we summarize}
-A simple automated analysis is done after all test cases finished, and a summary list is saved to a result\underline{ }check file for display. The complete result file extracted and saved as described in the \ref{extraction_compare_section} is used for analysis. Several data are collected regarding the overall results, including the number and label of test cases that have unmatched C/RTL results, that has failed to terminate C program, that being forced timeout during RTL synthesis and simulation, that trigger assertion error, and that has pragma error. Those data are extracted by looping through the result file and will be displayed through the terminal.
+Although, our custom reduction gives us the freedom and control of how reduce buggy programs, it is arduous and require a lot of manual effort. 
+Hence, 
+Secondly, we also provide larger and more complex programs to Creduce~\cite{creduce}. 
+Creduce enables a more systematic and automatic reduction of C programs. 
+The downside with Creduce is that we are not in control of which lines and features are prioritised for removal. 
+As a consequence, we can easily end up with Creduce producing smaller programs that are not HLS-friendly and are unysnthesisable, despite being valid C. 
+To avoid this scenario, we \NR{bla... Yann can explain this better.}