3 files changed, 27 insertions, 2 deletions
diff --git a/evaluation.tex b/evaluation.tex
index c28b43f..7b1e6d6 100644
--- a/evaluation.tex
+++ b/evaluation.tex
@@ -13,8 +13,7 @@ Our evaluation is designed to answer the following three research questions.
 
 \paragraph{Choice of benchmarks.} We evaluate CoqUp using the PolyBench/C benchmark suite\footnote{\url{http://web.cs.ucla.edu/~pouchet/software/polybench/}}. PolyBench/C is a modern benchmark suite that has been previously used to evaluate HLS tools~\cite{choi+18}. For completeness, we use the full set of 24 benchmarks. We set the benchmark parameters so that all datatypes are integers (since CoqUp only supports integers) and all datasets are `small' (to fit into the small on-chip memories). A current limitation of CoqUp, as discussed in Section~\ref{?}, is that it does not support addition and subtraction operations involving integer literals not divisible by 4. To work around this, we lightly modified each benchmark program so that literals other than multiples of 4 are stored into variables before being added or subtracted. \JW{Any other notable changes to the benchmarks?}
 
-\paragraph{Experimental setup.} In order to generate a hardware implementation, the Verilog produced by the HLS tool-under-test must be synthesised to a netlist using a tool such as Yosys~\cite{yosys} or Intel Quartus~\cite{quartus}. The resultant netlist can then be placed-and-routed for a particular FPGA device. In the ideal experimental setup, we would use the same netlist synthesis tool for both CoqUp and LegUp.
-However, we found that neither Yosys nor Quartus worked well with \emph{both} HLS tools. Quartus could synthesise efficient hardware from LegUp-generated Verilog in part because it detects opportunities to replace large numbers of registers with small RAM blocks, yet on CoqUp-generated Verilog, this RAM inference failed, leading to designs too large to fit onto the FPGA. Yosys had the same problem, but with the HLS tools reversed. So, in an effort to avoid disadvantaging either HLS tool, we use LegUp with Quartus and CoqUp with Yosys. In both cases, we then use Quartus to place-and-route the synthesised netlists for a \ref{?} FPGA.
+\paragraph{Experimental setup.} In order to generate a hardware implementation, the Verilog produced by the HLS tool-under-test must be synthesised to a netlist; the resultant netlist can then be placed-and-routed for a particular FPGA device. We use Intel Quartus~\cite{quartus} for both of these tasks, and we target an Arria 10 \ref{?} FPGA.
 
 \subsection{RQ1: How fast is CoqUp-generated hardware?}
 
diff --git a/results/comparison.csv b/results/comparison.csv
new file mode 100644
index 0000000..7983363
--- /dev/null
+++ b/results/comparison.csv
@@ -0,0 +1,26 @@
+benchmark,coqupcycles,coqupfreqMHz,coquplogicutilisation,coqupregs,coquprams,coqupdsps,coqupwallclockMS,coquputilisation,legupcycles,legupfreqMHz,leguplogicutilisation,legupregs,leguprams,legupdsps,legupwallclockMS,leguputilisation
+durbin,21581,18.23,2941,4750,0,8,1.18,0.69,15106,188.61,2509,4008,0,8,0.08,0.59
+lu,2685403,19.21,54188,104005,0,6,139.79,12.68,482766,244.62,3116,4593,0,10,1.97,0.73
+ludcmp,2460807,18.78,58022,108157,0,10,131.03,13.58,470843,249.69,3605,5397,0,15,1.89,0.84
+trisolv,24501,19.1,28792,54716,0,4,1.28,6.74,35382,213.9,2412,3749,0,3,0.17,0.56
+2mm,381553,20.21,130339,75976,0,19,18.88,30.51,60088,226.5,1114,1936,0,7,0.27,0.26
+3mm,498115,19.74,48152,87308,0,18,25.23,11.27,204195,188.96,4210,4801,0,43,1.08,0.99
+atas,54037,21.75,29525,55787,0,6,2.48,6.91,126288,193.24,3026,3719,0,10,0.65,0.71
+bicg,47877,20.91,31721,57142,0,8,2.29,7.43,11907,303.4,308,537,0,6,0.04,0.07
+mvt,65779,20.71,33574,57552,0,6,3.18,7.86,16806,384.47,372,597,0,4,0.04,0.09
+doitgen,333951,20.14,20416,37147,0,4,16.58,4.78,57199,252.14,909,1402,0,2,0.23,0.21
+symm,218021,20.94,29518,52848,0,14,10.41,6.91,64903,284.41,2155,3170,0,10,0.23,0.50
+syrk,256315,20.81,71000,56199,0,8,12.32,16.62,57395,278.01,598,976,0,2,0.21,0.14
+syr2k,402419,20.82,116382,78402,0,12,19.33,27.24,125705,240.85,3149,3679,0,6,0.52,0.74
+trmm,134887,20.3,30757,33192,0,4,6.64,7.20,41432,281.61,610,990,0,4,0.15,0.14
+gemm,302137,18.64,33531,60552,0,16,16.21,7.85,83676,192.68,1029,1544,0,35,0.43,0.24
+gemver,106533,18.9,35603,63113,0,14,5.64,8.33,28087,303.49,1854,2380,0,8,0.09,0.43
+gesummv,34257,20.67,33930,61677,0,10,1.66,7.94,6634,310.46,298,504,0,4,0.02,0.07
+covariance,262265,18.73,30052,56122,0,4,14.00,7.03,109992,245.16,2098,3096,0,5,0.45,0.49
+fdtd-2d,826507,20.27,32133,60497,0,6,40.77,7.52,214153,262.61,2736,3801,0,2,0.82,0.64
+heat-3d,544497,20.38,34572,67057,0,13,26.72,8.09,41059,115.54,3132,2910,0,60,0.36,0.73
+jacobi-1d,17481,21.07,2478,2989,0,0,0.83,0.58,6914,386.25,1355,1885,0,0,0.02,0.32
+jacobi-2d,314377,20.75,31101,59410,0,4,15.15,7.28,84609,240.79,2347,3185,0,2,0.35,0.55
+seidel-2d,609259,18.8,27638,52840,0,2,32.41,6.47,345294,232.4,2128,3337,0,2,1.49,0.50
+floyd-warshall,4556081,17.2,60713,116401,0,2,264.89,14.21,1238764,235.52,1869,2367,0,2,5.26,0.44
+nussinov,881801,87.67,60649,119210,0,0,10.06,14.20,216467,273.07,1078,1431,0,2,0.79,0.25
+\ No newline at end of file
diff --git a/results/comparison.numbers b/results/comparison.numbers
new file mode 100755
index 0000000..88aa34d
--- /dev/null
+++ b/results/comparison.numbers