Skip to content

Latest commit

 

History

History
298 lines (188 loc) · 11 KB

artifact-evaluation.org

File metadata and controls

298 lines (188 loc) · 11 KB

Artifact Evaluation ASPLOS’24 AEC Spring

Here are the instructions to reproduce all of the figures in our paper.

Many of our experiments require large amounts of memory to complete, and access to proprietary tools to compile kernels for the Tensilica architecture and perform cycle estimation. Educational licenses are available for free here, but we also separate out the generation of the data from the making of plots. We provide all of the data that we used for the paper.

For each figure, we will split instructions into how to generate the data, and how to make the plot. Because our larger kernels can take a long time to run, we provide the data so that you can generate the plots without waiting for all the experiments to run.

Setup

Space requirements: 10 Gb

Linux and Mac OS are the only supported operating systems.

For both operating systems, you will need to install either Podman or Docker. I prefer podman, but both will work identically. If you have both installed, the scripts will automatically use podman.

Once you have one of these tools installed, make a directory to work in.

mkdir isaria
cd isaria

Then download the following script and make it runnable.

wget https://raw.githubusercontent.com/sgpthomas/comp-gen/main/aec/run.sh
chmod +x run.sh

This is a simple helper script that handles some of the more complicated configuration of running commands inside of a container. The first time you run it, it will download a container for you. Subsequent runs will be much faster.

Copy the provided key into the isaria directory that you made. Make sure that it is named exactly aec.pem. Then export an environment variable with the IP of the provided server.

export EXP_SERVER=<provided-ip>

Mac Specific

If you are on a Mac and using podman, then you also have to make sure that the podman virtual machine is downloaded and running.

podman machine init
podman machine start

Reproducing our figures

Estimated time: 1 hour

This section describes the commands to run to make our figures, and run shorter versions of our experiments. Running everything in this section will take about an hour. Because these are not the full experiments, the figures won’t look exactly the same as they do in the paper.

Performance and Compile Time (Figures 4 & 5)

Making figures

So that you can generate the plots without waiting for all of the experiments to finish, we have provided the data that we used inside of the container. You can use this to generate our figures.

To make figure 4 and 5, run the following commands.

./run.sh ./aec.py make figure4
./run.sh ./aec.py make figure5

This will put two pdfs into output/. Take a look at them there.

Now that we know how to make the figures, we can turn our attention to generating the data.

Running experiments (Estimated time: 10 minutes)

As part of the AEC committee, you have been provided with a server ip. Otherwise, you will need to setup an experiment server as described in the nitty gritty details section. Plug the server ip into the following command. This will run a short version of the experiments needed for figures 4 and 5.

./run.sh ./aec.py gen-data overall --ip $EXP_SERVER

Effects of Pruning (Figure 6)

The flow for generating figure 6 and the data for it, is identical to the previous figures.

Make figure

Run the following command to generate the figure using the pre-existing data.

./run.sh ./aec.py make figure6

Running experiments (Estimated time: 15 minutes)

Running the following command will launch the pruning experiments.

./run.sh ./aec.py gen-data pruning --ip $EXP_SERVER

Once the command has completed, you can regenerate the figure.

Exploration of time spent generating rules (Figure 7)

Make figure

Run the following command to generate the figure using the pre-existing data.

./run.sh ./aec.py make figure7

Running experiments (Estimated time: 10 minutes)

Running the following command will launch the ruleset_ablation experiments.

./run.sh ./aec.py gen-data ruleset_ablation --ip $EXP_SERVER

Once the command has completed, you can regenerate the figure.

Adding new instructions (Table 2)

Make table

Run the following command to generate the table using the pre-existing data.

./run.sh ./aec.py make table2

Running experiments (Estimated time: 10 minutes)

Running the following command will launch the new instructions experiments.

./run.sh ./aec.py gen-data new_instructions --ip $EXP_SERVER

Once the command has completed, you can regenerate the figure.

Exploring the effect of alpha and beta (Figures 8 & 9)

Make figure

Run the following commands to generate the figures using the pre-existing data.

./run.sh ./aec.py make figure8
./run.sh ./aec.py make figure9

Run experiments (Estimated time: 10 minutes)

Running the following command will launch the alpha beta ablation experiments.

./run.sh ./aec.py gen-data alpha_beta --ip $EXP_SERVER

Once the command has completed, you can regenerate the figures.

Running complete experiments

Above, we have shown how to reproduce smaller versions of the experiments that we present in our paper. To run the full version of experiments. You can pass the --all flag to every one of the commands above. There are also a few additional commands that you need to run to regenerate the rulesets used.

Cleanup

After you are done playing with the artifact, you can remove the container.

podman stop isaria
podman rm isaria

Nitty-Gritty details

This section is for the brave who want to build the experiment server container (or run the server outside of a container), and setup an xtensa installation for performing cycle estimates. This is not needed for the AEC committee because I have set everything up for them. However, if anybody else wants to run this in the future, these instructions will be necessary.

Building experiment server container with buildah

To build, you need buildah and a relatively up-to-date fedora machine. To keep the image as small as possible, we start the image from just a base file system and use the host package manager to install packages in the image. Running the aec/fedora-build-image.sh inside of a buildah unshare session should do all the hard-work for you.

buildah unshare ./aec/fedora-build-image.sh

If you want to build and run the server from scratch, read the fedora-build-image script to see what all the dependencies are.

Setting up XtensaTools

Setup XtensaTools

You first need to download the files. Login to the XPG cadence portal and then go to the XPG View tab to select the version that you want to install. We did our testing on RI-2021.8.

We don’t want to have to go through the Xplorer IDE to compile and simulate kernels. So we just need to download the tools/Xtensa Tools/Xtensa Tools 14.08 for Linux file by navigating to it, and then pressing the green download button.

You’ll also want to download refernece-cores/Fusion G3 DSP cores for Linux.

Finally, navigate to the XPG License Manager, and download the license server software. I’m using v11.15

You should now have the following files:

XtensaTools_RI_2021_8_linux.tgz
XRC_FUSIONG3-linux.xws
licserv_linux_x64_v11_15.tgz

Copy these files into a directory named xtensa. Extract all of them. The .xws file is a zip archive in disguise. You can use unzip to extract it’s contents.

tar xvf XtensaTools_RI_2021_8_linux.tgz
unzip XRC_FUSIONG3-linux.xws
tar xvf licserv_linux_x64_v11_15.tgz

Start License server

The last thing that we need to do, is get a license and start the license server.

Find machine host-id

You need the host-id of the machine you want to run the license server on to cut the license keys.

./x64_lsb/lmutil lmhostid

This will give you the host-id. If you get this error: ./x64_lsb/lmutil: No such file or directory, then you have to create a symlink as per the following instructions.

Fix dynamically linked binaries

The binaries distributed with the license server expect /lib64/ld-lsb-x86-64.so.3 to exist. For whatever reason, this doesn’t exist on the version of Ubuntu that I used. If you run into this, you can link /lib64/ld-linux-x86-64.so.2 to /lib64/ld-lsb-x86-64.so.3 which solves the problem.

sudo ln -sf /lib64/ld-linux-x86-64.so.2 /lib64/ld-lsb-x86-64.so.3

Cut License keys

In the XPG License Manager web interface, press Add new host, give it a name, use Linux MAC and floating server and then enter the host id that you found in the previous step. Then Save New Host.

Once the host is created, click manage. Add ( + 1 ) for the New allocation on this host for all rows, and then press Cut. Save to file, and then copy into the xtensa directory.

Open the license file, and make the following edits:

  1. Change the line starting with SERVER to SERVER <hostname> <host-id> 27010, filling in hostname with the hostname of the machine you are running on. The host-id should already be correct.
  2. Change the line starting with VENDOR xtensad so that it points to the x64_lsb directory inside of the xtensa directory. My VENDOR line is VENDOR xtensad /home/ubuntu/xtensa/x64_lsb/.

Start server

The server expects /usr/tmp/.flexlm to exist. You can create it with the command

sudo mkdir -p /usr/tmp/.flexlm

Finally, we are all setup to start the server.

./x64_lsb/lmgrd -c <LICENSE_FILE>

Starting the server

You first need to pull the container image from the registry.

podman pull ghcr.io/sgpthomas/isaria-aec:latest

Then you can start the server.

mkdir -p jobs completed
podman run --rm -it \
       -v ./completed:/root/comp-gen/server/completed:U \
       -v ./jobs:/root/comp-gen/server/jobs:U \
       -v ./xtensa:/root/xtensa:U \
       --network slirp4netns:allow_host_loopback=true \
       --name isaria \
       ghcr.io/sgpthomas/isaria-aec

Building plotting container with buildah

buildah unshare ./aec/fedora-build-figure-image.sh