figems

Concretely, suppose you a program that can be run with many different settings and each run generates some output files. figems will automatically place each run in a separate directory and provides a powerful web interface to view/organize these runs.

Tutorial

Setup

1. Download figems and unpack it (you've probably already done this).
2. Make sure you have Java 5 installed and in your path. Set the JAVA_HOME environment variable (usually something like /usr/lib/jvm/java-1.5.0-sun).
3. Run source rc (assume you're using bash) in the figems directory. This sets up the environment.
4. Run figems-start to start the Apache tomcat servlet. You should see something like this:

   Using CATALINA_BASE:   /home/pliang/figems/tomcat
   Using CATALINA_HOME:   /home/pliang/figems/tomcat
   Using CATALINA_TMPDIR: /home/pliang/figems/tomcat/temp
   Using JRE_HOME:       /usr/lib/jvm/java-1.5.0-sun
   

   Go to http://localhost:8080/figems and you should see a page that says "Welcome to figems!" at the top.

Running examples

1. Let's try out some examples. Go to figems/examples/bash. There should be one script called count. You can try running it (type ./count 5) to see that it prints some stuff out and generates some files.
2. Now type execwrapper $PWD/count 5 to run it in the figems execution environment. Note: $PWD/count expands to the absolute path of the script, which is needed because execwrapper runs the script in another directory. Now, you should see some screen output:

   Execution directory: /home/pliang/figems/examples/bash/state/execs/0.exec
   Running: cd /home/pliang/figems/examples/bash/state/execs/0.exec && (/home/pliang/figems/examples/bash/count 5)
   0
   1
   2
   3
   4
   Success
   Execution directory: /home/pliang/figems/examples/bash/state/execs/0.exec
   

   If we go into the state/execs/0.exec directory, we see that the files that count writes are written here. Also, there are some other files, which can be read by the servlet.
3. If we run execwrapper $PWD/count 10 again, it will write everything into state/execs/1.exec, and so on.

Using the web interface

At a high-level, the interface provides a way of navigating and operating on a virtual file system. Each item in this file system is identified by a trail. Each item also has any number of children. A standard set of navigation operations allows one to traverse the file system to the desired trail. At any item, pretty much the same set of operations for creating, deleting, copying items apply.

1. Now let's go to the servlet http://localhost:8080/figems. Essentially, the figems interface sits lightly on top of a virtual file system; each table displays a trail (path) to some item or view. We are currently at the root item. You should see a table with four rows: baskets, files, workers, and domains.
2. Let's go into domains by clicking anywhere on the row; note that the black arrow on the left points to that row. Click on that arrow. If you like the keyboard, use vi keys (h, j, k, l) to move around and then press o to open it.
3. Now, we are in the domains view, which should be empty. Each domain corresponds to a project you are working on, and concretely, a different state/execs directory. To create a new domain, hit shift-n (you can also find it in the actions menu). Type in bash for the item name. A new row should appear. We can edit the description by double-clicking on appropriate cell (row bash, column description)---or navigate over and press shift-e. This description will be saved on disk. If you don't believe me, check out figems/servlet/var/domains/bash.index. We also need to edit the domainDir field, which we should set to full path the state directory that we created, for example, /home/pliang/figems/examples/bash/state.
4. Let's proceed on into bash (o or click on the left arrow). Now, we are at the domain item identified by trail domains | bash. Proceed into execs. If you see all zeros, don't be alarmed; just hit shift-r to reload; you might have to do this more than once. The reason for this odd behavior is that the servlet doesn't keep you hanging while it goes out to disk to reloads, so it just gives you what it has, but your reload request prompts the servlet to load so next time, things should be up to date.
5. Go into (all), and we should see our execution along with some information such as the time, status, memory.
6. We can make notes about each run by modifying the note column at the end (double-click or type shift-e on it).

Adding new fields

1. First, let's start with adding another column to this table. Hit ctrl-d to replicate the table. (Use ctrl-n and ctrl-p to jump between tables.)
2. On the lower table, let's go back up the trail to the parent by hitting u or click the black upward-pointing arrow in the upper-left corner of the table.
3. We should be back at the domain item. Go into fieldSpecs. A fieldSpec (field specification) describes which columns of a table should be displayed.
4. Let's create a new fieldSpec by hitting shift-n and calling it default.
5. Now, we're going to copy (execs) into default. To do this, check (execs) by clicking on the checkbox or navigating over and hitting x. Type y to add it to the clipboard (basket). Go into default and press p to paste it. Hit shift-z to clear the clipboard.
6. Now, to create a new field, hit shift-n and call it result. Edit the data to be $count.map:result (double-click on it or press shift-e). This will tell the servlet to look in the count.map file and pull out the line with result as the key. Also edit displayName to be result.
7. Now, go to the table that's on the trail domains | bash | execs | (all) and hit shift-r to refresh. A new result column should show up with the value 12.

Manipulating executions

1. If we have a lot of executions, we want to group them into folders rather than looking at (all). Go up to domains | bash | execs by pressing u.
2. Hit shift-n to create a new execution view. If you're lazy, you can leave the name empty, and it will automatically fill it in with a distinct number (there's always the description field if you want to be informative).
3. Go into (all), check 0.exec, copy it to the clipboard by hitting y, go back into domains | bash | execs | 0 and paste it by hitting p. No files are actually copied: we are simply manipulating references.
4. We can also remove an execution from this view: check it and type a r enter (note that this is actually selecting the action from the actions menu). Important: for this removal to have any effect, we need to save it by typing a s enter. Note that removal doesn't delete anything on disk; it simply removes it from this view.
5. Purging is different. To purge an execution, check it and type a p enter followed by a s enter. After purging, we won't be able to see it on the web servlet anymore (but in fact, the file was just renamed to 0.exec.purged, which you have to go delete manually).
6. While an execution is running (go type execwrapper $PWD/count 1000 at the shell to get ome started), type a k enter. All this does is create a file called kill in the execution directory, which is picked up by execwrapper. Press shift-r to refresh and note that the status has changed from running to killed.

Other

Configuration

<!-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->
<Connector port="8080" maxHttpHeaderSize="8192"
           maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
           enableLookups="false" redirectPort="8443" acceptCount="100"
           connectionTimeout="20000" disableUploadTimeout="true" />