Machine Perception

Alex Teichman

Quickly Navigating Large Repositories From the Terminal

The tools grep, ack, and ag are great for searching repositories, but often we don’t just want to find a match, we want to navigate to a match and make changes. What follows are some bash functions for rapidly searching a repository and opening selected matches.

This is my primary method for navigating source repositories. I’ve written enough emails with this content that I decided it was time to write it up properly once and for all. I’ll discuss C++ in particular here, but the functions are easy to adapt to whatever language you work in.

cgrep and yank

Here’s a list of the most frequently used commands in my Bash history, with a count in the left column.

Most-used commands
1
2
3
4
5
6
7
17959 l
11922 cd
10079 make
4586 git
3986 hg
3405 cgrep      <------------
2874 cat

So what is this cgrep thing, and why is it my 6th-most-used command? It greps through all C++ source files and returns matches like this:

This is similar to the output of ack --nogroup --cpp imread, but with the addition that you can quickly open one of the matches and navigate to the appropriate line number using yank.

1
$ cgrep imread | yank 6

Running this will open ./src/program/image_cut.cpp in a running Emacs window and move to the imread call on line 253. I use emacsclient to make this possible, and presumably you can do something similar with Vim or whatever editor you use.

It looks like this.

More examples from my Bash history

Because this is all happening on the command line, you can apply whatever Unix filters might be useful. Here I wanted to see all usages of isinf except those in the PCL library.

1
$ cgrep isinf | grep -v './pcl_trunk/'`

Because cgrep is using grep under the hood, all the usual options you know and love still apply. Here I was showing usages of the Eigen library’s InnerIterator, with 10 following lines of context. This makes it easy to scan for errors, then jump to the location with yank to fix them.

1
cgrep InnerIterator -A10

We can also define a variation on cgrep to only search in header files. Here I was looking for compound assignment operators in class declarations.

1
hgrep 'operator[^=!]='`

Details

Here’s how cgrep and friends are implemented. You can paste this in your .bashrc. To hook up yank to emacsclient, you’ll also probably want export EDITOR='emacsclient -n' in your .bashrc and (server-start) in your Emacs init file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
function cfiles {
    find -regextype posix-egrep -regex '.*\.h$|.*\.hpp$|.*\.c$|.*\.cpp|.*\.cc$'
}

function cgrep {
   2>/dev/null grep -n "$@" $(cfiles) | nl | grep "$@"
}

function hfiles {
    find -regextype posix-egrep -regex '.*\.h|.*\.hpp'
}

function hgrep {
    2>/dev/null grep -n "$@" $(hfiles) | nl | grep "$@"
}

function pyfiles {
    find -regextype posix-egrep -regex '.*\.py$' | grep -v '#'
}

function pygrep {
   2>/dev/null grep -n "$@" $(pyfiles) | nl | grep "$@"
}

function yank {
    grep "^\s*$@\s" | awk '{print $2}' | sed 's/-\([0-9]*\)-/:\1:/g' | awk -F: '{print $1 " +" $2}' | awk '{print $2 " " $1}' | xargs ${EDITOR:?EDITOR must be set.}
}

Related, the cfiles function is handy for doing large search and replace operations.

1
sed -i 's/NameMapping2/NameMapping/g' `cfiles`

If you’re wondering how to easily get a list of your own most frequently used commands, you can use something like this. The awk command likely won’t translate to your machine because I am using a custom definition of HISTTIMEFORMAT, but you get the idea.

1
$ history | awk '{print $6}' | sort | uniq -c | sort -n

Changelog

  • 2013-12-10: Added screencast example, clarified that the point is fast navigation rather than just search.

Sensor Deformation and Distortion


Unsurprisingly, gross physical deformation of an RGBD sensor body will produce changes in the distortion pattern of the sensor. Surprisingly, you can cause this yourself quite easily without being aware of it.

We ran in to this when mounting an Asus Xtion Pro Live using velcro cinch straps around the sensor body. Pulling to hand tightness easily causes noticeable changes in the distortion pattern. If you’re trying to calibrate a sensor using our CLAMS work, this can really mess you up.

Here are some examples showing the change in the raw sensor output when tightening vs loosening the straps. Mount your sensors with care!

PrimeSense Distortion and View Angle

After presenting our work on unsupervised calibration of PrimeSense sensors at RSS2013, several people asked if the distortion is a function of view angle. In light of how these sensors work this is a reasonable concern. However, we’re not seeing evidence of this. If you are, please contact me.

What follows are animated gifs of the learned distortion model working at different view angles. If the depth distortion was significantly a function of view angle, I’d expect the results to appear unreasonable, and this isn’t the case.

Here’s a typical scene looking at a long table, wooden chairs, and a white wall. At 7 meters, the wall is highly distorted in the raw data, and applying the learned distortion model results in a reasonably flat wall.

Switching to a top-down view, here’s the result at about 40 degrees. Again, the wall becomes flat after applying the learned distortion model.

At about 20 degrees, the distortion is getting harder to see from top-down. This is because the distortion is a depth multiplier; at this angle the displacement of the points is mostly in line with the wall.

Viewing the same scene from the side is telling. We should be seeing straight lines at the intersection of the flat wall and the left & top edges of the view frustum. These straight lines only appear after applying the learned distortion model.

This corresponds nicely to the learned distortion model, which says the upper left corner peels away from the sensor. (Red means raw data must be pushed further away to produce accurate readings, blue means raw data must be pulled closer.)

Test Post

This is a test post to see if Octopress is suitable for my purposes. Conclusion: yes. There’s nothing here for general consumption, but I’ve left this page up in the offchance you’re wondering if Octopress is for you.

YouTube videos

You can more or less just paste the YouTube embed line directly into the markdown file. I needed to add ‘http:’ to the URL supplied by YouTube. Adding ?rel=0&vq=hd1080 to the end of the URL will prevent it from showing related video suggestions at the end and make it play in HD.

Images

Images should be placed in the octopress/source/images directory. A link path looks like /images/2013-07-15_oblique_angle_test/00-animated.gif. When the site is generated, they’ll end up in the octopress/public/images directory.

alt text

Math

Latex is not supported by default but it is quite easy to add:

Terminal output

1
2
3
4
5
6
7
8
9
10
11
12
13
14
teichman@aluminum:~/octopress$ ls -l
total 60
-rw-rw-r--  1 teichman teichman  1303 Jul 15 17:55 CHANGELOG.markdown
-rw-rw-r--  1 teichman teichman   413 Jul 15 17:55 config.rb
-rw-rw-r--  1 teichman teichman   643 Jul 15 17:55 config.ru
-rw-rw-r--  1 teichman teichman  2865 Jul 15 18:06 _config.yml
-rw-rw-r--  1 teichman teichman   450 Jul 15 17:55 Gemfile
-rw-rw-r--  1 teichman teichman  1412 Jul 15 17:57 Gemfile.lock
drwxrwxr-x  2 teichman teichman  4096 Jul 15 17:55 plugins
drwxrwxr-x  7 teichman teichman  4096 Jul 15 18:09 public
-rw-rw-r--  1 teichman teichman 15660 Jul 15 17:55 Rakefile
-rw-rw-r--  1 teichman teichman  2839 Jul 15 17:55 README.markdown
drwxrwxr-x  6 teichman teichman  4096 Jul 15 18:01 sass
drwxrwxr-x 10 teichman teichman  4096 Jul 15 18:04 source

Code

bash code example
1
2
3
function cgrep {
  2>/dev/null grep -n "$@" $(cfiles) | nl | grep "$@"
}
c++ code example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
class Trajectory : public Serializable
{
public:
  Trajectory() {}
  //! Deep copy
  Trajectory(const Trajectory& other);
  //! Deep copy
  Trajectory& operator=(const Trajectory& other);
  ~Trajectory();

  void clear();
  //! Destroys everything inside.
  void resize(size_t num);
  void set(size_t idx, const Eigen::Affine3d& transform);
  const Eigen::Affine3d& get(size_t idx) const;
  bool exists(size_t idx) const;
  void remove(size_t idx);
  size_t size() const { return transforms_.size(); }
  size_t numValid() const;
  std::string status(const std::string& prefix) const;

  void serialize(std::ostream& out) const;
  void deserialize(std::istream& in);

protected:
  std::vector<Eigen::Affine3d*> transforms_;
};
python code example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/usr/bin/python

import os, sys, string, argparse, smtplib
from email.mime.text import MIMEText

# -- Parse args.
parser = argparse.ArgumentParser()
parser.add_argument('--fr', nargs='+')
parser.add_argument('--to', nargs='+')
parser.add_argument('--subj', nargs='+')
parser.add_argument('--body', nargs='*')
args = parser.parse_args()

to = ', '.join(args.to)
subj = ' '.join(args.subj)
if args.body is None:
    body = ''
else:
    body = ' '.join(args.body)

print 'To:   ' + to
print 'Subj: ' + subj
print 'Body: ' + body

# -- Set up the message.
msg = MIMEText(body)
msg['Subject'] = subj
msg['To'] = to

stanford = True
if stanford:
    server = smtplib.SMTP('smtp-unencrypted.stanford.edu')
    username = args.fr
else:
    # -- Connect to hard-coded gmail account.  Gross.
    username = 'anon.aoeu@gmail.com'
    password = '<YOUR PASSWORD>'
    server = smtplib.SMTP('smtp.gmail.com:587')
    server.starttls()
    server.login(username, password)

# -- Send.
fail = server.sendmail(username, args.to, msg.as_string())
if fail:
    print fail
else:
    print 'Email sent.'

server.quit()

GitHub Gists

Not as pretty as I’d like but they’ll do:

Lists

The indentation isn’t quite showing up correctly in my browser, but this can probably be fixed.

  • This is a list item
  • Something else
    • Foo
    • Bar

Long text

Bacon ipsum dolor sit amet sirloin corned beef jowl prosciutto flank kielbasa, fatback salami bacon shankle hamburger pork chop tongue beef. Ham hock strip steak hamburger jowl, shoulder bacon fatback. Biltong meatloaf prosciutto bacon drumstick ground round venison swine cow jerky tri-tip tongue fatback rump. Tenderloin ribeye venison pork belly shankle. Short loin ham hock tenderloin strip steak leberkas jowl pancetta swine drumstick chuck.

Turkey filet mignon hamburger ball tip brisket beef ribs corned beef tenderloin andouille capicola kielbasa. Venison shoulder hamburger, capicola swine drumstick short ribs beef ribs pig corned beef pork pork loin meatloaf. Salami andouille meatball, biltong jerky bacon ball tip. Doner shoulder ground round tri-tip turducken chicken t-bone pork shank. Bresaola pig ribeye strip steak ham hock brisket boudin flank, bacon short ribs pancetta turducken biltong meatball rump. Kielbasa pastrami cow, meatloaf bacon short ribs tail tongue swine andouille rump ground round.

Short loin fatback jerky hamburger ribeye bacon swine bresaola. Corned beef shankle pork chop, pork fatback leberkas salami chuck ribeye sausage rump bresaola meatball tri-tip capicola. Hamburger chuck tongue capicola turkey ground round short loin pork meatball andouille tail cow. Pork loin sirloin frankfurter, turducken pastrami andouille short ribs tongue shank spare ribs flank. Chicken flank tenderloin short loin capicola sausage bresaola venison boudin prosciutto swine cow jerky. Tongue biltong boudin pig fatback, ribeye beef turducken. Tongue shank pork chop sirloin jowl.

Swine ham hock ribeye, meatball tri-tip salami ham pork pastrami tail beef. Drumstick bacon prosciutto ham hock pork belly beef, strip steak pork. Boudin strip steak flank tri-tip brisket ribeye ham. Pancetta meatball short loin tri-tip hamburger swine filet mignon pig kielbasa ribeye.

Jowl swine shankle turkey ham turducken pastrami prosciutto short ribs ground round beef ribs pig. Pancetta bacon fatback boudin kielbasa turducken. Sausage t-bone prosciutto jerky bresaola ground round leberkas jowl meatball capicola chuck. Boudin jerky bresaola rump pork loin prosciutto sirloin swine drumstick.

Thanks to Bacon Ipsum.