Alex Kahn Blog

Working Through Git Conflicts in Vim

2011-10-12T00:00:00-04:00

In this post I present a workflow I use to help me efficiently go through Git merge conflicts and correct them. It eliminates some of the typing and tedium involved in this process.

It starts with a shell alias, conflicts. The alias is defined as:

alias conflicts="git ls-files --unmerged | cut -f2 | uniq"

In Git, unmerged means a file that couldn’t be merged because of a conflict. So this Git command lists all of the objects containing conflicts. I pipe that command’s output to cut -f2 to get the second column of its output, the filenames. I then remove repeat entries using uniq, since this ls-files command outputs multiple objects for each conflict.

When I want to start resolving conflicts, I issue the command conflicts | xargs mvim. xargs is a powerful tool that takes its standard input – in this case, a list of files – and and provides that to a command as arguments. So, MacVim receives the list of conflicting files as arguments and loads the files into its buffer list.

I then fix the first conflict. If I’m dealing with a particularly long file, I use / to search for <<<< or other such conflict markers that Git places in the file.

When I’m done fixing the conflict, I execute :Git add % to add the file to the Git index. fugitive.vim, Time Pope’s essential Git intergation tool for Vim, provides the :Git command which passes through your command to the git command line. On the Vim command line % refers to the current file.¹

Having fixed a conflict and added a file to the Git index, I’m ready to deal with the next conflict. To get to the next conflicted file I use :bdelete, or :bd for short. Since we started Vim with a series of files as command line arguments, the buffer list is loaded with all the files we need to edit. :bdelete discards the current one and loads the next one. I repeat this process until there are no buffers left.

Once there are no remaining conflicts, I close my Vim window, go back to a terminal, and complete the merge using gc. What is gc? It’s an alias for git commit --verbose. Verbose commit output is awesome because it shows exactly what you’re about to commit. It saves me the step of doing git diff before each commit. I never want to commit without --verbose.

And that’s it. If this tended to be a long process or one I undertook more often, I could automate it further by creating a Vim command that combines the git add and :bdelete steps. As it stands, this workflow eliminates typing out long filenames and introduces a pleasant rhythm to this process. I hope this post has inspired you to script the tedious workflows in your Unix life. What tedious operations would you like to automate away?

% is a great feature that I recommend reading up on. Also check out the bash-like filename modifiers that you can use with %. Most often, I use the “head” modifier, :h, to change to the directory of the current file: :cd %:h. Or, if I’m editing a new file in a directory that doesn’t exist yet, I can create the directory: :!mkdir -p %:h.
↩

ConnectionAttempt: Another Rails Refactoring

2011-10-03T00:00:00-04:00

In the last few months I’ve been learning a lot about object-oriented design. By following the conversations and writings of Avdi Grimm, Corey Haines, Steve Klabnik, Gary Bernhardt and others (even going way back to James Golick), I’ve been able to identify some design issues in the codebase I work on every day at Paperless Post. These are issues that I had sensed previously, but had lacked a vocabulary with which to discuss them. In this post I consider a refactoring I undertook recently in our codebase. Through this discussion, I hope to help spread the good word on design in Rails applications.

The code I’m working with here is for Paperless Post’s integration with Eventbrite. Recently, we launched a feature that allows users to connect their account with an Eventbrite account. This allows users to link invitations on our site with ticketed events on Eventbrite. As a new requirement emerged, my original implementation became inadequate.

The Old Code

The following controller is responsible for connecting a user’s account with their account on Eventbrite:

class EventbriteAccountConnectionsController < ApplicationController
  def create
    if connection = EventbriteAccountConnection.connect(
        :account_id => current_account.id,
        :email => params[:email],
        :password => params[:password]
      )
      client_messenger.add_notice "Successfully logged in to Eventbrite."
      client_messenger[:connection_id] = connection.id
    else
      client_messenger.add_error "Sorry, we were unable to log you in to Eventbrite."
    end
    send_messages
  end
end

There are two details to note here. One is that client_messenger and send_messages are methods that deal with a ClientMessenger, an object for preparing and sending JSON responses to the client. I’ll use this again later on. The other detail is that when connecting accounts is successful, the client needs the ID of the connection record, so the controller must include this data in the response.

This is the model code that is called by the controller:

class EventbriteAccountConnection < ActiveRecord::Base
  def self.connect(credentials)
    resp = EventbriteClient.user_get(credentials[:email], credentials[:password])
    return if resp['error']
    self.create(
      :account_id => credentials[:account_id],
      :code => resp['user']['user_key']),
      :remote_id => resp['user']['user_id']
    )
  end
end

This code attempts to fetch a user from Eventbrite’s API and, if successful, creates and returns a connection record. If the API request responds with an error, the method returns nil. This code worked reliably, but eventually our team wanted to convey error messages from Eventbrite to users rather than just indicating success or failure. This way we would be able to distinguish between a failure with the user’s Eventbrite account, or a mistake on our site.

The most immediate solution to this problem would have been to modify the model method to return a more descriptive response in the case of failure. Then the controller could receive the error message and pass it along to users. This means that in case of success, the method would return an EventbriteAccountConnection whereas in failure it would return a String. Maybe it’s the Haskell talking, but I tend to think of wildly differing return types as a code smell. If the types respond to the same messages, it’s all good: duck-typing should be embraced in our code. But in this case there would be little recourse other than to check the type of the response: if the controller gets back a String, there was an error; if it gets an EventbriteAccountConnection, success. Ick.

A better option would be to for EventbriteAccountConnection.connect to return a Struct subclass that has a slot for an error message and a slot for an EventbriteAccountConnection instance. If the connection instance is present, that indicates success. If it’s missing, the login failed, and we should display the struct’s error attribute.

This could be a workable solution, but it sidesteps some fundamental design problems in EventbriteAccountConnection.connect. The most obvious issue to me was that it is a class method. I don’t think, as some do, that class methods are evil. However, they can indicate an attempt to use a class as a “bucket of methods,” ignoring or avoiding the advantages of encapsulating data and behavior in an object. In this case, I defined this as a class method because I could sense that the ActiveRecord model couldn’t provide the encapsulation I needed. But at the time, I didn’t take the further step of defining a new class to abstract out the connection logic.

The New Code

Given the additional requirement of improved error propagation, I took the opportunity to extract this account connection logic into a new domain object. Here is the code I arrived at, with some details omitted:

class EventbriteAccountConnection::ConnectionAttempt
  def initialize(account, email_address, password)
    @account = account
    @response = EventbriteClient.user_get(email_address, password)
    @connection = save_connection
  end

  def as_messenger(messenger = ClientMessenger.new)
    if success?
      messenger << "Successfully logged in to Eventbrite."
      messenger[:connection_id] = @connection.id
    else
      messenger.add_error(error_message)
    end

    messenger
  end
end

This my not be the greatest Ruby class ever, but it’s a significant improvement in design. To use the terminology of Hal Abelson and Gerald Sussman in Structure and Interpretation of Computer Programs, I have created a data abstraction, binding together an account and some login information into a bundle that has some meaning. By introducing the name ConnectionAttempt I have brought to the surface – I have reified – a concept that was previously submerged. This will aid future readers and maintainers of this code. After all, the goal of code is not just to provide instructions for the computer to carry out – it is also to communicate meaning to other humans. In this sense, this refactoring is a success.

The new service object also yields a simpler controller:

class EventbriteAccountConnectionsController < ApplicationController
  def create
    render :json => EventbriteAccountConnection::ConnectionAttempt.new(
      current_account, params[:email], params[:password]).as_messenger
  end
end

This is my ideal controller: it contains no logic. It merely calls out to a collaborator, passing it some parameters, and returns a response. While, in many cases, building a response directly in the controller is appropriate, in this case, given the non-trivial logic for generating the response, I’ve made this the responsibility of ConnectionAttempt. This prevents the controller from having to reach into its collaborator to query its internal state. It tells the object what it wants rather than asking.

The entirety of the ActiveRecord model is now:

class EventbriteAccountConnection < ActiveRecord::Base
end

This svelte class now respects the Single Responsibility Principle. Its sole purpose is to persist account connections to the database.

Because ConnectionAttempt has a narrow interface to EventbriteAccountConnection, it’s easy to mock out this external dependency. This makes the tests faster since it eliminates the need to persist ActiveRecord objects to the database. It also allows running the test without loading the entire Rails application. But there’s an additional, more subtle advantage to this use of mocking. By isolating ConnectionAttempt from the EventbriteAccountConnection that it ultimately attempts to create, the test becomes clearer because it takes advantage the abstraction barrier between the two domain concepts, allowing the test to focus more strictly on the object under test.

Discussion

It’s a bit silly how revelatory this feels to me. Object-oriented design has never been kept a secret, yet it seems this knowledge has been forgotten in the context of Rails applications. At the same time, to some to some, this design knowledge has been a given for years. In my case, although I read James Golick’s post on the topic almost two years ago, when I was starting to use Ruby and Rails, I didn’t piece it all together until recently. Now I know that by paying attention to design, we can build more maintainable, more understandable Rails apps, with faster test suites.

I’d like to hear from you. What do you think of this code? How would you continue this refactoring? What design strategies have you applied in your Rails apps, and what lessons have you learned?

My Haskell Reading List

2011-04-18T00:00:00-04:00

Lately I’ve been spending some time experimenting with Haskell, attempting to learn the fundamentals of functional programming. This post is meant to share some of the interesting resources I’ve come across in exploring this new world. How did I, a humble Rubyist, get mixed up with Haskell? There were a couple forces pushing me in the functional direction. One was Jonas Westerlund. Jonas is a Swedish hacker who works in Python and JavaScript by day and Haskell by night. I eventually became attracted to the folds, lazy evaluation, and infinite lists he showed off continually in IRC. Reading the source for his IRC bot, norby, especially its IRC parser and the main bot module – which responds to commands in separate threads using one little forkIO call – has been a gentle way to get acquainted with some real world Haskell code.

I was also exposed to functional programming ideas recently when I read Gregory Brown’s Ruby Best Practices. This is a fantastic book and definitely helped me level up in Ruby. Relevant here is that this book dedicates a whole chapter to discussing functional programming techniques in Ruby. The author concludes that while Ruby is by no means a functional language, its functional influences have yielded some very powerful abilities. The chapter goes through several examples of idiomatic Ruby that take functional approaches to problem-solving. Some examples are lazy evaluation (with a discussion of MenTaLguY’s lazy.rb), memoization, Enumerable#inject (and why folds are not very efficient in Ruby), infinite lists, and some snazzy Symbol#to_proc magic. This chapter is a testament to the elegance of functional approaches to solving problems and to the power Ruby unlocks by tapping into some of Smalltalk’s functional features. Flip through it and you might find a new appreciation for some of Ruby’s design decisions.

Another reason for my exploration of functional programming is Node.js. The explosive popularity of Node and the resulting backlash by concurrency curmudgeons has been interesting to follow but has not always been illuminating. The “events versus threads” debate has been going on for something like twenty years: which side is right? It turns out that, after brushing aside the trolls, flamebait and hype, this isn’t the right question to ask. Threads and evented programming are both tools that can be used to write concurrent programs; which technology is correct depends on the problem at hand. This is illustrated most clearly in a paper that Coda Hale pointed me to. In Combining Events And Threads For Scalable Network Services, published in 2007, Peng Li and Stephan Zdancewic describe building a network service that takes a hybrid approach by using threads and events. In the end, these don’t have to be competing techniques; they can compliment each other. Equally important, this paper demonstrates that callbacks are not a necessary byproduct of event-driven code. Indeed, the system the paper describes abstracts away the callback spaghetti that can often make up Node programs.

Much of the rest of my reading has been around concurrency. Continuing on the subject of non-blocking IO, Gregory Collins’ talk at the recent QCon London, High Performance Web Applications in Haskell (warning: 18 MB PDF), discussed Haskell’s threading model at length. From his slides, we learn that Haskell uses lightweight threads that are mapped N:M to OS threads. These threads are scheduled by the runtime system and have very low overhead in terms of memory use and context switching. All system calls in Haskell are non-blocking. This system yields performance similar to that of evented frameworks, but steers clear of the need for deeply nested callbacks.

Haskell has no problem dealing with tens of thousands threads, but what about synchronization? One option is software transactional memory (STM). STM allows the programmer to treat shared resources as they would a database transaction, rolling back changes if another thread has modified a resource. I first encountered this construct in David Nolen’s blog posts on using STM for bulk inserts with CouchDB and comparing concurrency in Clojure and Node.js. More recently I came across a 2006 interview with Simon Peyton-Jones and Tim Harris laying out their research and approach to implementing STM in Haskell. Other languages have follewed suit in implementing STM.

However, STM is not a silver bullet. Users of the feature have found that its performance can be unpredictable. In this discussion between Rich Hickey (creator of Clojure) and Cliff Click, Click describes a problem where a program written using STM becomes dependent on the runtime system and the hardware it’s run on. Changing the hardware or using a version of Clojure with updated STM internals resulted vastly different performance, sometimes requiring a rewrite of the program. This discussion, from 2008, illustrates some of the pitfalls of STM, and how it should be used appropriately. I’m curious to find out if and how these issues have been addressed in the years since this discussion.¹

This covers some ground on the subject of concurrency on a single machine. But what about using Haskell for writing distributed programs? In Haskell for the cloud, a wonderfully clear and well-written paper, Jeff Epstein, Andrew P. Black, and Simon Peyton-Jones describe implementing a distributed system that emulates the message passing functionality of Erlang, with some added Haskell flair. An interesting diversion from Erlang is the implementation of typed channels, reminiscent of Go (thanks to Ilya Grigorik for his explanation of typed channels in Go, which better suit Haskell’s strong typing than the Erlang message passing model, where a message of any type can be dispatched to any process. The code for this system, known as Cloud Haskell, is available on GitHub.

Another great discussion of Erlang, Haskell, and concurrency is a talk given by Simon Peyton-Jones, one of the creators of Haskell, at Erlang Factory London 2009 entitled Haskell and Erlang: growing up together. This entertaining presentation provides a brief overview of the history of each language, describes the features that distinguish them, and argues that despite their differences, both languages (and functional programming generally) are very relevant for solving today’s engineering problems. The talk provides a great survey of concurrency in Haskell, including a brief mention of data parellelism, an approach I still need to read more on. Be sure to follow along with the slightly cheesy but information-rich slides.

There are many more articles and resources that I have found interesting in my functional journey, but I’m out of space. I want to close with a shoutout to Miran Lipovača for writing Learn You a Haskell for Great Good, a phenomenal book. Don’t be fooled by the illustrations – this is a more straightforward book than Why’s (Poignant) Guide to Ruby. It’s a gentle introduction to functional programming and it presents some fairly mind-bending ideas in a way that somehow manages to be accessible. For example, the book presents monads only after walking readers through functors, applicative functors and monoids. By the time you reach monads, you are fully comfortable with the idea of a value wrapped in a context. The book recognizes that programming in a purely functional language requires new ways of thinking for someone with an imperative, object-oriented background and teaches in a way that is comfortable for users of traditional languages while still remaining true to Haskell. If you’re interested in buying the book, I have a one-time-use 30% off coupon code for No Starch press. Let me know you’re interested and I’ll share the code with you.

For additional reading on STM, check out Hans-J. Boehm’s Transactional Memory Should Be an Implementation Technique, Not a Programming Interface and MenTaLguY’s implementation of STM in Ruby (and thanks to MenTaLguY for referring me to all of the STM links in this post).
↩

Tracking Events With Google Analytics

2010-11-12T00:00:00-05:00

Did you know you can track custom events with Google Analytics? It’s true; you can. It’s easy. Just take your Google Analytics asynchronous queue object, _gaq, and push an array onto it:

_gaq.push(['_trackEvent', 'accounts', 'signup', 'homepage']);

_trackEvent is the method, accounts is the category, signup is the action, and homepage is the label. There is an additional parameter we can provide, value, if our event has a numerical value that we’d like to view in aggregate later on.

Typically, you’d set this up to happen upon a certain user action. For example, using jQuery:

$('#signup-form').submit(function() {
  _gaq.push(['_trackEvent', 'accounts', 'signup', 'homepage']);
});

When a user submits the signup form, we track the event in Google Analytics.

If you need to handle many events, a more robust solution would be to describe these events in markup rather than JavaScript. In this example, for tracking clicks that download files, I place a data-attribute on links whose clicks you’d like to track:

// Given HTML like this:
// <a href="/download.pdf" data-track>Download as PDF</a>

$('a [data-track']).click(function(event) {
  var label = this.innerHTML,
      file  = this.pathname;
  _gaq.push(['_trackEvent', 'downloads', file, label);
});

Now our data about this download is read from the markup, rather than specified up-front in our JavaScript.

Tracking Pageviews

The stock Google Analytics snippet contains a call to _trackPageview. On typical web pages we call this function once, for each page the user loads in their browser. On fancy single-page apps that navigate based on the URL hash, we may want to track each hash-based navigation as a pageview. In a Sammy app, for example, we could do this using the after hook:

$.sammy(function() {
  // Set up some routes

  this.after(function() {
    _gaq.push(['_trackPageview']);
  });
});

Now we’re tracking navigation within the single-page app: our after function is called each time the user navigates to another route in the Sammy app. Simple!

Wrapping Up

For more detail, check out Google’s trackEvent reference. Also, be sure to read Mathias Bynens’ excellent post on optimizing Google’s stock analytics snippet. Mathias also does an excellent job demystifying the the _gaq object. One last thing: check out custom variables. This feature is an excellent way to keep track of different categories of visitors. At Paperless Post, we use custom variables to help us understand the behavior of anonymous visitors versus logged in users.

Gotham Ruby Conference 2010: A Recap

2010-05-26T00:00:00-04:00

Over the weekend I attended the fourth Gotham Ruby Conference (GoRuCo). The conference was well-organized, the content was top-notch and there were tons of great people there.

Photo by sd

The day opened with a welcome from a Pace University’s Jonathan Hill. Jonathan, who is assistant dean of Pace’s Computer Science program, described the longstanding relationship between Pace and the New York Ruby community, pointing out that GoRuCo has been held at Pace since its first year. Pace uses Ruby and Rails extensively in its Computer Science program and, in turn, its graduates often find employment through this community. Jonathan expressed real appreciation for the health and energy of the Ruby community in New York. It was neat to feel this connection and know that Pace was more than just a rented space for the conference.

Nick Gauthier presented the first talk, Grease Your Suite, on making slow test suites run faster. This talk demonstrated how he improved a test suite in a Rails project from 13 minutes down to under 20 seconds. To do this, Nick used lifo’s fast_context, his own Hydra for distributing tests across CPU cores or machines, and Ruby Enterprise Edition for its ability to tune Ruby’s garbage collection. He also gained speed by mocking PaperClip’s image manipulation, using fixtures to replace truly-resized images. Nick got further performance by tweaking a journaling setting for the ext4 filesystem (Nick is a staunch Linux-on-the-desktop user), as well as by disabling atime. Nick noted that the improvements from these tweaks gain as much in speed as running tests on a solid state drive. With such a fast test suite, the Continuous Integration server can return to its true calling—ensuring a green build in a production-like environment. While I don’t have any projects that are burdened by tests that take too long, I nonetheless left this talk with strategies to grease my suite. Get all the details in Nick’s slides.

The next talk, by Aman Gupta, was on a new Ruby profiling tool, Memprof. Aman provided a tour of the tools he’d worked with for profiling Ruby’s memory use: ObjectSpace.each_object, gdb.rb, bleak_house, and patching Ruby in order for it to provide heap dumps. These tools are all helpful, he said, but they require writing patching or running special versions of Ruby, or don’t provide complete enough information on the contents of the objects in memory or their context in code. Memprof, created by Joe Damato with contributions from Aman, solves the above problems: Installing it is simple (gem install memprof), use is simple (Memprof.dump_all, for example), it outputs to JSON, and provides all the necessary information and context about objects in memory, including references between objects. Memprof’s JSON output makes it easy to dump its output into a database like MongoDB, which memprof.com does. Aman’s talk struck at just the right level: deep enough to keep us interested, but not so deep as to lose the audience. And if you’ve ever checked out Aman and Joe’s blog you’ll know how deep this this rabbit hole goes. My thanks to Aman for that.

Managing Rubyists, a talk by Luke Melia, was directed at Ruby developers who manage a team. This was the only non-technical talk of the day and it was one of the highlights for me. In the talk, Luke laid out 15 principles or techniques for managers to follow for their team and its members to achieve happiness, motivation, productivity and growth. I was blown away by the care Luke took with this subject and the regard he has for the developers he works with. Luke takes his craft as a manager as seriously as Rubyists take their code. Check out Luke’s slides and notes for the details. If you’re a manager, be sure to check out Luke’s recommended reading.

After lunch, Chris Williams (of JSConf fame) presented Rails Caching à la JavaScript. In his talk, Chris made the case a straightforward approach to working around performance issues with Rails: using page caching to avoid the Rails stack altogether for most requests, and making JSONP AJAX requests to retrieve the personalized aspects of a page. Even though this AJAX request is handled by Rails, its rendering overhead is offloaded to the client (Chris recommends mustache.js for client-side rendering). Despite Chris’ preface that the technique described is not ideal for most Rails applications, the talk drew skepticism and ire from the audience. Nonetheless, the talk was enjoyable, humorous, and made a strong argument for a straightforward caching strategy.

Next, Alex MacCaw showcased Bowline (pronounced “bowlin’” like the knot), a new framework for creating cross platform desktop apps using Ruby, JavaScript and WebKit. Alex gave a tour of developing an app with Bowline, showing off its in-memory model layer, SuperModel, its bindings between Ruby and JavaScript and neat features such as seamless background updates for installed applications and synchronizing data across remote client apps. In all, Bowline looks to be a very ambitious and promising project.

Paul Dix, author of the forthcoming Service Oriented Architecture with Ruby on Rails, presented on how to create web service clients using ActiveModel to scale an app. This is not scaling an app for traffic, but how to scaling as complexity and team size grow. Paul demonstrated how to leverage ActiveModel to implement client models with validations, dirty tracking, callbacks and even persistence. If you are struggling to deal with complexity in a large application and to abstract components into separate services, Paul offers an excellent approach. Definitely look out for his book this August.

The last talk of the day, Scaling to Hundreds of Millions of Requests, was presented by James Golick. In this talk, James outlined how he improved performance and scaled a very active community site. James shared his experience moving from Amazon EC2 to physical hardware and the immense performance increase this brought. James also went over his trials with MongoDB and using MySQL “non-relationally” with Friendly before finally settling on Cassandra. As James argued in a recent blog post Cassandra is one of the NoSQL databases that truly scales horizontally. Look for James speaking further on this topic at Railsconf.

The final hour of the conference comprised of “Rejectconf,” a round of lightning talks and a chance to speak for people whose talks weren’t accepted for the conference program. Some highlights were Pat Nakajima presenting his Bookmarklet Factory (“how to downvote 100 Gizmodo comments”); Edward Ocampo-Gooding presenting on open data in Ottawa; Aaron Quint presenting on Jim, a JavaScript manager, bundler and compressor; and Michael Bernstein presenting on Ohm, the Redis object mapper with an extensive contrib collection. Then it was beer time.

In all it was a great conference. Thanks to the conference organizers and hello to all the folks I met. And hello to all the folks I didn’t get a chance to meet but wish I had. Keep your eyes peeled on the conference site; videos will be posted in the next two weeks.

Up and Running With LaTeX

2010-05-13T00:00:00-04:00

Recently I’ve had the pleasure of getting acquainted with LaTeX. For those that don’t know, LaTeX is a typesetting system that turns marked-up text files into precise, readable, print-ready documents. LaTeX is specialized for creating technical documents containing math and equations, but it’s also well-suited for any structured document. LaTeX is not just the command-line tool for procesing .tex files, though—it’s an entire ecosystem consisting of libraries, packages and tools for myriad purposes. The size of this ecosystem is staggering. At times, it’s frustrating. But I’ve grown to appreciate it the more I’ve learned.

What’s so great about LaTeX? I love that it demands explicitness. It doesn’t try to guess what you mean. Want a hyphen? Use -. Want a slightly longer dash, suitable for numerical ranges? That’s --. Using a dash for punctuation? --- inserts an em dash. The same explicitness applies to quotes. How does LaTeX know what is an opening quote and what is a closing quote? You tell it. Use two backticks (``) for an opening quote and two apostrophes for a closing quote (''). These are two examples of the attention to detail that LaTeX promotes.

Some things are more automatic. Hyphenation, for example, is handled by LaTeX. And it’s smart.¹ The system sometimes misses possible hyphenation points, but it’s very good at avoiding incorrect hyphenation, such as “new-spapers.” It also allows manual control of hyphenation. For example, put \- in the middle of a word to identify a possible hyphenation point. I also found some of these line breaking commands helpful.

LaTeX excels at working with tables, references, bibliographies, footnotes, indexes (and math, of course)—features whose use ranges from tedious to hellish in word processing software. Perhaps its greatest strength is that unlike word processing software, LaTeX forces authors to focus on the content and structure of a document, rather than its presentation. LaTeX takes care of what it looks like. It’s the anti-WYSIWYG.

Installing LaTeX

The easiest way to get started with LaTeX on Mac OS X is to install MacTeX. Once you have MacTeX installed, take a look in /usr/local/texlive (MacTeX is a specialized version of the TeX Live distribution) to see what was installed. You’ll want to add /usr/local/texlive/2009/bin/universal-darwin to your $PATH. Also make note of /usr/local/texlive/2009/texmf-local/tex/latex—this is where you’ll install additional LaTeX packages.

The Memoir Class

You can get far with LaTeX’s built in document classes: book, report, article, etc. But for my document, I needed a bit more flexibility. I turned to the Memoir document class. The goal of this class is to combine LaTeX’s built-in book class with some of the more common packages that help customize the design of a document. The result is a large class, but one that offers convenience, since it reduces the need for installing additional packages.

It’s difficult to summarize what Memoir offers because it does so much. Instead, I’ll just share my own experience: Memoir made laying out my document a breeze. I continually got the impression that the creators of Memoir had thought of everything. Refer to Memoir’s exhaustive, 500-page manual (PDF) to learn about what it offers. It’s epic.

To install Memoir (or any document class or package), download it from CTAN, compile it using the instructions in the README, and then copy the entire memoir directory to the installation directory mentioned above. Then, after running texhash, you should be able to compile documents using the Memoir class.

Printing 2-Up

The document I was preparing was a long packet of reading. I wanted to print this document with two pages per sheet, as a photocopied packet from a book would be layed out. Although there are ways to bend LaTeX’s strong desire to print one logical page per physical sheet of paper (among these are booklet and twoup), a more prudent alternative is pdfpages. pdfpages is a package for embedding—you guessed it—PDF pages into LaTeX documents. Conveniently, it has the ability to embed multiple pages of a PDF document into a single page of a LaTeX document. The end result: multiple logical pages per physical sheet. After producing a PDF of your original document, create another LaTeX document which embeds the compiled PDF, manipulating it to render two PDF pages on one LaTeX page. Some code might make this clearer:

\documentclass[landscape]{article}
\usepackage{pdfpages}

\begin{document}
\includepdf[nup=2x1,pages=-]{my-document.pdf}
\end{document}

The document class used here is irrelevant, but it’s important to set landscape orientation. The interesting line here is the \includepdf directive. The nup=2x1 tells pdfpages to place two pages from the PDF file side-by-side in our document. pages=- tells pdfpages to include all pages of the PDF file. When you compile this document, the output contains two logical pages laid out on a landscape-oriented physical sheet.

For optimal 2-up printing, you’ll need to adjust your original document’s margins. LaTeX assumes that you are printing a book, and thus need a large inner margin (where the binding would be) and a smaller outer margin. Memoir makes it easy to override these defaults:

% Set inner and outer margins to .5 and 1 inch, respectively
\setlrmarginsandblock{.5in}{1in}{*}
% Set top and bottom margins to .8 inches
\setulmarginsandblock{.8in}{.8in}{*}
% The {*} parameter tells LaTeX to use the measurements
% provided rather than attempt to calculate a ratio

pdfpages has numerous options for arranging booklet-style documents, which you can read about in its manual (PDF). Get the package here.

Better Typography

One point of frustration I had with LaTeX was its font handling. The routine for preparing a typeface to be used with LaTeX was so elaborate that it scared me away. But then I came across XeLaTeX, a project that brings modern, convenient typography to LaTeX.

Using XeLaTeX solved two problems for me. One was using Unicode characters in my documents. For some reason, LaTeX doesn’t support this by default, requiring external packages for this purpose. Using XeLaTeX, it was as simple as using Unicode characters in an HTML document. XeLaTeX also makes it dropdead simple to use any OpenType or TrueType typeface installed on your system. Replacing LaTeX’s handsome but staid Computer Modern with the dignified, refined Baskerville was as simple as \setmainfont[Mapping=tex-text]{Baskerville}.

Editing LaTeX in Vim

The Latex-Suite plugin for Vim is a lifesaver. It provides syntax highlighting, folding (handy for hiding sections or chapters you’re not currently working on) and numerous mappings that I haven’t taken the time to learn. One exception is the mapping for compiling. To compile your document to PDF, use ,ll. The compilation log is displayed in a preview window, and any compilation errors pop up in a QuickFix window. To set Latex-Suite to use xelatex to compile your document, add let g:Tex_CompileRule_pdf = 'xelatex -interaction=nonstopmode $*' to your .vimrc.

Notes on Redis: Data Modeling, Hashes, and Namespaces

2010-03-30T00:00:00-04:00

What is Redis?

Redis is a simple database. It stores its data in memory and saves a snapshot to disk in the background periodically. It can be used as a simple key-value store like Memcached, but it also supports more complex data types such as lists, sets, sorted sets and hashes. These will come in handy. Redis is versatile and is used for a plethora of tasks: as a job queue, as a cache store, for tracking downloads of Ruby Gems, for storing A/B testing results, and even as the backend for a chat server.

Redis has been receiving a lot of praise lately and is the subject of lot of excitement online. This praise is deserved – it’s an excellent piece of software. I’ve had the pleasure of using Redis recently and here are some of my findings.

Our Use Case

On mediaFAIL, we wanted to track how many mentions a fail receives on Twitter. We wanted to show a count of a fail’s Twitter mentions and roll that number into a fail’s votes – this feature lies somewhere between act.ly and Tweetmeme. Couldn’t we just store each tweet in our votes table? Perhaps. But this would break our “user has many fails through votes” association. How do I store this data in Redis?

Data Modeling

My first pass at the problem took advantage Redis’s sets. For each fail that was mentioned on Twitter, I created a set containing the users that mention a fail. So, key fail:1:tweets contained the set ['alexanderkahn', 'levjoy', 'TimKarr']. Since elements in sets must be unique, I wouldn’t have to worry about a user making multiple mentions of a fail getting counted more than once. This was an elegant solution: it allowed looking up the number of mentions a fail has received using SLEN (set length) and retrieving all the Twitter usernames for a fail using SMEMBERS (set members).

As I continued work on this feature, I realized that I wanted to link to the specific tweet where a Twitter user mentioned a fail. One way to do that would be to set a string key for each member of a set that corresponds to a member of the tweet:1:tweets set. So, for the above example, I would also set fail:1:tweet:alexanderkahn to “12330508057”, the tweet ID for my mention of the fail. But this would cause a lot of keys to be created and mean that looking up usernames and tweet IDs for a fail would require a great deal of queries for a popular fail. Thanks to Redis’s new hash data structure, there was a cleaner way. In a hash, I can store both the username and the tweet ID under the same fail:1:tweets key. The hash for this key would look like {"alexanderkahn" => "12330508057", "levjoy" => "12330508067", "TimKarr" => "12330518057"}. Since hash keys have to be unique, a Twitter mention can’t be counted twice, just like with a set. Now I can look up how many mentions a fail has received with HLEN (hash length) and fetch both usernames an tweet IDS with one HGETALL (hash get all) query. Groovy.

Redis and Ruby

The Ruby library for Redis, written by Ezra Zygmuntowicz, is a pleasure to use. It has certain touches that make it feel like Redis was born for Ruby. One example of this quality is how it handles Redis’s MULTI/EXEC transactions:

r = Redis.new
r.multi do
  r.set 'foo', 'bar'
  r.incr 'baz'
end

In the above code, if an exception is raised during the execution of the block, none of the operations inside are committed. This is just a basic transaction like ActiveRecord provides when working with a SQL database, but is neatly implemented in idiomatic Ruby. The block returns the result of each Redis operation in an array.

Another nice Ruby touch is the way the library translates a Redis hashes into a Ruby hashes. When retrieving the entire contents of a hash using HGETALL, rather than returning a flat list (1. key, 2.value, 3. secondkey, 4. secondvalue) as the Redis internal protocol does, the library turns this list into a Ruby hash using Hash::[]. I had never run into this way of creating a Ruby hash out of an array, but its use here struck me as clever. Check out ri Hash::[] for more on this.

Another good project for working with Redis in Ruby is redis-namespace by Chris Wanstrath. This library helps you compartmentalize your Redis keys to keep different sets of data separate. We’re using this to keep tweets, described above, separate from our A/B testing data. To use it, you just interact with a Redis::Namespace object rather than a Redis::Client object.

redis = Redis.new
namespaced = Redis::Namespace.new(:my_feature, :redis => redis)

# Interact with namespaced Redis as you would a normal Redis client:
namespaced.set "foo", "bar"
namespaced.get "foo"
# => 'bar'
namespaced.keys "*"
# => ['foo']

# The actual key in Redis is prefixed with our namespace:
redis.keys "*"
# => ['my_feature:foo']

Nice of Chris to take care of that for us, no?

One last note. How am I accessing Redis from within my Rails app? Currently, I’m taking a page out of Nick Quaranto’s book and setting a global variable (in environment.rb):

$redis = Redis.new # Or a namespaced equivalent

This variable provides access to Redis from anywhere in the application. This is not necessarily the best way, but for now it works. I’m happy to hear how I could improve upon this.

If you haven’t had a chance to explore Redis yet, I recommend doing so. I hope this post helps push you in the right direction. Some handy resources will be the Redis command reference, this PDF cheat sheet, and following @antirez, Redis’s author, on Twitter.

A Sinatra App Wrapping Flickr, Plus HTTP Caching and Heroku

2010-02-22T00:00:00-05:00

In the last several months, Esther has produced a large number of monotype prints at a local printmaking studio. She’s been sharing her work online by scanning it and posting it to Flickr. Recently she experimented with using Tumblr to create a stylish gallery of this work. But this process — scanning, adding metadata, uploading to Flickr, followed by posting each piece to Tumblr — proved to be laborious. So we sought to streamline this workflow, hoping to eliminate a step or two.

Titles, descriptions, tags and sets are all easy to manage on Flickr. What if there was a way to take advantage of Flickr’s great management interface, taking images and metadata and putting them on an external site? I took on this problem with a Sinatra application that wraps the Flickr API in a warm embrace. The app makes an API request to Flickr asking for all the images from Esther’s printmaking photoset and renders them as thumbnails, each image linking to a page for that image. On the individual image page, the app requests information on an image – title, description, and the next and previous image – and displays this information along with a larger image of the piece. Now, Flickr itself is the CMS and the gallery is just a pretty shell for information already stored on Flickr. No more managing metadata in multiple places. Have a look at the site.

What about performance? Isn’t it slow to make an API request to Flickr for every visit to the site? This is a definite concern: making an API call to Flickr and then responding takes the application about 500ms – respectable but not exactly speedy. But the content of the site is basically static. Once an image is uploaded and described on Flickr, it is unlikely to change. This means I can cache the entire page with a gateway cache¹. To do this, I set Cache-Control headers that say that my pages are allowed to be cached for long time, and the cache, Varnish, serves up a static representation of the page that the app produced. Once a page has been requested once, subsequent requests for that page are lightning-fast – under 100ms. After six hours, the cached representation of the homepage is considered stale, and a new one will be generated. After two days, the cached representation of an individual page is considered stale. This way, updates to the photoset or changes to images do eventually get served to visitors.

I should note that I’m hosting this site on Heroku. It’s been said elsewhere and should perhaps be obvious by now, but I’ll say it again: Heroku is awesome. Their use of Varnish made it so easy to make this gallery app fast. Deploying code with just a git push is seriously great. I recommend hosting an app on Heroku if you haven’t before. It’s easy and, for a simple app like this one, is free.

1: For more on HTTP caching, see Ryan Tomayko’s excellent article Things Caches Do

FakeWeb and Regular Expressions

2009-12-16T00:00:00-05:00

FakeWeb is a great way to speed up your tests and remove dependencies on external web services. Using FakeWeb is simple: specify a URL for FakeWeb to respond to, and give it the response you expect from the request. For example:

FakeWeb.register_uri(:get, "http://www.google.com", :body => "Stubbed Google!")
HTTParty.get("http://www.google.com") # => "Stubbed Google!"

Now FakeWeb will prevent your code from making an actual HTTP request and instead it will return the response you specified. But what if your application makes requests to URLs with query parameters? It would be tedious to register each URL with FakeWeb:

FakeWeb.register_uri(:get, "http://www.google.com/search?q=fakeweb", :body => "Stubbed Google!")
FakeWeb.register_uri(:get, "http://www.google.com/search?q=excited+kitten", :body => "Stubbed Google!")
FakeWeb.register_uri(:get, "http://www.google.com/search?q=ponies", :body => "Stubbed Google!")

Thankfully, FakeWeb has a (slightly underdocumented) solution for this. Instead of passing a string to FakeWeb.register_uri, pass it a regular expression (using Ruby’s %r|| syntax to avoid having to escape forward slashes):

FakeWeb.register_uri(:get, %r|http://www.google.com/search\?q=|, :body => "A bunch of results")
HTTParty.get("http://www.google.com/search?q=tunisia") # => "A bunch of results"
HTTParty.get("http://www.google.com/search?q=vegan+latkes") # => "A bunch of results"

By registering a regular expression, FakeWeb will respond to any HTTP GET request whose URL matches the pattern you’ve supplied. Your test suite can make requests with all different query parameters without having to register each individual URL. This feature adds a lot of flexibility to FakeWeb and makes it feasible to stub out your application’s HTTP requests, even when complex, changing query parameters are involved.

For a general overview of FakeWeb, as well a some far-less-contrived examples of its use, check out Josh Nichols’ post, Stop Net::HTTP dead in its tracks. Also check out FakeWeb on GitHub.

Pretty Feed Items with Facebook Connect

2009-11-06T00:00:00-05:00

At Free Press, when a user signs a petition or shows support for our issues on one of our sites, we want them to spread the word. We want to make it easy for users to share the story they told, or the petition they signed, and to encourage others to do the same. We usually do this with links to Facebook’s sharer.php and Twitter’s status parameter.

But sharer.php left us frustrated with its awkward interface and lack of branding. What about the pretty updates that Facebook applications produce? We wanted a feed item that looked branded, looked good, and compelled a user’s friends to also take action. This is where Facebook Connect comes in.

Using Facebook Connect for this task turned out to be simple. Surprisingly simple. Here’s the routine:

Create a Facebook application
Create a file to allow cross-domain JavaScript communication
Include Facebook’s JavaScript library
Write some JavaScript

Steps 1-3 are fairly well documented in the links above. Step 4 is the interesting part. Take a look at the code below.

var AK = {
  apiKey:  "de24e7cfd602218871fc8e30b3dd8a5f",
  message: "Facebook Connect is easy!",
  attachment: {
    name: "Updating from akahn.net",
    href: "http://www.akahn.net",
    caption: "So cool!",
    media: [{
      type: "image",
      src:  "http://akahn.net/images/shoes.jpg",
      href: "http://www.akahn.net"
    }]
  },
}
FB.init(AK.apiKey, "/xd_comm.html");
FB.ensureInit(function() {
  FB.Connect.streamPublish(AK.message, AK.attachment);
});

What’s going on here? First, I’m creating an object, AK, to store the various pieces of data I need. This includes an API key, the default message that the user will be posting to their feed, along with some other bits of text, and an image.

I then get the API ready, using FB.init, specifying my API key and the location of my cross-domain communication file.

Then I make sure that the API has finished loading, using FB.ensureInit. I pass in a function, to be executed once the API has finished loading.

In the function, I call FB.Connect.streamPublish. I pass in the message and the attachment object that I defined earlier. If the user is logged in on Facebook, they’re presented with a modal popup inviting them to post to their feed. If not, they’re prompted to log in before they are invited to create a feed item. Try it out below:

Post to my Facebook wall!

And that’s it! It takes a lot of setup just to make one little call to FB.Connect.streamPublish. But the smooth user experience and the customized, pretty feed item make it all worthwhile.