mg.to

I’m doing an IAmA on Reddit:

I was one of the first people Steve Jobs ever fired

Geek history! :-)

AnnoTube is a jQuery plugin that makes it easy to embed a YouTube video in your page along with an index and notes that are displayed and synchronized with the video as it plays. Each note can be an HTML snippet, or a URL to be loaded into an IFRAME, or even a JavaScript function to be run when a specific time in the video is reached.

Here’s a demo: the Mapping the Votes talk I gave at Google in April, with annotations provided by AnnoTube. This solves a mistake I made in this talk: Like many speakers, I left the text too small in my code examples. It looked fine to the people in the room, but the code is really hard to read in the YouTube video. But with the annotations I can put big, readable text right next to the video at the right time.

Fair warning: At this moment, I’ve only annotated the first 15 minutes of the video, and all of the notes so far are actually web pages in an IFRAME. Also, the AnnoTube plugin itself isn’t quite ready for general use. This is all a bit of work in progress, which started with something I put together for fun during the Google I/O conference. I’m only posting now because the YouTube API team is mentioning it in a post of their own, so please watch this page for updates over the next few days.

For a start, here’s what the timeline for my talk looks like (with URLs shortened to avoid long lines):

As you can see, it’s just a simple list of times, titles, and links or HTML snippets. AnnoTube takes care of connecting the events and watching for the times you specify.

More details soon… Thanks for your interest and patience!

I want to thank everyone who came to my Mapping the Votes talk at Google. The talk is available on YouTube - with apologies for the small font size in the code samples!

Here are some links and information that I referred to in the talk.

Maps and mapplets

Decision 2008 - the current election mapplet
Decision 2008 Gadget - the election map as a Google Gadget
Iowa Republican Caucus - an early API map
Iowa mapplet - an early mapplet
Twitter election map - the Super Tuesday twitter map (showing tweets from that day)
Campaign Trail - candidate calendars
New Hampshire in Google Earth - a KML file

Editors and desktop tools

The editor I used for the code samples is the one I use every day, Komodo IDE. Komodo’s debuggers for Ruby, Python, and PHP make it really easy to test my batch/script/server code. I’m especially fond of coding in the debugger. For the code that converts shapefiles and vote data into JSON output, I’d write the input part first, set a breakpoint and stop in the debugger after it reads the data, then write the conversion code with live data to look at while I code. Komodo also has a JavaScript debugger that works equally well, but most of the time I just use Firebug because of its simplicity.

Komodo IDE isn’t cheap, but I figure it paid for itself really fast. There’s also a free Komodo Edit that everyone should install even if you already have a favorite editor. Both versions have real-time syntax checking, where you get squiggly red underlines for syntax errors and squiggly green underlines for warnings, just like the spelling and grammar checkers in a word processor. This has saved me literally thousands of page reloads when testing, since Komodo catches my syntax errors before I even save the file. Komodo runs on Linux, Mac, and Windows.

One nice thing about GUI editors is that the basic editing works the same in all of them (or should), so it’s easy to switch back and forth if some other editor has a feature you want to take advantage of. Besides Komodo, I also use PSPad (free, Windows only), mostly because of its nice HTML/XML pretty-printer. It cleans up unreadable web page source code real quick.

Another expensive-but-well-worth-it tool for Windows and Mac is Araxis Merge, a terrific file compare and merge program with live editing. I use Merge as the diff/merge program for TortoiseSVN, which makes source control a dream.

A couple of free Windows tools I use every day are Zoom+ for screen zooming and my own JKLmouse for precise cursor control with the keyboard of your notebook computer. With JKLmouse, I can use the TrackPoint for fast cursor motion and then the keyboard for fine pixel-by-pixel movement, seamlessly and with no “modes”. (Sorry, I had to brag!)

Source code

The election map code is open source and is in two Google Code projects. The current code is in the primary-maps-2008 project, and the code for earliest caucuses and primaries is in the gmaps-samples project. (We moved the code to a new project to avoid filling up gmaps-samples!)

If you look at the code, go easy on me: much of it was written under severe time pressure. I asked if the elections could be delayed when I wasn’t quite ready, but even the mighty Google couldn’t seem to arrange that.

Also, if you read the code using the links provided here, there’s an awful lot of indentation, thanks to Google Code displaying my tab indentation using 8 spaces per tab. Shades of K&R! (So, why do I use tabs instead of two-space indents like everyone else? Well, one of the other benefits of Komodo is that unlike most code editors, it lets me edit in a proportional font. Two spaces in a proportional font is almost like not indenting at all.)

Shapefiles

Shapefiles are a wacky file format used for geographic data. Be thankful that other people have already written programs to pick them apart, so you and I don’t have to.

At first, I was using shp2text to convert shapefiles to an easy-to-use XML format (using the --gpx option), but this loses some of the information in the shapefile. More recently, Zachary Forest Johnson, author of the interesting indiemaps blog, wrote shpUtils.py, which decodes shapefiles into usable Python data.

I extended shpUtils.py to calculate correct centroids, area and other information about the shapes, and to fix a few bugs. The updated version is in the primary-maps-2008 project.

Centroids

The election maps use the centroids of the state and county polygons to position markers for those states.

Centroids are one of those things that you think you understand and then find out you were completely wrong. My first guess was the same as Zachary’s, to take the arithmetic mean of all the points (X and Y separately). The Wikipedia article even seems to say this, but it’s talking about the centroid of the points, not the centroid of the polygon that those points define. If you read it carefully, the article does give the correct algorithm, but it’s better explained on this page, along with sample implementations in various languages.

Census bureau shapefiles

The state and county outlines in the election maps come from shapefiles provided by the Census Bureau. Most states report votes by county, but a few New England states report by town (County Subdivisions in the Census Bureau page), and a few other states report by congressional district.

Shapefile simplification

D’oh! I completely forgot to talk about this important topic. The Census Bureau shapefiles have too much detail to be usable in a browser-based map. If you draw polygons from them, it will be much too slow. A tile layer can handle more detail, but the graphic files will be larger than they could be, because of the excess detail.

MapShaper is a free online tool to simplify shapefiles. It is pretty neat—you can see the effect of your simplification in realtime as you try different settings. I used MapShaper for the election maps, with various levels of simplification: simpler for JavaScript and more detailed for tile layers. More recently I discovered the Map Simplification Program which looks ideal for programmed simplification.

The code that processes shapefiles for the election maps is in makepolys.py which generates JSON output, and maketiles.py which generates tiles from that JSON data using ImageMagick.

Votes and delegates

The code to convert vote data from the latest primaries is in voter.py. This processes CSV files provided by the Boston Globe and converts them to JSON data.

Twitter map

The Ruby script that gathers the Twitter updates uses the Jabber::Simple module written by Blaine Cook to create a custom Jabber client that talks to Twitter, and uses the Twittervision API to get geographic information. It parses the XML data with sweet Hpricot, then generates JSON data (but you probably saw that coming). If you like jQuery, you’ll like Hpricot.

Mapplet code

The election mapplet code is in decision2008.xml and map.js. The code for the Campaign Trail mapplet is in campaign-trail.xml and campaign-trail.js. The latter file has the latest versions of the Array.mapjoin(), Array.index(), Object.sort(), S(), and related functions that I talked about. They are at the top of the file, and not yet documented, but you can find examples of each in the code.

More to come

That’s it for now! I’ll be posting more detailed articles on some of these topics. If there is a particular area you’re interested in, please let me know in the comments.

Thanks!

The Ajaxians are talking about a new $E function that is supposed to make it easier to create DOM elements. The example given creates the equivalent of this HTML code:

using this code to do it:

That seems just a tad complicated! Let’s see how it would look using my DOM creator for jQuery and Prototype:

Ah, that is quite a bit simpler. It’s also more flexible. Suppose you want to save a reference to that A tag in the middle. There’s no way to do that with $E, but it can be as easy as this:

Here is a jQuery plugin that makes it easy to build up a tree of DOM nodes. It lets you write code like this:

Basically, each function such as $.TABLE creates a DOM node and takes the following arguments:

The first argument is an object that list any attributes to be set on the node. You can specify the className attribute in a few different ways depending on your taste:

Any additional arguments after the first one represent child nodes to be created and appended. These arguments can be DOM elements themselves (e.g. inline $.FOO calls as above), or they can be numbers or strings which are converted to text nodes, or they can be arrays, in which case each element of the array is handled in this same way.

This interface is inspired by Bob Ippolito’s MochiKit DOM API, although it doesn’t implement all of the features of that one (yet).

The code predefines most of the common tags; you can add additional tags by calling:

Or simply add the tag names to the tags list in the code.

One last definition is $.NBSP which defines a non-breaking space (same as   in HTML).

$._createNode is an internal helper function used by the $.FOO functions. I would have hidden it away as a nested function, but I wanted to avoid any unnecessary closures.

This code doesn’t actually depend on any of the features of jQuery except for the presence of the $ function—and it uses $ only as a way to avoid cluttering the global namespace. I haven’t tested it with Prototype.js, but it should work equally well there. Or the code can be used with no library, by preceding it with:

Here is the source code, or you can download it:

Update 2007-09-13: As of version 1.2, the jQuery core now supports cross-domain JSONP downloads as part of the native Ajax support. I suggest you use this support instead of the plugin.

jQuery is a nifty new JavaScript library by John Resig. It features a $() function like the one in Prototype.js, but beefed up with CSS and XPath selectors, and with the ability to chain methods to do interesting things with concise code.

Unlike Prototype, jQuery doesn’t mess around with built-in JavaScript objects. It’s new—too new to have a version number!—but I’ve been writing some code with it and enjoying it.

jQuery provides an easy way to write plugin methods to extend the $ function. For you JSON fans out there, here is a JSON plugin for jQuery which lets you write code like this:

You can of course use an anonymous function if you prefer:

Or, using jQuery’s method chaining, you can combine calls like this code which displays a “Loading…” message when it starts loading the JSON resource:

To install the plugin, simply paste this code into a .js file and load it after loading jquery.js:

This adds a json() method to the $ function. The first argument is the URL to the JSON resource, with the text {callback} wherever the JSON callback method should be provided. In a JSONP URL, you would use jsonp={callback}; in a Yahoo! JSON URL you would use format=json&callback={callback}.

The second argument is the callback function itself. When the JSON resource finishes loading, this function will be called with a single argument, the JSON object itself. Inside the callback function, this is a reference to the HTML element found by the $ function. (If $ found more than one element, the callback function is called for each of them.)

The callback function is required, so this code won’t work with plain JSON APIs like del.icio.us that don’t let you specify a callback function. This would be easy enough to fix; I didn’t need it for the code I was writing, and didn’t think of it until just now. :-)

The code goes to a bit of extra work to create both an array entry and a unique global name for each callback. The global name is what is substituted into the {callback} part of the URL. It uses this name instead of the array reference to ensure compatibility with any JSON APIs that don’t allow special characters in the callback name. In fact, in the current code the callbacks[] array entries are not really used, but I figured it could be handy to have an array of all outstanding callbacks.

Update: John Resig suggested a couple of improvements to the code, so it’s updated, simpler and better now.

Update 2: Code updated to include Stephen and Brent’s fixes from the comments.