Fixing up our Atlassian suite.

2013-08-13T00:00:00-05:00

At the consulting firm I work at we use a handful of pieces of the Atlassian suite: JIRA, FishEye, HipChat.

HipChat we've only just started using. It's exciting for me since we didn't have a good group chat solution prior to this. As a long-time IRC junkie, I jumped on the idea of having bots and in-channel notifications and such. Admittedly a lot of this kind of stuff can be of no true value beyond it being kind of fun, but there were a couple interactions that ended up being of value. The highest priority HipChat integration for me was getting Git announcing pushes to our HipChat rooms.

Rather than using the managed Git repository features of FishEye (or something like Gitorious, Stash, BitBucket, GitHub), we have our own central Git repository called vm-repos. It's just a collection of bare repositories accessible via SSH which serves as the origin for most of our private code. Nothing fancy, but as a consulting firm we have lots of different projects, and lots of different repos.

Something we had been wanting to set up for a while was getting FishEye to automatically pull in new Git repositories that had been added to vm-repos. We mostly use FishEye as a code search utility, so getting repos added was less crucial than if we were using the full Crucible/FishEye setup for code reviews and commenting. Manually adding our projects was tedious, and since it was low priority too, it meant that a lot of projects never ended up in FishEye. This lead to a vicious cycle of FishEye neglect.

Exacerbating the problem, we don't have a DevOps team, which means that unless our internal infrastructure gets critically broken, we typically don't have time to work on it. Our developers are busy working on client work, or more interesting internal stuff (like HipChat bots, apparently).

Setting up HipChat integrations finally gave me the motivation to execute this.

HipChat and Git Integration

This ended up being really easy thanks to the efforts of some other folks in the community. I wanted a post-receive hook that would notify HipChat of new commits. I started from a HipChat post-receive hook that I found on GitHub. I dropped this in place and then I just set up a little shell script that installs a post-receive hook in all of our bare repositories that don't have a hook already:

find /data/repos/git -name hooks -type d \! -exec test -e '{}/post-receive' \; \
    -exec cp ~/serverscripts/vm-repos/post-receive.template {}/post-receive \;

The post-receive.template file is:

while read oldrev newrev refname
do
        echo $oldrev $newrev $refname | /bin/bash ~/serverscripts/vm-repos/notifyhipchat.sh
        echo $oldrev $newrev $refname | /bin/bash ~/serverscripts/vm-repos/notifyfisheye.sh

        # Other post-receive hooks can go here, following the same pattern.
done

The main thing I had to fix with the HipChat post receive hook was that this script only notified single rooms. We typically have one HipChat room per project, and we didn't want everything going all to the same place. (Although we did create a room just for push announcements; that's the $DEFAULT_ROOM you see below).

So I updated the script to read from a rooms.conf file, and depending on the repo that had just been pushed to, announce to different rooms or to multiple rooms:

# Near the top
ROOM_IDS=$DEFAULT_ROOM,$(grep "^$REPO=" rooms.conf | grep -o '[0-9,]\+$')

# Then at the bottom of the file:
IFS=',' ; for HIPCHAT_ROOM in $ROOM_IDS; do
    echo "$MSG" | $HIPCHAT_SCRIPT -c "$HIPCHAT_COLOR" -t "$HIPCHAT_TOKEN" -r "$HIPCHAT_ROOM" -f "$HIPCHAT_FROM" -n 1 > /dev/null
done

And then rooms.conf is just a simple mapping of repo directories to HipChat room IDs.

repo1=10001
repo2=10002
repo3=10003,10005

The next change was FishEye support, so that repo names and changesets linked to FishEye:

# We have some directory structures which we want to include in the FishEye project names.
# Replace / with _ for paths
REPOPATH=$(pwd | sed 's/\/data\/repos\/git\///')
REPONAME=$(echo "$REPOPATH" | sed 's/\//_/g')

# repo links:
REPO_LINK="<a href=\"http://$FISHEYE/browse/$REPONAME\">$REPO</a>"

# changeset links:
LOG=`$GIT_EXEC log $OLDREV..$NEWREV --format="- %s (<a href=\"http://$FISHEYE/changelog/$REPO?cs=%H\">%h</a>)"`

This is what lead into finally fixing our FishEye setup. A lot of these FishEye links being generated were bogus because our repo names in FishEye didn't line up with our repo names on vm-repos, or in some cases, we just didn't have the FishEye repos created at all.

FishEye Automatic Discovery of Git Repos

There are some FishEye plugins out there that theoretically do this, like the FishEye Directory Scanner. We're on FishEye 3.x though, and this plugin hasn't been updated yet. The REST API for FishEye also doesn't seem to support the creation of repos. I was poking around at FishEye to try to figure out how to do this. I was about to do this with browser automation using Fluent Automation and ScriptCS, but noticed that repos are actually defined in plain text in the config.xml file. I tested it out by adding a repo and sure enough, after restarting FishEye, it picked this up and indexed it right away.

So I wrote another script that would find all of our Git repositories and rewrite FishEye's config.xml to include them. Caveat: I'd never call bash scripting an area of expertise; this is a quick-and-dirty hack and I freely admit that, but it works great!

XMLFILE=/data/appdata/fisheye/config.xml

ALLREPOS=$(ssh git@vm-repos find /data/repos/git -name "config")
DATA=""

IFS=$'\n'; for LINE in $ALLREPOS; do
    # Replace out ugly path bits to just get the name
    REPOPATH=$(echo "$LINE" | sed 's/\/data\/repos\/git\///' | sed 's/\/config$//')

    # We have some directory structures here, convert the paths to underscores for the FishEye name.
    REPONAME=$(echo "$REPOPATH" | sed 's/\//_/g')
    
    # Load the description from the description file in git
    DESCPATH=$(echo "$LINE" | sed 's/config$/description/')
    DESC=$(ssh git@vm-repos cat $DESCPATH)

    # Sidenote, I felt dirty writing this:
    DATA+="<repository name=\"$REPONAME\" description=\"$DESC\" store-diff=\"true\" enabled=\"true\">
    <git location=\"git://vm-repos/$REPOPATH\" renames=\"none\"/>
    <security use-defaults=\"true\" allow-anon=\"false\" allow-logged-in=\"false\" allow-anon-write=\"false\" allow-logged-in-write=\"false\">
        <required-groups/>
        <required-write-groups/>
    </security>
</repository>
"
done

# Delete old repos
XMLDATA=$(xmlstarlet ed -d "/config/repository" $XMLFILE)

IFS=$'\n'; for line in $XMLDATA; do
    echo $line

    # Insert our new XML after the </repository-defaults> closing tag.
    if [[ "$line" == *\/repository-defaults* ]]; then
        echo "$DATA"
    fi
done

This script echoes out a new config.xml. A parent script just shuts down FishEye, executes this and drops the new config.xml in place, and then restarts FishEye.

Directory syncing, done!

Post receive hooks for FishEye updates

FishEye was previously polling repos every 15 minutes for changes. This is pretty ugly and inflicts some server load. Thanks to our newly perfectly-aligned FishEye and Git repo names though, writing a post-receive hook to update FishEye via the API became trivial:

# Replace directory separators with underscores again.
REPOPATH=$(pwd | sed 's/\/data\/repos\/git\///')
REPONAME=$(echo "$REPOPATH" | sed 's/\//_/g')

curl -s -X POST -H "X-Api-Key: $APIKEY" http://$FISHEYE/rest-service-fecru/admin/repositories-v1/$REPONAME/scan > /dev/null

Harmonious coexistence

Now when we do a Git push, we get a message like the one depicted. Project names link to the FishEye project, JIRA ticket numbers link correctly, commit hashes link to the changeset in FishEye, and FishEye's post-receive hook helps make sure that the commit number links are working in a matter of seconds. Server load is down on both the FishEye and Git servers since we're not polling every 15 minutes on every repository anymore. And our FishEye coverage is now 100%.

Huzzahs are in order.

Disqus Discover

2013-05-20T00:00:00-05:00

The internet's ubiquitous SaaS commenting widget, Disqus, has a new-ish feature called Discover which they're pushing heavily for their new sign-ups. Discover is meant to increase your traffic and make money from your website's engagement. It's really a value-added service whereby your comments section links to related featured/paid content on other sites, earning revenue for you. A kind of AdSense for the Disqus network.

I love it; this is a great idea. Content producers get wider exposure, site owners earn revenue, readers get linked to relevant related content, and Disqus monetizes.

Emphasis on great idea, though. The execution is just terrible. Check out the kind of content I'm getting:

As far as I can tell, the service is making no attempt to drive users to related content. And why are these all so seedy? Is there something in my browser history? I checked with a new IP and a clean browser history, nope, same garbage.

Thankfully this feature is optional. And as of today, it's turned off. But in the interests of making this a constructive post ...

What could Disqus do to make this better?

I think a lot of this is fairly obvious, and I'm surprised Disqus considered Discover viable enough to launch in its current state, even as an MVP for a VAS.

Content Matching

Scrape the content of the page, look at HTTP referrers for search keywords. Use this to make sure that Disqus is linking to related content. Granted, this is not a trivial thing to implement, but there are tools nowadays that make it so it's not truly hard anymore. To massively over-simplify: Hadoop and a couple Python libraries, and you're done.

Once you have Content Matching working, a great way give site owners control of the ads showing on their site would be a "Greed Level" slider. A scale of 1-to-10 of how tightly the site owner want their ads/featured content bound to the current page's content. Some users want full-on monetization, and they're okay with linking off to largely unrelated spam posts. Then for someone like me who doesn't really care about monetization, but thinks that relevant related content would be cool, you could set that to 1 and only show on-topic related posts.

Filtering Inappropriate Content

A quick survey of my pages shows a majority of the featured content consists of trashy articles about sex, and nude beaches and such. At the bare minimum they should be providing an option to limit related content to things that could be construed as professional. One of the linked pages was something like the 25 best laptops on the market. Okay, it's still totally off-topic, but at least it's not seedy.

Content Categorization

This is a pretty standard feature for ad networks, to have different topics that the site owner can specifically include or exclude. In lieu of content matching, this would be a good start. This is totally absent from Discover. In fact there are no options whatsoever other than something that seems to only control whether the ad content shows above or below the comments box.

Outcome

The outcome is that for now I've turned this brilliant feature off. I didn't want to, because I really like the idea, but it's just too poorly executed to leave on.

This post will serve as a reminder to me to try this again later, once Disqus has had some time to improve the feature. But for now, Disqus has missed the boat by launching prematurely. I have to imagine a lot of users have tried this, become disenfranchised, and won't turn it back on.

Setting up S3 and CloudFront for Static Site Hosting

2013-05-19T00:00:00-05:00

(This is a companion post to my post about my S3/CloudFront static site deployment script, _jaws)

S3 and CloudFront pair up really nicely to form a static site hosting solution. You can upload your site to S3, avoiding the need for real hosting, and then you can use CloudFront as a CDN to make things fast, all over the world.

If your blog gets little or no traffic, the total cost will be in the vicinity of $0.10 a month depending on your settings.

I love this setup for my blog since it means I don't need to maintain any infrastructure. Sure, I have server assets and VPSes lying around, but worrying about a WordPress blog being hacked is way down on my list of things I want to be doing. And while Amazon's datacenters have had downtime here and there, just like my Linodes, I have a pretty high level of confidence that things will come back up in a reasonable amount of time and "just work" with this setup. Push it, forget it.

I've also spent a lot of time traveling and living outside of the USA (I'm writing this blog post from my girlfriend's place in Hong Kong) which has made me particularly sensitive to sites that lack edge servers. It sounds petty, but I really don't want that extra 300ms of latency if I can avoid it.

So let's talk about how to set this up.

Step 1: S3

S3 setup is super easy. All you have to do is log into AWS and create a bucket.

There is one trick though. To make things place nice with CloudFront (and not have all your paths have index.html in them), you have to use the "Static Website Hosting" option.

If you don't do this, you're going to get confusing AccessDenied XML error messages from CloudFront down the road.

Make note of the Endpoint that it shows you.

You're now ready to go ahead and upload your files to your S3 bucket.

Step 2: CloudFront

CloudFront is a bit less obvious. Start off by creating a download distribution. Most of the settings are pretty straightforward, but one very easy mistake to make is when you're setting the "Origin Domain Name". Make sure to specify the HTTP URL to your S3 bucket, rather than one of the pre-filled S3 buckets. Otherwise, again, you'll get those AccessDenied messages.

Most settings can be left as the default but I highly recommend:

Set your Default Root Object to index.html.
Setting "Forward Query Strings" to Yes. This can be huge for proper caching of linked assets like JavaScript and CSS.
Setting a couple alternate CNAMEs. One for www.example.com, one for static.example.com, one for images.example.com. The main benefit here is that you can spread your HTTP requests over several different domains, for HTTP efficiency. Not really needed.
For Object Caching, I select Use Origin Cache Headers. This is because I know I'm setting these properly via _jaws; if you're not doing this, you'll want to Customize and set your own Minimum TTL.

After you submit, CloudFront may take a few minutes for things to finish setting up. Once it's done, add a CNAME record into your DNS such that www.example.com references your CloudFront domain name (mine for example is d1bx34tfj1fr6j.cloudfront.net).

You should be all set now.

A couple next steps:

Use a deployment script (like my aforementioned _jaws) to upload your site with proper compression and caching.
Make sure your naked/apex (e.g. example.com) domain 301 redirects to www.example.com. You can't do a CNAME on your domain apex, so CloudFront won't work nicely from there. You can use something like wwwizer. I use nginx redirect on Linode I have lying around for some other stuff. S3 can also do this with the "Redirect all requests to another host name" option pictured above.

From here, you should be in really good shape. Happy AWSing.

Introducing _jaws

2013-05-18T00:00:00-05:00

I've been hosting my blog on Amazon's S3 and CloudFront for a while; I also host a couple other static sites using a similar setup. It's a wonderful setup once you have it working. Blazing fast, dirt cheap, and the complicated moving parts are all abstracted away.

The problem was always getting everything up on CloudFront properly. That's why I wrote _jaws: a comprehensive, drop-in, AWS deployment script for Jekyll blogs or other static sites.

In this post I'll walk you through the problems that _jaws solves, and how it solves them.

Problem: Caching and cache-busting linked assets.

You want your truly static assets like CSS, JavaScript, and images to be cached forever. You don't even want the browser to wait for a 304 Not Modified response. This is easy to do with a Cache-Control: max-age=... header ... until you have to change your assets. The fix is to cache-bust using the query string - e.g., /example.js?v=1 - using a version number or a file modification time or something. You can do this per-asset with Jekyll, by having each asset you include have a ?ts={{site.time}} on the end. But this is tedious.

Solution: _jaws finds every script include, stylesheet include, and image, and inserts the current git commit on the end. Everything is versioned based on when the code actually changes, and is permanently cached. We also do a separate s3cmd sync pass to pass a proper Cache-Control header for JS/CSS.

Problem: Separate hostnames.

Admittedly a minor problem. It can be more efficient if CSS, JavaScript and images are on a separate domain. It means these assets can download asynchronously while the HTML continues to load.

Solution: Simple enough, _jaws does a full pass over HTML assets and inserts a fully-qualified domain name on any included JS, CSS, or image.

Problem: Minification.

It's just easier for development to use un-minified JavaScript, CSS, and HTML. But in a production environment, you really want to strip out all that extra weight. Yeah, you could have both minified and un-minified assets and swap between them (for JavaScript), and for CSS you're probably using something like SASS, LESS or Stylus that will generate minified CSS for you. For HTML you're almost certainly not minifying (and you probably don't care). But it's a good optimization, and it's nice to have it just work.

Solution: _jaws includes YUI Compressor and HtmlCompressor, and runs these on every appropriate asset in your project. Nice, tiny assets.

Problem: Compression.

This is actually where things get tricky and interesting. When you're serving text assets, you really want those to be gzipped. Neither S3 nor CloudFront do this for you. You can upload a gzipped file and tell S3 to include a Content-Encoding: gzip header though.

As an extra problem, for things like images, audio, or video files, you almost certainly don't want them to be gzipped, because they're already compressed. The additional compression will just slow things down.

Solution: The trick is to do several different s3cmd sync commands. One for gzipped content, one for non-gzipped-content. _jaws finds all the files that make sense to compress, and gzips them either using gzip -9 or with zopfli (bonus points for extra compression).

Problem: CloudFront Cache Invalidation

Sometimes you need your changes to happen now. Well, bad news. CloudFront won't let you do that. But 5-10 minutes is attainable using a CloudFront invalidation request.

Luckily s3cmd has a --cf-invalidate command that does this for us. So that was easy.

In Conclusion

After all this, we now have a properly compressed, minified, and optimized site, served off an enterprise CDN network ... for less than a dollar a month. And now, with _jaws, you can do this with two lines:

git clone git@github.com:pauljz/_jaws.git
cp _jaws/config.sample.sh _jaws/config.sh

Change a couple configuration settings and you're done! For more information and to get started, take a look at:

_jaws on GitHub.
My post on setting up S3 and CloudFront for static sites.

I also wanted to throw out a special thank-you to Tobias Sjösten whose excellent blog post inspired this whole script.

Always Google your error messages

2013-04-07T00:00:00-05:00

Watching other people debug is fascinating. I occasionally find myself spotting my coworkers or friends as they try to find the cause of a bug or the fix for a crash. Next time you get called over to someone's desk, really watch as they show you the problem, and show you what they've tried. Of course, you're helping to look for the fix, but also pay attention to them. There's a big opportunity here if you study their approach as well.

You can learn a lot about how a person thinks about and internalizes code and architecture by watching the way they try to fix a bug. Methodologies and thought-processes are more different than you would ever expect. I respect a healthy difference in debugging approaches — it means a friend is more likely to find a fix for something that I overlooked, and vice-versa — but there's one thing that always bothers me:

When people don't Google their error messages.

The number of times I've seen someone get a cryptic error message, say What the hell does that mean?!, and then go back to rummaging in their code ... disturbing.

Unless it's an error from your in-house code, or you're working on something incredibly niche, chances are that someone has had the same problem as you in the past. Any time it looks like your error might be due to anything other than your own software, plug it into Google! It works!

Thanks in large part to StackOverflow and the other StackExchange sites, but also to a proliferation of domain-specific forums, you'll almost always get results. And you'll often get an answer too.

Maybe a quarter of the time you'll find threads where someone has had the problem, and a few people have discussed it, and no fix has shown up. Even then it's not a wash, since you just invested all of 2 minutes to learn a handful of things that don't work.

Of course, every now and then you get a soul-crushing defeat, but in the past few years this has become even rarer for me.

Let's do a case study, just for fun.

This one was just today. I was debugging an error in a new client's mod_perl/Embperl application. All the client has is their production server — their original dev environment was eaten by a former contractor — so the first thing we did was build out a proper dev environment again. If you've ever installed someone else's Perl application, you're probably familiar with the hell of missing, undocumented CPAN module dependencies. This was no exception. About 40 CPAN modules later we had a working environment (had to Google a few errors there for unresolved dependencies). Except we found out a few AJAX requests would crash with this:

/usr/sbin/apache2: symbol lookup error: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: apreq_handle_apache2

Huh? Version incompatibility between Perl's Apache2.so and our Apache or mod_perl version? Bug in the CPAN module? Library misuse? A Google search later yielded this thread, where I found out I just had to run sudo a2enmod apreq. I didn't even know there was an apreq module for Apache. But I now had my fix in about 30 seconds of seeing the error.

I'll try to keep appending more case studies as they come up, but feel free to add your own in the comments!

Embedding escaped code in a Jade template

2012-08-06T00:00:00-05:00

A while back I had built a bit of a blogging platform using the ubiquitous node.js stack of Express and Jade templates. It was pretty straight-forward — blog posts were just written in Jade and served with Express — but syntax highlighting ended up being a problem.

I had opted to use google-code-prettify, which takes an escaped <code> block of HTML like this and highlights it:

<pre class="prettyprint">
    <code>
        &lt;html&gt;
            &lt;head&gt;&lt;/head&gt;
            &lt;body&gt;&lt;/body&gt;
        &lt;/html&gt;
    </code>
</pre>

I really didn't want to pre-escape my code, of course. That's horrible. And as far as I can tell, Jade has no way to automatically escape HTML like this, unless it's already stored in a variable.

Solution: Custom Filters

Jade's documentation briefly mentions filters. The syntax is just a word prefixed with a colon. There are a few canned ones like :markdown. I jumped in and wrote my own custom filter, :code, to strip out significant HTML characters and replace them with < > and the like.

require('jade').filters.code = function( block ) {
    return block
        .replace( /&/g, '&amp;' )
        .replace( /</g, '&lt;' )
        .replace( />/g, '&gt;' )
        .replace( /"/g, '&quot;' );
}

Cryptic errors ensued.

SyntaxError: Unexpected token ILLEGAL
    at Object.Function (unknown source)
    at Object.compile (/Users/paul/Code/paulzumbrun.com/node_modules/jade/lib/jade.js:176:8)
    at Function.compile (/Users/paul/Code/paulzumbrun.com/node_modules/express/lib/view.js:68:33)

Turns out the result still gets parsed by Jade; it's not just a string that gets dropped in like I thought. This is actually a good thing because it means you can chain filters, but we have to work around it. To make the parser happy with me again I had to strip out a few extra characters. Newlines needed to become \n. Other escape sequences needed double-escaped. Hashes need stripped too, since they're used to execute JavaScript inline #{thusly}.

This is the final working filter.

require('jade').filters.code = function( block ) {
    return block
        .replace( /&/g, '&amp;'  )
        .replace( /</g, '&lt;'   )
        .replace( />/g, '&gt;'   )
        .replace( /"/g, '&quot;' )
        .replace( /#/g, '&#35;'  )
        .replace( /\\/g, '\\\\'  )
        .replace( /\n/g, '\\n'   );
}

Now when you go to use this in a template, all you have to do is this. Pretty close to my original goal.

pre.prettyprint
    code.lang-html
        :code
            <html>
                <head></head>
                <body></body>
            </html>

There may be a cleaner way to do this, but I sure couldn't find it with the search terms I was using.