SuperPumpup (dot com)
General awesomeness may be found here.

23 December 2014

Packaging a Vert.X/Rails Application (Part II)

Alright, step 2. This will cover:

  • Bundling your gems
  • Packaging your app "better"
  • Putting your packages in a repository
  • Put your repository in S3
  • Get your packages back from the repository

To torture the metaphor @sigil66 is fond of, we don't want to ship the factory with the car, so are going to isolate the "building" part of software from "running" it.

Getting gems bundled correctly ($GEM_PATH and $GEM_HOME)

So this is a bit of a nightmare. I've been doing a lot of experimentation to try to understand bundler and gem environments. Nothing is as straightforward as I'd have thought and it really makes me take for granted how easy this has been for me to date.

But things are mostly straightened out now. I'll show you the steps on my build box that will package up the app - the key steps are:

jruby -S gem install -i vendor/gem_home --no-rdoc --no-ri bundler

So this installs Bundler into a cordoned off zone.

GEM_HOME=vendor/gem_home GEM_PATH=vendor/gem_home jruby -S bundle install --path=vendor/bundle --binstubs --without development test

And then bundles up all the gems!

GEM_HOME=vendor/gem_home GEM_PATH=vendor/gem_home jruby -S bin/rake assets:precompile RAILS_ENV=staging

Precompiling assets is a pretty good "canary".

It's not super complicated, and in theory I could probably export the GEM_HOME and GEM_PATH variables to clean up the command lines, but I was just getting inconsistent results from having the exported variables.

Packaging up the Rails app

All of this goes into a script I finally wrote to package up my Rails application. This runs in a Vagrant instance with my code directory mounted at /src, and note that it copies it to a /tmp directory and deletes a bunch of things. I found this to be easier than trying to get fpm's -x flag to properly exclude some of the files from getting packaged up.

#!/bin/bash

set -e

CWD=`pwd`
VERSION=`if [ -f web-version ]; then cat web-version; else echo 1.0; fi`
ITERATION=`if [ -f web-iteration ]; then cat web-iteration; else echo 1; fi`

cp -R /src/Exchange /tmp/web-build
cd /tmp/web-build

# Clean up cruft
rm log/* || true
rm -rf .git
rm -rf .bundle
rm -rf tmp/*
rm -rf vendor/bundle

# Build assets/dependencies
sudo npm install
cd webpack && BUILD_DEV=0 webpack -p --config webpack.rails.config.js && cd ..

export EXECJS_RUNTIME='Node' JRUBY_OPTS="-J-d64 -X-C"

jruby -S gem install -i vendor/gem_home --no-rdoc --no-ri bundler
GEM_HOME=vendor/gem_home GEM_PATH=vendor/gem_home jruby -S gem list
GEM_HOME=vendor/gem_home GEM_PATH=vendor/gem_home jruby -S bundle install --path=vendor/bundle --binstubs --without development test
GEM_HOME=vendor/gem_home GEM_PATH=vendor/gem_home jruby -S bin/rake assets:precompile RAILS_ENV=staging

# Now clean up the asset install tools
sudo rm -rf node_modules

# Build the package
cd $CWD

fpm -s dir -t deb -n obsidian-web -a noarch -f \
  -C /tmp/web-build \
  --version $VERSION \
  --iteration $ITERATION \
  .=/usr/local/web

if [ $? -ne 0 ]
then
  echo ""
  echo "Build failed. Exiting. Check and clean up as needed."
  exit 2
fi
expr $ITERATION + 1 > web-iteration

rm -rf /tmp/web-build

echo ""
echo "Build successful"
exit 0

cd $CWD

Repository-ing up Your Packages

So what is an apt repository? I have no fucking clue. From what I can tell it's a tree of directories on your filesystem. Here's mine:

pool/
└── dists
    └── trusty
        ├── Release
        ├── Release.gpg
        └── staging
            └── binary-all
                ├── backbone_1.0_all.deb
                ├── md5-results
                │   ├── obsidian-backbone_1.0-2_all.deb
                │   ├── obsidian-cookbooks_1.3-22_all.deb
                │   ├── obsidian-web_1.0-7_all.deb
                │   ├── obsidian-web_1.0-8_all.deb
                │   ├── obsidian-web_1.0-9_all.deb
                │   ├── obsidian-web_--iteration_all.deb
                │   ├── vertxpackages_1.0-3_all.deb
                │   ├── vertxpackages_1.0_all.deb
                │   └── web_1.0_all.deb
                ├── obsidian-backbone_1.0-2_all.deb
                ├── obsidian-cookbooks_1.3-22_all.deb
                ├── obsidian-web_1.0-7_all.deb
                ├── obsidian-web_1.0-8_all.deb
                ├── obsidian-web_1.0-9_all.deb
                ├── obsidian-web_--iteration_all.deb
                ├── Packages
                ├── Packages.gz
                ├── Release
                ├── vertxpackages_1.0-3_all.deb
                ├── vertxpackages_1.0_all.deb
                └── web_1.0_all.deb

So basically, what makes this function as the repository is the package (say, vertxpackages_1.0-3_all.deb), the Packages file - here's mine:

vagrant@vagrant-ubuntu-trusty-64:~$ cat pool/dists/trusty/staging/binary-all/Packages
Package: vertxpackages
Version: 1.0-3
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 192758
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/vertxpackages_1.0-3_all.deb
MD5sum: 2530240341dc026f20a3179854ca4541
Size: 178261524

Package: vertxpackages
Version: 1.0
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 189204
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/vertxpackages_1.0_all.deb
MD5sum: dcfb0db6ef053f634ba266f757b8c269
Size: 175044474

Package: obsidian-cookbooks
Version: 1.3-22
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 4487
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-cookbooks_1.3-22_all.deb
MD5sum: 302322915340afd68df6d41b6b7471cd
Size: 4536716

Package: obsidian-backbone
Version: 1.0-2
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 274130
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-backbone_1.0-2_all.deb
MD5sum: e10b45e74eb1c390ba13cf48bfe818b7
Size: 267225700

Package: backbone
Version: 1.0
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 274096
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/backbone_1.0_all.deb
MD5sum: 6cf4a12dfd0baf1d1203ecc5b9b5e8bd
Size: 267207408

Package: obsidian-web
Version: 1.0-9
License: unknown
Vendor: vagrant@vagrant-ubuntu-trusty-64
Architecture: all
Maintainer: <vagrant@vagrant-ubuntu-trusty-64>
Installed-Size: 562455
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-web_1.0-9_all.deb
MD5sum: bff3be238e0c419fb898203f1a157ae8
Size: 269526812

Package: web
Version: 1.0
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 718139
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/web_1.0_all.deb
MD5sum: 94e0988f912f68f692470c880fbd029b
Size: 331714612

Package: obsidian-web
Version: 1.0-8
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 732843
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-web_1.0-8_all.deb
MD5sum: ea30d51e51d7c8b4f42abf956b83a87d
Size: 333915052

Package: obsidian-web
Version: 1.0-7
License: unknown
Vendor: litch@The-Box-3.local
Architecture: all
Maintainer: <litch@The-Box-3.local>
Installed-Size: 723448
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-web_1.0-7_all.deb
MD5sum: 97e2c7811e120c6d2097dcddfb650a4b
Size: 329463556

Package: obsidian-web
Version: --iteration
License: unknown
Vendor: vagrant@vagrant-ubuntu-trusty-64
Architecture: all
Maintainer: <vagrant@vagrant-ubuntu-trusty-64>
Installed-Size: 985631
Section: default
Priority: extra
Homepage: http://example.com/no-uri-given
Description: no description given
Filename: dists/trusty/staging/binary-all/obsidian-web_--iteration_all.deb
MD5sum: 5668749a3b44f1985fc0f39bacffe13a
Size: 472612812

Of course there is the gzipp'd version of that. And then there is that Release file, which is apparently empty:

vagrant@vagrant-ubuntu-trusty-64:~$ cat pool/dists/trusty/staging/binary-all/Release
vagrant@vagrant-ubuntu-trusty-64:~$

MD5 hashes of the packages:

vagrant@vagrant-ubuntu-trusty-64:~$ cat pool/dists/trusty/staging/binary-all/md5-results/obsidian-web_1.0-8_all.deb
ea30d51e51d7c8b4f42abf956b83a87d

There is another Release file:

vagrant@vagrant-ubuntu-trusty-64:~$ cat pool/dists/trusty/Release
Date: 2014-12-23 00:16:19 UTC
Suite: trusty
MD5Sum:
 d41d8cd98f00b204e9800998ecf8427e                  0 Release
 fa72c833d4068a9444a26ee6bbbaf3b5                  4012 staging/binary-all/Packages
 7d032058b76a04883994caddf50e45eb                  775 staging/binary-all/Packages.gz
 d41d8cd98f00b204e9800998ecf8427e                  0 staging/binary-all/Release

And the signature of that file:

vagrant@vagrant-ubuntu-trusty-64:~$ cat pool/dists/trusty/Release.gpg
<binary gibberish>

So that's not too bad. Now we know. We can build all of this with the PRM tool.

There isn't a "usage guide" per se on the prm site that I could find, mostly I just experimented with stuff until it worked. This is the command I use to build my repo:

prm --type deb --path pool --component staging --release trusty --arch all --directory /src/obsidian-packages -k
# note the -k has to be last to just grab the default gpg key

So if you run that command, it will go look in the --directory that you specify and add all the packages it can find (that have the arch all) into the repository that you specify with the --path component. For some reason mine is called pool. I don't know why, I must have just cargo-culted that.

Getting Your Repo to a Place You Can Access It

This is the easiest part. Mostly because the documentation is written at a level that's accessible to mere mortals. There's a good description here.

The key command, after installing s3cmd and configuring it is:

s3cmd sync pool/ s3://your-fancy-repository-bucket-name

Checkpoint

Ok, so now we have packaged up the Rails application, put it into an apt repository, and put that out on the internet. Phew. Now all that remains is to get it back off.

Retrieving Packages From An S3 Apt Repository

This was one of the most confusing parts, as there is a lot of documentation about how to do this distributed over about 6 years. Which means, most of it is wrong. Go internet.

So, on your build box, compile the apt-transport-s3 project using the directions from:

https://github.com/sigil66/apt-transport-s3

It's basically make/make install (phew).

Then package it up:

fpm -s dir -t deb -a amd64 -n apt-s3 src/s3+http=/usr/lib/apt/methods/s3 src/s3+https=/usr/lib/apt/methods/s3+https
# note a sneaky rename here to change the s3+http to s3

Get that package on your production server (or any package consumers), and then you can just

dpkg -i <apt-s3-whatever-version>.deb

And then add to your sources.list:

#/etc/apt/sources.list
deb [arch=all] s3://<access-key>:[<secret>]@s3.amazonaws.com/your-fancy-repository-bucket-name trusty staging

Though it's maybe blindingly obvious, the trusty there refers to the distribution, which you set in prm with the --release flag, and the staging refers to the "component" you set in the prm command.

That's it!

And now you can apt-get update and apt-get install to your heart's content.

Holymoly!