SuperPumpup (dot com)
General awesomeness may be found here.

25 February 2014

Agh! People are starting to use my App! Now what?

This is a slightly beefed up version of the lightning talk I gave at Austin on Rails last night - Feb 25, 2014.

I had intended to give this talk on things I've learn about infrastructure as I've been able to help stabilize and grow at a my last two jobs. I've had a lot of experience working on deployment infrastructure, from Dreamhost, to Rackspace, to Heroku, AWS, and now am mananging our infrastructure on Amazon OpsWorks using Chef (which I've been writing about recently).

However, as I thought about it, the problems that have been hindering our growth have not been as much server infrastructure related, as much as they were visibility-related...

Continue reading


13 February 2014

Deploying a Multi-Rails-App OpsWorks Stack

I've written before about how OpsWorks kind of pushes you into a weird architecture because of their default behaviors for applications in a stack.

Namely, the system tries to push all the applications on all of the layers, and chaos ensues.

But, it turns out I needed to finally move our various apps into one stack, and after a few days of poking, prodding, waaaaaaaaaaiting for machines, I got it working, so I thought I'd document it.

Continue reading


06 February 2014

Setting Up OpsWorks Using Your Very Own (Shiny) VPC

I'm in the middle of changing our application architecture at work from being a bunch of OpsWorks stacks into one large stack - so that I can tell my individual application nodes what's going on where. I ran into one pretty big hiccup while porting my stack from "toy" that I got by following this excellent walkthrough.

Basically the demo stack uses public git repos - no custom cookbooks or private app code. That clearly doesn't work in the real world, though. We have a large set of custom cookbooks that we're working on supplementing so I need to pull those down. To do so, not only did I need to change the stack settings to point to the repo using the git@github... address (instead of git://github...).

OpsWorks VPC Private Cookbook Repo

But I also had to go to the NAT security group that I had set up using the CloudFormation and add port 22 to one that was allowing inbound and outbound connections. The defaults allow port 9418 (the git protocol), but that protocol does not allow you to use our deploy key for authentication:

OpsWorks VPC NAT Setup

So if you are having problems setting up your OpsWorks using custom cookbooks inside a VPC, then make sure you have port 22 forwarded on your NAT.

So much to learn...


25 December 2013

Such Doge. So Cloud.

Or, how to set up an EC2 box to mine 300 KH/s

So I thought that this DogeCoin was starting to sound more and more interesting, and, having access to some spare CPU cycles in the cloud and not wanting to burn up my Retina MBP GPU, I thought it would be fun to see if I could mine at a decent rate in on Amazon EC2. I started out just doing some CPU mining, but that was not terribly effective (only getting me a few Doge/hr), and since it's fun to play with other people's hardware, I thought this would be fun to explore.

Turns out, I kind of can - it's about 8 minutes to get up and running - read on for detailed walkthrough, or skip to the end for analysis (or just leave if you'd rather do that)...

Instance type

Poking around, it seems that this Amazon Linux AMI with NVIDIA GRID GPU Driver is a good candidate. Amazon Nvidia Banner

So register/sign in with Amazon however you need to and click the Continue button, then just leave the rest of the "Launch with 1-Click" settings alone and you should be set. Default settings work pretty goodAre you sure?Box coming online

Connect however you're going to connect. Walking you through connecting to an EC2 instance is beyond the scope of this, but you should be able to figure it out.

You should update packages...

sudo yum update
(select yes when prompted)
sudo yum install automake

Download the CudaMiner source code we'll be installing:

[ec2-user@ip-10-51-131-176 ~]$ wget https://github.com/cbuchner1/CudaMiner/archive/master.zip
--2013-12-25 11:13:45--  https://github.com/cbuchner1/CudaMiner/archive/master.zip
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/cbuchner1/CudaMiner/zip/master [following]
--2013-12-25 11:13:45--  https://codeload.github.com/cbuchner1/CudaMiner/zip/master
Resolving codeload.github.com (codeload.github.com)... 192.30.252.146
Connecting to codeload.github.com (codeload.github.com)|192.30.252.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 310982 (304K) [application/zip]
Saving to: ‘master.zip’

100%[========================================================================================>] 310,982     --.-K/s   in 0.008s

2013-12-25 11:13:45 (37.1 MB/s) - ‘master.zip’ saved [310982/310982]

[ec2-user@ip-10-51-131-176 ~]$ ls
master.zip
[ec2-user@ip-10-51-131-176 ~]$ unzip master.zip
Archive:  master.zip
e47cf1994a3021050bf64521acf6978d3a5c4f27
   creating: CudaMiner-master/

 ... lots of output ...

[ec2-user@ip-10-51-131-176 ~]$ cd CudaMiner-master/
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Now we've gotten and unzipped the code, let's start trying to build it.

Running naked ./configure fails with a configure: error: Missing required libcurl >= 7.15.2 which is annoying, so let's try autogen instead.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
configure.ac:116: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.

So let's see what else we can glean from what they've given us:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/usr/local/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ which cuda
/usr/bin/which: no cuda in (/opt/nvidia/cuda/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin)
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Well that's unfortunate - it looks like the ./configure.sh script is pointing to the wrong location (for us) for cuda.

Let's fix that:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ echo ./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/opt/nvidia/cuda > configure.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure CFLAGS=-O3 CXXFLAGS=-O3 --with-cuda=/opt/nvidia/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Trying ./configure.sh now gets further and it errors at:

checking for pthread_create in -lpthread... yes
./configure: line 7109: syntax error near unexpected token `,'
./configure: line 7109: `LIBCURL_CHECK_CONFIG(, 7.15.2, ,'`

So I'll try ensuring I have all my libcurls:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install libcurl-devel.x86_64

And try again. Nope, same error... But googling around suggests that this may have to do with the line endings - there's a difference between Windows and *nix systems with how they end lines, and this is kind of a common cause of errors, amazingly...

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install dos2unix
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ find . -type f -exec dos2unix {} \;
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./configure.sh

Success! The output of ./configure is now:

configure: creating ./config.status
config.status: creating Makefile
config.status: creating compat/Makefile
config.status: creating compat/jansson/Makefile
config.status: creating cpuminer-config.h
config.status: executing depfiles commands

So then now that that's configured I can actually build the thing:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ make

This dumps TONS of output to the console (including gobs of warnings), and takes a few minutes.

Built!

It looks like the make was successful on the first try, and I have the following output at the end:

ptxas info    : Used 32 registers, 336 bytes cmem[0]
g++  -O3 -pthread -L/opt/nvidia/cuda/lib64  -o cudaminer cudaminer-cpu-miner.o cudaminer-util.o cudaminer-sha2.o cudaminer-scrypt.o salsa_kernel.o spinlock_kernel.o legacy_kernel.o fermi_kernel.o kepler_kernel.o test_kernel.o titan_kernel.o -lcurl   compat/jansson/libjansson.a -lpthread  -lcudart -fopenmp
make[2]: Leaving directory `/home/ec2-user/CudaMiner-master'
make[1]: Leaving directory `/home/ec2-user/CudaMiner-master'

So now you can test it out like so:

./cudaminer -o stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Feel free to swap out your own pool and worker name/password, or just do a few hashes for me. =)

It will take the system a few minutes to get the GPU figured out and autotuned, but while you wait you can get set up to take advantage of the 8 CPU's that will be sitting idle once we get this GPU hashing.

For reference, here is my output:

*** CudaMiner for nVidia GPUs by Christian Buchner ***
                 This is version 2013-12-18 (beta)
    based on pooler-cpuminer 2.3.2 (c) 2010 Jeff Garzik, 2012 pooler
           Cuda additions Copyright 2013 Christian Buchner
       My donation address: LKS1WDKGED647msBQfLBHV3Ls8sveGncnm

[2013-12-25 11:32:32] 1 miner threads started, using 'scrypt' algorithm.
[2013-12-25 11:32:32] Starting Stratum on stratum+tcp://doge.poolerino.com:3333
[2013-12-25 11:32:32] Stratum detected new block
[2013-12-25 11:32:57] GPU #0: GRID K520 with compute capability 3.0
[2013-12-25 11:32:57] GPU #0: the 'K' kernel requires single memory allocation
[2013-12-25 11:32:57] GPU #0: interactive: 0, tex-cache: 0 , single-alloc: 1
[2013-12-25 11:32:57] GPU #0: Performing auto-tuning (Patience...)
[2013-12-25 11:32:57] GPU #0: maximum warps: 459
[2013-12-25 11:34:47] GPU #0:  244.68 khash/s with configuration K18x21
[2013-12-25 11:34:47] GPU #0: using launch configuration K18x21
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 0.09 khash/s
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 116.83 khash/s[2013-12-25 11:34:54] GPU #0: GRID K520, 1439424 hashes, 231.48 khash/s
[2013-12-25 11:34:54] accepted: 1/1 (100.00%), 231.48 khash/s (yay!!!)
[2013-12-25 11:34:56] GPU #0: GRID K520, 508032 hashes, 228.09 khash/s
[2013-12-25 11:34:56] accepted: 2/2 (100.00%), 228.09 khash/s (yay!!!)
[2013-12-25 11:34:59] GPU #0: GRID K520, 556416 hashes, 228.73 khash/s

Etc etc etc. So every line or two it is accepting a ... whatever it is. I really don't understand the mining process that well, but I'm learning.

CPU Hash

This is a lot easier because we can just go grab a precompiled 64-bit binary and run it.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cd
[ec2-user@ip-10-51-131-176 ~]$ wget http://downloads.sourceforge.net/project/cpuminer/pooler-cpuminer-2.3.2-linux-x86_64.tar.gz?r=&ts=1387953566&use_mirror=softlayer-ams
[ec2-user@ip-10-51-131-176 ~]$ tar xvf pooler-cpuminer-2.3.2-linux-x86_64.tar.gz\?r\=
[ec2-user@ip-10-51-131-176 ~]$ ./minerd --url stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Ta-Da!

And that, step-by-step is how you can run a pretty decent DogeMining box in the cloud. So Cloud. So Doge.

Does this make any sense to do?

I don't know. Like this, totally 'naked' it is hashing at about 300 KH/s, which is generating me on the order of a few hundred coins/hr at this pool right now.

Math! 200 coins at $0.65/hr. That would mean that I would not return unless the price of a coin was about $0.003 (point three cents), and I think that the current (5:41AM on Christmas Morning) price is a factor of three below that. So it doesn't really seem to make a TON of sense to do economically, but hey, it's fun! I'm mining in the cloud! And maybe DogeCoin and BitCoin will go up in price 1000000000000000000%!

So I think that by setting this up, I'm now losing about $0.40/hr. Which is... Less-than-effective, but hey! With the scalabiltily, and not needing to worry about the hardware, I can always make it up in volume!

Better automating the process

A more efficient set of commands could definitely be built, and I'd toyed with the idea of "burning" this AMI so that I can just click and have it up and running, but I haven't decided the best way to (or if I'm going to) automate this more.

Lots of tools I'm really interested in in the configuration management space would make this helpful, but the learning curve on those is pretty steep and it probably won't help a ton.

In any case, if you want to send me Doge, that's much appreciated, too!

DAJa5tjgjrnJ3iMJb8khpDuE8tW8KNBWqG

10 December 2013

Paperclip and The Legend of Zelda

Figuring out where a Paperclip attachment is a bit like playing the Legend of Zelda sometimes.

These are things you need to figure out, from three different sources:

1 - The filename of the actual file (face.png) - source your database

1a - The name of the model and model id that you have attached this to. This is different information, but generally in the DB

2 - The pathing strategy that Paperclip uses - this is typically in a global paperclip config file (maybe config/initializers/paperclip.rb)

3 - Where the eff in the cloud it is - this is the biggest pain in the butt there. In a sloppily-configured prod setup, you'll be lucky enough to have this in a file like s3.yml or maybe paperclip.yml. But with good configuration-management, that file will only live on the server it's supposed to because, you know, security.

So once you have all three of these pieces of information from different sources, and have rescued the princess, what can you build? A string. The information to construct a string that's like 100 characters long is scattered across the corners of your application's world and that's just nuts.

WORSE! If someone ever decides to touch corner 2 of the triforce there and change anything about how paths are calculated, your application just forgets where to find its attachments. All your links are broken, and you're off trying to figure out how to set up "compatibility modes" for some of the files. Because apparently that string changed? Wut? The location of this file is mutable all of a sudden? (facepalm)

This came up for me as I'm building a Persistence as a Service module for an application and I started putting in Carrierwave (since it's a touch saner than Paperclip) and then realized, "Self, wtf are you doing?" Shouldn't this just be holding a string? And maybe (maybe?!) aren't you using a modern database that will let you have a field hold an array of strings? And can't that array of strings hold all of the immutable data?

Yep. So that's what I've got now. If someone wants the Foo object to hold some attachments, they send something like this:

"foo": {
  "name": "The ultimate Foo",
  "attachments": [
    {
      "type": "image",
      "image_content_type": "image/png",
      "small_url": "some_absolute_path_on_s3/picture_small.png",
      "medium_url": "some_absolute_path_on_s3/picture_medium.png",
      "large_url": "some_absolute_path_on_s3/picture_large.png",
      "original_url": "some_absolute_path_on_s3/picture_small.png"
    }
  ]
}

So there we go. That's at that end, but why can't that level of treatment of the location of these files as immutable be pushed down into the actual image processing library? What if I decide to start saving new pictures in a different bucket? Can't my database just point two different things at two different buckets? What if I want to push some of the files immediately out to a CDN and only reference that location of theirs? Piece of cake! It's just strings! URI's are just strings. The only thing your application is ever going to care about in most cases is "what is the URI of this file." So keep those concerns separate.


15 October 2013

On writing well

"On Writing Well" somehow appeared at the top of my Amazon wish list the other day. I must have put it there in some kind of 'moment of clarity' and then forgotten it, but for whatever reason I decided to pull the trigger.

I've really enjoyed reading it and have found fantastic correlations between his suggestions for writing good nonfiction, and writing good code (which is, hopefully, also nonfiction).

Refactoring

I don't like writing; I like having written. But I love to rewrite. I especially love to cut: to press the delete key and see an unnecessary word or phrase or sentence vanish into the electricity. On Writing Well, p. 87

I share Professor Zinsser's views on writing and having written prose, but disagree on code. I love writing code. I love the kind of frantic desperation and looming 'done' that I have when a failing test is reminding me of its presence at every pause.

But then, when I'm finally green, I find, the art comes in. I'll not try to compete with the imagery of seeing things 'vanish into the electricity' so will merely repeat it. So gratifying.

The best code is the code you don't have to maintain.

Form

Zinsser discusses paragraphs and their visual appeal (at a high level) on a page and I've found that though a wall of text compared to a cluster of paragrahs IS indeed more approachable, the disparity between well- and poorly-factored (even just from a high level) code is even more dramatic.

When I open the file with a 200 line controller method, I have to struggle so hard to not go to the water cooler.

Code of corresponding quality but broken into methods LOOKS so much more approachable. Even if its' not particualrly well-factored code, it's less intimidating - maybe enough so that some brave soul will come in and refactor it.

Verbs

Use strong, and evocative verbs in your prose. Send strong and clear messages in your code. Is this name intention-revealing enough? Ask that constantly.

I was at lunch with a colleague the other day and he mentioned that he had written a class where the main method was named, annhilate. He said he got such a kick out of calling that method. It's a wonderful name - it is perfectly descriptive of what it is going to do, and evokes an appropriate sense of danger.

Just like passive verbs or "nouned" verbs make bad prose, ambiguous, vapid, and weak method names make bad code. Send good messages. Name your methods well.

Nouns

On the same token, name your objects well. Zinsser mentinons "creeping nounism" - he mentions having seen a "communication facilitation skills development intervention". Five 'concept nouns' where one - workshop perhaps? - would do splendidly.

I've made my share of poorly named objects, and I know we all see (and probably work with) them every day. Name your objects well.

The Quickest Fix

The quickest fix when struggling with a word or a phrase in nonfiction prose is to delete it. I remember when I finally came to this understanding with Rails when I was learning - maybe struggling with the Rails 2 router to route to a non-RESTful action? - that maybe if it's hard, I'm doing the wrong thing. I wasn't doing it wrong, I was doing the wrong thing. As I learn more and develop more, it seems more consistent, not less, that if it's hard, I need to step back and examine why. If I can't come up with a compelling why, then it's time for the DELETE key and watch the mess 'vanish into the electricity'.

It's been amazing to see the 'carryover' of concepts from nonfiction prose to code, and I can't recommend this book enough to someone who wants to improve, but is burned out on reading SO MANY programming books. And hell, maybe my nonfiction prose will improve, too.


14 October 2013

OpsWorks and System Architecture

TL;DR Moving to OpsWorks has been very helpful for us to get our system infrastructure represented and reproducible as code. I don't think it's a great tool for us to stay on as things grow in complexity, or we just want better ease-of-use. It's a great stepping stone.

At OwnLocal, I've been involved in porting our application infrastructure from EC2 instances controlled by Capistrano to Amazon's new product, OpsWorks.

I was impressed by the Capistrano scripts, though some things caused us a lot of pain:

  • We did not always get consistent deploys
  • We had to hard-code in IP addresses in our deploy files
  • Server configuration was unreliable (some of our servers had customized configurations that were problematic to rebuild)

Understanding what I do about Opsworks has taken a lot of effort, and I'm certainly no expert (yet?) but we now have servers that spin up on weekdays as our system comes under load and down as the load abates without any intervention.

The first time I saw my load-based servers spin up was amazing

However, I have felt some tension between what I consider good system design (an ecosystem of small applications) and the way that OpsWorks "Stacks" work. Oddly, when you add an App to this server that is deployed using their friendly GUI, it adds the app to all the servers, and then a deploy command is triggered on ALL instances by default. Which is odd. There's no obvious way to deploy several different Rails app layers within the same stack (so communicating between service applications require hard-coding addresses).

This design makes it seem easier to leave all your logic in One Big App

I also have been frustrated that though I have to write my own cookbooks to run other than the most trivial vanilla "Stack", I do not have access to the most powerful Chef concepts like "Search". Plus, debugging is an absolute nightmare. Amazon does a lot of "crafting" of the settings used on lifecycle events that are not easily (possibly?) replicated in something like a Vagrant environment. Therefore the feedback cycle on customizing scripts is ungodly slow at times (tens of minutes for some apsects of the life cycle).

Now I'm at a place where I have the set of cookbooks written to be able to deploy my infrastructure as code (GO ME!) and OpsWorks helped me get there. However, I'm really looking for a justifiable way to jump ship and go to a hand-rolled cloud (maybe VPC) that runs Chef (probably paying Opscode) with more modern tools like knife and search. In fact, I'll probably achieve this the way all great change is done - slowly and deliberatly, one service at a time.

Goals for moving forward:

  • Search for nodes (or settings) in a sane way
  • Reconfigure multiple "stacks" more quickly (to install package the "OpsWorks way", I click a lot, which is "convenient", though it's a pain to do on two stacks - like Production and Staging)
  • Have better visualization tools
  • Have a better deployment tool ("cap -S staging deploy branch=test_something_dangerous")

So in all, OpsWorks is great. If you want an easy-ish way to move to infrastructure as code, I encourage you to check it out. However, it's not the be-all and end-all, and it won't save you from learning Chef. So if your application has some significant complexity already, it may not be right for you.


22 September 2013

OpsWorks -> NewRelic Deployment Notification

NewRelic is awesome.

Amazon OpsWorks is awesome.

However, getting OpsWorks to work with other things is sometimes not awesome.

I wanted these pretty notations on my graphs of when deployments happened, and I spent a lot of time working on it before coming upon a simple solution. I'll spare you the pain and just show the simple solution:

NewRelic Deployment Notification

#(opsworks-cookbooks)/newrelic/recipes/deploy.rb
include_recipe "deploy"

node[:deploy].each do |application, deploy|
  deploy = node[:deploy][application]

  execute "notify newrelic" do
    cwd deploy[:current_path]
    environment ({'RAILS_ENV' => deploy[:rails_env]})
    command "bundle exec newrelic deployments"
  end
end

I'll probably add a few things to use more of the command line options:

deployments [OPTIONS] [description] OPTIONS:
   -a, --appname=DIR            Set the application name.
                                Default is app_name setting in newrelic.yml
   -e, --environment=name       Override the (RAILS|MERB|RUBY)_ENV setting
   -u, --user=USER              Specify the user deploying.
   -r, --revision=REV           Specify the revision being deployed
   -c, --changes                Read in a change log from the standard input
   -h                           Print this help

Notably the user and revision (branch name), but I am still working on extracting those out of my chef json.

To use this, I simply added a call to this recipe in the Deploy group in that layer's Custom Chef Recipes:

OpsWorks Custom Chef Recipes

Update

HOLY MOTHER OF GOD THAT SUCKED.

I just went through the pain of trying to get the username attribute out of the OpsWorks deploy script and want to spare any other humans from it.

OpsWorks Deploy Attributes

To dump the attributes in something like a readable form, I was able to do this:

log JSON.pretty_generate node[:deploy]

Deploying User

OpsWorks doesn't support the search command, for whatever reason, which is totally evil, but I was able to track down the deploying_user and then parse it out like so:

long_user_string = node[:deploy][application][:deploying_user]
#something like: "arn:aws:iam::(long-number):user/username"
username = long_user_string.split('/').last

So then my log deployment command became:

command "bundle exec newrelic deployments -u #{username}"