SuperPumpup (dot com)
General awesomeness may be found here.

22 July 2014

Docker & RackConnect

To enable Docker containers to connect to the external network in a RackConnected environment, touch /etc/rackconnect-allow-custom-iptable. Otherwise RackConnect will destroy all Docker's forwarding rules.

Longer Version

So I just got bitten by a little issue using Docker & RackConnect. I had a nice fresh server brought up with lots of containers running and everything was grand. Ish.

I found two different problems, both with the same solution. On the one hand, I would start a service container (RabbitMQ), and then link to it with a "web" container, and then go to my RackConnect config and open a port to test connecting to the "web" container (that would have a port number like 49156 or whatever), but then all of a sudden that container wouldn't be able to access the service container.

Also, from within any container, I couldn't access the network.

curl google.com or ping 8.8.8.8 were just timing out.

I couldn't find a direct answer to this question, but it turns out that docker writes a ton of iptables FOWARD entries during the course of doing its thing, and those periodically get clobbered by RackConnect when it runs.

This article tells how to let RackConnect know to leave your custom rules alone - it does a merge rather than a force-overwrite. So touch that file, restart Docker, and be on your way!


24 June 2014

Log RabbitMQ to STDOUT (Because Docker)

Docker and Log aggregation. Not a solved problem. But I think the strategy I'm going to adopt for my current venture is having all of my containers log everything possible to STDOUT, and then just my log collection agents ingest the docker logs of each container. That way I can keep my containers simple (a single process without cheats like supervisord), and the actual operations also remains simple.

Continue reading


25 February 2014

Agh! People are starting to use my App! Now what?

This is a slightly beefed up version of the lightning talk I gave at Austin on Rails last night - Feb 25, 2014.

I had intended to give this talk on things I've learn about infrastructure as I've been able to help stabilize and grow at a my last two jobs. I've had a lot of experience working on deployment infrastructure, from Dreamhost, to Rackspace, to Heroku, AWS, and now am mananging our infrastructure on Amazon OpsWorks using Chef (which I've been writing about recently).

However, as I thought about it, the problems that have been hindering our growth have not been as much server infrastructure related, as much as they were visibility-related...

Continue reading


13 February 2014

Deploying a Multi-Rails-App OpsWorks Stack

I've written before about how OpsWorks kind of pushes you into a weird architecture because of their default behaviors for applications in a stack.

Namely, the system tries to push all the applications on all of the layers, and chaos ensues.

But, it turns out I needed to finally move our various apps into one stack, and after a few days of poking, prodding, waaaaaaaaaaiting for machines, I got it working, so I thought I'd document it.

Continue reading


06 February 2014

Setting Up OpsWorks Using Your Very Own (Shiny) VPC

I'm in the middle of changing our application architecture at work from being a bunch of OpsWorks stacks into one large stack - so that I can tell my individual application nodes what's going on where. I ran into one pretty big hiccup while porting my stack from "toy" that I got by following this excellent walkthrough.

Basically the demo stack uses public git repos - no custom cookbooks or private app code. That clearly doesn't work in the real world, though. We have a large set of custom cookbooks that we're working on supplementing so I need to pull those down. To do so, not only did I need to change the stack settings to point to the repo using the git@github... address (instead of git://github...).

OpsWorks VPC Private Cookbook Repo

But I also had to go to the NAT security group that I had set up using the CloudFormation and add port 22 to one that was allowing inbound and outbound connections. The defaults allow port 9418 (the git protocol), but that protocol does not allow you to use our deploy key for authentication:

OpsWorks VPC NAT Setup

So if you are having problems setting up your OpsWorks using custom cookbooks inside a VPC, then make sure you have port 22 forwarded on your NAT.

So much to learn...


25 December 2013

Such Doge. So Cloud.

Or, how to set up an EC2 box to mine 300 KH/s

So I thought that this DogeCoin was starting to sound more and more interesting, and, having access to some spare CPU cycles in the cloud and not wanting to burn up my Retina MBP GPU, I thought it would be fun to see if I could mine at a decent rate in on Amazon EC2. I started out just doing some CPU mining, but that was not terribly effective (only getting me a few Doge/hr), and since it's fun to play with other people's hardware, I thought this would be fun to explore.

Turns out, I kind of can - it's about 8 minutes to get up and running - read on for detailed walkthrough, or skip to the end for analysis (or just leave if you'd rather do that)...

Instance type

Poking around, it seems that this Amazon Linux AMI with NVIDIA GRID GPU Driver is a good candidate. Amazon Nvidia Banner

So register/sign in with Amazon however you need to and click the Continue button, then just leave the rest of the "Launch with 1-Click" settings alone and you should be set. Default settings work pretty goodAre you sure?Box coming online

Connect however you're going to connect. Walking you through connecting to an EC2 instance is beyond the scope of this, but you should be able to figure it out.

You should update packages...

sudo yum update
(select yes when prompted)
sudo yum install automake

Download the CudaMiner source code we'll be installing:

[ec2-user@ip-10-51-131-176 ~]$ wget https://github.com/cbuchner1/CudaMiner/archive/master.zip
--2013-12-25 11:13:45--  https://github.com/cbuchner1/CudaMiner/archive/master.zip
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/cbuchner1/CudaMiner/zip/master [following]
--2013-12-25 11:13:45--  https://codeload.github.com/cbuchner1/CudaMiner/zip/master
Resolving codeload.github.com (codeload.github.com)... 192.30.252.146
Connecting to codeload.github.com (codeload.github.com)|192.30.252.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 310982 (304K) [application/zip]
Saving to: ‘master.zip’

100%[========================================================================================>] 310,982     --.-K/s   in 0.008s

2013-12-25 11:13:45 (37.1 MB/s) - ‘master.zip’ saved [310982/310982]

[ec2-user@ip-10-51-131-176 ~]$ ls
master.zip
[ec2-user@ip-10-51-131-176 ~]$ unzip master.zip
Archive:  master.zip
e47cf1994a3021050bf64521acf6978d3a5c4f27
   creating: CudaMiner-master/

 ... lots of output ...

[ec2-user@ip-10-51-131-176 ~]$ cd CudaMiner-master/
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Now we've gotten and unzipped the code, let's start trying to build it.

Running naked ./configure fails with a configure: error: Missing required libcurl >= 7.15.2 which is annoying, so let's try autogen instead.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
configure.ac:116: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.

So let's see what else we can glean from what they've given us:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/usr/local/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ which cuda
/usr/bin/which: no cuda in (/opt/nvidia/cuda/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin)
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Well that's unfortunate - it looks like the ./configure.sh script is pointing to the wrong location (for us) for cuda.

Let's fix that:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ echo ./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/opt/nvidia/cuda > configure.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure CFLAGS=-O3 CXXFLAGS=-O3 --with-cuda=/opt/nvidia/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Trying ./configure.sh now gets further and it errors at:

checking for pthread_create in -lpthread... yes
./configure: line 7109: syntax error near unexpected token `,'
./configure: line 7109: `LIBCURL_CHECK_CONFIG(, 7.15.2, ,'`

So I'll try ensuring I have all my libcurls:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install libcurl-devel.x86_64

And try again. Nope, same error... But googling around suggests that this may have to do with the line endings - there's a difference between Windows and *nix systems with how they end lines, and this is kind of a common cause of errors, amazingly...

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install dos2unix
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ find . -type f -exec dos2unix {} \;
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./configure.sh

Success! The output of ./configure is now:

configure: creating ./config.status
config.status: creating Makefile
config.status: creating compat/Makefile
config.status: creating compat/jansson/Makefile
config.status: creating cpuminer-config.h
config.status: executing depfiles commands

So then now that that's configured I can actually build the thing:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ make

This dumps TONS of output to the console (including gobs of warnings), and takes a few minutes.

Built!

It looks like the make was successful on the first try, and I have the following output at the end:

ptxas info    : Used 32 registers, 336 bytes cmem[0]
g++  -O3 -pthread -L/opt/nvidia/cuda/lib64  -o cudaminer cudaminer-cpu-miner.o cudaminer-util.o cudaminer-sha2.o cudaminer-scrypt.o salsa_kernel.o spinlock_kernel.o legacy_kernel.o fermi_kernel.o kepler_kernel.o test_kernel.o titan_kernel.o -lcurl   compat/jansson/libjansson.a -lpthread  -lcudart -fopenmp
make[2]: Leaving directory `/home/ec2-user/CudaMiner-master'
make[1]: Leaving directory `/home/ec2-user/CudaMiner-master'

So now you can test it out like so:

./cudaminer -o stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Feel free to swap out your own pool and worker name/password, or just do a few hashes for me. =)

It will take the system a few minutes to get the GPU figured out and autotuned, but while you wait you can get set up to take advantage of the 8 CPU's that will be sitting idle once we get this GPU hashing.

For reference, here is my output:

*** CudaMiner for nVidia GPUs by Christian Buchner ***
                 This is version 2013-12-18 (beta)
    based on pooler-cpuminer 2.3.2 (c) 2010 Jeff Garzik, 2012 pooler
           Cuda additions Copyright 2013 Christian Buchner
       My donation address: LKS1WDKGED647msBQfLBHV3Ls8sveGncnm

[2013-12-25 11:32:32] 1 miner threads started, using 'scrypt' algorithm.
[2013-12-25 11:32:32] Starting Stratum on stratum+tcp://doge.poolerino.com:3333
[2013-12-25 11:32:32] Stratum detected new block
[2013-12-25 11:32:57] GPU #0: GRID K520 with compute capability 3.0
[2013-12-25 11:32:57] GPU #0: the 'K' kernel requires single memory allocation
[2013-12-25 11:32:57] GPU #0: interactive: 0, tex-cache: 0 , single-alloc: 1
[2013-12-25 11:32:57] GPU #0: Performing auto-tuning (Patience...)
[2013-12-25 11:32:57] GPU #0: maximum warps: 459
[2013-12-25 11:34:47] GPU #0:  244.68 khash/s with configuration K18x21
[2013-12-25 11:34:47] GPU #0: using launch configuration K18x21
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 0.09 khash/s
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 116.83 khash/s[2013-12-25 11:34:54] GPU #0: GRID K520, 1439424 hashes, 231.48 khash/s
[2013-12-25 11:34:54] accepted: 1/1 (100.00%), 231.48 khash/s (yay!!!)
[2013-12-25 11:34:56] GPU #0: GRID K520, 508032 hashes, 228.09 khash/s
[2013-12-25 11:34:56] accepted: 2/2 (100.00%), 228.09 khash/s (yay!!!)
[2013-12-25 11:34:59] GPU #0: GRID K520, 556416 hashes, 228.73 khash/s

Etc etc etc. So every line or two it is accepting a ... whatever it is. I really don't understand the mining process that well, but I'm learning.

CPU Hash

This is a lot easier because we can just go grab a precompiled 64-bit binary and run it.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cd
[ec2-user@ip-10-51-131-176 ~]$ wget http://downloads.sourceforge.net/project/cpuminer/pooler-cpuminer-2.3.2-linux-x86_64.tar.gz?r=&ts=1387953566&use_mirror=softlayer-ams
[ec2-user@ip-10-51-131-176 ~]$ tar xvf pooler-cpuminer-2.3.2-linux-x86_64.tar.gz\?r\=
[ec2-user@ip-10-51-131-176 ~]$ ./minerd --url stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Ta-Da!

And that, step-by-step is how you can run a pretty decent DogeMining box in the cloud. So Cloud. So Doge.

Does this make any sense to do?

I don't know. Like this, totally 'naked' it is hashing at about 300 KH/s, which is generating me on the order of a few hundred coins/hr at this pool right now.

Math! 200 coins at $0.65/hr. That would mean that I would not return unless the price of a coin was about $0.003 (point three cents), and I think that the current (5:41AM on Christmas Morning) price is a factor of three below that. So it doesn't really seem to make a TON of sense to do economically, but hey, it's fun! I'm mining in the cloud! And maybe DogeCoin and BitCoin will go up in price 1000000000000000000%!

So I think that by setting this up, I'm now losing about $0.40/hr. Which is... Less-than-effective, but hey! With the scalabiltily, and not needing to worry about the hardware, I can always make it up in volume!

Better automating the process

A more efficient set of commands could definitely be built, and I'd toyed with the idea of "burning" this AMI so that I can just click and have it up and running, but I haven't decided the best way to (or if I'm going to) automate this more.

Lots of tools I'm really interested in in the configuration management space would make this helpful, but the learning curve on those is pretty steep and it probably won't help a ton.

In any case, if you want to send me Doge, that's much appreciated, too!

DAJa5tjgjrnJ3iMJb8khpDuE8tW8KNBWqG

10 December 2013

Paperclip and The Legend of Zelda

Figuring out where a Paperclip attachment is a bit like playing the Legend of Zelda sometimes.

These are things you need to figure out, from three different sources:

1 - The filename of the actual file (face.png) - source your database

1a - The name of the model and model id that you have attached this to. This is different information, but generally in the DB

2 - The pathing strategy that Paperclip uses - this is typically in a global paperclip config file (maybe config/initializers/paperclip.rb)

3 - Where the eff in the cloud it is - this is the biggest pain in the butt there. In a sloppily-configured prod setup, you'll be lucky enough to have this in a file like s3.yml or maybe paperclip.yml. But with good configuration-management, that file will only live on the server it's supposed to because, you know, security.

So once you have all three of these pieces of information from different sources, and have rescued the princess, what can you build? A string. The information to construct a string that's like 100 characters long is scattered across the corners of your application's world and that's just nuts.

WORSE! If someone ever decides to touch corner 2 of the triforce there and change anything about how paths are calculated, your application just forgets where to find its attachments. All your links are broken, and you're off trying to figure out how to set up "compatibility modes" for some of the files. Because apparently that string changed? Wut? The location of this file is mutable all of a sudden? (facepalm)

This came up for me as I'm building a Persistence as a Service module for an application and I started putting in Carrierwave (since it's a touch saner than Paperclip) and then realized, "Self, wtf are you doing?" Shouldn't this just be holding a string? And maybe (maybe?!) aren't you using a modern database that will let you have a field hold an array of strings? And can't that array of strings hold all of the immutable data?

Yep. So that's what I've got now. If someone wants the Foo object to hold some attachments, they send something like this:

"foo": {
  "name": "The ultimate Foo",
  "attachments": [
    {
      "type": "image",
      "image_content_type": "image/png",
      "small_url": "some_absolute_path_on_s3/picture_small.png",
      "medium_url": "some_absolute_path_on_s3/picture_medium.png",
      "large_url": "some_absolute_path_on_s3/picture_large.png",
      "original_url": "some_absolute_path_on_s3/picture_small.png"
    }
  ]
}

So there we go. That's at that end, but why can't that level of treatment of the location of these files as immutable be pushed down into the actual image processing library? What if I decide to start saving new pictures in a different bucket? Can't my database just point two different things at two different buckets? What if I want to push some of the files immediately out to a CDN and only reference that location of theirs? Piece of cake! It's just strings! URI's are just strings. The only thing your application is ever going to care about in most cases is "what is the URI of this file." So keep those concerns separate.


15 October 2013

On writing well

"On Writing Well" somehow appeared at the top of my Amazon wish list the other day. I must have put it there in some kind of 'moment of clarity' and then forgotten it, but for whatever reason I decided to pull the trigger.

I've really enjoyed reading it and have found fantastic correlations between his suggestions for writing good nonfiction, and writing good code (which is, hopefully, also nonfiction).

Refactoring

I don't like writing; I like having written. But I love to rewrite. I especially love to cut: to press the delete key and see an unnecessary word or phrase or sentence vanish into the electricity. On Writing Well, p. 87

I share Professor Zinsser's views on writing and having written prose, but disagree on code. I love writing code. I love the kind of frantic desperation and looming 'done' that I have when a failing test is reminding me of its presence at every pause.

But then, when I'm finally green, I find, the art comes in. I'll not try to compete with the imagery of seeing things 'vanish into the electricity' so will merely repeat it. So gratifying.

The best code is the code you don't have to maintain.

Form

Zinsser discusses paragraphs and their visual appeal (at a high level) on a page and I've found that though a wall of text compared to a cluster of paragrahs IS indeed more approachable, the disparity between well- and poorly-factored (even just from a high level) code is even more dramatic.

When I open the file with a 200 line controller method, I have to struggle so hard to not go to the water cooler.

Code of corresponding quality but broken into methods LOOKS so much more approachable. Even if its' not particualrly well-factored code, it's less intimidating - maybe enough so that some brave soul will come in and refactor it.

Verbs

Use strong, and evocative verbs in your prose. Send strong and clear messages in your code. Is this name intention-revealing enough? Ask that constantly.

I was at lunch with a colleague the other day and he mentioned that he had written a class where the main method was named, annhilate. He said he got such a kick out of calling that method. It's a wonderful name - it is perfectly descriptive of what it is going to do, and evokes an appropriate sense of danger.

Just like passive verbs or "nouned" verbs make bad prose, ambiguous, vapid, and weak method names make bad code. Send good messages. Name your methods well.

Nouns

On the same token, name your objects well. Zinsser mentinons "creeping nounism" - he mentions having seen a "communication facilitation skills development intervention". Five 'concept nouns' where one - workshop perhaps? - would do splendidly.

I've made my share of poorly named objects, and I know we all see (and probably work with) them every day. Name your objects well.

The Quickest Fix

The quickest fix when struggling with a word or a phrase in nonfiction prose is to delete it. I remember when I finally came to this understanding with Rails when I was learning - maybe struggling with the Rails 2 router to route to a non-RESTful action? - that maybe if it's hard, I'm doing the wrong thing. I wasn't doing it wrong, I was doing the wrong thing. As I learn more and develop more, it seems more consistent, not less, that if it's hard, I need to step back and examine why. If I can't come up with a compelling why, then it's time for the DELETE key and watch the mess 'vanish into the electricity'.

It's been amazing to see the 'carryover' of concepts from nonfiction prose to code, and I can't recommend this book enough to someone who wants to improve, but is burned out on reading SO MANY programming books. And hell, maybe my nonfiction prose will improve, too.