SuperPumpup (dot com)
General awesomeness may be found here.

12 August 2014

ReactJS and Vert.x

I'm building a new project in Vert.x and it's amazing to be able to separate concerns well. However, I "miss" having a Rails project to fall back on. Here I'll talk a bit about Vert.x and exploring using the ReactJS framework to build the UI.

Continue reading


12 August 2014

Mode Analytics Is Helping Me Become A Better Developer

Like most other "software writers" who primarily write Rails code I've met with, I don't have any formal CS training - or any formal training at all, really. I just started learning how to build things, which was hard and horrible in PHP and spectacularly easy in Rails. Love it or hate it, it's SO EASY to build systems in Rails when you have no idea what you're doing. I'll always argue that, and think it's amazing.

That said, I never learned SQL well...

Continue reading


22 July 2014

Docker & RackConnect

To enable Docker containers to connect to the external network in a RackConnected environment, touch /etc/rackconnect-allow-custom-iptable. Otherwise RackConnect will destroy all Docker's forwarding rules.

Longer Version

So I just got bitten by a little issue using Docker & RackConnect. I had a nice fresh server brought up with lots of containers running and everything was grand. Ish.

I found two different problems, both with the same solution. On the one hand, I would start a service container (RabbitMQ), and then link to it with a "web" container, and then go to my RackConnect config and open a port to test connecting to the "web" container (that would have a port number like 49156 or whatever), but then all of a sudden that container wouldn't be able to access the service container.

Also, from within any container, I couldn't access the network.

curl google.com or ping 8.8.8.8 were just timing out.

I couldn't find a direct answer to this question, but it turns out that docker writes a ton of iptables FOWARD entries during the course of doing its thing, and those periodically get clobbered by RackConnect when it runs.

This article tells how to let RackConnect know to leave your custom rules alone - it does a merge rather than a force-overwrite. So touch that file, restart Docker, and be on your way!


24 June 2014

Log RabbitMQ to STDOUT (Because Docker)

Docker and Log aggregation. Not a solved problem. But I think the strategy I'm going to adopt for my current venture is having all of my containers log everything possible to STDOUT, and then just my log collection agents ingest the docker logs of each container. That way I can keep my containers simple (a single process without cheats like supervisord), and the actual operations also remains simple.

Continue reading


25 February 2014

Agh! People are starting to use my App! Now what?

This is a slightly beefed up version of the lightning talk I gave at Austin on Rails last night - Feb 25, 2014.

I had intended to give this talk on things I've learn about infrastructure as I've been able to help stabilize and grow at a my last two jobs. I've had a lot of experience working on deployment infrastructure, from Dreamhost, to Rackspace, to Heroku, AWS, and now am mananging our infrastructure on Amazon OpsWorks using Chef (which I've been writing about recently).

However, as I thought about it, the problems that have been hindering our growth have not been as much server infrastructure related, as much as they were visibility-related...

Continue reading


13 February 2014

Deploying a Multi-Rails-App OpsWorks Stack

I've written before about how OpsWorks kind of pushes you into a weird architecture because of their default behaviors for applications in a stack.

Namely, the system tries to push all the applications on all of the layers, and chaos ensues.

But, it turns out I needed to finally move our various apps into one stack, and after a few days of poking, prodding, waaaaaaaaaaiting for machines, I got it working, so I thought I'd document it.

Continue reading


06 February 2014

Setting Up OpsWorks Using Your Very Own (Shiny) VPC

I'm in the middle of changing our application architecture at work from being a bunch of OpsWorks stacks into one large stack - so that I can tell my individual application nodes what's going on where. I ran into one pretty big hiccup while porting my stack from "toy" that I got by following this excellent walkthrough.

Basically the demo stack uses public git repos - no custom cookbooks or private app code. That clearly doesn't work in the real world, though. We have a large set of custom cookbooks that we're working on supplementing so I need to pull those down. To do so, not only did I need to change the stack settings to point to the repo using the git@github... address (instead of git://github...).

OpsWorks VPC Private Cookbook Repo

But I also had to go to the NAT security group that I had set up using the CloudFormation and add port 22 to one that was allowing inbound and outbound connections. The defaults allow port 9418 (the git protocol), but that protocol does not allow you to use our deploy key for authentication:

OpsWorks VPC NAT Setup

So if you are having problems setting up your OpsWorks using custom cookbooks inside a VPC, then make sure you have port 22 forwarded on your NAT.

So much to learn...


25 December 2013

Such Doge. So Cloud.

Or, how to set up an EC2 box to mine 300 KH/s

So I thought that this DogeCoin was starting to sound more and more interesting, and, having access to some spare CPU cycles in the cloud and not wanting to burn up my Retina MBP GPU, I thought it would be fun to see if I could mine at a decent rate in on Amazon EC2. I started out just doing some CPU mining, but that was not terribly effective (only getting me a few Doge/hr), and since it's fun to play with other people's hardware, I thought this would be fun to explore.

Turns out, I kind of can - it's about 8 minutes to get up and running - read on for detailed walkthrough, or skip to the end for analysis (or just leave if you'd rather do that)...

Instance type

Poking around, it seems that this Amazon Linux AMI with NVIDIA GRID GPU Driver is a good candidate. Amazon Nvidia Banner

So register/sign in with Amazon however you need to and click the Continue button, then just leave the rest of the "Launch with 1-Click" settings alone and you should be set. Default settings work pretty goodAre you sure?Box coming online

Connect however you're going to connect. Walking you through connecting to an EC2 instance is beyond the scope of this, but you should be able to figure it out.

You should update packages...

sudo yum update
(select yes when prompted)
sudo yum install automake

Download the CudaMiner source code we'll be installing:

[ec2-user@ip-10-51-131-176 ~]$ wget https://github.com/cbuchner1/CudaMiner/archive/master.zip
--2013-12-25 11:13:45--  https://github.com/cbuchner1/CudaMiner/archive/master.zip
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/cbuchner1/CudaMiner/zip/master [following]
--2013-12-25 11:13:45--  https://codeload.github.com/cbuchner1/CudaMiner/zip/master
Resolving codeload.github.com (codeload.github.com)... 192.30.252.146
Connecting to codeload.github.com (codeload.github.com)|192.30.252.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 310982 (304K) [application/zip]
Saving to: ‘master.zip’

100%[========================================================================================>] 310,982     --.-K/s   in 0.008s

2013-12-25 11:13:45 (37.1 MB/s) - ‘master.zip’ saved [310982/310982]

[ec2-user@ip-10-51-131-176 ~]$ ls
master.zip
[ec2-user@ip-10-51-131-176 ~]$ unzip master.zip
Archive:  master.zip
e47cf1994a3021050bf64521acf6978d3a5c4f27
   creating: CudaMiner-master/

 ... lots of output ...

[ec2-user@ip-10-51-131-176 ~]$ cd CudaMiner-master/
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Now we've gotten and unzipped the code, let's start trying to build it.

Running naked ./configure fails with a configure: error: Missing required libcurl >= 7.15.2 which is annoying, so let's try autogen instead.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
configure.ac:116: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.

So let's see what else we can glean from what they've given us:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/usr/local/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ which cuda
/usr/bin/which: no cuda in (/opt/nvidia/cuda/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin)
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Well that's unfortunate - it looks like the ./configure.sh script is pointing to the wrong location (for us) for cuda.

Let's fix that:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ echo ./configure "CFLAGS=-O3" "CXXFLAGS=-O3" --with-cuda=/opt/nvidia/cuda > configure.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cat configure.sh
./configure CFLAGS=-O3 CXXFLAGS=-O3 --with-cuda=/opt/nvidia/cuda
[ec2-user@ip-10-51-131-176 CudaMiner-master]$

Trying ./configure.sh now gets further and it errors at:

checking for pthread_create in -lpthread... yes
./configure: line 7109: syntax error near unexpected token `,'
./configure: line 7109: `LIBCURL_CHECK_CONFIG(, 7.15.2, ,'`

So I'll try ensuring I have all my libcurls:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install libcurl-devel.x86_64

And try again. Nope, same error... But googling around suggests that this may have to do with the line endings - there's a difference between Windows and *nix systems with how they end lines, and this is kind of a common cause of errors, amazingly...

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ sudo yum install dos2unix
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ find . -type f -exec dos2unix {} \;
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./autogen.sh
[ec2-user@ip-10-51-131-176 CudaMiner-master]$ ./configure.sh

Success! The output of ./configure is now:

configure: creating ./config.status
config.status: creating Makefile
config.status: creating compat/Makefile
config.status: creating compat/jansson/Makefile
config.status: creating cpuminer-config.h
config.status: executing depfiles commands

So then now that that's configured I can actually build the thing:

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ make

This dumps TONS of output to the console (including gobs of warnings), and takes a few minutes.

Built!

It looks like the make was successful on the first try, and I have the following output at the end:

ptxas info    : Used 32 registers, 336 bytes cmem[0]
g++  -O3 -pthread -L/opt/nvidia/cuda/lib64  -o cudaminer cudaminer-cpu-miner.o cudaminer-util.o cudaminer-sha2.o cudaminer-scrypt.o salsa_kernel.o spinlock_kernel.o legacy_kernel.o fermi_kernel.o kepler_kernel.o test_kernel.o titan_kernel.o -lcurl   compat/jansson/libjansson.a -lpthread  -lcudart -fopenmp
make[2]: Leaving directory `/home/ec2-user/CudaMiner-master'
make[1]: Leaving directory `/home/ec2-user/CudaMiner-master'

So now you can test it out like so:

./cudaminer -o stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Feel free to swap out your own pool and worker name/password, or just do a few hashes for me. =)

It will take the system a few minutes to get the GPU figured out and autotuned, but while you wait you can get set up to take advantage of the 8 CPU's that will be sitting idle once we get this GPU hashing.

For reference, here is my output:

*** CudaMiner for nVidia GPUs by Christian Buchner ***
                 This is version 2013-12-18 (beta)
    based on pooler-cpuminer 2.3.2 (c) 2010 Jeff Garzik, 2012 pooler
           Cuda additions Copyright 2013 Christian Buchner
       My donation address: LKS1WDKGED647msBQfLBHV3Ls8sveGncnm

[2013-12-25 11:32:32] 1 miner threads started, using 'scrypt' algorithm.
[2013-12-25 11:32:32] Starting Stratum on stratum+tcp://doge.poolerino.com:3333
[2013-12-25 11:32:32] Stratum detected new block
[2013-12-25 11:32:57] GPU #0: GRID K520 with compute capability 3.0
[2013-12-25 11:32:57] GPU #0: the 'K' kernel requires single memory allocation
[2013-12-25 11:32:57] GPU #0: interactive: 0, tex-cache: 0 , single-alloc: 1
[2013-12-25 11:32:57] GPU #0: Performing auto-tuning (Patience...)
[2013-12-25 11:32:57] GPU #0: maximum warps: 459
[2013-12-25 11:34:47] GPU #0:  244.68 khash/s with configuration K18x21
[2013-12-25 11:34:47] GPU #0: using launch configuration K18x21
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 0.09 khash/s
[2013-12-25 11:34:48] GPU #0: GRID K520, 12096 hashes, 116.83 khash/s[2013-12-25 11:34:54] GPU #0: GRID K520, 1439424 hashes, 231.48 khash/s
[2013-12-25 11:34:54] accepted: 1/1 (100.00%), 231.48 khash/s (yay!!!)
[2013-12-25 11:34:56] GPU #0: GRID K520, 508032 hashes, 228.09 khash/s
[2013-12-25 11:34:56] accepted: 2/2 (100.00%), 228.09 khash/s (yay!!!)
[2013-12-25 11:34:59] GPU #0: GRID K520, 556416 hashes, 228.73 khash/s

Etc etc etc. So every line or two it is accepting a ... whatever it is. I really don't understand the mining process that well, but I'm learning.

CPU Hash

This is a lot easier because we can just go grab a precompiled 64-bit binary and run it.

[ec2-user@ip-10-51-131-176 CudaMiner-master]$ cd
[ec2-user@ip-10-51-131-176 ~]$ wget http://downloads.sourceforge.net/project/cpuminer/pooler-cpuminer-2.3.2-linux-x86_64.tar.gz?r=&ts=1387953566&use_mirror=softlayer-ams
[ec2-user@ip-10-51-131-176 ~]$ tar xvf pooler-cpuminer-2.3.2-linux-x86_64.tar.gz\?r\=
[ec2-user@ip-10-51-131-176 ~]$ ./minerd --url stratum+tcp://doge.poolerino.com:3333 -u superjustin.superpumpupworker -p workwork &

Ta-Da!

And that, step-by-step is how you can run a pretty decent DogeMining box in the cloud. So Cloud. So Doge.

Does this make any sense to do?

I don't know. Like this, totally 'naked' it is hashing at about 300 KH/s, which is generating me on the order of a few hundred coins/hr at this pool right now.

Math! 200 coins at $0.65/hr. That would mean that I would not return unless the price of a coin was about $0.003 (point three cents), and I think that the current (5:41AM on Christmas Morning) price is a factor of three below that. So it doesn't really seem to make a TON of sense to do economically, but hey, it's fun! I'm mining in the cloud! And maybe DogeCoin and BitCoin will go up in price 1000000000000000000%!

So I think that by setting this up, I'm now losing about $0.40/hr. Which is... Less-than-effective, but hey! With the scalabiltily, and not needing to worry about the hardware, I can always make it up in volume!

Better automating the process

A more efficient set of commands could definitely be built, and I'd toyed with the idea of "burning" this AMI so that I can just click and have it up and running, but I haven't decided the best way to (or if I'm going to) automate this more.

Lots of tools I'm really interested in in the configuration management space would make this helpful, but the learning curve on those is pretty steep and it probably won't help a ton.

In any case, if you want to send me Doge, that's much appreciated, too!

DAJa5tjgjrnJ3iMJb8khpDuE8tW8KNBWqG