Project angelhair: Building a queue on cassandra

A few weeks ago my work had a hack day and I got together with some of my coworker friends and we decided to build a queue on top of Cassandra.

For the impatient, give it a try (docker hub):

docker run -it \
    -e CLUSTER_NAME="" \
    -e KEYSPACE="" \
    -e CONTACT_POINTS="" \
    -e USERNAME="" \
    -e PASSWORD="" \
    -e USE_SSL="" \
    -e DATA_CENTER="" \
    -e METRICS_GRAPHITE "true" \
    -e GRAPHITE_URL=""  \

The core features for what we called Project Angelhair was to handle:

- long term events (so many events that AMQ or RMQ might run out of storage space)
- connectionless – wanted to use http
- invisibility – need messages to disappear when they are processing but be able to come back
- highly scaleable – wanted to distribute a docker container that just did all the work

Building a queue … Read more

Dynamic HAProxy configs with puppet

I’ve posted a little about puppet and our teams ops in the past since my team has pretty heavily invested in the dev portion of the ops role. Our initial foray into ops included us building a pretty basic puppet role based system which we use to coordinate docker deployments of our java services.

We use HAProxy as our software load balancer and the v1 of our infrastructure managment had us versioning a hardcoded haproxy.cfg for each environment and pushing out that config when we want to add or remove machines from the load balancer. It works, but it has a few issues

  1. Cluster swings involve checking into github. This pollutes our version history with a bunch of unnecessary toggling
  2. Difficult to automate swings since its flat file config driven and requires the config to be pushed out from puppet

Our team did a little brainstorming and came up with … Read more

Adventures in pretty printing JSON in haskell

Today I gave atom haskell-ide a whirl and wanted to play with haskell a bit more. I’ve played with haskell in the past and always been put off by the tooling. To be fair, I still kind of am. I love the idea of the language but the tooling is just not there to make it an enjoyable exploratory experience. I spend half my time in the repl inspecting types, the other half on hoogle, and the 3rd half (yes I know) being frustrated that I can’t just type in package names and explore API’s in sublime or atom or wherever I am. Now that I’m on a mac, maybe I’ll give leksah another try. I tried it a while ago it didn’t work well.

Anyways, I digress. Playing with haskell and I thought I’d try poking with Aeson, the JSON library. Like Scala, you have to define your objects … Read more

Automating deployments with salt, puppet, jenkins and docker

I know, its a buzzword mouthful. My team has had good first success leveraging jenkins, salt, sensu, puppet, and docker to package and monitor distributed java services with a one click deployment story so I wanted to share how we’ve set things up.

First and foremost, I’ve never been an ops guy. I’ve spent time on build systems like msbuild and fake, but never a full pipeline solution that also had to manage infrastructure, but there’s a first for everything. Most companies I’ve worked at have had all this stuff set up already, and out of the minds of developers, but the place I am at now does not. I actually think it’s been a great opportunity to dive into the full flow of how do you get your damn code out the door. I think if you are developing an application and don’t know how it gets from git … Read more

Testing puppet with docker and python

In all the past positions I’ve been in I’ve been lucky enough to have a dedicated ops team to handle service deployment, cluster health, and machine managmenent. However, at my new company there is much more of a “self serve” mentality such that each team needs to handle things themselves. On the one hand this is a huge pain in my ass, since really the last thing I want to do is deal with clusters and machines. On the other hand though, because we have the ability to spin up openstack boxes in our data centers at the click of a button, each team has the flexibility to host their own infrastructrure and stack.

For the most part my team and I are deploying our java services using dockerized containers. Our container is a centos7 base image with a logstash forwarder in it and some other minor tooling, and we … Read more

Converting akka scala futures to java futures

Back in akka land! I’m using the ask pattern to get results back from actors since I have a requirement to block and get a result (I can’t wait for an actor to push at a later date). Thats fine, but converting from scala futures to java completable futures is a pain. I also, (like mentioned in another post) want to make sure that my async responses capture and set the MDC for proper logging.

My final usage should look something like:

private <Response, Request> Future<Response> askActorForResponseAsync(Request source) {
    final FiniteDuration askTimeout = new FiniteDuration(config.getAskForResultTimeout().toMillis(), TimeUnit.MILLISECONDS);

    final Timeout timeout = new Timeout(askTimeout);

    final scala.concurrent.Future<Object> ask = Patterns.ask(master.getActor(), new PersistableMessageContext(source), timeout);

    return FutureConverter.fromScalaFuture(ask)
                          .thenApply(i -> (Response) i);

The idea is that I’m going to translate a scala future with a callback into a completable future java promise.

Next up, the future converter:

public class FutureConverter {
    public static <T> … Read more
Shareable zsh environment: EnvZ

Introducing EnvZ.

What is Envz?

During the course of normal production development we all tend to write a bunch of shell scripts and other useful command line utilities that help us out. Usually the end up being a bunch of one offs or stored in one mega .zshrc file. However, there’s something to be said about having a small framework to share environment utilities and to use as a jump off to “version” a shared set of utilities with team mates.

With that in mind I’ve been building out a small zsh bootstrapper that builds on tools that I like and use (like yadr) and gives a way to add pluggable team modules into it. All a pluggable module is is a sym link to another directory that auto loads .sh files on shell start. And while it sounds like a small thing, it’s actually really nice to be … Read more

Adding MDC logging to akka

I’ve mentioned before, but I’m working heavily in a project that is leveraging akka. I am really enjoying the message passing model and so far things are great, but tying in an MDC for the SLFJ logging context proved complicated. I had played with the custom executor model described here but hadn’t attempted the akka custom dispatcher.

I was thinking that a custom dispatcher would work great to pass along the MDC since then you’d never have to think about it, but unfortunately I couldn’t get it to work. Akka kept failing to instantiate the dispatcher. I was also worried about configuration data and possible tuning that you might lose giving akka your own dispatcher configurator.

So, given that I wasn’t quite sure what to do. What I ended up with however was a little extra work but turned out well. I went with an augmented dispatcher/subscriber model. Basically for … Read more

Getting battery percentage in zsh

I’m on osx maverick still at home on my laptop and I spent part of today dicking around customizing my zsh shell. I wanted to be able to show my battery percentage in the shell and it’s really pretty easy.

First, the main shell function

function get_battery()
    current_battery=`system_profiler SPPowerDataType | grep -i "charge remaining" | awk '{print $4}'`

    max_battery=`system_profiler SPPowerDataType | grep -i "full charge capacity" | awk '{print $5}'`

    percent=`bc <<< "scale=4; ${current_battery}/${max_battery} * 100"`

    printf '%.0f' $percent 

This queries from the profiler the charge current and remaining, uses bc to get a floating point division, and then just shows the integer value of that (we could round it too but I was lazy).

Now, we just need to tie it into the prompt. I’m using the steef prompt by default and just tweaked it a bit:



datetime="%F{yellow}[%D{%a %I:%M:%S %p}]%f" # yellow [Day hh:mm:ss am/pm] … Read more
Handling subclassed constraints with a DSL in java 8

I really like doing all of my domain modeling with clean DSL’s (domain specific languages). Basically I want my code to read like a sentence, and to hide all the magic behind things. When things read clearly even a non professional can determine if something is wrong. The ideal scenario is to have your code read like pseudocode since nobody really cares what the internals are, what matters is your general solution.

I found myself recently in a scenario where I have some methods that do work, and they all return a subtype of a response root. Something like:

class Response extends ResponseRoot {}

class Worker{
   Response doWork(Request request) { // }

And I need to create a dynamic callback given the context of the request. So basically I want

Consumer<ResponseRoot> callback = generateCallback(request);

Response response = doWork(request);


However I am going to have a lot of this … Read more