Uncategorized

Infra graphs with neo4j

I spent some time recently mucking around with neo4j attempting to model infrastructure, incidents, teams, users, etc. Basically what does it take to answer questions about organizations.

Getting neo4j set up with go was non trivial and thankfully someone had documented how to do it already (instructions in the readme: https://github.com/devshorts/graphql). In the sample API I exposed we can

  • Find related incidents. The pathway here is incidentA is failing because of infraA. incidentB is failing because of infraB. InfraB depends on some pathway that ends up infraA. This means that from InfraB -InfraA there is a relationship, and so that implies that IncidentA and IncidentB are related.
  • Find betweeness of the graph. This shows graph nodes that have heavy flow (high connections) and can be potential hot spots
  • Find communities in the graph. This shows clusterability of infrastructure/teams/etc.

The API exposed in the github is meant to model … Read more

Tracking batch queue fanouts

Edit: This code now exists at https://github.com/paradoxical-io/carlyle

When working in any resilient distributed system invariably queues come into play. You fire events to be handled into a queue, and you can horizontally scale workers out to churn through events.

One thing though that is difficult to do is to answer a question of when is a batch of events finished? This is a common scenario when you have a single event causing a fan-out of other events. Imagine you have an event called ProcessCatalog and a catalog may have N catalog items. The first handler for ProcessCatalog may pull some data and fire N events for catalog items. You may want to know when all catalog items are processed by downstream listeners for the particular event.

It may seem easy though right? Just wait for the last item to come through. Ah, but distributed queues are loosely ordered. If an … Read more

CassieQ at the Seattle Cassandra Users Meetup

Last night Jake and I presented CassieQ (the distributed message queue on cassandra) at the seattle cassandra users meetup at the Expedia building in Bellevue. Thanks for everyone who came out and chatted with us, we certainly learned a lot and had some great conversations regarding potential optimizations to include in CassieQ.

A couple good points that came up where how to minimize the use of compare and set with the monoton provider, whether we can move to time UUID’s for “auto” incrementing monotons. Another interesting tidbit was the discussion of using potential time based compaction strategies that are being discussed that could give a big boost given the workflow cassieq has.

But my favorite was the suggestion that we create “kafka” mode and move the logic of storing pointer offsets out of cassieq and onto the client, in which case we could get enormous gains since we no longer … Read more

, ,

AngularJS for .Net developers

A few months ago I was asked to be a technical reviewer on a new packt pub book called AngularJS for .Net developers. It mostly revolves around ServiceStack (not web API) and building a full stack application with angular. I actually really enjoyed reading it and thought it touched on a lot of great points that a developer who is serious needs to know about.

Unfortunately I think the book doesn’t do a very good job at explaining angular in general. It’s certainly geared to the experienced developer who has worked with angular and servicestack/c# REST before.

Still, if you are interested in using angular as a .net developer its an informative and quick read! … Read more

Leveraging message passing to do currying in ruby

I’m not much of a ruby guy, but I had the inkling to play with it this weekend. The first thing I do when I’m in a new language is try to map constructs that I’m familiar with, from basic stuff like object instantiation, singletons, inheritance, to more complicated paradigms like lambdas and currying.

I came across this blog post that shows that ruby has a way to auto curry lambdas, which is actually pretty awesome. However, I was a little confused by the syntax

a.send(fn, b)

I’m more used to ML style where you would do

fn a b 

So what is a.send doing?

Message passing

Ruby exposes its dynamic dispatch as a message passing mechanism (like objective c), so you can send “messages” to objects. It’s like being able to say “hey, execute this function (represented by a string) on this context”.

If you think of it that … Read more

,

Machine Learning with disaster video posted

A few weeks ago we had our second DC F# meetup with speaker Phil Trelford where he led a hands on session introducing decision trees. The goal of meetup was to see how good of a predictor we could make of who would live and die on the titanic. Kaggle has an excellent data set that shows age, sex, ticket price, cabin number, class, and a bunch of other useful features describing Titanic passengers.

Phil followed Mathias‘ format and had an excellent .fsx script that walked everyone through it. I think the best predictor that someone made was close to 84%, though it was surprisingly difficult to exceed that in the short period of time that we had to work on it. I’d implemented my own shannon entropy based ID3 decision tree in C# so this wasn’t my first foray into decision tree’s, but the compactness of the tree … Read more

, ,

Thinking about haskell functors in .net

I’ve been teaching myself haskell lately and came across an interesting language feature called functors. Functors are a way of describing a transformation when you have a boxed container. They have a generic signature of

('a -> 'b) -> f 'a -> f 'b

Where f isn’t a “function”, it’s a type that contains the type of 'a.

The idea is you can write custom map functions for types that act as generic containers. Generic containers are things like lists, an option type, or other things that hold something. By itself a list is nothing, it has to be a list OF something. Not to get sidetracked too much, but these kinds of boxes are called Monads.

Anyways, let’s do this in C# by assuming that we have a box type that holds something.


public class Box<T>
{
    public T Data { get; set; }   
}

var boxes = 
Read more

,

ParsecClone on nuget

Today I published the first version of ParsecClone to nuget. I blogged recently about creating my own parser combinator and it’s come along pretty well. While FParsec is more performant and better optimized, mine has other advantages (such as being able to work on arbitrary consumption streams such as binary or bit level) and work directly on strings with regex instead of character by character. Though I wouldn’t recommend using ParsecClone for production string parsing if you have big data sets, since the string parsing isn’t streamed. It works directly on a string. That’s still on the todo list, however the binary parsing does work on streams.

Things included:

  • All your favorite parsec style operators: < |>, ., ., |, etc. I won’t list them all since there are a lot.
  • String parsing. Match on full string terms, do regular expression parsing, inverted regular expressions, etc. I
Read more

, ,

Machine learning from disaster

If any of my readers are in the DC/MD/VA area you should all come to the next DC F# meetup that I’m organizing on september 16th (monday). The topic this time is machine learning from disaster, and we’ll get to find out who lives and dies on the Titanic! We’re bringing in guest speaker Phil Trelford so you know its going to be awesome! Phil is in the DC area on his way to the F# skills matters conference in NYC a few days later. I won’t be there but I expect that it will be top notch since all the big F# players are there (such as Don Syme and Tomas Petricek)!.

For more info check out our meetup page.… Read more

F# and Machine learning Meetup in DC

As you may have figured out, I like F# and I like functional languages. At some point I tweeted to the f# community lamenting that there was a dearth of F# meetups in the DC area. Lo and behold, tons of people replied saying they’d be interested in forming one, and some notable speakers piped up and said they’d come and speak if I set something up.

So, If any of my readers live in the DC metro area, I’m organizing an F# meetup featuring Mathias Brandewinder. We’ll be doing a hands on F# and machine learning coding dojo which should be a whole buttload of fun. Here’s the official blurb:

Machine Learning is the art of writing programs that get better at performing a task as they gain experience, without being explicitly programmed to do so. Feed your program more data, and it will get smarter at handling

Read more

, , ,

Previous Posts