If you have done any DevOps work on Ama­zon Web Ser­vices (AWS) then you should be famil­iar with Ama­zon Cloud­Watch, a ser­vice for track­ing and view­ing met­rics (CPU, net­work in/out, etc.) about the var­i­ous AWS ser­vices that you con­sume, or bet­ter still, cus­tom met­rics that you pub­lish about your service.

On top of that, you can also set up alarms on any met­rics and send out alerts via Ama­zon SNS, which is a pretty stan­dard prac­tice of mon­i­tor­ing your AWS-hosted appli­ca­tion. There are of course many other paid ser­vices such as Stack­Driver and New Relic which offer you a host of value-added fea­tures, per­son­ally I was impressed with some of the pred­ica­tive fea­tures from Stack­Driver.

The built-in Ama­zon man­age­ment con­sole for Cloud­Watch pro­vides the rudi­men­tary func­tion­al­i­ties that lets you browse your met­rics and view/overlap them on a graph, but it falls short once you have a decent num­ber of metrics.

For starters, when try­ing to browse your met­rics by name­space, you’re capped at 200 met­rics so dis­cov­ery is out of the ques­tion, you have to know what you’re look­ing for to be able to find it, which isn’t all that use­ful when you have hun­dreds of met­rics to work with…

image

Also, there’s no way for you to fil­ter met­rics by the recorded dat­a­points, so to answer even sim­ple ques­tions such as

‘what other times­pan met­rics also spiked at mid-day when our ser­vice dis­cov­ery latency spiked?’

you now have to man­u­ally go through all the rel­e­vant met­rics (and of course you have to find them first!) and then visu­ally check the graph to try and find any correlations.

 

After being frus­trated by this man­ual process for one last time I decided to write some tool­ing myself to make my life (and hope­fully oth­ers) a bit eas­ier, and in comes Amazon.CloudWatch.Selector, a set of DSLs and CLI for query­ing against Ama­zon Cloud­Watch.

 

Amazon.CloudWatch.Selector

With this sim­ple library you will get:

  • an inter­nal DSL which is intended to be used from F# but still usable from C# although syn­tac­ti­cally not as intuitive
  • an exter­nal DSL which can be embed­ded into a com­mand line or web tool

 

Both DSLs sup­port the same set of fil­ters, e.g.

Name­spaceIs Fil­ters met­rics by the spec­i­fied namespace.
Name­space­Like Fil­ters met­rics using a regex pat­tern against their namespaces.
NameIs Fil­ters met­rics by the spec­i­fied name.
Name­Like Fil­ters met­rics using a regex pat­tern against their names.
Uni­tIs Fil­ters met­rics against the unit they’re recorded in, e.g. Count, Bytes, etc.
Aver­age Fil­ters met­rics by the recorded aver­age data points, e.g. aver­age > 300 looks for met­rics whose aver­age in the spec­i­fied time­frame exceeded 300 at any time.
Min Same as above but for the min­i­mum data points.
Max Same as above but for the max­i­mum data points.
Sum Same as above but for the sum data points.
Sam­ple­Count Same as above but for the sam­ple count data points.
Dimen­sion­Con­tains Fil­ters met­rics by the dimen­sions they’re recorded with, please refer to the Cloud­Watch docs on how this works.
Dur­in­gLast Spec­i­fies the time­frame of the query to be the last X minutes/hours/days. Note: Cloud­Watch only keeps up to 14 days worth of data so there’s no point going any fur­ther back then that.
Since Spec­i­fies the time­frame of the query to be since the spec­i­fied time­stamp till now.
Between Spec­i­fies the time­frame of the query to be between the spec­i­fied start and end timestamp.
Inter­valOf Spec­i­fies the ‘period’ in which the data points will be aggre­gated into, i.e. 5 min­utes, 15 min­utes, 1 hour, etc.

Here’s some code snip­pet on how to use the DSLs:

 

In addi­tion to the DSLs, you’ll also find a sim­ple CLI tool as part of the project which you can start by set­ting the cre­den­tials in the start_cli.cmd script and run­ning it up. It allows you to query Cloud­Watch met­rics using the exter­nal DSL.

Here’s a quick demo of using the CLI to select some CPU met­rics for Ela­si­Cache and then plot­ting them on a graph.

 

As a side note, one of the rea­sons why we have so many met­rics is because we have made it super easy for our­selves to record new met­rics (see this recorded webi­nar for more infor­ma­tion) to gives our­selves a very gran­u­lar set of met­rics so that any CPU-intensive or IO work is mon­i­tored as well as any top-level entry points to our services.

 

Links

Share

Icon for package Filbert

Some time ago I put together a small BERT seri­al­izer and BERT-RPC client for .Net called Fil­bert (which is another name for Hazel­nut, that has the word bert and the let­ter F and at the time every F# library has a lead­ing F in its name!).

As an exper­i­men­tal project admit­tedly I hadn’t given too much thought to per­for­mance, and as you can see from below, the num­bers don’t make for a flat­ter­ing read­ing even against the BCL’s binary formatter!

I finally found some time to take a stab at improv­ing the dread­ful dese­ri­al­iza­tion speed and with a cou­ple of small changes I was able to halve the dese­ri­al­iza­tion time for a very sim­ple bench­mark for a sim­ple object, and there are still a cou­ple of low hang­ing fruits that can improve things further.

image

The new ver­sion is up on Nuget so feel free to play around with it, and do let me know if you have any feed­backs and raise any issues or bugs on the Issues page for the project.

Share

Just under three weeks ago, Ama­zon announced the pub­lic avail­abil­ity of their new Kine­sis ser­vice, a ser­vice which is designed to allow real-time pro­cess­ing of stream­ing big data.

As an exper­i­ment I have put together a sim­ple, actor-based cus­tomer appen­der for log4net which allows you to pub­lish your log mes­sages into a con­fig­ured Kine­sis stream. You can then have another clus­ter of machines to fetch the data from the stream and do what­ever pro­cess­ing or aggre­ga­tion you like to do.

You can down­load and install the appen­der from Nuget here or check­out the source code here.

The imple­men­ta­tion is done in F# in 100 lines of code, and as you can see is very sim­ple, easy to rea­son with, fully asyn­chro­nous and thread-safe.

 

Once you have pushed your log mes­sages into the stream, you’ll need to use the AWSSDK to fetch the data and process them. For Java, there’s a client appli­ca­tion which takes care of most of the heavy lift­ing – e.g. track­ing your progress, han­dling failovers and load bal­anc­ing. Unfor­tu­nately, at the time of writ­ing, there’s no equiv­a­lent of such client appli­ca­tion in the cur­rent ver­sion of the .Net AWSSDK.

So to help make it eas­ier for us .Net folks to build real-time data pro­cess­ing appli­ca­tions on top of Ama­zon Kine­sis, I had started a Rx-based .Net client library called Reac­toKi­ne­siX (I really wanted to get RX into the name!), more details to follow.

 

I think the intro­duc­tion of Kine­sis is very excit­ing and opens up many pos­si­bil­i­ties, and at the cur­rent pric­ing model it also rep­re­sents a very cost effec­tive alter­na­tive to some of the other com­pet­ing and more pol­ished ser­vices out there.

Share

I work with Ama­zon S3, and indeed many of the Amazon’s cloud ser­vices, and one of the things that is often frus­trat­ing in my day-to-day work­flow is the need to keep jump­ing out of the IDE, find the files I’m look­ing for in a S3 explorer, grab the key to the file and then go back to the IDE and write code against the AWS SDK to read the data from S3 and do some­thing with that data.

This inef­fi­ciency in my devel­op­ment process is mag­ni­fied when work­ing against buck­ets with large num­ber of keys and/or buck­ets which has ver­sion­ing turned on.

With these in mind, I wanted to cre­ate a type provider which is:

  • able to inter­ac­tively nav­i­gate buck­ets, fold­ers and files in S3
  • able to retrieve data eas­ily in either binary of sim­ple text format
  • able to search for files easily

plus all the usual good­ness that comes with F# type providers, such as:

  • not hav­ing to leave the IDE and let the provider do all the hard work for me
  • have sta­tic type safety with regards to the file struc­ture in S3 (if some­one deletes/renames a file or folder I will be informed via a com­pile time error rather than a run­time exception!)

 

After a few nights of work I finally have some­thing that’s of usable shape, here is a quick demo video I’ve pre­pared of the type provider in action:

 

The source code for the type provider is open source and can be found here, you can also grab it from Nuget here.

Go on, give it a play and let me know if you have any feedbacks!

 

Enjoy!

Share

Just a quick note to say that slides for my F# in Social Gam­ing talk at the recent CodeMesh con­fer­ence is now avail­able on SlideShare and that record­ing of the talk will be avail­able on InfoQ at a later date.

Share