Introducing log4net.Kinesis, a log4net appender for Amazon Kinesis

Just under three weeks ago, Ama­zon announced the pub­lic avail­abil­i­ty of their new Kine­sis ser­vice, a ser­vice which is designed to allow real-time pro­cess­ing of stream­ing big data.

As an exper­i­ment I have put togeth­er a sim­ple, actor-based cus­tomer appen­der for log4net which allows you to pub­lish your log mes­sages into a con­fig­ured Kine­sis stream. You can then have anoth­er clus­ter of machines to fetch the data from the stream and do what­ev­er pro­cess­ing or aggre­ga­tion you like to do.

You can down­load and install the appen­der from Nuget here or check­out the source code here.

The imple­men­ta­tion is done in F# in 100 lines of code, and as you can see is very sim­ple, easy to rea­son with, ful­ly asyn­chro­nous and thread-safe.

 

Once you have pushed your log mes­sages into the stream, you’ll need to use the AWSSDK to fetch the data and process them. For Java, there’s a client appli­ca­tion which takes care of most of the heavy lift­ing – e.g. track­ing your progress, han­dling failovers and load bal­anc­ing. Unfor­tu­nate­ly, at the time of writ­ing, there’s no equiv­a­lent of such client appli­ca­tion in the cur­rent ver­sion of the .Net AWSSDK.

So to help make it eas­i­er for us .Net folks to build real-time data pro­cess­ing appli­ca­tions on top of Ama­zon Kine­sis, I had start­ed a Rx-based .Net client library called Reac­toKi­ne­siX (I real­ly want­ed to get RX into the name!), more details to fol­low.

 

I think the intro­duc­tion of Kine­sis is very excit­ing and opens up many pos­si­bil­i­ties, and at the cur­rent pric­ing mod­el it also rep­re­sents a very cost effec­tive alter­na­tive to some of the oth­er com­pet­ing and more pol­ished ser­vices out there.