Note: don’t for­get to check out the Bench­marks page to see the lat­est round up of binary and JSON serializers.

A lit­tle while ago I put together a quick per­for­mance test com­par­ing the BCL’s Bina­ry­For­mat­ter with that of Marc Grav­ell’s protobuf-net library (.Net imple­men­ta­tion of Google’s pro­to­col buffer for­mat). You can read more about my results here.

There’s another fast binary seri­al­iza­tion library which I had heard about before, Mes­sagePack, which claims to be 4 times faster than pro­to­col buffers which I find a lit­tle hard to believe, so nat­u­rally, I have to see for myself!

Tests

The con­di­tions of the test is very sim­i­lar to those out­lined in my first post, except Mes­sagePack is now included in the test.

I defined two types to be serialized/deserialized, both con­tain the same amount of data, one exposes them as pub­lic prop­er­ties whilst the other exposes them as pub­lic fields:

image image

The rea­son for this arrange­ment is to try and get the best per­for­mance out of Mes­sagePack (see Note 2 below).

100,000 iden­ti­cal instances of the above types are seri­al­ized and dese­ri­al­ized over 5 runs with the best and worst results excluded, results of the remain­ing 3 runs are then aver­aged to give the final results.

Results

image

image

image

A cou­ple of obser­va­tions from these results:

1. Bina­ry­For­mat­ter per­forms bet­ter with fields – faster seri­al­iza­tion and smaller payload!

2. Protobuf-net per­forms equally well with fields and prop­er­ties – because it’s dri­ven by the [DataMem­ber] attributes.

3. Mes­sagePack per­forms sig­nif­i­cantly bet­ter with fields – over 10% faster seri­al­iza­tion and deserialization!

4. Mes­sagePack is NOT 4x faster than Pro­to­col Buffer! – at least not with the two imple­men­ta­tions (Msg­Pack and Profotbuf-net) I have tested here, speed-wise there’s not much sep­a­rat­ing the two and depend­ing on the data being seri­al­ized you might find a slightly dif­fer­ent result.

5. Mes­sagePack gen­er­ates big­ger pay­load than Pro­to­col Buffer – again, depend­ing on the data being seri­al­ized you might arrive at a slightly dif­fer­ent result.

Source code

If you’re inter­ested in run­ning the tests your­self, you can find the source code here. It uses my Sim­ple­SpeedTester frame­work to orches­trate test runs but you should be able to get the gist of it fairly easily.

Clos­ing thoughts…

Although Mes­sagePack couldn’t back up its claim to be 4x faster than Pro­to­col Buffer in my test here, it’s still great to see that it offers com­pa­ra­ble speed and pay­load, which makes it a viable alter­na­tive to Pro­to­col Buffer and in my opin­ion hav­ing a num­ber of dif­fer­ent options is always a plus!

In addi­tion to seri­al­iza­tion Mes­sagePack also offers a RPC frame­work, and is also widely avail­able with client imple­men­ta­tions avail­able in quite a num­ber of dif­fer­ent lan­guage and plat­forms, e.g. C#, JavaScript, Node.Js, Erlang, Python and Ruby, so it’s def­i­nitely worth con­sid­er­ing when you start a new project.

On a dif­fer­ent note, although you’re able to get a good 10% improve­ment in per­for­mance when you use fields instead of prop­er­ties to expose pub­li­cally avail­able data in your type, it’s impor­tant to remem­ber that the idiomatic way in .Net is to use pub­lic properties.

As much as I think per­for­mance is of para­mount impor­tance, main­tain­abil­ity of your code is a close sec­ond, espe­cially in an envi­ron­ment where many devel­op­ers will be work­ing on the same piece of code. By fol­low­ing the ‘rec­og­nized’ way of doing things you can more eas­ily com­mu­ni­cate your inten­tions to other devel­op­ers work­ing on the same code.

Imag­ine you’ve inten­tion­ally used pub­lic fields in your class because you know it will be seri­al­ized by the Mes­sagePack seri­al­izer but then a new devel­oper joins the team and, not know­ing all the intri­cate details of your code, decides to con­vert them all to pub­lic prop­er­ties instead..

So if you’re will­ing to step out of the idiomatic .Net way to get that per­for­mance boost, be sure to com­ment your code well in all the places where you’re diverg­ing from the norm!

NOTE 1:

The Sim­pleOb­ject class in this post is slightly dif­fer­ent from the one I used in my pre­vi­ous post, the ‘Scores’ prop­erty is now an int array instead of List<int>, doing so dras­ti­cally improved per­for­mance for ALL the seri­al­iz­ers involved.

This is because the imple­men­ta­tion of List<T> uses an array inter­nally and the array is resized when capac­ity is reached. So even though you only have x num­ber of items in a List<T> the inter­nal array will always have more than x num­ber of spaces. Which is why Sim­pleOb­ject in my first post (where ‘Scores’ is of type List<int>) is seri­al­ized into 708 bytes by the Bina­ry­For­mat­ter but the Sim­pleOb­ject class defined in this post can be seri­al­ized into 376 bytes.

Sim­i­larly, ini­tial­iz­ing an instance of Dictionary<TKey, TValue> with the intended capac­ity allows you to add items to it much more effi­ciently, for more details, see here.

Note 2:

One caveat I came across in the Mes­sagePack test was that, if you instan­ti­ate Com­piled­Packer with­out any para­me­ter, the default behav­iour is to seri­al­ize pub­lic fields only and that meant none of the pub­lic prop­er­ties I’ve defined on the Sim­pleOb­ject type will get seri­al­ized. As a result, to serialize/deserialize instances of Sim­pleOb­ject I needed to use an instance of Com­piled­Packer that seri­al­izes pri­vate fields too, which accord­ing to the MessagePack’s Get­tingStarted guide, is slower.

So in order to get the best per­for­mance (speed) out of Mes­sagePack, I defined a sec­ond type (see Sim­pleOb­jectWith­Fields type above) to see how well it does with a type that defines pub­lic fields instead of prop­er­ties. In the inter­est of fair­ness, I repeated the same test for Bina­ry­For­mat­ter and protobuf-net too, just to see if you get bet­ter per­for­mance out of them too.

Share

One Response to “Performance Test – Binary serializers Part II

  1. […] my last round of bench­marks on binary seri­al­iz­ers, there’s a new player in town – MessageShark, […]

Leave a Reply