Performance Test – Binary serializers Part II

Note: don’t for­get to check out the Bench­marks page to see the lat­est round up of bina­ry and JSON seri­al­iz­ers.

A lit­tle while ago I put togeth­er a quick per­for­mance test com­par­ing the BCL’s Bina­ry­For­mat­ter with that of Marc Grav­ell’s pro­to­buf-net library (.Net imple­men­ta­tion of Google’s pro­to­col buffer for­mat). You can read more about my results here.

There’s anoth­er fast bina­ry seri­al­iza­tion library which I had heard about before, Mes­sagePack, which claims to be 4 times faster than pro­to­col buffers which I find a lit­tle hard to believe, so nat­u­ral­ly, I have to see for myself!

Tests

The con­di­tions of the test is very sim­i­lar to those out­lined in my first post, except Mes­sagePack is now includ­ed in the test.

I defined two types to be serialized/deserialized, both con­tain the same amount of data, one expos­es them as pub­lic prop­er­ties whilst the oth­er expos­es them as pub­lic fields:

image image

The rea­son for this arrange­ment is to try and get the best per­for­mance out of Mes­sagePack (see Note 2 below).

100,000 iden­ti­cal instances of the above types are seri­al­ized and dese­ri­al­ized over 5 runs with the best and worst results exclud­ed, results of the remain­ing 3 runs are then aver­aged to give the final results.

Results

image

image

image

A cou­ple of obser­va­tions from these results:

1. Bina­ry­For­mat­ter per­forms bet­ter with fields – faster seri­al­iza­tion and small­er pay­load!

2. Pro­to­buf-net per­forms equal­ly well with fields and prop­er­ties – because it’s dri­ven by the [DataMem­ber] attrib­ut­es.

3. Mes­sagePack per­forms sig­nif­i­cant­ly bet­ter with fields – over 10% faster seri­al­iza­tion and dese­ri­al­iza­tion!

4. Mes­sagePack is NOT 4x faster than Pro­to­col Buffer! – at least not with the two imple­men­ta­tions (Msg­Pack and Pro­fot­buf-net) I have test­ed here, speed-wise there’s not much sep­a­rat­ing the two and depend­ing on the data being seri­al­ized you might find a slight­ly dif­fer­ent result.

5. Mes­sagePack gen­er­ates big­ger pay­load than Pro­to­col Buffer – again, depend­ing on the data being seri­al­ized you might arrive at a slight­ly dif­fer­ent result.

Source code

If you’re inter­est­ed in run­ning the tests your­self, you can find the source code here. It uses my Sim­ple­SpeedTester frame­work to orches­trate test runs but you should be able to get the gist of it fair­ly eas­i­ly.

Closing thoughts…

Although Mes­sagePack couldn’t back up its claim to be 4x faster than Pro­to­col Buffer in my test here, it’s still great to see that it offers com­pa­ra­ble speed and pay­load, which makes it a viable alter­na­tive to Pro­to­col Buffer and in my opin­ion hav­ing a num­ber of dif­fer­ent options is always a plus!

In addi­tion to seri­al­iza­tion Mes­sagePack also offers a RPC frame­work, and is also wide­ly avail­able with client imple­men­ta­tions avail­able in quite a num­ber of dif­fer­ent lan­guage and plat­forms, e.g. C#, JavaScript, Node.Js, Erlang, Python and Ruby, so it’s def­i­nite­ly worth con­sid­er­ing when you start a new project.

On a dif­fer­ent note, although you’re able to get a good 10% improve­ment in per­for­mance when you use fields instead of prop­er­ties to expose pub­li­cal­ly avail­able data in your type, it’s impor­tant to remem­ber that the idiomat­ic way in .Net is to use pub­lic prop­er­ties.

As much as I think per­for­mance is of para­mount impor­tance, main­tain­abil­i­ty of your code is a close sec­ond, espe­cial­ly in an envi­ron­ment where many devel­op­ers will be work­ing on the same piece of code. By fol­low­ing the ‘rec­og­nized’ way of doing things you can more eas­i­ly com­mu­ni­cate your inten­tions to oth­er devel­op­ers work­ing on the same code.

Imag­ine you’ve inten­tion­al­ly used pub­lic fields in your class because you know it will be seri­al­ized by the Mes­sagePack seri­al­iz­er but then a new devel­op­er joins the team and, not know­ing all the intri­cate details of your code, decides to con­vert them all to pub­lic prop­er­ties instead..

So if you’re will­ing to step out of the idiomat­ic .Net way to get that per­for­mance boost, be sure to com­ment your code well in all the places where you’re diverg­ing from the norm!

NOTE 1:

The Sim­pleOb­ject class in this post is slight­ly dif­fer­ent from the one I used in my pre­vi­ous post, the ‘Scores’ prop­er­ty is now an int array instead of List<int>, doing so dras­ti­cal­ly improved per­for­mance for ALL the seri­al­iz­ers involved.

This is because the imple­men­ta­tion of List<T> uses an array inter­nal­ly and the array is resized when capac­i­ty is reached. So even though you only have x num­ber of items in a List<T> the inter­nal array will always have more than x num­ber of spaces. Which is why Sim­pleOb­ject in my first post (where ‘Scores’ is of type List<int>) is seri­al­ized into 708 bytes by the Bina­ry­For­mat­ter but the Sim­pleOb­ject class defined in this post can be seri­al­ized into 376 bytes.

Sim­i­lar­ly, ini­tial­iz­ing an instance of Dictionary<TKey, TVal­ue> with the intend­ed capac­i­ty allows you to add items to it much more effi­cient­ly, for more details, see here.

Note 2:

One caveat I came across in the Mes­sagePack test was that, if you instan­ti­ate Com­piled­Pack­er with­out any para­me­ter, the default behav­iour is to seri­al­ize pub­lic fields only and that meant none of the pub­lic prop­er­ties I’ve defined on the Sim­pleOb­ject type will get seri­al­ized. As a result, to serialize/deserialize instances of Sim­pleOb­ject I need­ed to use an instance of Com­piled­Pack­er that seri­al­izes pri­vate fields too, which accord­ing to the MessagePack’s Get­tingStart­ed guide, is slow­er.

So in order to get the best per­for­mance (speed) out of Mes­sagePack, I defined a sec­ond type (see Sim­pleOb­jectWith­Fields type above) to see how well it does with a type that defines pub­lic fields instead of prop­er­ties. In the inter­est of fair­ness, I repeat­ed the same test for Bina­ry­For­mat­ter and pro­to­buf-net too, just to see if you get bet­ter per­for­mance out of them too.