Performance | theburningmonk.com

JSON serializers benchmark updated – including MongoDB Driver

Just a quick note to say that I have updated the JSON serializers benchmark to use the latest Nuget versions of ServiceStack.Text, Json.Net and JsonFX.

I have also included the JSON and BSON serializers from the MongoDB C# Driver in the test, and since BSON is a binary format I have included protobuf-net as a reference since it’s the fastest binary serializer I know of on the .Net platform:

Compared to the last set of results, you can see that the latest version of JsonFX seems have gone much much slower on deserialization, whilst ServiceStack.Text is still the fastest JSON serializer around. The MongoDB C# Driver’s JSON and BSON both held up pretty well in the test too.

Performance Test – Json Serializers Part III

Note: Don’t forget to check out Benchmarks page to see the latest round up of binary and JSON serializers.

Following on from my previous test, I have now included JsonFx and as well as the Json.Net BSON serializer in the mix to see how they match up.

The results (in milliseconds) as well as the average payload size (for each of the 100K objects serialized) are as follows.

image[4]

Graphically this is how they look:

I have included protobuf-net in this test to provide more meaningful comparison for Json.Net BSON serializer since it generates a binary payload and as such has a different use case to the other JSON serializers.

In general, I consider JSON to be appropriate when the serialized data needs to be human readable, a binary payload on the other hand, is more appropriate for communication between applications/services.

Observations

You can see from the results above that the Json.Net BSON serializer actually generates a bigger payload than its JSON counterpart. This is because the simple POCO being serialized contains an array of 10 integers in the range of 1 to 100. When the integer ‘1’ is serialized as JSON, it’ll take 1 byte to represent as one character, but an integer will always take 4 bytes to represent as binary!

In comparison, the protocol buffer format uses varint encoding so that smaller numbers take a smaller number of bytes to represent, and it is not self-describing (the property names are not part of the payload) so it’s able to generate a much much smaller payload compared to JSON and BSON.

Lastly, whilst the Json.Net BSON serializer offers a slightly faster deserialization time compared to the Json.Net JSON serializer, it does however, have a much slower serialization speed.

Disclaimers

Benchmarks do not tell the whole story, and the numbers will naturally vary depending on a number of factors such as the type of data being tested on. In the real world, you will also need to take into account how you’re likely to interact with the data, e.g. if you know you’ll be deserializing data a lot more often than serializing them then deserialization speed will of course become less important than serialization speed!

In the case of BSON and integers, whilst it’s less efficient (than JSON) when serializing small numbers, it’s more efficient when the numbers are bigger than 4 digits.

Takeaways from Gael Fraiteur’s multithreading talk

After watching Gael’s recent SkillsMatter talk on multithreading I’ve put together some notes from a very educational talk:

Hardware Cache Hierarchy

Four levels of cache

L1 (per core) – typically used for instructions
L2 (per core)
L3 (per die)
DRAM (all processors)

Data can be cached in multiple caches, and synchronization happens through an asynchronous message bus.

The latency increases as you go down the different levels of cache:

Memory Reordering

Cache operations are in general optimized for performance as opposed to logical behaviour, hence depending on the architecture (x86, AMD, ARM7, etc.) cache loads and store operations can be reordered and executed out-of-order:

To add to this memory reordering behaviour at a hardware level, the CLR can also:

cache data into register
reorder
coalesce writes

The volatile keyword stops the compiler optimizations, that’s all, it does not stop the hardware level optimizations.

This is where memory barrier comes in, to ensure serial access to memory and to force data to be flushed and synchronized across all the local cache, this is done via the Thread.MemoryBarrier method in .Net.

Atomicity

Operations on longs cannot be performed in an atomic way on a 32-bit architecture, it’s possible to get partially modified value.

Interlocked

Interlocks provides the only locking mechanism at hardware level, the .Net framework provides access to these instructions via the Interlocked class.

On the Intel architecture, interlocks are typically implemented on the L3 cache, a fact that’s reflected by the latency associated with using Interlocked increments compared with non-interlocked:

CompareExchange is the most important tool when it comes to implemented lock-free algorithms, but since it’s implemented on the L3 cache, in a multi-processor environment it would require one of the processor to take out a global lock, hence why the contented case above takes much longer.

You can analyse the performance of your application at a CPU level using Intel’s vTune Amplifier XE tool.

Multitasking

Threads do not exist at a hardware level, CPU only understands tasks and it has no concept of ‘wait’. Synchronization constructs such as semaphores and mutex are built on top of interlocked operations.

One core can never do more than 1 ‘thing’ at the same time, unless it’s hyper-threaded in which case the core can do something else whilst waiting on some resource to continue executing the original task.

A task runs until interrupted by hardware (I/O interrupt) or OS.

Windows Kernel

A process has:

private virtual address space
resources
at least 1 thread

A thread is:

a program (sequence of instructions)
CPU state
wait dependencies

Threads can wait for dispatcher objects (WaitHandle) – Mutex, Semaphore, Event, Timer or another thread, when they’re not waiting for anything they’re placed in the waiting queue by the thread scheduler until it is their turn to be executed on the CPU.

After a thread has been executed for some time, it is then moved back to the waiting queue (via a kernel interrupt) to give some other thread a slice of the available CPU time. Alternatively, if the thread needs to wait for a dispatcher object then it goes back to the waiting state.

Dispatcher objects reside in the kernel and can be shared among different processes, they’re very expensive!

Which is why you don’t want to use kernel objects for waits that are typically very short, instead they’re best used when waiting for something that takes longer to return, e.g. I/O.

Compared to other wait methods (e.g. Thread.Sleep, Thread.Yield, WaitHandle.Wait, etc.) Thread.SpinWait is an odd ball because it’s not a kernel method, it resembles a continuous loop (it keeps ‘spinning’) but it tells a hyper-threaded CPU that it’s ok to do something else. It’s generally useful when you know the interrupt will happen very quickly and hence saving you from an unnecessary context switch. If the interrupt does not happen quickly as expected, the SpinWait will be transformed into a normal thread wait (Thread.Sleep) to avoid wasting CPU cycles.

.Net Framework Thread Synchronization

The lock Keyword

start with interlocked operations (no contention)
continue with ‘spin wait’
create kernel event and wait

Good performance if low contention.

Design Patterns

Thread unsafe
Actor
Reader-Writer Synchronized

This is where the PostSharp multithreading toolkit comes to the rescue! It can help you implement each of these patterns automatically, Gael has talked more about the toolkit in this blog post.

F# – Speed test iter and map operations with array vs list

I have heard a few people argue that when it comes to performance critical code you should prefer arrays over other collections (such as F#’s lists) as it benefits from sequential reads (which is faster than seeks) and offers better memory locality.

To test that theory somewhat, I wanted to see if there is any difference in how fast you can iterate through an array versus a list in F#, and how much faster you can map over an array compared to a list:

The result is a little surprising, whilst I wasn’t expecting there to be a massive difference in the iterating through the two types of collections, I didn’t think mapping over a list would be quite as slow in comparison. I knew that constructing a list is much heavier than constructing an array, but I didn’t think it’d take 22x as long in this case.

What was even more surprising was how much slower the Seq.iter and Seq.map functions are compared to the Array and List module equivalents! This is, according to John Palmer:

Once you call in to Seq you lose the type information — moving to the next element in the list requires a call to IEnumerator.MoveNext. Compare to for Array you just increment an index and for List you can just dereference a pointer. Essentially, you are getting an extra function call for each element in the list.

The conversions back to List and Array also slow the code down for similar reasons

Update 2012/06/04:

As a work around, you COULD shadow the Seq module with iter and map functions that adds simple type checking and in the case of an array or list simply call the corresponding function in the Array or List module instead:

Whilst this approach will work to a certain extend, you should be careful with which functions you shadow. For instance, it’s not safe to shadow Seq.map because it can be used in conjunction with other functions such as Seq.takeWhile or Seq.take. In the base implementation, a line of code such as:

arr |> Seq.map incr |> Seq.take 3

will not map over every element in the source array.

With the shadowed version (see above) of Seq.map, however, this would first create a new array by applying the mapper function against every element in the source array before discarding all but the first three elements in the new array. This, as you can imagine, is far less efficient and requires much more memory space (for the new array) and defeats the purpose of using Seq module functions in most cases.

Performance Test – String.Contains vs String.IndexOf vs Regex.IsMatch

To find out if a string contains a piece of substring, here are three simple ways of going about it in C#, just to name a few:

String.Contains
String.IndexOf where the return value is >= 0
Regex.IsMatch

Out of curiosity I wanted to see if there was any noticeable difference in the performance of each of these options.

Given a simple string “Mary had a little lamb”, let’s find out how long it takes to test whether or not this string contains the terms ‘little’ (the match case) and ‘big’ (the no match case) using each of these approaches, repeated over 100k times:

As you can see, Regex.IsMatch is by far the slowest option in this test, although using RegexOptions.Compiled yielded slightly faster execution time. What was also interesting is that String.Contains turned out to be significantly faster than String.IndexOf.

If you take a look at the implementation for String.Contains in a reflector you will see:

So that explains the difference between the execution times for String.Contains and String.IndexOf, and indeed if I change the String.IndexOf test to use StringComparison.Ordinal (default is StringComparison.CurrentCulture) then I get an identical result to String.Contains.

With all that said, String.Contains and String.IndexOf is only useful for checking the existence of an exact substring, but Regex is much more powerful and allows you to do so much more. However, you do end up paying for them even when you don’t need those additional capabilities!

theburningmonk.com

a blog for Yan and Yinan

JSON serializers benchmark updated – including MongoDB Driver

Performance Test – Json Serializers Part III

Takeaways from Gael Fraiteur’s multithreading talk

F# – Speed test iter and map operations with array vs list

Performance Test – String.Contains vs String.IndexOf vs Regex.IsMatch

Recent Posts

Categories

Archives