Cassandra Datastax C# Driver problems - solution

Please find the updated version of this post here: https://piotr.westfalewicz.com/blog/2016/09/cassandra-datastax-c-sharp-driver-problems---solution/

In the previous post I've described a strange problem related to Cassandra Datastax C# Driver which was happening once in the production environment. It's time to reveal the mystery.

Two root causes

1.

One of the hidden, but important metric, which you won't find usually in your logs is the CPU usage. What important is that the connection setup to the Cassandra cluster consists of many small steps. In production, when there was a very high CPU usage (around 100% - for reason known and already eliminated), the connection setup process was timing out in such a moment, that the final result was reported as NoHostAvailableException. This shows, how important is to track and prevent 100% CPU usage.

2.

But why, let me quote myself:

Things get back to normal after the client restart... and gets back to madness few hours later, at higher load. Incredible high number of NoHostAvailableExceptions, like almost any connection to the Cassandra fails.

The problem is here:

private readonly Cluster _cluster;
private readonly ConcurrentDictionary<string, Lazy<ISession>> _sessions; //lockless session cache

public ISession GetSession(string keyspaceName)
{
    if (!_sessions.ContainsKey(keyspaceName))
    {
        _sessions.GetOrAdd(keyspaceName, key => new Lazy<ISession>(() => _cluster.Connect(key)));
    }
    var result = _sessions[keyspaceName];
    return result.Value;
}

Do you see it? It turns out that when an exception is thrown in _cluster.Connect(key) method, the Lazy<T> will cache this exception. Therefore all invocations to Lazy<T>.Value will result in the same, cached exception instead of retrying the connection to the Cassandra cluster. If you are planning to use the Lazy<T> class, there are more "gotchas". Read the documentation on MSDN.

Lessons learned?

1. CPU usage is very, very important and critical metric. Do not ignore it, as it may lead to numerous, strange errors.
2. RTFM! Read the documentation when using any class for the first time. Especially when copying&pasting code from the StackOverflow.

The performance of setting T[] vs. List by index

Source: wikimedia.org

Let's compare asymptotic time complexity of two following loops:

int[] _array = Enumerable.Range(0, n).ToArray();
List<int> _list = Enumerable.Range(0, n).ToList();

//ListChangeByIndex
for (var i = 0; i < n; i++)
{
    _list[i] = i + 1;
}

//ArrayChangeByIndex
for (var i = 0; i < n; i++)
{
    _array[i] = i + 1;
}

How do you think, which one is faster?

.

.

.

.

.

.

.

.

Many developers think the fist one will be slower, because in each loop computer is forced to visit all nodes from 0 to i to finally set the variable.
However, that's not the case. Both loops have O(n) complexity. That's because in the .NET source we can clearly see that's underlying data structure for a List<T> is an array: list.cs. Therefore, those two loops are essentially equal.

Messing with C# types. Making type1.FullName==type2.FullName, but not type1==type2!

Please find the updated version of this post here: https://piotr.westfalewicz.com/blog/2016/07/the-performance-of-setting-t-vs.-list-by-index/

Given the following method:

private static void CompareTypes(Type type1, Type type2)
{
    Console.WriteLine($"type1.FullName = {type1.FullName}");
    Console.WriteLine($"type2.FullName = {type2.FullName}");
    Console.WriteLine($"type1.FullName {(type1.FullName == type2.FullName ? '=' : '!')}= type2.Fullname");
    Console.WriteLine($"type1.AssemblyQualifiedName = {type1.AssemblyQualifiedName}");
    Console.WriteLine($"type2.AssemblyQualifiedName = {type2.AssemblyQualifiedName}");
    Console.WriteLine($"type1.AssemblyQualifiedName {(type1.AssemblyQualifiedName == type2.AssemblyQualifiedName ? '=' : '!')}= type2.AssemblyQualifiedName");
    Console.WriteLine($"type1.GUID = {type1.GUID}");
    Console.WriteLine($"type2.GUID = {type2.GUID}");
    Console.WriteLine($"type1.GUID {(type1.GUID == type2.GUID ? '=' : '!')}= type2.GUID");

    Console.WriteLine("o1 = Activator.CreateInstance(type1)");
    Console.WriteLine("o2 = Activator.CreateInstance(type2)");
    var o1 = Activator.CreateInstance(type1);
    var o2 = Activator.CreateInstance(type2);
    Console.WriteLine($"o1 == {o1}");
    Console.WriteLine($"o2 == {o2}");

    Console.WriteLine();
    Console.WriteLine($"but... type1 {(type1 == type2 ? '=' : '!')}= type2");
}

Is it possible to get the following result?

type1.FullName = MyLibrary.MyPrecious
type2.FullName = MyLibrary.MyPrecious
type1.FullName == type2.Fullname
type1.AssemblyQualifiedName = MyLibrary.MyPrecious, MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
type2.AssemblyQualifiedName = MyLibrary.MyPrecious, MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
type1.AssemblyQualifiedName == type2.AssemblyQualifiedName
type1.GUID = cacf8c0d-b903-3da6-808f-024a3070ab9d
type2.GUID = cacf8c0d-b903-3da6-808f-024a3070ab9d
type1.GUID == type2.GUID
o1 = Activator.CreateInstance(type1)
o2 = Activator.CreateInstance(type2)
o1 == MyLibrary.MyPrecious
o2 == MyLibrary.MyPrecious

but... type1 != type2

As it turns out, it is. Doing such a hell is relatively easy:

private static Assembly LoadAssemblyByName(string name)
{
    var myPreciousAssemblyLocation = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), name);
    using (var fs = new FileStream(myPreciousAssemblyLocation, FileMode.Open, FileAccess.Read))
    {
        var data = new byte[fs.Length];
        fs.Read(data, 0, data.Length);
        fs.Close();
        var assembly = Assembly.Load(data);
        return assembly;
    }
}

static void Main()
{
    var type1 = typeof (MyPrecious);
    var myLibraryAssembly = LoadAssemblyByName("MyLibrary.dll");
    var type2 = myLibraryAssembly.GetType("MyLibrary.MyPrecious", true);

    CompareTypes(type1, type2);
}

The code above compares type1 from referenced project to type2 from the same assembly, but loaded again through Assembly.Load(byte[]). That makes the library loaded twice in the AppDomain. Now when a call to AppDomain.CurrentDomain.GetAssemblies() is made, the assemblies are:

AppDomain.CurrentDomain.GetAssemblies:
mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Microsoft.VisualStudio.HostingProcess.Utilities, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Microsoft.VisualStudio.HostingProcess.Utilities.Sync, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Microsoft.VisualStudio.Debugger.Runtime, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
vshost32, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
ConsoleApplication1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Accessibility, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3

Even in such a small, console application it is quite confusing. So, let's make it more confusing... What's the output of the following code?

var myPreciousAssemblyLocation = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "MyLibrary.dll");
var myLibraryAssemblyLoadFrom = Assembly.LoadFrom(myPreciousAssemblyLocation);
var type3 = myLibraryAssemblyLoadFrom.GetType("MyLibrary.MyPrecious", true);
CompareTypes(type1, type3);

Now, surprisingly, its:

type1.FullName = MyLibrary.MyPrecious
type2.FullName = MyLibrary.MyPrecious
type1.FullName == type2.Fullname
type1.AssemblyQualifiedName = MyLibrary.MyPrecious, MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
type2.AssemblyQualifiedName = MyLibrary.MyPrecious, MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
type1.AssemblyQualifiedName == type2.AssemblyQualifiedName
type1.GUID = cacf8c0d-b903-3da6-808f-024a3070ab9d
type2.GUID = cacf8c0d-b903-3da6-808f-024a3070ab9d
type1.GUID == type2.GUID
o1 = Activator.CreateInstance(type1)
o2 = Activator.CreateInstance(type2)
o1 == MyLibrary.MyPrecious
o2 == MyLibrary.MyPrecious

but... type1 == type2

Hint

A nice hint is shown, when you try to execute the following code:

var o1 = Activator.CreateInstance(type1);
var o2 = Activator.CreateInstance(type2);
MyPrecious p1 = (MyPrecious) o1;
try
{
    MyPrecious p2 = (MyPrecious)o2;
}
catch (Exception e)
{
    Console.WriteLine(e);
}

System.InvalidCastException: [A]MyLibrary.MyPrecious cannot be cast to [B]MyLibrary.MyPrecious. Type A originates from 'MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'LoadNeither' in a byte array. Type B originates from 'MyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location 'c:\users\pwdev\documents\visual studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\bin\Debug\MyLibrary.dll'. at ConsoleApplication1.Program.Main() in c:\users\pwdev\documents\visual studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\Program.cs:line 49

Explanation

Yes, it's all about the load contexts. There are three different assembly load contexts: Load, LoadFrom, Neither. Usually there is no need to load the same library twice and get the strange behavior written above, but sometimes there might be. There are many advantages and disadvantages of using different Assembly.Load(From/File) methods. Take a look: Choosing a Binding Context. Furthermore, consider what's happening to assembly dependencies when you load an assembly. There are best practices described on MDSN for loading assemblies: Best Practices for Assembly Loading. I have to say, in my whole career I've been loading assemblies by hand twice, and from time perspective, both two cases were wrong.

TypeHandle

Instead of comparing the types in the examples above by == operator, there is a possibility to compare them by the TypeHandle:

TypeHandle encapsulates a pointer to an internal data structure that represents the type. This handle is unique during the process lifetime. The handle is valid only in the application domain in which it was obtained.

Source: MDSN. Well, I can't think of an interesting usage for the TypeHandles for now, but it's good to know.

Little-known, useful, charming and beautiful algorithms - part 2

Please find the updated version of this post here: https://piotr.westfalewicz.com/blog/2016/03/little-known-useful-charming-and-beautiful-algorithms---part-2/

Image source: https://goo.gl/EGDuX7

This is the second post about charming algorithms, the first post is here: Little-known, useful, charming and beautiful algorithms - part 1

.NET Timers

How many timers there are in the .NET framework? It's not 2 or 3. It's 5:

1. System.Timers.Timer
2. System.Threading.Timer
3. System.Windows.Forms.Timer
4. System.Web.UI.Timer
5. System.Windows.Threading.DispatcherTimer

Here is the comparison of 3 most important of them:
Comparing the Timer Classes in the .NET Framework Class Library.

Which timer would you use for implementing Speculative query execution (which, BTW. it is a nice idea too)? The answer should be: none of them.

.NET timers: internals&assumptions

As you can read in the .NET source code of the System.Threading.Timer, each AppDomain in .NET has one single native timer, supplied by the VM, to fire all managed timers in the AppDomain. The performance considerations are:

We assume that timers are created and destroyed frequently, but rarely actually fire. There are roughly two types of timer:

  - timeouts for operations.  These are created and destroyed very frequently, but almost never fire, because the whole point is that the timer only fires if something has gone wrong.

  - scheduled background tasks.  These typically do fire, but they usually have quite long durations. So the impact of spending a few extra cycles to fire these is negligible.

Because of this, we want to choose a data structure with very fast insert and delete times, but we can live with linear traversal times when firing timers.

The data structure we've chosen is an unordered doubly-linked list of active timers.  This gives O(1) insertion and removal, and O(N) traversal when finding expired timers.

Because of the nature of speculative query execution, we actually care about finding expired timers, and O(N) time is not acceptable. Imagine 10000 read requests per second per one single node to the Cassandra cluster in your web server (which isn't really a big deal, rather normal situation in high traffic scenarios). If you use speculative query execution, you get 10000 timers to manage. Stop, start and expiry processing operations aren't the big deal here, but the per-tick bookkeeping is. Fortunately, there is a better structure for keeping timers, than doubly-linked list.

Hashed and Hierarchical Timing Wheels

Adrian Colyer described all kinds of different timer structures in his blog post: Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility. Go ahead and read it, as he done really good job explaining academic paper.

Real world applications

I've found two real world applications for those structures:

• One is already mentioned Speculative Query Execution, in Cassandra
• Second is request purgatory in Kafka: Apache Kafka, Purgatory, and Hierarchical Timing Wheels

If you are looking for the C# implementation, the one and only implementation of this (which I've found) is here: HashedWheelTimer.cs (many thanks to Cassandra Datastax C# Driver team).