Software. Efficiency. Scalability.

New languages

New languages are popping up almost every other day. If you consider languages being under major development (including core syntax and library changes) there is Go, Rust, Scala, F#, Gosu, just to name a few. In the wake of the immense popularity of dynamic languages such as Ruby, Groovy and Python, these new languages are statically typed.  It is quite obvious that there are some issues that current languages don’t address. These issues are pressing enough for people to undertake such an expensive exercise as creating a new language.

To understand things better I look at 5 major categories: memory management, libraries, readability, scalability, and deployment.

• Memory management has arguably brought about the biggest productivity gain in the last 2 decades. Major imperative languages, such as Java and C# gained Garbage Collector (GC) and made developers stop about allocating memory.
• Libraries are important for writing meaningful applications. Good libraries make a world of difference. For instance, WinForms is head and shoulders better than Swing. It has nothing to do with the language per say (even though C# support for events makes a huge difference for GUI libraries), but it certainly reflects on the language. One of the major problem for the new languages is lack of libraries.
• Readability of the code is paramount for a language. If developers have hard time reading code written in the language, they will be much less likely to use it. On top of that, languages that are hard to read make support and maintenance more expensive. If you have ever spent hours trying to figure out what a piece of code does, you understand what I mean.
• Scalability implies that software written in your language should be able to scale on modern hardware. For instance, a language with a single threaded model can’t take advantage of multi-core processors. A language without support for concurrency will make writing scalable apps much harder.
• Deployment is important because it’s when the software actually gets to work. There is always cost associated with deployment. Choosing Ruby will likely require more servers. Choosing C# or F# will require Windows licenses when you deploy. C# will enable you to deploy on Windows Azure , while Ruby will give you an option of Heroku.

Applying these 5 categories to the mainstream languages makes it easier to see why there is a need for new languages.

C

• Unmanaged
• Good basic libraries
• Poor readability
• Fast code, no built-in concurrency support
• Runs everywhere

Java

• Managed (GC)
• Great deal of libraries and frameworks
• Poor readability, verbose
• lack of proper generics, closures, lambda expressions, events, etc.
• VM + JIT, no built-in concurrency support
• Runs everywhere

C#

• Managed (GC)
• Has limited number of libraries and frameworks, Microsoft dominated
• Great readability, has support of most modern features (lambda, closures, generics, etc) as well as LINQ
• VM + JIT, no built-in concurrency support
• Only runs on Windows, license costs might be prohibitive for larger projects. Cloud deployment available with Azure.

Ruby

• Managed (GC)
• Has a good deal of libraries and frameworks
• Great readability, expressive and concise syntax
• VM, slow dynamic code,  no built-in concurrency support
• Cheap deployment but known to cause problems in larger deployments (e.g. Twitter)

Python

• Managed (GC)
• Has a great deal of libraries and frameworks
• Good readability
• VM, slow dynamic code, no built-in concurrency support
• Cheap linux deployment but can require more servers for large scale sites

Notice a common theme? Check category number 4 for all listed languages. No built-in concurrency support.

When all these languages were designed the requirements for the concurrent requests execution were much lower than what we have now. There has been a tremendous shift in volume of requests. 

Concurrency: CGI

Massive need for concurrency first happened when web applications became available. The first answer to running concurrent scripts was Common Gateway Interface (CGI). It made an HTTP requests look like it was executed by a user, mapped requests parameters to the environment variables, passed input to stdin and sent stdout back to the browser. First implementations would start a process for each request. For instance, if you had a python script it would execute Pyton interpreter for every request. The model relied on OS security and access control and was comfortable at the time as many tools operated with stdin/stdout. However, starting a process for each requests quickly becomes prohibitively expensive.

Concurrency: FastCGI

The answer was found in FastCGI, where a process wasn’t closed at the end of the requests but instead re-used to process next request. FastCGI allowed for a pull of processes to serve a lot of incoming requests, allowing for a better scalability. This model is still widely used for many deployments. However, you quickly find out that switching processes is an expensive task and that inter-process communication (IPC) is also expensive. Another problem is sharing of resources. For instance, each process needs its own database connection pool, instead of having a shared one.

Imagine having 100 front end servers each running 5 FastCGI processes. If no SQL queries are executed in parallel you would need 500 connections to the database. Depends on your database, it can be very expensive to have that many simultaneous connections open.

Concurrency: Thread Pool

The answer to the problem was found in using thread pools. Instead of having multiple processes we run one process per application but we create multiple threads withing the process. Threads are much lighter than processes so context switches are much cheaper, you can run many threads per process (normally anywhere between 50 and 250) , and you can have access to shared memory and resources such as DB connection pool. However, for large scale web applications even this model has its limitations. Your code is  expected to be synchronously executed on a single thread that is dedicated to processing your requests. The code blocks your thread while executing. If you have a to access a resource with a slow response time (for instance a web service such as google, facebook, twitter, etc.) your thread has to wait until the sub-requests is executed. If underlying resources are slow, your thread pool will quickly becomes starved: every available thread will be waiting for sub-requests, and no new requests will be processed. This is a very common problem.

Concurrency: Fibers

Many people are coming to realize that thread pools are quite ineffective for applications that have massive load. Many people also realize that you don’t need to be next google to experience these issues; you can have a popular iPhone app or mentioned on Reddit/Digg/Slashdot and find our that your services are bogged down.

The problem with scalability is that you need to be able to survive the burst. It can be extremely cost ineffective to scale hardware to that level. For instance, most of the time your web farm will be doing nothing, but you need to maintain it for that moment when you get mentioned on Twitter and the traffic comes your way.

I mentioned iPhone where a simple free application can become available to millions of users instantly and setting up a web farm with many servers in it is simply not an option. One solution would be to utilize the super powerful servers that you have to the max of their potential. You achieve this by following one simple rule: don’t waste resources on waiting if there is something else to work on.

If you want to achieve maximum throughput you have no choice but to be asynchronous. Instead of waiting for that Google API request to come back you can start processing new incoming requests and get back to serving the first requests once the data is available. It requires a primitive lesser than a thread. You might have heard names such as “fiber”, “tasklet”, “channel”, “coroutine”. Let’s call it “fiber”. Fibers are micro-threads that are executed concurrently. When your fiber encounters a long running operation it schedules it to be executed and yields to another fiber. Essentially, you execute cooperative multitasking within preemptive multitasking.

An ideal application will execute on top of a thread pool with N threads where N = number of CPU cores. You don’t need extra threads, as your operations never block. In reality you might want more threads to smooth things out.

Concurrency: Libraries

This is by no means a novel idea. There are plenty of solutions written to address this specific issue. There is Tornado, Node.js, any many others. I personally think the fact that JavaScript in the browser requires developers to only use non-blocking (another name for asynchronous) calls has advanced the cause greatly. The performance is there, so what’s the problem? Readability and control flow. Consider this code:

function updateStatus() {
try {
  var status = twitter.getStatus();
  displayStatus( status );
}
catch( error ) {
  displayError( error );
}

pretty straight forward, huh? Now, imagine writing  this code in a non-blocking way:

function updateStatus() {
  twitter.getStatusAsync( onUpdateStatusSuccess, onUpdateStatusError );
}

function onUpdateStatusSuccess( status ) {
  displayStatus( status );
}

function onUpdateStatusError( error ) {
  displayError( error );
}

Not nearly as hot. Now imagine that displayStatus and displayError are non-blocking calls either. You will quickly end up with a very fragmented code.

Languages that support first class functions or lambdas allow for writing success/error functions inline, however it generally leads to the continuation-passing  style which has same problem across the languages: non-blocking calls make natural language constructs such as structured exception handling, loops, condition statements, etc. ineffective.

This is a very important thing to understand. Languages designed as they are with a certain control flow in mind. They have control structures in place to assist the developer in accomplishing certain task. For instance, foreach loop can usually be replaced with for and switch/case/default can be replaced with if/elseif/else construct, but they are there for a reason. They serve to increase productivity through providing more concise and expressive code. Non-blocking libraries render you unable to use a lot of the language constructs and produce poorly structured code.

Concurrency: Language support

It’s quite clear that if you want to achieve the best possible scalability and maintain readability, you not only need a non-blocking library, but also a language that supports this specific style of development.

Let me outline this: to enable  wide-spread, practical use of non-blocking paradigm a programming language needs to provide built-in support for it.

This is the theme of all new languages. Whether it is co-routines in Go, actors in Scala, tasks in Rust, or new async/await keywords in C#, all these languages are trying to get a crack at the same problem: make a concurrent development easier and more available to developers.

All new languages (with notable exception of C# 5 approach)  implement a form of a message exchange. You can send message from one part of your program to another. Unfortunately, I don’t think this address the issue properly and, while a step forward, is far from the desired solution.

C# is taking a different approach that will reduce code fragmentation a lot better. However it will be a limited success due to the need to explicitly define functions as async and use  of Task<T> and thread pool.

I think there is a better solution to the concurrency issue,  one that could be applied to the existing languages by introducing libraries and compiler magic around those libraries. However, this is a topic for a different conversation.

I run Windows 7 on my workstation with SSD + RAID1 setup. The SSD contains Windows, Visual Studio, and some other tools that I use. The mirrored HDDs contain all data files and a backup copy of the SSD. This setup has been working fairly well for me because it gives an outstanding performance of the SSD drive (Windows 7 boots up in under 8 sec) and a stability of RAID1 which prevents any significant data loss.

The only disappointing issue so far has been Windows mirror fail once in a while. It’s hard to say why, but it seems that sometimes the mirror just breaks and needs to be restored. It takes a LONG time. By that I mean it’s been running for over 30 hours now and it’s 48% complete. No data has been lost, but it’s still rather annoying when it happens. Last time I blamed a power failure but this time there was none. The mirror just broke while my workstation was running. I am not sure what’s the deal with it as the HDDs are new and checkdisk doesn’t seem to find any issues.

As promised,here is  a little example of how to use SyncQueue with TaskGroup when you need to queue multiple requests and wait until all of them come back. This is a fairly common situation that is not addressed by the standard BackgroundWorker control.

Here is a piece of working code from one of the projects I’ve been working on. It searches and fetches all health care claims that belong to a specific hospital visit and displays the visit once all the claims are fetched:

private void FetchVisitData() {
  // create new SyncTaskGroup in the constructor
  Global.TaskQueue.Append( new SyncTask( this, new SyncTaskGroup() ) {
    Action = task => {
      // find all claims that belong to the visit
      var claims = Global.Claims.Search( Data.Claims.ClaimSearchAction.VisitId,
                      VisitId, 500, Data.Claims.ClaimSearchKind.Equals );
      ListData list = new ListData();
      foreach( Data.Claims.ClaimSearchResult info in claims.Items ) {
        // when we queue a task, pass the group from the parent task as the first argument
        Global.TaskQueue.Append( task.Group, new SyncTask( this ) {
          Context = info,
          Action = task2 => {
            // fetch claim data from the server
            MapData map = GetClaimData( (Data.Claims.ClaimSearchResult)task2.Context );
            // add claim to the list, lock as this happens concurrently
            lock( list ) { list.Add( map ); }
          },
          Error = Global.ReportTaskError,
          // completion of the child task notifies the group
          Complete = task.Group.TaskComplete
        } );
      }
      task.Result = list;
      // in a concurrent environment (especially multi-CPU) the first group task can
      // complete BEFORE the 2nd task is queued, therefore triggering a premature
      // completion of the whole group. To address this a task group needs to be manually
      // released when all child tasks are queued
      task.Group.Release();
    },
    Error = Global.ReportTaskError,
    Complete = task => {
      // we will only get here when all the claims have been fetched and the group task is completed
      Visit = new MapData( "Claims", (ListData)task.Result );
      RenderVisit();
    }
  });
}

As promised, here is a little library that allows for background queuing of long running synchronous operations.  Unlike the standard BackgroundWorker component, it’s not a visual component. Additionally,  it allows a little bit more control over what’s going on.  Also, I personally prefer lambdas over events, because  it keeps all the code relevant to the background process in the same place.

Here is the code:
http://code.google.com/p/syncqueue/source/browse/syncqueue.cs

Here is how you can use it:

// create a new queue with 4 worker threads
TaskQueue queue = new TaskQueue( 4 );

// 'this' must inherited from System.Windows.Forms.Control
SyncTask task = new SyncTask( this )  {

    Action = task => {
       // task action, execute your background task here
       // executed on a worker thread
       // use task.Result to return values
    },

    Complete = task => {
      // task complete without errors, called on the main thread
    },

    Error = task => {
      // an error occurred during execution, called on the main thread
      // exception is stored in task.Exception
      // if an exception if thrown in this method it will be ignored
    },

    Finally = task => {
      // if exists, always called at the end on the main thread
    }

};

// en-queue the task for execution
queue.Append( task );

The library also allows for spawning sub-tasks using SyncTaskGroup class. I will illustrate how it can be used in the next post.

After an extended period of silence I am back. A lof has happened but I am back in business of blogging.

To compensate for the missing posts, I will publish a little but handy C# library for proper handling of the background operations.  The built-in BackgroundWorker component is rather weak. I just need to put proper comments and figure out a free license for its distribution.

Stay tuned!

In the last post I mentioned some code samples from Interesting Finds by Jason Haley. Specifically, the Code to Look At section of the blog.

Here is yet another example of an interesting piece of code I found in that section: The .NET Asynchronous I/O Design Pattern.

The name implies that this is the way to get async done in .NET. After all, that’s what “pattern” means — a customary way to solve a particular problem.

Recently, the term lost its meaning since literally everything is a pattern now. Small solutions, big solutions, even custom solutions are patterns now. Sometimes, when I hear developers talk I can’t help but think of the scene from Being John Malkovich where John Malkovich gets inside his own head:

Let’s get back to the article. There are two big things that are wrong it.

1) The whole point of async is to avoid thread blocking. It’s in the first words of the Wikipedia page: “Asynchronous I/O, or non-blocking I/O…”

Non-blocking IO. Instead of blocking your thread, you queue an operation (or a set of operations) and allow thread to do more work. You don’t sit around and wait, you go and start processing other requests. When the async operation is completed you will be called via a callback. That’s the whole point. Fire and forget, until it’s done.

It is easy to understand if you are doing async right. A pure async application does not need to start more OS threads than physical CPU cores. While you can start more threads to increase responsiveness in certain scenarios (long calculations, for instance) it will not improve your RPS.

2) The code author uses to illustrate his “pattern” is a damn good example of how not to write code.

Take a look at this code from MultiHostLookup method (this method queues multiple async requests and waits for the completion):

lock (addressList)
      {
        // ensure all lookups have returned, otherwise wait
        while (addressList.Count != hosts.Count)
        {
          Monitor.Wait(addressList);
        }
      }

So we lock addressList and we wait under the lock. Well, guess what happens when an operation is completed? According to the “pattern” this is what happens in GetHostAddressesCallback method:

// we need to ensure updates to the address list are threadsafe
      lock (addressList)
      {
        addressList.Add(address);

        // notify listeners that another address has been added
        Monitor.PulseAll(addressList);
      }

The thread will try and acquire a lock on addressList. Which, if you remember, is locked by the thread executing endless while loop in MultiHostLookup.

One thread waits for another under a lock. The other thread uses the same lock to signal its completion back. A wonderful example of “how to deadlock your threads” pattern.

Here is a simple rule for writing multi-threaded apps: NEVER EVER WAIT UNDER A LOCK. That’s a sure way to get your application deadlocked.

It is extremely frustrating to see such async 101 mistakes. IO Completion Ports have been around since NT 3.5- more than 15 years! If you are interested in how to actually write high performance async applications, that’s a good place to start getting the concept.