Software. Efficiency. Scalability.

New languages

New languages are popping up almost every other day. If you consider languages being under major development (including core syntax and library changes) there is Go, Rust, Scala, F#, Gosu, just to name a few. In the wake of the immense popularity of dynamic languages such as Ruby, Groovy and Python, these new languages are statically typed.  It is quite obvious that there are some issues that current languages don’t address. These issues are pressing enough for people to undertake such an expensive exercise as creating a new language.

To understand things better I look at 5 major categories: memory management, libraries, readability, scalability, and deployment.

• Memory management has arguably brought about the biggest productivity gain in the last 2 decades. Major imperative languages, such as Java and C# gained Garbage Collector (GC) and made developers stop about allocating memory.
• Libraries are important for writing meaningful applications. Good libraries make a world of difference. For instance, WinForms is head and shoulders better than Swing. It has nothing to do with the language per say (even though C# support for events makes a huge difference for GUI libraries), but it certainly reflects on the language. One of the major problem for the new languages is lack of libraries.
• Readability of the code is paramount for a language. If developers have hard time reading code written in the language, they will be much less likely to use it. On top of that, languages that are hard to read make support and maintenance more expensive. If you have ever spent hours trying to figure out what a piece of code does, you understand what I mean.
• Scalability implies that software written in your language should be able to scale on modern hardware. For instance, a language with a single threaded model can’t take advantage of multi-core processors. A language without support for concurrency will make writing scalable apps much harder.
• Deployment is important because it’s when the software actually gets to work. There is always cost associated with deployment. Choosing Ruby will likely require more servers. Choosing C# or F# will require Windows licenses when you deploy. C# will enable you to deploy on Windows Azure , while Ruby will give you an option of Heroku.

Applying these 5 categories to the mainstream languages makes it easier to see why there is a need for new languages.

C

• Unmanaged
• Good basic libraries
• Poor readability
• Fast code, no built-in concurrency support
• Runs everywhere

Java

• Managed (GC)
• Great deal of libraries and frameworks
• Poor readability, verbose
• lack of proper generics, closures, lambda expressions, events, etc.
• VM + JIT, no built-in concurrency support
• Runs everywhere

C#

• Managed (GC)
• Has limited number of libraries and frameworks, Microsoft dominated
• Great readability, has support of most modern features (lambda, closures, generics, etc) as well as LINQ
• VM + JIT, no built-in concurrency support
• Only runs on Windows, license costs might be prohibitive for larger projects. Cloud deployment available with Azure.

Ruby

• Managed (GC)
• Has a good deal of libraries and frameworks
• Great readability, expressive and concise syntax
• VM, slow dynamic code,  no built-in concurrency support
• Cheap deployment but known to cause problems in larger deployments (e.g. Twitter)

Python

• Managed (GC)
• Has a great deal of libraries and frameworks
• Good readability
• VM, slow dynamic code, no built-in concurrency support
• Cheap linux deployment but can require more servers for large scale sites

Notice a common theme? Check category number 4 for all listed languages. No built-in concurrency support.

When all these languages were designed the requirements for the concurrent requests execution were much lower than what we have now. There has been a tremendous shift in volume of requests. 

Concurrency: CGI

Massive need for concurrency first happened when web applications became available. The first answer to running concurrent scripts was Common Gateway Interface (CGI). It made an HTTP requests look like it was executed by a user, mapped requests parameters to the environment variables, passed input to stdin and sent stdout back to the browser. First implementations would start a process for each request. For instance, if you had a python script it would execute Pyton interpreter for every request. The model relied on OS security and access control and was comfortable at the time as many tools operated with stdin/stdout. However, starting a process for each requests quickly becomes prohibitively expensive.

Concurrency: FastCGI

The answer was found in FastCGI, where a process wasn’t closed at the end of the requests but instead re-used to process next request. FastCGI allowed for a pull of processes to serve a lot of incoming requests, allowing for a better scalability. This model is still widely used for many deployments. However, you quickly find out that switching processes is an expensive task and that inter-process communication (IPC) is also expensive. Another problem is sharing of resources. For instance, each process needs its own database connection pool, instead of having a shared one.

Imagine having 100 front end servers each running 5 FastCGI processes. If no SQL queries are executed in parallel you would need 500 connections to the database. Depends on your database, it can be very expensive to have that many simultaneous connections open.

Concurrency: Thread Pool

The answer to the problem was found in using thread pools. Instead of having multiple processes we run one process per application but we create multiple threads withing the process. Threads are much lighter than processes so context switches are much cheaper, you can run many threads per process (normally anywhere between 50 and 250) , and you can have access to shared memory and resources such as DB connection pool. However, for large scale web applications even this model has its limitations. Your code is  expected to be synchronously executed on a single thread that is dedicated to processing your requests. The code blocks your thread while executing. If you have a to access a resource with a slow response time (for instance a web service such as google, facebook, twitter, etc.) your thread has to wait until the sub-requests is executed. If underlying resources are slow, your thread pool will quickly becomes starved: every available thread will be waiting for sub-requests, and no new requests will be processed. This is a very common problem.

Concurrency: Fibers

Many people are coming to realize that thread pools are quite ineffective for applications that have massive load. Many people also realize that you don’t need to be next google to experience these issues; you can have a popular iPhone app or mentioned on Reddit/Digg/Slashdot and find our that your services are bogged down.

The problem with scalability is that you need to be able to survive the burst. It can be extremely cost ineffective to scale hardware to that level. For instance, most of the time your web farm will be doing nothing, but you need to maintain it for that moment when you get mentioned on Twitter and the traffic comes your way.

I mentioned iPhone where a simple free application can become available to millions of users instantly and setting up a web farm with many servers in it is simply not an option. One solution would be to utilize the super powerful servers that you have to the max of their potential. You achieve this by following one simple rule: don’t waste resources on waiting if there is something else to work on.

If you want to achieve maximum throughput you have no choice but to be asynchronous. Instead of waiting for that Google API request to come back you can start processing new incoming requests and get back to serving the first requests once the data is available. It requires a primitive lesser than a thread. You might have heard names such as “fiber”, “tasklet”, “channel”, “coroutine”. Let’s call it “fiber”. Fibers are micro-threads that are executed concurrently. When your fiber encounters a long running operation it schedules it to be executed and yields to another fiber. Essentially, you execute cooperative multitasking within preemptive multitasking.

An ideal application will execute on top of a thread pool with N threads where N = number of CPU cores. You don’t need extra threads, as your operations never block. In reality you might want more threads to smooth things out.

Concurrency: Libraries

This is by no means a novel idea. There are plenty of solutions written to address this specific issue. There is Tornado, Node.js, any many others. I personally think the fact that JavaScript in the browser requires developers to only use non-blocking (another name for asynchronous) calls has advanced the cause greatly. The performance is there, so what’s the problem? Readability and control flow. Consider this code:

function updateStatus() {
try {
  var status = twitter.getStatus();
  displayStatus( status );
}
catch( error ) {
  displayError( error );
}

pretty straight forward, huh? Now, imagine writing  this code in a non-blocking way:

function updateStatus() {
  twitter.getStatusAsync( onUpdateStatusSuccess, onUpdateStatusError );
}

function onUpdateStatusSuccess( status ) {
  displayStatus( status );
}

function onUpdateStatusError( error ) {
  displayError( error );
}

Not nearly as hot. Now imagine that displayStatus and displayError are non-blocking calls either. You will quickly end up with a very fragmented code.

Languages that support first class functions or lambdas allow for writing success/error functions inline, however it generally leads to the continuation-passing  style which has same problem across the languages: non-blocking calls make natural language constructs such as structured exception handling, loops, condition statements, etc. ineffective.

This is a very important thing to understand. Languages designed as they are with a certain control flow in mind. They have control structures in place to assist the developer in accomplishing certain task. For instance, foreach loop can usually be replaced with for and switch/case/default can be replaced with if/elseif/else construct, but they are there for a reason. They serve to increase productivity through providing more concise and expressive code. Non-blocking libraries render you unable to use a lot of the language constructs and produce poorly structured code.

Concurrency: Language support

It’s quite clear that if you want to achieve the best possible scalability and maintain readability, you not only need a non-blocking library, but also a language that supports this specific style of development.

Let me outline this: to enable  wide-spread, practical use of non-blocking paradigm a programming language needs to provide built-in support for it.

This is the theme of all new languages. Whether it is co-routines in Go, actors in Scala, tasks in Rust, or new async/await keywords in C#, all these languages are trying to get a crack at the same problem: make a concurrent development easier and more available to developers.

All new languages (with notable exception of C# 5 approach)  implement a form of a message exchange. You can send message from one part of your program to another. Unfortunately, I don’t think this address the issue properly and, while a step forward, is far from the desired solution.

C# is taking a different approach that will reduce code fragmentation a lot better. However it will be a limited success due to the need to explicitly define functions as async and use  of Task<T> and thread pool.

I think there is a better solution to the concurrency issue,  one that could be applied to the existing languages by introducing libraries and compiler magic around those libraries. However, this is a topic for a different conversation.