Concurrency basics

Please refer to multi-core computing for some very basic overview on the topic.

Multi-processing or multi-threading
It is easy to scale using multi-processing when there is no need to share data. For example, we can have a splitter of an input file and have multiple formatting jobs running. At the end, a merge function need to consolidate the data back into one unit. We need a scheduler to manage the dependencies and scheduling, either by a script file or a tool.

On the other hand, we can have a program that has different threads for different purposes and controlling all the dependencies and scheduling of the threads within the program. Starting a thread is also much faster and less resource intensive than starting a new process.

Threading considerations
Let's review some key areas.

Tread safety
This is the most important aspect of concurrent processing. We should not have race conditions, or data corruptions. The classic example on race condition is the simultaneous deposit/withdraw used in many textbooks. Moreover, we cannot have thread A handling the data for client A overlap and corrupt data by thread B of Client B. Think clearly on global, class, instance and local variables and their scopes and uses for each thread.

Locking mechanism
Semaphores, Mutex, Monitor, Lock.. Each one of them has different uses and its pros and cons in concurrency. When we are using a library to implement concurrent data access, we should understand which locking mechanism they used and why.

Deadlock and livelock
We need to avoid both. We can use prevention (e.g. by ordering) or detection (and kill to recover) mechanisms.

How many threads to run
In general we should have the number of threads less than or equal to the number of CPUs. However in some cases, we can increase the number of threads somewhat if we know some threads are not CPU bound and can be interrupted for I/O or other activities. Do some testings and performance benchmarking and determine the optimal number.

Operating system support
It is important for the kernel to have native support for multi-threading. Sometimes, the language or tool opt for user level threads instead of kernel level threads. User level threads in general cannot utilize the multiple CPUs as the kernel treats it as one process. There may also be different threading library implementations for the same operating system. In Linux, there are LinuxThreads and NPTL. In LinuxThreads (found in older Linux versions), each thread actually has a unique PID, so we need to take that into consideration in coding.

Code review, testing and logging
It is not easy to spot and catch every possible threading issues. Reviewing codes carefully, focusing on the business logic, variables scope and locking mechanism can help. Performance testing may detect some bugs due to luck. Since it is almost impossible to reproduce any production issues which involved the timing of execution of different threads, detail logging may be desired for important critical sessions.

Java
The most widely used concurrency control in Java is the "synchronized" keyword, which is a basically a monitor. However, we should only synchronized the critical sections of the codes. In the extreme case, if you synchronized one big function, all threads running that function will be serialized. You can also increase concurrency if you use wait() and notify() correctly.