Can Janitors Make Structural Changes?

They clearly can sometimes do so. In fact, I am a case in point — some decades ago, I worked as a janitor. But the unfortunate fact is that the odds are not always good, and so you might want to think twice before staking your project such a change. So, what to do?

The first thing to do is to double-check whether parallelization is required. If single-threaded execution is fast enough, be happy with what you have.

If the application needs to go faster, are there any simple single-threaded optimizations that could be brought to bear? Perhaps some of these optimizations could safely be carried out by the current team.

If you need to apply the performance optimization of parallelization, you have a number of choices, in roughly increasing order of difficulty:

1. Run multiple instances of your existing application, perhaps through use of virtualization. You might divide up the incoming load randomly, or you might need to partition based on some sort of transaction or user ID. In some cases, this sort of approach can safely be carried out by administrators, while in other cases invasive surgery will be required.
2. Replace the entire application with software (be it open-source or proprietary) that has been designed to get the job done with adequate performance and scalability.
3. Link your application against an existing parallel library, for example, a parallel sort function. In the best case, this can be done simply by providing the library, using an existing binary of your application, courtesy of dynamic linking. This approach can work well, but only if much of the application's execution time is consumed by the task to be parallelized.
4. Redesign your application to use existing parallel middleware, for example, a relational database. This approach can also work well, however, not only must your application's execution be concentrated in the code to be replaced, but the speedup due to parallelization must exceed the overhead of the newly added communication.
5. Redesign the bottleneck portion of your application to use explicit parallelism. This can require major surgery, and will require some serious parallel-programming skills on your team.
6. Rewrite the application from scratch, using parallelism. This is tantamount to demolishing a building and constructing a new building in its place — not a task to be undertaken lightly.

The first option, running multiple instances of your application, is very attractive in that little or no change is required to the application itself.

The second option requires detailed knowledge of what the application does, but little or know knowledge of how it goes about doing it. Unfortunately, these two kinds of knowledge often go together. That said, this can be an extremely effective approach.

The third option requires some knowledge of the application's performance, which might be obtained from profiling tools. Of course, profiling be quite challenging given a large application that is distributed across a large datacenter, but crafty systems administrators are usually up to this sort of task.

The remaining options require much deeper knowledge of the application, the equivalent of the blueprints for a building. Although simple code broswers such as “cscope” can help, they do have their limits.

One colleague from a former life used an interesting technique for dealing with of complex software that would otherwise have been well beyond his capabilities. He used a large roll of paper, on which he drew the data structures, important portions of the code, and how they related. When the diagram got too messy, he would tear it off, unroll some more paper, and copy the diagram over, prettying it up in the process. This approach can be extremely effective, although some people may wish to do the diagramming electronically, especially given today's large monitors.

For the not-uncommon case of applications that are too large and complex to be drawn out neatly, more automation may be required, as might the hiring of expert help.