"Smart Automation": How to Speed up Software Development without Losing Control

Introduction: Application Development Isn't Pulling Its Weight

At the beginning of the 21st Century, we live in the Age of Software. More and more everyday activities are dependent on computer programs. Every time we use a mobile phone or land line, browse the Web, receive or pay a bill, make a booking, travel by car or by air, make a financial transaction, or deal with our government, we are using software in one way or another. Many industries and enterprises could not even survive without software.

Yet there is a frustrating bottleneck in the supply of that essential commodity. While computer hardware gets cheaper and faster year by year, software does not. If anything, it is getting harder to deliver applications that work correctly, on time and within budget. Organisations do not have a very positive view of application development - with good reason. Quoting the Standish Group 1,

• Application development is expensive: 44% of software projects run over budget
• Application development is complex and risky: 30% of software projects are cancelled
• Application development seldom delivers according to expectations: 60% of software projects are not considered a success by the business
• Application development is almost always late: 90% of software projects are delivered later than expected

It's Time to Speed Things Up with Smart Automation!

It's clear that application development has to change radically to keep up with the demands of business. Many tasks have been automated, using software, with excellent results - from surgical procedures to landing Jumbo jets. Now it is time to automate the creation of software itself, always remembering Albert Einstein's saying: "Everything should be as simple as possible, but no simpler".

Just as Henry Ford automated the manufacture of cars by replacing craftsmanship with the assembly line, we can move towards the "manufacture" of software by automating application development. Among the immediate benefits are faster turnaround, more consistent quality, and a new division of labour - slashing a path through the complexity that all too often baffles programmers.

It is easy to design code generators that can produce application software at a tremendous rate. But we must be aware of the tradeoffs between speed of development, and other desirable factors such as quality, security, maintainability, runtime efficiency and reliability. We do not want to generate mass-produced, inefficient software that fails to meet the needs of its users. The trick is to combine the right amount of automation with the right amount of human intelligence, so that both human and computer are fully occupied doing the work for which they are best suited.

1. Fortunately, there are some factors that make this challenge easier. For instance: The ever-growing framework of industry standards, complex and frustrating as it may be, has solved many fundamental problems that previously blocked progress.
2. Even outside the arena of formal standards, a huge amount of practical software experience has been gained - and, as we shall see, this can be tapped in the form of patterns.

For example, there are several "standard" ways of storing data in a database, and retrieving it later. It is attractive to automate the programming of storage and retrieval routines, as this can often be done by plugging in long sequences of instructions that differ in only a few locations.

Experienced developers have been doing this kind of thing for years. Often, they will recognise that the function they have to write is similar to one they previously created. So they reuse their old code, usually by copy-and-paste. But this technique has serious weaknesses: it is error prone, and does not provide any benefits in the maintenance phase of the application.

Speeding up the production of code by generating it automatically, or reusing it selectively, does not solve the worst problem of application development. The trouble is that the developer's mind is being overloaded with too many facts. To write a program that solves a given business requirement, developers must understand that requirement. At the same time, they have to keep up to date with a range of technical software issues. And that is assuming they have just one single program to write! In practice, developers have to write dozens of applications, which require them to understand many different business processes. How can they be expected to keep track? To make matters worse, all those programs are going to need ongoing maintenance, enhancements, and bug fixes.

To stay on top of this mass of complexity, developers need intelligent software tools. But the editors, compilers, debuggers and even integrated development environments (IDEs) of today have little, if any, intelligence. In addition most tools provide after-the-fact intelligence: the developer has to make mistakes, before the tool can identify and correct those mistakes. More useful are tools that prevent developers from making mistakes, instead of correcting them afterwards.

Modelling and Code Generation

Just as programmers have been writing source code for decades, some of them have also been designing their software by drawing graphical models. To start with, this was just an extra activity: after finishing the model, a programmer still had to write all the code to implement it. The model was just a way of making sure the design was clear and consistent before getting down to the real work.

Great progress has been made in modelling, especially since the industry agreed to standardise on the Unified Modelling Language (UML) as a universal notation. Bearing in mind the advantages of automation, modelling tool vendors have provided the ability to generate some code sequences directly from UML models. More recently, a new technique called Model Driven Architecture (MDA) has emerged, in which high-level (analysis) models are transformed into more detailed lower-level (design) models for "platforms" such as J2EE and .NET. IBM has already thrown its weight behind model-driven development, and Microsoft is moving in a similar direction.

There is powerful synergy between MDA and frameworks like J2EE and .NET. Starting from models of business requirements, it is possible to generate source code for typical commercial applications through a series of successive translations, as follows:

1. User interviews are translated into requirements.
2. Requirements are translated into business models.
3. Business models are translated into architectural software designs.
4. Architectural designs are translated into source code
5. Source code is translated into executable applications. See Figure 3.

The beauty of this scheme is that some of the pieces are already in place. Step 5, for instance, calls for nothing more than an ordinary compiler. On the other hand, steps 1 and 2 are still best done by human experts. That leaves Steps 3 and 4, where MDA makes its biggest contribution.

The best thing about MDA is that it eliminates "developer overload". Why is this? Well, the problem was that the developer had to juggle too much information: user requirements, the details of the relevant business processes, technical architecture and programming techniques. MDA introduces a "separation of concerns" which assigns these responsibilities to those best able to handle them.

Steps 1 and 2 are carried out by domain experts, such as business analysts. In steps 3 and 4, business models are automatically translated into architectural models, and the latter into source code. Before compilation (step 5), experienced application experts write the critical sections of code that call for intelligent, creative programming. Meanwhile technology experts, such as software engineers, take on the role of monitoring, and where possible enhancing, the automated transformation steps.

Once MDA is up and running, all sorts of constraints disappear. When applications must be modified to reflect new business conditions, it is no longer necessary to wait for weeks or months while the change requests work their way up the queue. Instead, domain experts can update the models as the need arises to support new or changed business processes. Then a "first cut" version of the modified application's source code can be produced by applying the translations of steps 3 and 4.

The technology experts, for their part, do not have to worry about keeping up to date with business processes and organisational details. Instead, they can devote their time to measuring, tuning and optimising the MDA tools.

How Patterns Can Help

The best way of implementing an MDA tool is by enlisting the help of patterns - a powerful idea popularised by the book "Design Patterns"2 . A pattern is "a common solution to a common problem in a given context"; in other words, it is a way of reusing software ideas rather than code. To take a simple example, the "Singleton" pattern is useful where a class must have one, and only one, instance. There should usually be just one file system, one window manager, one print spooler. Other patterns, such as "Business Delegate" and "Data Access Object", are more complex. But every pattern represents a tried and tested solution to a common problem, which is guaranteed to work well if appropriately implemented.

Although patterns were originally suggested as an aid to conventional programming, they can be applied more consistently and reliably through automation. There are many ways to transform a business model into source code; patterns thin out that huge, wasteful variety, leaving only those transformations that are known to lead to efficient, reliable and robust applications. Fortunately, both Sun and Microsoft - the creators of J2EE and .NET - have published sets of patterns specifically designed to help developers working with those frameworks. (Sun Core J2EE Design Patterns; Microsoft Integration Patterns and Enterprise Solution Patterns).

In the context of MDA, patterns help especially with Steps 3 and 4. An experienced software engineer will always translate a given model construct into the same equivalent design construct. This amounts to the use of a pattern - except that it is done "in private", inside the engineer's head. It is better to write the pattern down, and keep it in a repository where it can be used by MDA tools. For example, a given pattern might specify how to translate a piece of a business model - say the maintenance and updating of customer bank accounts - into a particular J2EE construct, such as an Entity Bean. Then, a second pattern could dictate how the Entity Bean should be described in terms of Java source code.

Each transformation pattern can be written as a set of rules. Then, if a pattern needs to be changed for any reason, all that needs to be done is to add, remove, or rewrite some of the rules. Take, for example, the common situation where an organization decides to migrate its applications from EJB 2 to EJB 3.

Using conventional methods, this would be a tremendous upheaval, involving the rewriting of large amounts of code, which would take many months. With transformation patterns, however, all that needs to be done is to replace the patterns for translating business models to EJB 2 with patterns for translating to EJB 3. Each change is made only once, and then all the models can be run through the new transformation patterns - and hey presto, a complete set of EJB 3 application code!

You might ask how much difference it really makes - after all, doesn't the change to EJB 3 have to be made one way or another? Well, the answer is that it makes a very great difference. In the first place, it is far easier to go through all the changes in the new EJB specification and update the transformation patterns, rather than to pick through a far larger volume of source code, trying to remember which changes need to be applied to which statements.

More important still, this is where the separation of concerns comes into play. The people who change the patterns are not application experts, who are used to working under pressure, and juggling a thousand programming techniques with the ever-changing complexity of the business domain they are trying to understand. Instead, the highly specialised job of updating the patterns is done by a small team of very experienced technology experts, who are intimately familiar with the EJB patterns and with the language in which the pattern rules are written.

Conclusions

More than fifty years into the Computer Era, we are now living in the Age of Software. More and more essential and useful functions depend entirely on the proper working of software, and organisations are increasingly constrained by their ability to create and modify business-critical (or mission-critical) applications. We need a quicker way to produce efficient, secure reliable software.

Inevitably, the key must lie in automating the development process. Yet indiscriminate, unintelligent automation risks losing control of those other vital factors - quality, efficiency and reliability. Moreover, the key problem lies in the overloading of the developer's mind. No matter how intelligent and experienced, a human being can only do one thing at a time.

Through the magic of separation of concerns, MDA breaks the development process into three independent sets of activities. Domain experts, such as business analysts, work steadily away defining business problems and solutions in terms of UML models. Application experts, freed from the drudgery of writing the same routine code segments over and over again, use their creativity and skills to produce innovative, efficient answers to the challenges of development.

Meanwhile, the technology experts keep an eye on the software machinery that automatically translates from models to designs, and from designs to source code. If they see a wisp of smoke, they hurry over with an oil-can. Occasionally, they might turn off a section of the production line, in order to replace a whole set of machinery. And they are always on the look-out for new optimisations that could make the whole operation run more smoothly and efficiently.

Using MDA, organisations can change their application development methods to keep up with the demands of business. They can deliver applications in less time, saving money while helping to meet deadlines. Because so much of the routine code is generated by patterns which incorporate best practice, its quality can be improved. And when changes become necessary, they can be quickly implemented with minimal risk of unintended side effects.

1 Computerwire, 14 September 2004
2 "Design Patterns", E. Gamma, R. Helm, R. Johnson, J. Vlissides (Addison-Wesley, 1995)