In Search of Quality
From Computer Business Review Online - April 2005Testing software is not new, but its role in helping to boost software quality is more vital than ever. CBR reports.
When a boiler goes wrong, it is a relatively easy matter to find the fault. Software, however, can fail in a dazzling number of imaginative ways, because unlike a boiler, the faults are built into the design, not the manufacturing process. And that software design can be an incredibly complex animal. Given an infinite budget and an infinite timescale, developers could probably produce totally fault-free software. However, business people only ever want better software delivered faster, which means extra pressure on IT departments to keep pace with demand.
This pressure to perform is one of the reasons that software goes live with potentially damaging faults or bugs, as companies fail to build in enough time to test their applications.
The statistics on application failure make grizzly reading. Gartner estimates that 40% of a development team's time will be spent resolving application problems, and this can escalate to 100% of their time at certain stages of the lifecycle. The Yankee Group estimates that 80% of applications are either not tested or tested manually before being delivered to production, leaving the quality of such software open to speculation.
With statistics like these, you would expect companies to be putting a lot of time and effort into tackling software development, particularly testing it for bugs. But the reality is that companies' attitudes to software testing have little improved over the last decade. Those lessons have not been learned.
"I don't think the failure rate is lowering. High-profile failures tend to become failures when you get to the testing bit," says Neil Allcock, a partner at consulting practice Deloitte. "For some reason, IT organisations or companies tend not to give testing the consideration it needs."
The twin problems of time and money are largely behind this failure. "It's money that drives it," says Allcock. "If you've got a -�20m [$37m] project using internal and external resources, they will look to minimise budget and one of the easy ways to do that is testing."
When budgets and time are under pressure, testing is the area where companies find it easiest to cut corners. Partly that is because testing tends to happen at the end of the development process, so if other parts of the development lifecycle have taken longer than expected then companies may have little choice but to limit the testing phase.
"What happens," explains Sarah Saltzman, technology manager at Compuware, "is the end date doesn't change. If you're using a manual testing team, then you have finite resources that can easily run out."
But there is a simple way to avoid this bottleneck and improve application reliability: bring testing right up to the front of the development lifecycle, rather than leave it as an afterthought. As soon as the requirements are defined, the testing plan should be mapped out at the same time.
Failure mitigation
Testing tool and performance management company Mercury Interactive questioned leading UK organisations in financial services, media, telecommunications, IT and the public sector throughout February 2005 to reveal why software failures happen. The survey revealed that 55% of respondents feel software designs are rushed through too quickly due to inadequate planning, while another 42% said the primary reason behind sub-standard project management was that initial designs were not sufficiently verified at the outset."Companies today are leaving application testing far too late in the development lifecycle, leaving them at risk of large-scale software failures," says Colin Robb, product marketing manager at Mercury Interactive. "When testing is conducted after applications go live, fixes and adjustments are costly and wasteful. Companies must migrate their testing culture towards the beginning of the development cycle when problems can still be fixed." In other words, the later you find problems, the more expensive it is to fix them.
But part of the reason testing is at the end of the development process is due to methodologies such as the Waterfall method of software development, where each stage from analysis and design through to implementation is carried out in a strict sequence. The output of one step is the input into the next; and testing comes right at the end of the cycle.
"The issues we still see at the moment are that many organisations still think of development as a segmented cycle: a waterfall type approach," explains Jon Harrison, EMEA Java product manager for Optmizeit at Borland. "What happens is that testing tends to get relegated to the end of the development lifecycle which is not good, because it doesn't leave enough time. Testing has for a long time been something we can do at the end if we have time."
The limitations of the waterfall style approach are well known, but it is still widely used. "It's very comfortable and people are used to working that way," says Cyndi Mitchell, director of UK operations at ThoughtWorks. There is also the fear factor. People like using it, because it is what they are used to. ThoughtWorks says it has a different approach to software development, advocating the agile method, which brings testing right up to the start of the development process.
Unlike most development teams, which work separately from testing people from the outset, developers, testers, business people and project managers work together, producing iterative versions of the software. Every bit of code is put into a central repository and then tested immediately, so problems are sorted out quickly.
"You end up with systems that have a much simpler design," says Mitchell. "It also means that there's 'collective ownership' of the code: it's not just down to an individual developer."
Risk-based testing
Another pragmatic way of tackling testing further up the development cycle is through risk-based testing, which brings together the business analysts, stakeholders and technology specialists at the start of a project to analyse which parts of the application are most business critical."Traditional testing would involve manual techniques to see if the software answered requirements. But what we're seeing is there's never enough time to complete all testing: the reality is it's rarely achieved," says Compuware's Saltzman. "If you accept you're not going to test everything, then you have to make sure testing is done on the right parts of the application."
Compuware, Telelogic, Mercury Interactive and others have taken on-board the need for requirements management and testing to be an intrinsic part of the application development lifecycle, weaving the ability to carry out these functions earlier on into their application lifecycle management frameworks.
"Best practice dictates that you do the riskiest things first, but those aren't necessarily the things people like to do or the most interesting things to do," says Matthew Hopkins, managing director at Dunstan Thomas Consulting. "There's a real gap between best practice and common practice. Things have got better but not completely, and organisations need to start seeing software teams as professionals: you wouldn't run your accounting department the way most companies run their software desks," says Hopkins.
Automation tools
Of course, you could outsource the whole testing process, but that does not solve the problem of testing. You will still hit the same bottlenecks and time constraints if you have not allowed sufficient time for testing. Although it will not speed up development, Allcock says: "It kind of helps and there's an independence about it: and you do need to separate your development and your testing."One way of improving the quality - and taking the grunt-work out of testing - is using automation tools. "What we're trying to do is make it easier for people to do testing as a natural part of their day-to-day work," says Borland's Harrison. "One thing is to integrate the ability to do testing. If you're a developer, your job is to develop code, but if you have to stop and run tests it makes it more difficult. But if you can integrate that and a single mouse-click makes it happen, then you're much more likely to do it."
There are many different testing tools available to help all stages of the development cycle, from code audits that help analyse the complexity of code, to unit tests, where the code is evaluated to see if it does what it is supposed to do. One area where automated tools can really help is with the weekly peer reviews, when developers check over each other's code. "It's boring and laborious and it can be neglected, so automation can really help there," says Harrison.
But as ThoughtWorks' Mitchell points out, although there are tools out there to tackle every stage of development, you still have to build time into the development time to actually use them. "Pretty much every organisation I've been into has invested in automation tools, but they just aren't using them," she says. "It's a big corporate IT phenomenon. We buy packages because it makes us feel we've done something about a problem."
Barry Varley, chief executive of independent testing consultancy Acutest, believes that companies need to invest in training people, and creating effective testing processes, as well as aligning the testing activities with the business needs. "What surprises us most, however, is not poor tool usage. It's the number of development shops that would benefit from spending less time and resources on testing," Varley points out. '''"Time and money is wasted testing application behaviour that would have little impact on business if it did fail. In one large software developer, we found three quarters of the testing effort on one product was being spent on testing low to medium-impact behaviour, where the likelihood of failure was similarly low to medium. And do the business managers want to delay launch while this goes on? Of course not.
"If you really want to integrate testing into the development lifecycle effectively, start with its governance: take a long look at how aligned the testing is with the business objectives and risks, and provide business managers with more control over testing."'''
So without the management and organisational impetus to use them, as well as the time in the development process to use them, tools alone cannot solve your testing needs. For, as Hopkins says: "Automation is just a third of the equation of people, process and technology."
The impetus for change must come from the top. A respondent in the Mercury survey, for example, indicated that testing software applications against performance criteria before they went live was regularly ignored by senior management, who instructed them to 'fix it when it goes live'. What is needed is strong project management to prevent this kind of attitude and to ensure that testers and developers communicate throughout the project.
Here again tools can help. Instead of the QA team coming up with a text description to point out problem areas to developers, automated tools can pinpoint the problem. "The QA people are always able to effectively communicate back to developers: the developers are technical, but the QA people don't necessarily have the same level of technical ability, therefore we offer a product where you can capture a snapshot of what's happening when something occurs. That means the QA team can analyse the information and pass it to the development team," says Borland's Harrison.
