Agile turns traditional (waterfall oriented) testing on its head. Organizations that transition to Agile methods typically find themselves very confused – What do we do with traditional testing functions? How do we integrate non-functional types of testing? Do you just leave it all up to the team? Does Agile somehow magically take care of everything? How does Agile handle all of our compliance rules and enterprise risks?
How does Agile handle all of our compliance rules
and enterprise risks?
There is no single answer to any of these questions.
The reason is that the approach to testing depends on (among others):
• The nature of the requirements.
• The types and degree of risks perceived by the organization.
• Who the stakeholders are.
• The types of environments that can be obtained for testing.
• The skills of the development teams.
For example, consider Morticia’s website:
We are on a small team building a website for my grandmother, Morticia, to sell used odds and ends. Morticia expects a traffic of about five purchases per day – mostly from our many cousins around the world. My grandmother fulfills all of her orders herself from items in her attic and basement, has Lurch and Thing package them, and employs my uncle Pugsley and his kids (who are Uber drivers) to take the packages to the post office every day. It is very simple, and so we write the user stories for the website, with the acceptance criteria expressed in Gherkin, code the tests in Ruby, and we are done! We are not worried too much about security, because all of the payment on the site will be handled by PayPal, which is a pretty self-contained service, and not a-lot is at stake anyway since most of the items are priced under $10.
Now let’s consider another (fictitious) example, EFT Management Portal:
We are on a team in a six team program to build a website that enables banks to manage their electronic funds transfer (EFT) interconnections. There are lots of bank partners that will connect to this system via their own back end systems. We also have to comply with Sarbanes Oxley regulations, as well as all applicable State laws for each bank’s EFT endpoints. There is a great risk of fraud – both internal and external – given the nature of the application. The technology stacks include several open source, commercial, and proprietary stacks as well as mainframe systems, and there are some non-IP protocols (e.g., SWIFT) over private leased networks. As a result, we have some experts on the teams who know about these things. Finally, while the work can be done incrementally, and a one year roadmap of incremental releases has been defined, each release has to be rock solid – there can be no screw-ups. There is simply too much at stake. The first release will therefore not be live, serving more as a production quality demonstrator.
The test plan for the EFT management portal is quite a bit more involved than the one for my grandmother’s website. Certainly, “trusting the team” will not fly with management. In fact, management has hired an independent risk management company to verify that testing will be sufficient, and that all risks are being managed. This was insisted on by the boards of two of the primary banking partners. This risk management team wants to meet with the program manager and team leads next week to discuss the risk team’s information needs, which will certainly including knowing what the test plan is.
I think that the testing approach for my grandmother’s website will be quite a bit different from the one for the EFT management portal – don’t you?
What is the whole point of testing?There are alternatives to testing. For example, the IBM “clean room” method substitutes careful review of source code, and its effectiveness is reported to be comparable to other forms of testing. From a Software Engineering Institute report,
Improvements of 10-20X and more over baseline performance have been achieved. Some Cleanroom-developed systems have experienced no errors whatsoever in field use. For example, IBM developed an embedded, real-time, bus architecture, multiple-processor device controller product that experienced no errors in two years use at over 300 customer locations.
The goal is therefore not to test, but to achieve a sufficient level of assurance that the code does what the customer needs it to do. I said “need” instead of “want” because an important aspect of Agile methods is to help the customer to discover what they actually need, through early exposure to working software, which constitutes a form of exploratory testing. There has also been a great deal of recent work in attempts to apply formal methods to Agile. This is especially valuable in high assurance systems such as aircraft, medical devices, and systems with very large capital costs such as the microcode that executes in mass produced hardware. Formal methods are also of great interest for mass produced consumer devices - especially security critical devices such as routers, for which people do not often update the firmware. As the “Internet Of Things” (IOT) becomes reality, formal methods are being considered as a way to increase the reliability of the myriad devices that will be operating throughout our environment – things will be quite a mess if all of those devices are buggy, require constant updates, and all present security risks.
The goal is not to test, but to achieve a
sufficient level of assurance.
The goal is therefore assurance. Testing is not a goal by itself. The question is, what are the Agile practices that help us to build software with sufficient assurance that it meets the need, including the various needs related to dependability? If you go to your bank’s website and transfer money from one account to another, do you expect to have a very high confidence that the money will not simply disappear from your accounts? If you make a purchase on the Internet, do you expect to have high confidence that the purchase will actually be fulfilled after your credit card has been charged? Thus, assurance is important even for the common activities that everyone does every day. Reliability affects a company’s reputation. When you code, do you think in terms of the required reliability of the end product?
Planning is important in Agile!First of all, we need to dispel the notion that creating a plan is not Agile. As Eisenhower said, “Plans are worthless, but planning is everything.” Implicit in this is that creating plans is important – that is what planning does (creates a plan), but plans always change – that’s why plans are “worthless” – because they end up needing to be changed. Of course, saying that they are “worthless” is hyperbole: it would be more accurate to say that they are only a starting point. Plans are simply the recorded outcomes of a planning meeting – the decisions that were made, and any models that were used. The planning meeting is what is most important, and the plan – the recorded outcomes – is important too, because people forget.
An Agile test plan is not externally driven –
it is internally driven, by the team.
One of the main Agile push-backs on test planning is that a traditional (waterfall) test plan is something created ahead of time by someone who is not on the development team. That is a non-starter for Agile projects, because Agile teams need to decide how they do their work, and testing is a central element of software development. Thus, an Agile test plan is not externally driven – it is internally driven, by the team. It is their test plan. It is the output of their discussions on how they plan to test the application.
That said, other stakeholders should be present for those discussions – stakeholders such as Security, Architecture, or whatever external parties exist who have an interest in the quality of the delivered product. Further, those external parties have every right to form their own opinions on the efficacy of the test plan – that is, whether the tests will actually prove that the requirements of the stakeholders are covered.
The question is not whether to plan, but what an Agile test plan and Agile test planning process really looks like. Brett Maytom, manager of the LinkedIn Scrum Practitioners group, has said in a discussion in that group,
“The bigger questions are, when to prepare it, how to represent it, how far in advance is the plan, who creates it, in what format is the test plan.”
Shuchi Singla, an Agile QA practitioner, commented in that same discussion,
“…we do need test plans, but unlike traditional formats they should be quick reference points. Rather than having 50 page bulky document a crisp sticky or may be white board information like RACI, platforms would suffice…we cannot let test plans just go. They have their importance and should stay though in KISS layout.”
There is a problem though: the very phrase “test plan” conjures up memories of waterfall test plans: large documents with all of the details, right down to the test scripts to be used – and with little input from the developers, and not able to evolve during development. We don’t want to do that – not anything close. As Shuchi Singla said, we want our test plan to be lightweight: we want our test plans to be test strategies. So from now on, we will refer to it as a test strategy. The strategy’s written form should also be highly maintainable – such as a set of wiki pages – so that it can be kept up to date as the team’s decisions about testing evolve over time. An Agile test strategy is always being refined and adjusted as the team learns more and more about the application and the things that actually need to be tested.
The very phrase “test plan” conjures up
memories of waterfall test plans, so from now on,
we will refer to it as a test strategy.
In collaborating with a colleague Huett Landry, one of us (Cliff) found it to be very effective to have a test strategy consisting primarily of a table, with columns of essentially “What, Why, Where, When, How/Who, Coverage”. In a recent LinkedIn discussion, Eugene Joseph, an Engineering Program Manager at Lockheed Martin, said,
“We had a Sys Admin team that was constantly being pulled in different directions by agile teams. We stood up a Kanban board a prioritized task that we put on the board. Someone will have to prioritize the task across your agile teams. One key thing we did was made sure that tasks entered had a clear specification of what, where, when, who, and how support was provided and a clear definition of done. This helped reduce communication delay since the client needing support CLEARLY specified what the needed.”
Thus, if you have existing silos, establishing a way of interacting with them – perhaps documented in a table that lists who will do what, when, and how – is especially important for enabling the development team to operate in an Agile manner.
If you have existing silos, establishing a way of
interacting with is especially important.
A diagram is also very useful for identifying the up-stream and down-stream dependencies of a development and testing release train. As W. Edwards Deming said, “You can see from a flow diagram who depends on you and whom you can depend on. You can now take joy in your work.” ☺ This enables you to see at a glance the types of testing that occur at each stage of the release train, the environments that each will need (and any contention that might result), and the latency that might occur for each testing activity. After all, continuous delivery is a pipeline, and to treat it as such, it helps to view it as such. Figure 1 shows such a diagram. Note that the various environments are identified, along with the activities or events pertinent to each environment. The horizontal dashed lines are not a data flow – they represent a logical sequence. Many loops are implied in that sequence.
|Figure 1: A delivery pipeline (aka “release train”).|
Stories that focus on outcomes – not just functionalityOne of us (Scott) who has done a-lot of Agile work in the embedded software space, has had great success using an evolutionary approach for exploratory testing. Awareness of the desired outcome – from a functional as well as a reliability perspective – provides the tester with a clarity of focus, so that he or she can push the application or device harder and find more issues. Scott has found this approach to be very effective as a general paradigm: creating a focus on desirable outcomes, from the perspective of both functional and non-functional requirements.
Develop the testing strategy as an
output of story development.
To implement this approach, story development should include analysts, developers and testers: develop the testing strategy as an output of story development. Some of the strategies end up as rows in the test strategy table, but others might be specific to a story. Being able to read specifically how something is going to be tested at the story level (because tests are part of the story detail) helps implementers of the story focus on the outcome that the user needs. The story should mention the types of non-functional testing that will occur. That way, the developer has the non-functional requirements at the top of their mind when they implement the story.
End Of Part 1In the next article we will talk about the various kinds of testing and how to make sure that they occur – in an Agile manner!