Real Agile Testing, In Large Organizations – Part 3

(Continued from Part 2)

What about TDD and unit testing?

As promised, let’s talk about how Test-Driven Development (TDD) fits into the over testing regime that we have been presenting.

TDD is a powerful technique for achieving high coverage unit tests and code with low coupling. TDD is a complex issue and there is quite a bit of controversy around it, so we will not take sides here. TDD is absolutely not a mandatory or necessary practice of Agile, but many very smart people swear by it, so be your own judge. It is even possible that TDD works for certain personality types and styles of thinking: TDD is an inherently inductive process, whereas top-down test design is a deductive process, and so TDD possibly reflects a way of thinking that is natural for its proponents. (See also this discussion of top-down versus bottom-up design. The book The Cathedral and the Bazaar also discusses the tension between top-down and bottom up design and development.)

The controversy between top-down and bottom-up approaches – and the personalities that are attracted to each – might even have analogy to the well known division in the sciences between those who are theorists and those who are experimentalists at heart: these two camps never seem to see eye-to-eye, but they know that they need each other. Thus, instead of getting into the TDD or no-TDD debate, we will merely explain TDD’s place in an overall testing approach, and a few things to consider when deciding whether to use it or not. Most importantly: do not let proponents of TDD convince you that TDD is a necessary Agile practice, or that teams that do not use TDD are inherently less “advanced”. These assertions are not true: TDD is a powerful design and test strategy, but there are other competing strategies (e.g., object-oriented analysis and design, functional design – both of which are completely compatible with Agile – and many others).

TDD operates at a unit test level (i.e., on individual code units or methods) and does not replace acceptance tests (including paradigms such as acceptance test-driven development (ATDD) and behavior-driven development (BDD)), which operate at a feature (aka story, or scenario) level. Unit testing is what most programmers do when they write their own tests for the methods and components that they write – regardless of whether or not they use TDD. Unit testing is also well suited for testing “failure mode” requirements such as that bad data should not crash the system. Used in conjunction with TDD and a focus on failure modes, failure mode issues can be resolved far sooner which is certainly very Agile.

Acceptance level testing is still critically important. Unit testing cannot replace acceptance tests, because one of the most important reasons for acceptance tests is to check that the developer’s understanding of the requirements is correct. If the developer misunderstands the requirements, and the developer writes the tests, then the tests will reflect that misunderstanding, yet the tests will pass! Separate people need to write a story’s acceptance tests and the story’s implementation. Therefore, TDD is a technique for improving test coverage and improving certain code attributes, and many see it as an approach to design – but it is not a replacement for acceptance tests.

One type of unit level testing that is very important,

regardless of TDD, is interface level testing.

One type of unit level testing that is very important, regardless of TDD, is interface level testing. Complex systems usually have tiers or subsystems, and it is very valuable to have high coverage test suites at these interfaces. In the TDD world, such an interface is nothing special: it is merely a unit test on those components. In a non-TDD world, it is viewed as an interface regression test and one specifically plans for it. For example, a REST based Web service defines a set of “endpoints” that are essentially remote functions, and those functions define a kind of façade interface for access to the server application. There should be a comprehensive test suite at that interface, even if there are user level (e.g., browser based, using Selenium) acceptance tests. The reason is that the REST interface is a reusable interface in its own right, and is used by many developers, so changes to it have a major impact. Leaving it to the user level tests to detect changes makes it difficult to identify the source of an error. In this scenario mocking is often the most advantageous way of unit level testing interfaces.

Another, much more important reason to have high coverage tests on each major interface is that the user level tests might not exercise the full range of functionality of the REST interface – but the REST level tests should, so that future changes to the user level code will not access new parts of the REST interface that have not been tested yet – long after the REST code has been written. The REST interface can also be tested much more efficiently, without having to run them in a browser. In fact, the performance tests will likely be performed using that interface instead of the user level interface.

Detection of change impact, at a component level, is in fact one of the arguments for Unit Testing (and TDD): if a change causes a test to fail, the test is right at the component that is failing. That helps to narrow down the impact of changes. The cost of that, however, is maintaining a large set of tests, which introduce a kind of impedance to change. Be your own judge on the tradeoff.

TDD can also impact the group process: it is generally not feasible in a shared code ownership environment to have some people using TDD and others not. Thus, TDD really needs to be a team-level decision.

It is possible that the preference (or not) for TDD

should be a criteria in assembling teams.

Legacy code maintenance is often a leading challenge when it comes to unit testing. TDD helps greatly to identify the impact when changes are made to an existing code base, but at the cost of maintaining a large body of tests, which can impede refactoring. Another example of a real challenge to utilizing TDD techniques is model-based development (see also this MathWorks summary) – often used today for the design of real time software e.g., in embedded controllers, using tools such as Simulink. These techniques are used because of the extremely high reliability of the generated code. There are ways of applying TDD in this setting (such as writing .m scripts for Simulink tests), but that is not a widespread practice. Acceptance Test Driven Development (ATDD) is potentially a better approach when using model-based development.

Finally, TDD seems to favor certain types of programmers over others. By adopting TDD, you might enable some of your team to be more effective, but you might also hinder others. It is therefore possible that the preference (or not) for TDD should be a criteria in assembling teams. Making an organization-wide decision, however, might be a mistake, unless you intend to exclude deductive thinkers from all of your programming teams.

The jury is still out on these issues, so you will have to use your own judgment: just be sure to allow for the fact that people think and work differently – from you. Do not presume that everyone thinks the way that you do. Do not presume that if you have found TDD to be effective (or not), that everyone else will find the same thing after trying it for long enough.

Some other types of testing

There are still many types of testing that we have not covered! And all are applicable to Agile teams!

Disaster Recovery

In our EFT management portal example (see Part 1), the system needed to be highly secure and reliable, comply with numerous laws, and our development process must satisfy Sarbanes Oxley laws and the information demands of an intrusive oversight group. Most likely, there is also a “continuity” or “disaster recovery” requirement, in which case there will have to be an entire repeatable test strategy for simulating a disaster with failover to another set of systems in another data center or another cloud provider. That is one case where a detailed test plan is needed: for testing disaster recovery. However, one could argue that such a plan could be developed incrementally, and tried in successive pieces, instead of all at once.

Security

Nowadays, security is increasingly being addressed by enumerating “controls” according to a security control framework such as NIST FISMA Security Controls. For government systems, this is mandatory. This used to be executed in a very document-centric way, but increasingly it is becoming more real time, with security specialists working with teams on a frequent basis – e.g., once per iteration – to review controls. Most of the controls pertain to tools and infrastructure, and can be addressed through adding scanning to some of the CI/CD pipeline scripts, to be run a few times per iteration. These scans check that the OSes are hardened and that major applications such as Apache are hardened. In addition, the security folks will want to verify that the third party components in the project binary artifact repository (Nexus, etc.) are “approved” – that is, they have been reviewed by security experts, are up to date, and do not pose a risk. All this can be done using tools without knowing much about how the application actually works.

Unfortunately, we cannot test for

careful secure design: we can only build it in.

However, some controls pertain to application design and data design. These are the hard ones. Again, for Morticia’s website (see Part 1), we don’t need to worry about that. But for the other end of the spectrum, where we know that we are a juicy target for an expert level attack – such as what occurred for Target, Home Depot, and Sony Pictures in the past 12 months – we have no choice but to assume that very smart hackers will make protracted attempts to find mistakes in our system or cause our users to make mistakes that enable the hackers to get in. To protect against that, scanning tools are merely a first step – a baby step. The only things that really work are a combination of,
1.    Careful secure design.
2.    Active monitoring (intrusion detection).

Unfortunately, we cannot test for careful secure design: we can only build it in. To do that, we as developers need to know secure design patterns – compartmentalization, least privilege, privileged context, and so on. For monitoring, all large organizations have monitoring in place, but they need the development team’s help in identifying what kinds of traffic are normal and what are not normal – especially at points of interconnection to third party or partner systems. Teams should conduct threat modeling, and in the process identify the traffic patterns that are normal and those that might signify an attack. This information should be passed to the network operations team. Attacks cannot be prevented, but they can often be stopped while they are in progress – before damage is done. To do that, the network operations team needs to know what inter-system traffic patterns should be considered suspicious.

Compliance with laws

Compliance with laws is a matter of decomposing the legal requirements and handling them like any other – functional or non-functional, depending on the requirement. However, while it is important for all requirements to be traceable (identifiable through acceptance criteria), it is absolutely crucial for legal compliance requirements. Otherwise, there is no way to “demonstrate” compliance, and no way to prove that sufficient diligence was applied in attempting to comply.

Performance testing

There are many facets to performance testing. If you are doing performance testing at all, four main scenario types are generally universal:
1.    Normal usage profile.
2.    Spike profile.
3.    Break and “soak” test.
4.    Ad-hoc tests.

Normal usage includes low and high load periods: the goal is to simulate the load that is expected to occur over the course of normal usage throughout the year. Thus, a normal usage profile will include the expected peak period loads. One can usually run normal load profile tests for, say, an hour – this is not long duration testing. It is also not up-time testing.

The purpose of spike testing is to see what happens if there is an unusual load transient: does the system slow down gracefully, and recover quickly after the transient is over? Spike testing generally consists of running an average load profile but overlaying a “spike” for a brief duration, and seeing what happens during and after the spike.

Break testing is seeing what happens when the load is progressively increased until the system fails. Does it fail gracefully? This is a failure mode, and will be discussed further below. Soak testing is similar, in that lots of load is generated for a long time, to see if the system starts to degrade in some way.

The last category, “ad-hoc tests”, are tests that are run by the developers in order to examine the traffic between internal system interfaces, and how that traffic changes under load. E.g., traffic between two components might increase but traffic between two others might not – indicating a possible bottleneck between the first two. Performing these tests requires intimate knowledge of the system’s design and intended behavior, and these tests are usually not left in place. However, these tests often result in monitors being designed to permanently monitor the system’s internal operation.

In an Agile setting, performance tests are best run in a separate performance testing environment, on a schedule, e.g., daily. This ensures that the results are available every day as code changes, and that the tests do not disrupt other kinds of testing. Cloud environments are perfect for load testing, which might require multiple load generation machines to generate sufficient load. Performance testing is usually implemented as a Jenkins task that runs the tests on schedule.

Testing for resiliency

Acceptance criteria are usually “happy path”: that is, if the system does what is required, then the test passes. Often a few “user error” paths are thrown in. But what should happen if something goes wrong due to input that is not expected, or due to an internal error – perhaps a transient error – of some kind? If the entire system crashes when the user enters invalid input or the network connection drops, that is probably not acceptable.

Failure modes are extremely important to explicitly test for. For example, suppose Morticia’s website has a requirement,

Given that I am perusing the product catalog,
When I click on a product,
Then the product is added to my shopping cart.

But what happens if I double-click on a product? What happens if I click on a product, but then hit the Back button in the browser? What happens if someone else clicks on that product at the same instant, causing it to be out of stock? You get the idea.

Generally, there are two ways to address this: on a feature/action basis, and on a system/component basis. The feature oriented approach is where outcomes based story design comes into play: thinking through the failure modes when writing the story. For example, for each acceptance scenario, think of as many situations that you can about what might go wrong. Then phrase these as additional acceptance criteria. You can nest the criteria if you like: languages like Gherkin support scenario nesting and parameterized tables to help you decompose acceptance criteria into hierarchical paths of functionality.

Testing for resiliency on a component basis is more technical. The test strategy should include intentional disruptions to the physical systems with systems and applications highly instrumented, to test that persistent data is not corrupted and that failover occurs properly with minimal loss of service and continuing compliance with SLAs. Memory leaks should be watched for by running the system for a long time under load. Artifacts such as exceptions written to logs should be examined and the accumulation of temporary files should be watched for. If things are happening that are not understood, the application is probably not ready for release. Applying Agile values and principles to this, this type of testing should be developed from the outset, and progressively made more and more thorough.

Concurrency testing is a special case of functional testing.

Driving the system to failure is very important for high reliability systems. The intention is to ensure that the system fails gracefully: that it fails gradually – not catastrophically – and that there is no loss or corruption of persistent data and no loss of messages that are promised to be durable. Transactional databases and durable messaging systems are designed for this, but many web applications do not perform their transactions correctly (multiple transactions in one user action) and are vulnerable to inconsistency if a user action only partially completes. Tests should therefore check that as the system fails under load, nothing “crashes”, and each simulated update request that failed does not leave artifacts in the databases or file system, and as the system recovers, requests that completed as the system was failing do not get performed twice.

Concurrency testing is a special case of functional testing, but it is often overlooked. When I (Cliff) was CTO of Digital Focus (acquired by Command Information in 2006) we used to put our apps in our performance lab when the developers thought that the apps were done. (We should have done it sooner.) We generally started to see new kinds of failures at around ten concurrent simulated users, and then a whole new set of failures at around 100 concurrent users. The first group – at around ten users – generally were concurrency errors. The second group had to do with infrastructure: TCP/IP settings and firewalls.

Regarding the first group, these are of the kind in which, say, a husband and wife have a joint bank account, and the husband accesses the account from home and the wife accesses it from her office, and they both update their account info at the same time. What happens? Does the last one win – with the other one oblivious that his or her changes were lost? Do the changes get combined? Is there an error? These conditions need to be tested for, because these things will happen under high volume use, and they will result in customer unhappiness and customer support calls. There need to be test scenarios written, with acceptance criteria, for all these kinds of failure mode scenarios. These should be run on a regular basis, using a simulated light load of tens of users, intentionally inducing these kinds of scenarios. This is not performance testing however: it is functional testing, done with concurrent usage.

Is all this in the Definition of Done?

The definition of done (DoD) is an important Agile construct in which a team defines what it means for a story – any story – to be considered done. Thus, the DoD is inherently a story level construct. That is, it is for acceptance criteria that are written for stories. DoD is not applicable to system-wide acceptance criteria, such as performance criteria, security criteria, general legal compliance criteria that might apply to the implementation of many stories, etc.

It is not practical to treat every type of requirement as part of the DoD. For example, if one has to prove performance criteria for each story, then the team could not mark off stories as complete until the performance tests are run, and each and every story would have to have its performance measured – something that is generally not necessary: typically a representative set of user actions are simulated to create a performance load. Thus, non-functional requirements or system-wide requirements are best not included in the DoD. This is shown in Figure 1 of Part 1, where a story is checked “done” after the story has passed its story level acceptance tests, has not broken any integration tests, and the user has tried the story in a demo environment and agrees that the acceptance criteria have been met. Ideally, this happens during an iteration – not at the end – otherwise, nothing gets marked as “done” until the end of an iteration. Thus, marking a story as “done” is tentative because that decision can be rejected by the Product Owner during the iteration review, even if a user has tried the story and thought that it was done. Remember that the Product Owner represents many stakeholders – not just the users.

Another technique we use with larger sets of teams (think portfolio) – and especially when there are downstream types of testing (e.g., hardware integration testing) – is definition of ready (DoR). The state of “ready” is a precursor to the state of being “done”. This helps to ensure that the DoD – which might include complex forms of testing – can be met by the team. The team first ensures that a story meets the DoR. These are other criteria such as that the story has acceptance criteria (DoD would say the acceptance criteria have been met), certain analysis have be completed, etc. – just enough so that development and testing have a much higher likelihood of being completed within an iteration. This works with teams and programs of all sizes. We do find that for larger programs, the DoR is almost always very useful. Defining a DoR with the Product Owner is also a great way of engaging the Product Owner on non-functional issues to increase their understanding of those issues and ensure the team is being fed high quality stories.

End Of Part 3

Next time in Part 4 we will connect the dots on the organizational issues of all this!


Authors (alphabetically):
Scott Barnes
Cliff Berg

Real Agile Testing, In Large Organizations – Part 2

(Continued from Part 1)

Last time we saw that there is no single answer to the level of test planning needed for Agile projects – it depends!

We also remembered that the whole point of testing is to achieve an acceptable level of assurance that the system meets the actual business need – in every way that matters to the organization.

This time we will look at a kind of template for the pieces of an Agile test strategy. You can then add and subtract from this template for your own project – and perhaps even dispense with it altogether for a very simple project – but in that case it at least provides food for thought.

What about technical stories?

Many teams use “technical stories” to specify non-functional requirements. This is ok, except that these are not really stories: you never finish them – they are actually cross-functional acceptance criteria. But casting non-functional requirements as acceptance criteria does not work perfectly either: that means that no story is done until all of the non-functional criteria are done, and that is not a practical way to run an iteration.

Create a “theme” for each type of

non-functional requirement.

Thus, while the above approaches can work, it is often better to treat non-functional requirements as just that: requirements. Don’t try to fit that round peg into the story square hole. Instead, create a “theme” for each type of non-functional requirement, e.g., “performance”, “security”, etc., with theme level acceptance criteria – i.e., requirements! Then write stories for the work that needs to be done; but do not skip creating a strategy for how to test the requirements for each of these themes. A strategy (high level plan) is needed too, in order to think through the non-functional testing and other activities in an end-to-end manner. This is a strategy that the team should develop. The strategy is the design for the testing aspects of the release train. Without it, you will find it difficult to discuss testing issues and dependencies that arise during testing, because there will be no conceptual context, and you will also find it difficult to communicate the testing strategies to stakeholders.

You can define exploratory testing activities
that provide feedback to the application’s monitoring theme.

There is a side benefit. If you treat the testing pipeline as a system, then you are in a good position to identify ways to monitor the application. For example, exploratory performance testing will reveal bottlenecks, and the application can then be enhanced to monitor those bottlenecks during operation of the system. Monitoring platforms such as Sensu can be used to consolidate the monitors across the many components of the application. Thus, in your testing strategy, you can define exploratory testing activities that provide feedback to the application’s monitoring theme, resulting in stories pertaining to operational monitoring. Identifying this ahead of time – at a large grain level – is important for making sure that this type of work is not a surprise to the Product Owner and that it receives sufficient priority. The key is to treat the testing pipeline as an aspect of the development pipeline, and design it with feedback loops, minimum latency, and sufficient coverage of each requirement category.

The key is to treat the testing pipeline as an aspect

of the development pipeline.

The What, Why, Where/When, How/Who, and Coverage

Let’s look at the What, Why, Where, When, How/Who, and Coverage.

The “What” is the category of testing, such as “functional acceptance testing”, “performance testing”, or “exploratory testing”. If you like, these can be grouped together according to the “testing quadrants”.

The “Why” is the aspect of requirements that this type of testing is intended to address, such as “story acceptance criteria”, “system-wide performance requirements”, or “anomaly identification”.

The “Where” is the environment(s) in which the testing will occur. In a CI/CD process, most types of testing will occur in multiple environments (as shown in Figure 1), but not necessarily all – e.g., in the example shown in Figure 1, performance testing is only being done in the “SCALE” environment. Your test strategy should reference a table or information radiator depicting all of the identified test environment types.

The “When” is the event(s) that will trigger the testing, and the frequency if the triggering event is calendar or time based. (Examples are shown in Table 1.) The “How” is the strategy to be used for those types of tests, such as “Use JBehave/Java, Selenium”, or “Use JMeter in cloud instances”.

The “How” should include “Who” – i.e., who will do what: that is, who will write the tests, who will perform them if there are manual tests, etc.

The Where/When and How/Who are

especially important if you interface with

specialized “enterprise” testing functions.

The Where/When and How/Who columns are especially important if you interface with specialized “enterprise” testing functions of any kind, e.g., Security, Performance Testing, Independent Acceptance Testing, etc.: you want to integrate these groups in an Agile “pipeline” manner so that no one is ever waiting on anyone, and that requires that everyone have a very clear idea of what everyone will be doing and when.

Integrate these groups in an Agile “pipeline” manner

so that no one is ever waiting on anyone.

Table 1: Sample lines of a test strategy table.

What	Why	Where, When	How (Strategy), Who	Coverage 1. How measured; 2. How assessed; 3. Sufficiency
Functional acceptance testing	Story acceptance criteria	• LOCAL (Workstation or personal cloud). Continually. • Cloud “CI”. When code pushed. • Cloud TEST. Daily.	Use JBehave/Java, Selenium. Acc test programmer must not be the story programmer.	• How meas: Cobertura. • How asses: Use Gherkin executable test specs, to ensure that no acc crit are missed. • Reqd: Need 100% coverage.
Performance testing	System-wide performance requirements	• “PERF” (in cloud). Nightly.	Use JMeter in cloud instances. Perf test team and architect.	QA will verify coverage of executable test specs.
Exploratory	To detect unanticipated anomalies	• DEMO. Any time.	Manual. Anyone who volunteers – but not the story’s programmer.	Amount of time/effort should be indicated by the story.

The final column, “Coverage”, is really about thoroughness. It has three parts: (1) how test coverage will be measured, (2) how coverage will be assessed, and (3) what level of coverage is considered to be sufficient. This gets into an important issue for testing: How do you know when you are done testing? How do you know how much testing is enough?

How much testing is enough?

In a traditional waterfall development setup, there is often a separate Quality Assurance (QA) function that independently assesses whether the test plan is adequate. This is usually implemented as a gate, such that the QA performs its assessment after the application has been tested. That whole approach is a non-starter for Agile projects – and even more so for continuous delivery (CD) – where working, tested software is produced frequently and can be deployed with little delay. But let’s not throw out the whole concept of QA – like “throwing the baby out with the bath water”. QA can play a vital role for Agile teams: independence.

QA can play a vital role for Agile teams: independence.

My mother Morticia knows the people who are building her website: they are our cousins, and she trusts them implicitly. But the EFT Management Portal is another matter. In that case, an external technical auditor has been engaged to provide independent assessment. But what about inbetween situations? What about run-of-the-mill mission critical applications developed by most organizations? Should you just “trust the team”?

To “trust the team” is not to have blind trust.

To answer that question, we need to clear up a common point of confusion. To “trust the team” is not to have blind trust: if there is a-lot at stake, then blind trust would be illogical and naïve. The Agile adage that one should “trust the team” does not mean to have blind trust: it means to give the team substantial (but not absolute) leeway to do the work in the way that it finds most effective. That does not relieve the team from explaining their processes, or from the responsibility to convince stakeholders that the processes (especially testing) will result in the required level of assurance. After all, some of those stakeholders are paying the bill – it’s their system.

Self-directing teams are never without leadership and vision. Leaders need to ensure that teams have a clear understanding of the end goal (product) and why the business needs the product (vision). When vision and goals are clear, acceptance criteria and intent become much clearer. By producing what stakeholders have described and by being provided a clear set of goals and a vision for a product, teams typically are able to build significant trust with their stakeholders and the business. This trust continues and the team feels empowered.

When clear goals and vision (leadership) are missing, there tend to be longer testing cycles because the testing starts to focus on ensuring the requirements are correct instead of ensuring the requirements are met.

Another consideration is that teams are under great pressure to create features for the Product Owner. If the Product Owner will not have to maintain the application, then the Product Owner will not be very concerned with how maintainable the system is – that is “IT’s problem”. (When Product Owners fulfill the role because they are the responsible person for an application, they are much better within this role. When Product Owners are not responsible for the product because produced but are only responsible for delivery of a project, they are no longer Product Owners and are now back to Project Managers.) Further, the Product Owner will not have the expertise to ask about things such as “concurrency testing”, for checking that the application works correctly when multiple users try to update the same data. In fact, some software teams do not know too much about that either – so should you simply “trust the team”? Teams cannot always be staffed with all of the skills sets that are needed – resources might be constrained. These reasons are why organizations need independent trustworthy assessment of testing – as a second pair of eyes on whether the testing has been sufficient. It is just common sense.

Have QA work with the team on a continuing basis 

– not as a “gate”.

If we don’t implement QA as a gate, then how should we do it? The Agile way to do it is to have QA work with the team on a continuing basis, examining the team’s test strategies, spot checking actual testing code, and discussing the testing strategies with the various stakeholders to get their thoughts. QA should have a frequent ongoing presence as development proceeds, so that when the application is ready for release, no assessment is needed – it has already been done – and in fact it has been used to help the team to refine its approach to testing. QA effectively becomes an independent test on the testing process itself – a feedback loop on the testing feedback loop.

How does QA decide how much testing is enough? I.e., how does QA decide what level of coverage is sufficient? That is a hard question. For functional testing there is a fairly straightforward answer: the organization should start tracking the rate at which bugs are found in production, and correlating that metric with the test coverage that was measured when the software was built. Over time, the organization will build a history of metrics that can provide guidance about how much coverage is effective in preventing production bugs – with sufficient assurance for that type of application.

In the previous article we mentioned that story development should include analysts, developers and testers: that one should consider testing strategy as an output of each story’s development, since each story might have unique testing requirements. We have found it very effective when testing or QA teams contribute to that discussion, so that the test plan evolves during the story writing rather than after software has been produced. The testers help write acceptance criteria during the story writing sessions. One of the great advantages of this is that the developer knows exactly how the story will be testing, thus helping implementation direction.

Accumulate operational robustness metrics

over time and use those to inform judgment

about the level of testing that is needed.

This is a little harder to do for other kinds of requirements, e.g., security requirements, performance requirements, maintainability requirements, and so on, but the concept is the same: accumulate operational robustness metrics over time and use those to inform judgment about the level of testing that is needed. We suggest that leveraging architectural themes will help teams keep an eye on key issues such as these.

Back to the table’s “Coverage” column. Consider the example for functional tests shown as row one in the table: we specify Cobertura for #1 (measuring coverage). But Cobertura checks code paths traversed: it does not check that you have actually coded everything that needs to be tested. Thus, #2 should be something like, “Express story level acceptance criteria directly in JBehave Gherkin”. That ensures that nothing gets left out. In other words, we will be using “executable test specs”, or “behavioral specifications”. Finally, as to what coverage is sufficient, we might specify that since we want a really, really robust application, we need 100% code coverage.

The real intent behind coverage is that the more important parts of the application are well covered. We typically do not see 100% coverage over entire application code bases, but that is a nice stretch goal. The most important part of coverage though is that coverage does not stagnate at anything less than 70% and steadily grows over time.

To test the response time, we can write “executable” specs.

Coverage is more difficult to specify for non-functional types of testing. For example, how would you do it for performance tests? The requirement is most likely expressed in SLA form, such as, “The system shall be up 99.99% of the time”, and “The response time will not be less than 0.1 second 99% of the time”.

To test the response time, we can write “executable” specs of the form, “Given that the system has been operating for one hour under a normal load profile (to be defined), when we continue that load for one hour, then the response time is less than 0.1 second for 99% of requests.” Of course, not all performance testing tools provide a language like Gherkin but one can still express the criteria in executable “if, when, then” form and then write matching scripts in the syntax required by the load testing tool.

Testing the up-time requirement is much harder: the only way to do it is to run the system for a long time, and to design in hot recovery mechanisms and lots of redundancy and elasticity. Defining coverage for these kinds of requirements is subjective and is basically a matter of checking that each requirement has a reasonable test.

The coverage requirement for Exploratory testing is interesting: In the example of Table 1, we list it as “Amount of time/effort should be indicated by the story”. In other words, for exploratory testing, decide this when the story is written: decide at that time how thoroughly the exploratory testing should be for that story. This gets back to writing stories that focus on outcomes, as we discussed in Part 1.

The test strategy wiki page is for recording decisions on how testing is actually being done. It is a living, evolving thing.

Most likely some narrative will be needed to explain the table entries, which need to be concise to fit in a cell. If the test strategy is maintained on a wiki (strongly encouraged), it is good Agile practice to use it as the scratchpad for thinking about testing and for recording decisions on how testing is actually being done. It is not a document that one creates and then forgets about: it is a living, evolving thing.

(Note: We consider Sharepoint to be a wiki if (a) everyone on a team can edit the pages, (b) everyone on the team can create new pages, and (c) full change history is maintained; but if you uses Sharepoint, don’t upload documents: create the content right in Sharepoint, as pages.)

The test strategy should inform the decisions on

what environments are needed.

The test strategy should inform the decisions on what environments are needed: this is an integrated decision that should be driven by the testing strategy. Since it takes time to provision environments, or to write scripts that provision cloud environments, this means that the testing strategy is something that is needed very early – enough to allow for the lead time of getting the environments set up. That is why testing strategy should be addressed during release planning, aka “sprint 0”. Ideally, a team continues to use a very similar testing process from one release to the next, or one project to the next, so that the environment architecture stays pretty much the same, and that way you always know what types of environments you will need.

What QA really does is inform us about the current state

of the system under test.

We believe that the term “QA” is a misnomer. We prefer the term “Quality Informers”. Due to the fact that the vast majority of people who make up QA teams are not allowed to touch source code, not allowed to hire and fire and are not allowed to impact or alter budgets, they clearly have no enforceable means of quality assurance. What they do really well though is inform us about the current state of system under test. This is an important point when you consider previous paragraphs where we talked about informing and feedback.

Not everything can be automated

Automation is central to Agile, and it is essential for continuous delivery. But not everything can be automated. For example, these things cannot usually be automated:
1.    Exploratory testing.
2.    Focus group testing and usability testing.
3.    Security penetration testing. (Basic automation is possible.)
4.    Up-time testing.
5.    Testing on every single target mobile platform.

To deal with these in the context of continuous integration (CI) and continuous delivery, the CI/CD process needs to focus on the tests that need to be run repeatedly, versus tests that can be done with sufficiently high confidence that things will not change too much. By automating tests that are repeatable, we free up more time for the type of testing that cannot be automated. For example, if usability testing is done once a month, that might be sufficient unless the entire UX paradigm changes. Security penetration (“pen”) testing should be done on a regular basis, but real (manual) penetration testing is an expensive process and so there is a cost/benefit tradeoff to consider – in many cases (depending on the level of risk), automated pen testing is sufficient, with perhaps expert manual pen testing done on a less frequent basis. Up-time testing can really only be tested in production, unless you are testing a new airplane’s system software before the first shipment and have the luxury of being able to run the software non-stop for a very long time.

Today’s mobile devices present a large problem for platform compatibility testing. There are so many different versions of Android out there, and versions of Apple’s iOS, and many versions of Android have significant differences. Fortunately, there are online services that will run mobile app tests on many different devices in a “cloud”. Even Apple’s OSX operating system can now be run in a cloud.

End Of Part 2

At this point, test-driven development (TDD) proponents are jumping up and down, feeling that their entire world has been sidestepped by this article, so in part 3 of this article we will start off with that. Also, while the Who column of the test strategy table provides a means for coordinating testing-related activities performed by multiple parties, we have not talked about who should do what. For example, some testing activities – such as performance testing and security testing and analysis – might require special skills; and if there are multiple teams, perhaps each working on a separate sub-component or sub-system (possibly some external teams), then how should integration testing be approached? We will examine these issues in a later part of this series.

As PDF.

Authors (alphabetically):
Scott Barnes
Cliff Berg

Real Agile Testing, In Large Organizations – Part 1

This article launches a new section on the challenges that organizations face when adopting Agile approaches to testing! We will start out with a four-part article series, and then add more articles over time.

Agile turns traditional (waterfall oriented) testing on its head. Organizations that transition to Agile methods typically find themselves very confused – What do we do with traditional testing functions? How do we integrate non-functional types of testing? Do you just leave it all up to the team? Does Agile somehow magically take care of everything? How does Agile handle all of our compliance rules and enterprise risks?

How does Agile handle all of our compliance rules

and enterprise risks?

There is no single answer to any of these questions.

The reason is that the approach to testing depends on (among others):
•    The nature of the requirements.
•    The types and degree of risks perceived by the organization.
•    Who the stakeholders are.
•    The types of environments that can be obtained for testing.
•    The skills of the development teams.

For example, consider Morticia’s website:

We are on a small team building a website for my grandmother, Morticia, to sell used odds and ends. Morticia expects a traffic of about five purchases per day – mostly from our many cousins around the world. My grandmother fulfills all of her orders herself from items in her attic and basement, has Lurch and Thing package them, and employs my uncle Pugsley and his kids (who are Uber drivers) to take the packages to the post office every day. It is very simple, and so we write the user stories for the website, with the acceptance criteria expressed in Gherkin, code the tests in Ruby, and we are done! We are not worried too much about security, because all of the payment on the site will be handled by PayPal, which is a pretty self-contained service, and not a-lot is at stake anyway since most of the items are priced under $10.

Source

Now let’s consider another (fictitious) example, EFT Management Portal:

We are on a team in a six team program to build a website that enables banks to manage their electronic funds transfer (EFT) interconnections. There are lots of bank partners that will connect to this system via their own back end systems. We also have to comply with Sarbanes Oxley regulations, as well as all applicable State laws for each bank’s EFT endpoints. There is a great risk of fraud – both internal and external – given the nature of the application. The technology stacks include several open source, commercial, and proprietary stacks as well as mainframe systems, and there are some non-IP protocols (e.g., SWIFT) over private leased networks. As a result, we have some experts on the teams who know about these things. Finally, while the work can be done incrementally, and a one year roadmap of incremental releases has been defined, each release has to be rock solid – there can be no screw-ups. There is simply too much at stake. The first release will therefore not be live, serving more as a production quality demonstrator.

The test plan for the EFT management portal is quite a bit more involved than the one for my grandmother’s website. Certainly, “trusting the team” will not fly with management. In fact, management has hired an independent risk management company to verify that testing will be sufficient, and that all risks are being managed. This was insisted on by the boards of two of the primary banking partners. This risk management team wants to meet with the program manager and team leads next week to discuss the risk team’s information needs, which will certainly including knowing what the test plan is.

I think that the testing approach for my grandmother’s website will be quite a bit different from the one for the EFT management portal – don’t you?

What is the whole point of testing?

There are alternatives to testing. For example, the IBM “clean room” method substitutes careful review of source code, and its effectiveness is reported to be comparable to other forms of testing. From a Software Engineering Institute report,

Improvements of 10-20X and more over baseline performance have been achieved. Some Cleanroom-developed systems have experienced no errors whatsoever in field use. For example, IBM developed an embedded, real-time, bus architecture, multiple-processor device controller product that experienced no errors in two years use at over 300 customer locations.

The goal is therefore not to test, but to achieve a sufficient level of assurance that the code does what the customer needs it to do. I said “need” instead of “want” because an important aspect of Agile methods is to help the customer to discover what they actually need, through early exposure to working software, which constitutes a form of exploratory testing. There has also been a great deal of recent work in attempts to apply formal methods to Agile. This is especially valuable in high assurance systems such as aircraft, medical devices, and systems with very large capital costs such as the microcode that executes in mass produced hardware. Formal methods are also of great interest for mass produced consumer devices - especially security critical devices such as routers, for which people do not often update the firmware. As the “Internet Of Things” (IOT) becomes reality, formal methods are being considered as a way to increase the reliability of the myriad devices that will be operating throughout our environment – things will be quite a mess if all of those devices are buggy, require constant updates, and all present security risks.

The goal is not to test, but to achieve a

sufficient level of assurance.

The goal is therefore assurance. Testing is not a goal by itself. The question is, what are the Agile practices that help us to build software with sufficient assurance that it meets the need, including the various needs related to dependability? If you go to your bank’s website and transfer money from one account to another, do you expect to have a very high confidence that the money will not simply disappear from your accounts? If you make a purchase on the Internet, do you expect to have high confidence that the purchase will actually be fulfilled after your credit card has been charged? Thus, assurance is important even for the common activities that everyone does every day. Reliability affects a company’s reputation. When you code, do you think in terms of the required reliability of the end product?

Planning is important in Agile!

First of all, we need to dispel the notion that creating a plan is not Agile. As Eisenhower said, “Plans are worthless, but planning is everything.” Implicit in this is that creating plans is important – that is what planning does (creates a plan), but plans always change – that’s why plans are “worthless” – because they end up needing to be changed. Of course, saying that they are “worthless” is hyperbole: it would be more accurate to say that they are only a starting point. Plans are simply the recorded outcomes of a planning meeting – the decisions that were made, and any models that were used. The planning meeting is what is most important, and the plan – the recorded outcomes – is important too, because people forget.

An Agile test plan is not externally driven –

it is internally driven, by the team.

One of the main Agile push-backs on test planning is that a traditional (waterfall) test plan is something created ahead of time by someone who is not on the development team. That is a non-starter for Agile projects, because Agile teams need to decide how they do their work, and testing is a central element of software development. Thus, an Agile test plan is not externally driven – it is internally driven, by the team. It is their test plan. It is the output of their discussions on how they plan to test the application.

That said, other stakeholders should be present for those discussions – stakeholders such as Security, Architecture, or whatever external parties exist who have an interest in the quality of the delivered product. Further, those external parties have every right to form their own opinions on the efficacy of the test plan – that is, whether the tests will actually prove that the requirements of the stakeholders are covered.

The question is not whether to plan, but what an Agile test plan and Agile test planning process really looks like. Brett Maytom, manager of the LinkedIn Scrum Practitioners group, has said in a discussion in that group,

“The bigger questions are, when to prepare it, how to represent it, how far in advance is the plan, who creates it, in what format is the test plan.”

Shuchi Singla, an Agile QA practitioner, commented in that same discussion,

“…we do need test plans, but unlike traditional formats they should be quick reference points. Rather than having 50 page bulky document a crisp sticky or may be white board information like RACI, platforms would suffice…we cannot let test plans just go. They have their importance and should stay though in KISS layout.”

There is a problem though: the very phrase “test plan” conjures up memories of waterfall test plans: large documents with all of the details, right down to the test scripts to be used – and with little input from the developers, and not able to evolve during development. We don’t want to do that – not anything close. As Shuchi Singla said, we want our test plan to be lightweight: we want our test plans to be test strategies. So from now on, we will refer to it as a test strategy. The strategy’s written form should also be highly maintainable – such as a set of wiki pages – so that it can be kept up to date as the team’s decisions about testing evolve over time. An Agile test strategy is always being refined and adjusted as the team learns more and more about the application and the things that actually need to be tested.

The very phrase “test plan” conjures up

memories of waterfall test plans, so from now on,

we will refer to it as a test strategy.

In collaborating with a colleague Huett Landry, one of us (Cliff) found it to be very effective to have a test strategy consisting primarily of a table, with columns of essentially “What, Why, Where, When, How/Who, Coverage”. In a recent LinkedIn discussion, Eugene Joseph, an Engineering Program Manager at Lockheed Martin, said,

“We had a Sys Admin team that was constantly being pulled in different directions by agile teams. We stood up a Kanban board a prioritized task that we put on the board. Someone will have to prioritize the task across your agile teams. One key thing we did was made sure that tasks entered had a clear specification of what, where, when, who, and how support was provided and a clear definition of done. This helped reduce communication delay since the client needing support CLEARLY specified what the needed.”

Thus, if you have existing silos, establishing a way of interacting with them – perhaps documented in a table that lists who will do what, when, and how – is especially important for enabling the development team to operate in an Agile manner.

If you have existing silos, establishing a way of

interacting with is especially important.

A diagram is also very useful for identifying the up-stream and down-stream dependencies of a development and testing release train. As W. Edwards Deming said, “You can see from a flow diagram who depends on you and whom you can depend on. You can now take joy in your work.” ☺ This enables you to see at a glance the types of testing that occur at each stage of the release train, the environments that each will need (and any contention that might result), and the latency that might occur for each testing activity. After all, continuous delivery is a pipeline, and to treat it as such, it helps to view it as such. Figure 1 shows such a diagram. Note that the various environments are identified, along with the activities or events pertinent to each environment. The horizontal dashed lines are not a data flow – they represent a logical sequence. Many loops are implied in that sequence.

Figure 1: A delivery pipeline (aka “release train”).

Stories that focus on outcomes – not just functionality

One of us (Scott) who has done a-lot of Agile work in the embedded software space, has had great success using an evolutionary approach for exploratory testing. Awareness of the desired outcome – from a functional as well as a reliability perspective – provides the tester with a clarity of focus, so that he or she can push the application or device harder and find more issues. Scott has found this approach to be very effective as a general paradigm: creating a focus on desirable outcomes, from the perspective of both functional and non-functional requirements.

Develop the testing strategy as an

output of story development.

To implement this approach, story development should include analysts, developers and testers: develop the testing strategy as an output of story development. Some of the strategies end up as rows in the test strategy table, but others might be specific to a story. Being able to read specifically how something is going to be tested at the story level (because tests are part of the story detail) helps implementers of the story focus on the outcome that the user needs. The story should mention the types of non-functional testing that will occur. That way, the developer has the non-functional requirements at the top of their mind when they implement the story.

End Of Part 1

In the next article we will talk about the various kinds of testing and how to make sure that they occur – in an Agile manner!

Authors (alphabetically):
Scott Barnes
Cliff Berg

An Agile Transformation Thanksgiving

How to become less agile through eating too much on Thanksgiving!

Transformation: Converting your waistline from loose casual slim and trim to post-holiday tightness and regret! This will make you less agile!

Self-organization: Everyone gather in the kitchen, and just start cooking! (The more cooks, the better!)

Planning Session: When planning the Thanksgiving meal, everyone gathers in the kitchen and talks through each dish they want. There are too many dishes, so write each dish on a post-it note, affix them all to the fridge, re-order them, stand back and make them all anyway!

Standup (before the meal): Once every hour, everyone stops what they are doing (hopefully cooking) and explains what they are cooking and which wine goes best with their dish. Now take a drink of that wine. Since you are a team, salute the person talking and everyone take a drink. If someone misses the drink, start over. Keep drinking until you get it right.

Standup (after the meal): Once every hour (unless you are asleep), everyone stops what they are doing (probably watching football), get up from your recliner (if you can) and go back to the kitchen. Tell each person (who manages to get there) what you’re going to eat next as you make your rounds.

Taste-driven cuisine (TDC): Taste the dish before you even make it! Pretend to dip your spoon into it, and bring it to your mouth, and prepare to go “ahh!” – but only to find the spoon empty. Keep doing that as you cook until you really do say “ahh!!!”

Crumb master: the dog gets everything that gets dropped! He’s always around, sniffing here and sniffing there, poking his nose everywhere, but doesn’t really do anything! (except provide an essential glue that holds the family together!)

Culture challenges: Get ready for making smalltalk with relatives who you see once a year. And be prepared to listen hard to those who you have trouble understanding, or who spit when they talk, and smile! Drink lots of wine – that helps!

Continuous delivery: “George – bring out the first turkey!”

Sprint 0: The “idea” you have that not eating before this meal will let you eat more. The more time you spend not eating the more you will be able to eat. Also, the amount of running that you can do after dinner (zero).

Executive leadership: Mom is in charge. There is no servant leadership here!

Dealing with interruptions: The dog just won’t stop barking! The doorbell keeps ringing! Kids keep running through the kitchen! “Oh no! – I just poured orange juice into the cake mix! – hey, that might be good!”

Automated deployment (pre-production): It’s 2014, and the guys still just sit at the table while the women serve them. ;-(

Automated Deployment (post-production): The rush of air felt when dinner is over and the mass exodus to the living room occurs.

Refactoring: When you run out of wine and have to switch to beer!

Sustainable pace: Eat slowly, or you will get full before the meal is over. Make sure you leave room for that final “Lardening Sprint” – dessert!

Retrospective: “I can’t believe I ate the whole thing! And now it’s time to clean up – gosh, now I wish we had used fewer pots!”

What are your Agile Transformation Thanksgiving ideas? (add as comments)

Transformed! - Not very agile anymore

Modes Of Leadership In Agile Transformation

“There are spear chuckers and spear carriers.”

That is what a friend of mine once told me. He had been captain of a nuclear submarine, and so he had a very well developed sense of what leadership was – at least in his world.

The metaphor bothered me, because I am not a spear chucker or a spear carrier. I told him that I was a spear maker. He gave me a blank stare: in his world, the spear makers are military contractors, and they do their work pretty much out of site – before anyone boards the submarine.

From http://arthistorysummerize.info/page/2/?s=stone+age

Software programmers are spear makers. We are not thought of as leaders. Yet Agile requires us to be leaders: Agile does not work unless team members show leadership. More on that later.

At the organization level, Agile transformation requires leadership as well. The notion that “executive leadership is necessary” is well established. And in the Agile spirit, it implies lots of self organization. But what does that look like? What types of leadership are in play?

Leadership context

Two important contexts of leadership are (1) individual/self, and (2) team/organization. Individual leadership is about personal development. Team leadership is about one’s interactions within a team. I will defer talking about individual leadership until the end of this article.

Team leadership context

If one has a position of explicit authority within a team – e.g., you are the team’s “boss” – then it is useful to talk about the relationships between the team leader and the subordinates. Leader-Member Exchange Theory (LMX Theory) is a widely used model for this. Relationships among peers within the team is also an important element. LMX Theory claims that an “in group” tends to form within a team. The “in group” is the portion of the team that the leader trusts more than the other team members. Getting into the “in group” requires demonstrated loyalty.

Leadership within a team does not always
take the form of explicit authority.

On the other hand, leadership within a team does not always take the form of explicit authority. One can play a leadership role by winning the confidence of others in the team. Further, there can be multiple leaders within a team.

Leadership in a hierarchy has other dimensions: the upward and sideways dimensions. If one has a leadership role within a team, one sometimes might have to represent the team to one’s boss, and to other parts of the organization. This time though, one is representing a team – not merely oneself. The table breaks all this down.

Table 1: Contexts of leadership

Individual context: 1.    Inward focused: Individual leadership (personal context). Team or organization context (LMX Theory): 1.    Inward focused team leadership: managing subordinates. 2.    Outward focused leadership: representing oneself and one’s team to peers and superiors.

The lines between these modes of leadership are actually very blurry. For example, what constitutes a “subordinate”? If a team has an assigned leader, with explicit authority, then that person is clearly a team leader and the others are subordinates. However, there might be other leaders on the team who have earned a de facto leadership position by demonstrating good ideas, or simply by being persuasive. Leadership is often informal. The official leader might even find him or herself competing with informal leaders, putting the official leader into the position of having to continually demonstrate outward focused leadership to maintain his or her credibility to the team. Forming an “in group” is a tactic that enables a leader to hold onto their authority, by creating a buffer of “informants” and loyal supporters. On the other hand, informal leaders can arise within a group and challenge the official leader. Informal leadership is a source of “cronyism” and corruption in organizations, but can also be a source of great strength.

All three of these leadership contexts are important for Agile transformation, as we will see.

Leadership styles

By “style”, I am referring to the way that you exercise leadership, e.g., by being a bully, or by “killing them with kindness” – or perhaps by simply being the person with the most actionable and understandable ideas. Some schools of thought distinguish leadership styles as “modes”, reflecting that an individual can assume different modes depending on the situation. This is sometimes called situational leadership.

There are limitless styles of leadership – as many as there are leaders. However, a particularly good taxonomy of styles is described in this article: 8 Common Leadership Styles. I summarize some of those styles in the table, and add a few.

Table 2: Leadership styles

Charismatic – Inspires. May be egotistical. People follow because they trust and believe. Examples: Steve Jobs. Adolf Hitler. Innovative – Able to capitalize on situations. Leads by creating opportunities for others. Example: Elon Musk. Command and Control – Intensely goal focused and demanding. Leads by explicit authority. Example: a military commander in a live mission situation. Laissez-Faire – Sets an example, and advocates. Leads through personal credibility attained through prior achievements, reputation, or status. Example: Lawrence Lessig. Pace Setter – Creates a high pressure environment of high expectations. Leads through setting an example of working extremely hard. Example: Ross Perot. Servant – Works by enabling others. Leads through explicit authority, but exercises it only by facilitating collaboration and removing obstacles. Example: a Scrum Master. Transformational – Creates change at a fundamental level, enabling “breakthrough” change to create behavioral patterns that are completely new to those involved. Leads through helping others to understand themselves better and understand their situation better. Example: a group therapist. Political – Adept at negotiating win/wins with others. Leads through identifying – or creating – mutually beneficial situations with others. Example: any successful politician. Competitive – Sees everything in terms of a zero-sum game. Leads through aggressively out-thinking others. Example: managers who compete for budget in an organization. Collaborative – Tries to find the solution that is best for everyone as a whole, and involves others to do so. Leads through arranging discussion forums, establishing new paths for communication, and creating a large social network. Example: the authors of the Agile Manifesto. Rule oriented – Needs explicit authority to lead, and focuses on having clear boundaries defined for who is supposed to make each type of decision. Leads through having authority granted by an accepted source of authority. Example: a police officer.

Again, these styles are not definitive any more than a list of personality types is definitive: these are only named generalizations to help us to discuss approaches to leadership. You might be able to think of some styles to add to the list. In practice, people use a combination of styles with their own unique personality traits thrown in.

Team Leadership

Inward Focused

Inward focused team leadership involves the interactions between the leader and the team. In other words, it is about the internal dynamics of the team – not the rest of the organization or the outside world.

Inward focused leadership tends to be relatively invisible outside the team except in its outcomes, unless there is such a severe problem that team members revolt.

Inward focused leadership is the mode of leadership that is most often discussed in articles on team leadership, especially in the Agile community, which has had an inward team focus from the beginning.

Transformational approach

Good Agile coaches use transformational leadership. IT executives who want to transform their organization should also use transformational leadership. This entails viewing things as a system, performing introspection on the causes and effects, and changing things in a strategic way so that the desired new behaviors are inevitable and sustainable. A very useful tool for this purpose is constraint analysis: What are the things that are preventing movement toward Agile? Then, consider the causes of those things, perhaps using the Five Whys method, and remove those constraints. And do this not on your own, but by leveraging the thoughts and ideas of the people in your organization.

One of the goals of strategic transformational
leadership is to convert strategic decisions
into repeatable tactical decisions.

One of the goals of strategic transformational leadership is to convert strategic decisions into repeatable tactical decisions that collectively advance the organization towards the strategic changes that are desired. For example, instituting Agile coaches to help teams is a repeatable tactical practice that will have a strategic impact over time.

Transformational leadership is so important for Agile transformation that we will have a series of articles about it.

Servant approach

Scrum Masters use servant leadership. A servant leader is someone who leads softly, by educating, by suggesting, by facilitating, and by enabling. A servant leader sees it as his or her job to remove obstacles for the team, to instruct the team, by advise the team, and to encourage the team to identify and discuss important issues and reach decisions.

Servant leadership is not an abdication of authority.

On the other hand, servant leadership is not an abdication of authority. As James Hunter puts it in his book The Servant, “The leader should never settle for mediocrity or second best – people have a need to be pushed to be the best they can be. It may not be what they want, but the leader should always be more concerned with needs than with wants.”¹

For example, a servant leader might propose – might even insist on – certain collaborative routines (such as standups and the discussions that should follow), or assessment practices (such as measuring team velocity), if the leader thinks that that is what the team needs to become more effective. In other words, the leader needs to exercise judgment. There might be teams that are very innovative and can come up with even better practices for achieving the goals, and a servant leader should have an open mind, and when in doubt, lean towards deferring to the team. That is not an absolute though.

Command and control approach

This approach is appropriate for situations in which unexpected things can occur and it is difficult to have discussion to collaboratively decide what to do. For example, in a situation of urgency, such as fighting a fire or a military operation, direct command and control is essential. When seconds count, there is no time to explain a decision or ask for ideas: immediate coordinated action is needed.

Collaboration is extremely important for military operations, but the collaboration occurs before and after the operation – not during. For example, Steve Crago – a former US Marine Corps member and now an Agile coach – says,

“What Scrum would call a Retrospective, the military labeled an After Action Report (AAR). AARs were conducted after every field exercise and/or mission. The difference between an AAR and a Retrospective are varied. AAR’s have a prescribed format and are usually attended by the leadership… each team would brainstorm ways in which to perform at a higher level. We spent hours and hours looking at various ways to do something; and our more experienced leaders at the Team, Squad, and Platoon level were continually showing us how to do something they had learned on previous missions and/or exercises. As I moved up the ranks, I added what I had learned experientially to the knowledge base that was imparted to me; and just like those who had gone before me, I began to train others to find a more efficient, effective way of performing tasks.”

Thus, it is often the case that different leadership styles are appropriate at different times. After a mission, a military leader transitions from command and control to a more collaborative approach. If you are a parent, you probably make these transitions all the time: you might generally want to be mostly a servant leader with your child, but if the child is in danger, you transition to a command mode and direct them what to do, and once the danger has passed, you might transition back and have a “retrospective” to discuss the dangerous situation with them and learn from it.

Command and control leadership can be very destructive if it is used in the wrong situation. It does not utilize the collective insights of the team, and it can make people feel disempowered and helpless. The “bad boss” is a common problem in organizations – managers who give orders but who do not seek the input of their team, or micro-manage how team members do their work. This causes low morale and poor productivity.

Other approaches

The right approach for leading a team is to do whatever works. If the team members are miserable and are not able to have their best ideas adopted, or are not able to work in ways that are most effective for them, then the team as a whole will not be effective. That said, leaders sometimes have unusual traits that compensate for what would otherwise be destructive behavior. Steve Jobs is a famous example of a leader who was autocratic and abusive, yet he was extremely popular within his company, generating great loyalty, and led Apple to great success – twice.

Charismatic leaders inspire others, creating a
belief that there is a worthwhile goal.

Richard Barrett pointed out to me once that Jobs’ success is not evidence that that bad behavior was not destructive: it is possible that if Jobs had been less abusive, then his organization might have been even more successful, or successful sooner. We cannot know. What we can know is that Jobs’ negative behaviors did not prevent success. So what was it about him that generated such a loyal following? Jobs was an example of a charismatic or inspirational leader. Charismatic leaders inspire others, creating a belief that there is a worthwhile goal. If people are committed to a goal, they will tolerate a leader’s quirks. Still, it might be that if those quirks were overcome, that people would be able to perform even better, so even charismatic leaders can have room for improvement.

You might also find that different approaches work at different stages of a person’s or team’s development. When my son was in his early teens, I found to my great disappointment that he only listened when I exhibited anger, so I had to present an attitude of anger to get his attention. That was unique to him. As he grew older that changed, and I was able to use calm reason and sensitivity instead, which was much more pleasurable. A leader needs to be sensitive to what is effective, given the personalities and capabilities of the team, and where they are in their development. This adaptive approach is central to the shuhari method, in which an autocratic leader demands that the student follow instruction and rituals rigidly but once the student has reached the Ha stage, the instructor also engages in introspective discussion with the student. Whether this rigid approach is appropriate for a software team is debatable, but it could be that it is the best approach for certain things, such as when particularly sensitive software modules must be built.

Outward Focused

Outward focused team leadership is about interactions with your environment. If you are a member of a team, but not its designated leader, then your outward focused leadership is about the way that you represent yourself to the other team members. If you lead a team, outward focused leadership is about representing and advocating for the team to the rest of the organization. It includes creating the team in the first place and defining its purpose (“charter”) and scope of action. It includes representing the team in discussions for resources and authority to act, and whatever else is needed to enable the team to perform its purpose.

Outward focused leadership is the mode of leadership that articles about business often emphasize because it is market facing, board facing, and it exhibits power – which is attention-getting. This article is a great example. From the article:

“The best human beings are collaborative, compassionate, empathetic and free of most defects of character. But the best leaders usually are not. By “best leaders,” I’m talking mainly about people who consistently show the ability to get things done—the ability to sell others on an idea, the ability to take them in new directions, the ability to talk their way out of a jam, the ability to come back from a setback, and so on.”

In terms of LMX Theory, outward focused leadership is about your relationships with your superiors and your peers.

Adapt to the situation

Servant leaders often have to take on a different leadership style when performing outward focused leadership. For example, if the team’s project competes with other projects for budget resources, the team’s leader needs to be somewhat aggressive on behalf of the team to ensure that its funding continues. The leadership styles that are potentially effective in that situation are charismatic, innovative, political, competitive, collaborative, and rule oriented. However, the styles that are likely to be most effective are charismatic (if you can pull it off), innovative (if you truly are), political (if you have the knack and like to schmooze), and collaborative (if you are very persuasive).

The right approach depends on your own
abilities as a leader.

So the right approach depends not only on the situation, but also on your own abilities as a leader. Scott Beilke, a principal consultant at Brighton Leadership Group, puts it this way:

“A Sweet Spot exists at the intersection of three areas of [leader’s skills and commitment to change, what change is needed, and the organization’s culture] for any business transformation (Change). It is critical for leaders to get context clarity on these areas to make the difference between a resounding successful change and a crashing failure.”

Team leaders also need to represent the team to a community. When Steve Jobs would speak about upcoming products, he was representing Apple to the world. His “One more thing” announcements were commitments about what Apple would be doing – exciting products to come. In that situation, it helps to sound authoritative, and to put on a charismatic, and possibly even a little command and control, attitude, since your objective is to sound authoritative.

Leaders need to represent their teams in a more mundane context as well. If a senior manager wants to know how the team is doing, the team leader is available to explain the progress and challenges for the team – sometimes information radiators are not enough. Managers like to have a point person with whom to discuss issues one-on-one, in order to circumvent politics and be frank: i.e., to have “powerful conversations”. We have to remember that while Agile promotes transparency, it is unrealistic to think that politics does not operate in all organizations – even Agile ones – and one-on-one discussions are essential for managers to be able to work together. To enable that, teams need leaders who can speak for the team.

Leading without authority

Personal outward focused leadership is very important for Agile teams because Agile relies on a large amount of self organization – i.e., no one is going to tell you what to do at every step. Some Agile teams do not have a designated leader who has actual authority within the team: the team is expected to identify problems, discuss them, and take action. Everyone needs to exhibit leadership for this to work optimally. If only a few people on a team exhibit leadership, then they will have an unfair burden and will not be able to spend as much time as they should on their own work, and also the team will very possibly miss many issues that need resolution.

Informal leadership can be constructive or destructive.

Informal leadership can be constructive or destructive. If a team member is able to cause others to trust him or her, that trust can be misused. In Jim Collins’ latest book, From Good To Great, he states that, “the first job of a new leader is to get the right people in the right seats.”²  The sources of group dysfunction are myriad. Also, groups tend to go through a series of stages in which power structures compete and eventually stabilize – and not always in a good way. In terms of the Tuckman model which is widely quoted in the Agile community, the team can end up in a “performing” state that is far from optimal. According to the Wikipedia article on the Tuckman model,

[The Storming] phase can become destructive to the team and will lower motivation if allowed to get out of control. Some teams will never develop past this stage. Supervisors of the team during this phase may be more accessible, but tend to remain directive in their guidance of decision-making and professional behavior. The team members will therefore resolve their differences and members will be able to participate with one another more comfortably.

In other words, a self-organizing group is not guaranteed to reach a healthy Performing stage without intervention by management.

Extroverts and introverts

People who are extroverts tend to naturally exhibit leadership in teams. People tend to like extroverts and develop trust for them. Introverts have a more difficult time establishing trust because they interact less, and often interact less effectively. Thus, introverts can sometimes find it difficult to exhibit informal team leadership, even if they have ideas that are worth listening to. This is why many teams need an explicit leader, to ensure that everyone on the team is able to be heard, contribute, and exhibit leadership.

This is especially important for Agile because IT people tend to be more introverted than the normal population. It is fascinating that in surveys in which IT people self-assess, they report that they are extroverted; but in objective surveys, IT people are actually introverted as a group. From a 2002 paper in the International Journal of Human-Computer Studies, Personality Types In Software Engineering,

“The personality type most prominent is a combination of introversion, sensing, thinking and judging. ISTJs assume responsibility readily and tend to persevere. From the data it can be deduced that the majority of software engineers (ISTJ) are technically oriented and prefers working with facts and reason rather than with people.”

Thus, as a group, software developers are introverts but do not know it!

Many introverts view Agile with some anxiety. From this blog post,

“…pair programming, bullpens, stand-ups, and scrums. We talk about close collaboration, about embedded customers… Who wouldn’t want to use such a system? Well… introverts.”

We need to consider this when thinking about the ways that Agile teams will self organize and be effective. In particular, if there are some extroverts on a team, we need to make sure that they do not dominate the group. We also need to make sure that discussions are occurring in a way that is effective for everyone, because introverts and extroverts think and discuss issues differently. Simply putting them all together around a whiteboard is not always the most effective approach.

Individual Leadership

This is the type of leadership that you exhibit yourself when you take the initiative to improve yourself. If you realize that you need to learn a new skill, do you set out to do so, or just think about it? Do you reflect on your abilities and personality traits, in the interest of personal growth? Do you listen carefully to others when they comment on your ideas or your character? These are all elements of individual leadership.

Individual leadership is extremely valuable for a happy, successful and productive life – no matter what profession you are in. As Richard Barrett says,

“Organizational transformation begins with the personal transformation of the leaders...The chief executive officer (CEO) or the leader of the organization must be willing and committed to his or her own personal transformation to change the culture. The leaders must live the change they want to see in the culture of the organization. For each member of the leadership group, personal alignment to the organization’s values and group cohesion around the vision, mission, values, and behaviors must become part of his or her personal journey.” ³

Taking the initiative to learn is part of individual leadership. Agile organizations need to have people who are continuous learners. Gone are the days when you learned a skill and used that skill for the duration of your career. Nowadays – at least in IT – skills change every single year. Today’s tools seem like the greatest thing, but three years from now only half them are still in use. Ways of working change too – Agile is an example of that. People can no longer wait to be trained: they have to take the initiative, and this is never-ending.

Ironically, the foundational skills that are durable are the most valuable, but in the IT profession companies do not usually seek those: things like experience with object oriented design, experience building secure and maintainable systems, experience with design patterns, experience at solving problems and learning new things – all those things will get skipped over on a resume. Recruiters scan for the current tools – today they are NodeJS, Cucumber, and whatever – tomorrow they are something else entirely. The truth is, programmers need to continually learn new things in order to stay relevant – they are accustomed to this. Being proactive in learning new things is a form of self leadership. Agile expects it, demands it.

In a future post I will look at the situations in an Agile transformation and the kinds of leadership that they require.


1. Page 66, “The Servant: A Simple Story About the True Essence Of Leadership”, by James Hunter, © 1998, published by Crown Business. 
2. Jim Collins. Good to Great. New York: HarperCollins, 2001. 
3. Barrett, Richard (2006-08-14). Building a Values-Driven Organization (Kindle Location 1143-1156). Taylor and Francis. Kindle Edition.