Interview with Madhur Kathuria

Madhur Kathuria has coached nearly 300 teams for almost 75 clients across the US, Europe, South East Asia, Malaysia and Thailand. In this interview he talks about some of the cultural challenges for agile adoption. Read it here.

Interview with Elena Yatzeck

Elena was Chief Agilist for JP Morgan Chase Treasury Services and is now a VP of Corporate Compliance Tech. Find out how JP Morgan Chase reconciles agile with compliance and risk management demands. Read it here.

Friday, October 24, 2014

What To Do About Distributed Teams

A recent client of of one of us had a dilemma: their procurement department had selected the lowest cost bidder on a software contract, and that bidder was located on the other side of the country (USA). When work on the “agile” project started, the kickoff consisted of a conference call.

It is a real problem that procurement is viewed as a separate function, apart from how the software development team will operate. Fixing that requires going way, way up in the organization – beyond the CIO – and if the organization is a subdivision of a large company, or a government agency, addressing the issue possibly requires going even beyond the organization itself to the owning organization, which sets procurement policies.

We don’t want to get into procurement here: that is not the topic of this article (it will be a future topic). Instead, we want to look at the very common, real, and unchangeable situation that we sometimes face: when a team – or worse, only part of a team – is remote.

Is it really a bad thing?

Not too long ago, Jeff Sutherland – the “father” of Scrum – published a paper on a software project that was highly distributed, which he claimed had been one of highest productivity distributed software projects ever, with a productivity comparable to that of a small co-located Scrum team. Some noteworthy aspects of the project’s organization included:

1.    There were multiple teams, spread across six geographic locations.
2.    Teams were split across the functional areas of the requirements.
3.    Team members were purposefully spread across geographic boundaries: thus, for any given team, the members were distributed. Any member of a team, at any location, can work on any item on that team’s backlog.
4.    The information to be shared in a team’s standup was written up before the standup and distributed to the team’s members. This helped to overcome communication challenges over long distance phone connections, as well as language barriers.
5.    Standups were done in real time, despite a fourteen hour time zone difference.
6.    Each team had a Product Owner assigned to it. All Product Owners were co-located, and met daily, and there was a “chief” Product Owner.
7.    All Scrum Masters were co-located, there was a “chief” Scrum Master, and there was a “Scrum of Scrums” meeting every Monday. The Scrum Masters were also co-located with the Product Owners.
8.    An Architecture group, also located with the Scrum Masters, met on Monday after the Scrum of Scrums meeting and controlled the direction of the project architecture through the Scrum meetings.
9.    A co-located Automation Test team created scripts for an automated testing tool.
10.    Tests were written simultaneously with code most of the time.
11.    During a Sprint, the Product Owner tested features that were in the Sprint backlog – rather than waiting to the end of the Sprint.
12.    A shared code repository, used by all teams.
13.    Integration of eXtreme Programming (XP) practices, to augment Scrum practices.
14.    Some continuous integration practices were used, such as hourly automated builds, but there was a ways to go in this, especially with regard to test automation for acceptance tests.

One of the reasons cited for the project’s high productivity was the caliber of the teams – made possible by the move to a distributed approach. This enabled the selection of a remote group in Russia, StarSoft, that had extensive experience with systems level software as well as experience using XP, rather than trying to use the best talent that could be found locally. According to Jeff Sutherland,
“The local talent pool is not sufficient to expand team size and salary costs are much higher than outsourced teams...The primary driver is enhanced technical capability resulting in dramatically improved throughput of new application functionality.”
In other words, using a distributed model enabled the project to recruit the best talent, rather than relying on what was available locally.

In contrast to this, many organizations choose remote teams in order to get the lowest cost. That can potentially undermine success. Instead, it is wiser to find the right skills, whether they are local or remote. Looking at a wider geography can enable you to find the right resources more easily. The cost objective is self-defeating if the project is not successful.

Co-location of “thought leadership”

The practice used by this project that is probably the most controversial for the agile community is that Scrum Masters were all located together, as were the team architects, and the Product Owners. There is a general feeling in the agile community that leaders should be no more than facilitators, and so leaders need to be present in person to be able to facilitate. This was not possible in this project, due to the fact that each team was distributed. For the Product Owners, putting them all together is less controversial: agile teams are accustomed to this; but putting all of the Scrum Masters together, and the architects as well, is controversial because doing this effectively creates a collaborative forum that excludes the team members. Agile relies on collaboration, and having an elite “architect” group collaborate privately is viewed as not very agile. In fact, the Scrum community does not even approve of having a role such as “architect”, which is ironic because Jeff Sutherland – the author of the article – seems to feel that in this instance it was fine.

Based on our own experience with both architects and agile teams, it is essential that an architect or technical lead work in a collaborative way. The kind of architect role that is effective for agile teams is one that does not base his or her work around documents, but rather discusses issues with the teams directly, as well as with other stakeholders such as Security, Release Management, the data center or cloud service group (if there is one), and so on. Architects can play a valuable role in that they (1) are aware of the organization’s larger architecture strategies, (2) are charged with thinking more long term and more end-to-end, and can do this throughout a project because they are not saddled with creating stories, (3) have (or should have) expertise in a wide range of aspects of IT beyond software development, including system architecture, scalability, security, maintainability, systems thinking, and design patterns. Certainly programmers should know these things, but they often do not, and an architect – if sufficiently qualified – can serve as a valuable mentor for the team.

The last part is important: an agile architect should be a mentor – not an autocratic decision-maker. Issues should be discussed with the team, rather than decided independently, and the architect should serve as a coach, teaching the team about architecture concerns. For a distributed team, in which all of the architects are located together, this is even more important, because the architect is less accessible physically, and the co-location of the architects makes it inevitable that they will collaborate privately – that is in fact an advantage if there are multiple architects, but it must be ensured that the architects also collaborate with their teams. The “chief” architect, as Jeff Sutherland referred to the role, should make sure that this occurs: the chief architect is also a coach for the architect team and must ensure that they work in a way that is compatible with agile practices. The same applies to the “chief” Scrum Master, and the “chief” Product Owner.

Table 1: Roles and their value for the fully distributed team cited.
RoleWhere LocatedValue of Role For Fully Distributed Teams
Scrum Master or Team LeadAll at central locationEnsure that remote communication practices are effective.
Product OwnerAll at central location(same as for any Scrum team)
ArchitectAll at central locationEnsure architectural consistency.
TesterAll at central locationManual testing; automated test script development; stress testing.
Chief Scrum MasterCentral locationLead the Scrum-Of-Scrums meetings.
Chief Product OwnerCentral locationLead daily Product Owner meetings.
Team MemberAt each location(same as for any Scrum team)
Tech LeadAt each locationCoordinate technical issues across teams.
Test LeadCentral locationCoordinate testing activities.

According to Jeff Sutherland, “SirsiDynix achieved strong central control of teams distributed across geographies by centrally locating ScrumMasters, Product Owners, and Architects. This enabled them to get consistent performance across all distributed teams.”

It should be noted that some of the lessons here conflict with sentiment in some parts of the agile community. The Scrum community in particular is uncomfortable with having specific lead roles such as “architect”, or indeed any “chief XYZ”. Even the concept of “control”, as expressed in the above quote by Jeff Sutherland, is objectionable to much of the Scrum community – which is ironic given Sutherland’s role in the creation of Scrum. The intention in this article is not to take a general position on these points, but to share what we have known to work given the unique challenges of a distributed team, recognizing that there might be other approaches that could also work. Effective leaders often are able to make any approach work: the leadership is what makes the difference.

Use of remote collaboration tools

Many organizations use remote meeting tools such as GoToMeeting, WebEx, Skype, and so on. These tools are really nice, in that they make it simple to set up a conference call and share screens, but they do not really solve the challenges with remote meetings. It is still hard to hear and understand people. When people are remote, they often feel that it is ok to join late, or they call in from their car (not ok if they need to see a shared screen or look at documents), or they multi-task, or they have background noise (another conversation going on in their cubicle), and so on – all of these distractions and interruptions can destroy the cadence and cohesion of a meeting: people start to emotionally disconnect. Here is a hilarious comedy portrayal of this. Preventing these kinds of interruptions requires very strong meeting management – a skill that most agile coaches do not have, since they are accustomed to working with teams in person.

One of the greatest drawbacks to a remote meeting is the inability to whiteboard. Many people think visually. Most agile coaches are accustomed to writing on a board, or putting up sticky notes, or conducting a “dot vote”. These visual techniques cannot be done easily on an ordinary computer screen, although there are touch screens that are available for this purpose, e.g, the “FlexTouch”. In addition, for a technical meeting, it is almost always necessary to draw on a whiteboard, and the drawings can become very expansive – spanning multiple boards. This is not easy to replicate on a computer screen, although that is changing with the advent of very large “4K” screens. Even so, one needs to be able to draw on the screen.

Given these challenges, a remote meeting must be facilitated very differently from an in-person meeting. Careful thought must be given to how diagramming will be done, if it is needed, and preparations and discipline must be applied to enable people to hear each other. When we facilitate a remote meeting, and someone is not easily understood, we ask them to speak up, or we repeat what they said to confirm that we understood it correctly. In the project described by Jeff Sutherland, team members were required to submit their standup statements in writing prior to the standup, enabling everyone to read it if they cannot understand what is being said. Coaches need to use these techniques. The meeting’s coach or facilitator also needs to set things up before the meeting, since it takes a few minutes (sometimes longer) to overcome technical glitches.

Meet in person at the beginning?

It is really essential to have the entire team meet in person for a few days – a week is recommended – at the beginning of a project so that they can collaborate most effectively on the key “conceptual pillars” of the project: the core vision and requirements, the core design concept, the team roles and collaborative processes, the development pipeline, and the testing strategies. (See this prior post.) These are all very complex issues and require lots of whiteboarding and active sharing of ideas. Doing this remotely will take weeks, and it would be difficult to resolve all of these things in a collaborative way by doing it remotely: one would likely have to fall back to the non-agile approach of having someone dictate much of this. Establishing the key conceptual pillars is extremely important to ensure that risks can be managed well and that things will go smoothly and evolve, instead of having radical and disruptive conceptual changes mid-stream.

How to go through stories remotely?

Agile replaces the traditional practice of writing a detailed requirements document with a more fluid real-time process of having stakeholders describe what they want to developers in person, noting those requirements down as brief “stories”, and then adding details to those, primarily in the form of acceptance criteria. Developers are encouraged to reach out to stakeholders as they program to ask for clarifications, and stakeholders are encouraged to try out the software as soon as a developer thinks that it is complete, followed by a formal demonstration at the end of each iteration. This interactive exchange is more difficult if the stakeholders and developer are not geographically collocated.

In the project cited above, stakeholders wrote requirements down with details ahead of time. This enabled developers to review those requirements before a sprint planning meeting, and then ask clarifying questions. This process seemed to work well for those teams. However, it is mentioned in the article that the remoteness of stakeholders made it necessary for some “multi-tasking” – that is, the developers sometimes had to put a story aside and work on another until they received feedback from a stakeholder about a question.

Regarding trying out software as it is developed, we have seen it work well when remote users try out an application during an iteration by logging in remotely, as soon as a developer thinks that a story is done, rather than waiting until the review at the end of the iteration. The application can be deployed into a test area specifically intended for users to try out stories as they are completed. This reduces the risk that a story will be marked as complete, but then not accepted by the Product Owner (representing the stakeholders) at the end of the iteration. It is important that remote stakeholders have a separate test environment for trying things out, so that they do not interfere in the test environments and disrupt the testing cycle of the team.

How to coach people you can’t see?

One of the greatest challenges for coaching remote teams is that you cannot simply stroll up to people and see how things are going for them. Calling them out of the blue is potentially disruptive, so that leaves email. Asking in an email to schedule a time is cumbersome, and seems “managerial” – something that a coach tries to avoid. We personally recommend going out of one’s way to establish a unique relationship with each team member as soon as possible at the start of a project. Reach out to each person – don’t skip anyone – ask for a time to chat on the phone, and take notes about their perspectives and challenges, because if you can’t see them, it is harder to remember those things and associate them with the person. Primarily ask questions and listen – don’t do the talking. Then, make it your business to remove obstacles for people. Check in on things that you know they are waiting for. Advocate for them. Connect them with people who they have been trying to reach, such as someone in a support function (Testing, Release Management, etc.) from whom they need something. And have a regular, say weekly, chat with each team member about how they feel things are going, on a personal level. Keep their confidence, and be on their side. And if at some point you visit their location in person, make it a point to stop by and chat.

Pair Programming is a great way to increase collaboration in Distributed teams. When teams are distributed in two or more locations, each location team will try to behave as a separate team by itself. They often try to “throw over the wall” to each other. Instead, it is very important to promote One Team behavior within such teams. One technique which we often use is Pair Programming, such that in each Sprint, every developer pairs with a developer at another location during the hours in which their work schedule overlaps. They both work on a single story until it is done, with a handoff from one to the other at the end of each other’s work day, but working together when their work period overlaps. One developer codes the story during the day, and later in the day he pairs up with the other developer, walks him or her through the code that was changed, and checks-in the code before leaving for the day. The other developer checks out the code and resumes coding, and records his findings in a small video (we use the Snagit tool to record) and puts the video in a shared folder which the other developer can access the next day. This way there is a collective code ownership between the team members who are not at the same location.

Some other techniques which we recommend for distributed teams are:
•    Share the pain caused by time zone differences
•    Make sure that everyone has a calendar that covers the regions and cultures present on the team, so that if someone will be out for a holiday, it will not be a surprise.
•    Have a daily or frequently recurring optional virtual meeting for discussing issues that come up during the day. This does not replace the standup, but is an open forum for sharing ideas. This helps to overcome the fact that people are not co-located and cannot overhear each other or just walk around to each other’s desk.
•    Do retrospectives (every iteration) with the entire team
•    Encourage cross-region communication (for example, by using pair programming)
•    Actively share feedback from stakeholders with the whole team
•    Focus on delivering the value stated in the stories

How to participate in ad-hoc discussions?

Another severe challenge for remote coach is that agile relies on ad-hoc discussions for resolving issues that come up. If you are not there in person, then you cannot observe these discussions happening, and so you can’t lean in and hear what is going on. You can become out of the loop on things pretty quickly.

In the project referenced earlier, the fully distributed nature of each team cleverly avoids this problem: each team member is apart from the rest of the team, so discussions have to be arranged. It is therefore the coach’s job to keep reminding the team that he or she wants to be included. Eventually they will get the message and will do so – as long as your participation is positive. Generally, a coach should not interfere in a discussion unless the discussion has become dysfunctional. A coach is first and foremost a facilitator. If a discussion is remote – over the phone, or via email – then a coach can play a valuable role in making sure that the communication works well, that people who are hard to understand speak up, that decisions get written down and disseminated (e.g., on the team’s wiki or – even better – on a wall in each of the remote offices), and that the discussions stay on topic (especially difficult for email exchanges). However, a coach’s presence should not be felt: it should be an enabling presence – not a contributing presence – kind of like the way ceiling lights help everyone to see in a room, but you don’t notice them per se.

Dealing with regional and cultural differences

Cultural differences is a topic in its own right, and we will have a separate article on that. For now, suffice it to say that the most dangerous mistake people make when leading a cross-cultural team is to assume that we’re more alike than we actually are. Some cultures are not accustomed to speaking their mind in a meeting, while others are – especially if there are people present who have a higher position (and a Scrum Master can be seen as such a person). In some cultures, people will not express an opinion unless they have had time to check their facts and do some thinking offline, or they might even consider it to be rude to speak honestly if it would be critical. Some cultures are accustomed to being given explicit work instructions while others welcome autonomy and self direction. Some cultures see their personal time as sacrosanct while others are perfectly willing to work weekends as needed. If a culture is accustomed to long summer holidays, make sure to plan around that. Some cultures value punctuality while others do not.

In addition, do not underestimate the challenges of language. Even when all parties speak the same language, it can be very difficult for people to understand each other when there are multiple dialects. For example, American English, UK English, Singaporean English, and Indian English are all sufficiently different that it can be difficult to communicate verbally in real time, especially if the communication system is imperfect – which is usually the case. Idioms can be especially problematic. The distractions of communication dropouts, coupled with soft speaking and multiple language dialects can all combine to create enough distraction so that people cannot focus on the issues being discussed. If communication is challenged, it is the job of the meeting facilitator to repeat what people say if it was not clear or not loud, and to verify that all parties are following the discussion. Ask people regularly to summarize their understanding and ask people for their opinions. Do not wait for them to speak up. Plan for the extra time that this consumes: it is a cost of having distributed teams.

Contributors (alphabetically):
Cliff Berg
Sekhar Burra
Nancy Settle-Murphy – facilitator, virtual team expert, cross-cultural trainer, OD consultant, and author of Leading Effective Virtual Teams

6 comments:

  1. Gary Toofany (https://www.linkedin.com/pub/gary-toofany/1/a06/634) wrote in LinkedIn (paraphrased):

    Whilst our teams were distributed across offices in the UK, we agreed within the team that we would endeavour to meet up face to face every week, either as a team, or in part, as much as reasonably possible.

    In all cases I ran a virtual stand-up, essentially a conference call with Web-Ex / Lync for sharing visuals, time-boxed to the usual 15 minutes and following the standard rules of thumb.

    For one project, for a large complex program, I ran a monster-scrum meeting surveying the an entire programme of work. Taking up to 1.5 hours, this allowed us to virtually be in the same room, giving each team member visibility of their team-mates workstack, issues, and blockers. It was extremely effective. The monster-scrum meeting was always followed by some minutes.

    I have not found distributed teams a blocker to success. They simply mean you have to think laterally about the social and practical aspects of team working.

    ReplyDelete
  2. Thad Scheer (https://www.linkedin.com/in/thadscheer) wrote on LinkedIn (excerpted):

    Where I see things go south is when some team members are co-located and others are distributed. Under that scenario things don't optimize they way they do on a 100% distributed Open Source project...and they don't optimize around the water cooler like an in-house studio project. You get the worst of both without the best of either.

    ReplyDelete
  3. Richard McCabe (https://www.linkedin.com/pub/richard-mccabe/1/69b/a03) wrote on LinkedIn:

    I've seen some of the worst cases: the team is distributed for cost reasons, and even worse due to availability (not, "We can't get the best people locally" but, "We can't get **any one** locally" or at least, can't get enough people locally). Often, these "teams" have never done agile even as a collocated team, much less as a distributed team. I have wondered whether such a team should even attempt to be agile. But given a charter, I've forged ahead. What I've been working with is an interaction where the local people work as an agile team, but the off site people work more as subcontractors. This can kinda sorta work as long as all the necessary skill sets and knowledge expertise is represented by onsite people who lead (actually manage "piece job" assignments to) the offsite people. It's not so great, but it still seems to be much better than pure waterfall approach.

    ReplyDelete
  4. Alistair Munro (https://www.linkedin.com/in/munroalistair) wrote on LinkedIn:

    I have run a number of agile projects involving globally distributed teams. The most recent was a team of 100 where the BAs sat in Sydney and Hong Kong, gather requirements from SMEs in Sydney, Hong Kong, London, Singapore, New York and Cape Town. The Dev was done in India, Test in Manila, then back to the SME for UAT, per story. It ran well, because a very strong emphasis was placed on high quality written requirements per story. We used Jira for storing the requirement and solution design, as well as workflow for the story lifecycle. Sure, you lose some efficiency in communication, with teleconference standups, but when delivering a system for globally distributed users, you don't really have a choice. So whether agile or not, if you nail the requirements, you at least give yourself a chance of succeeding.

    The solution design was relatively detailed. It was written up by the Business Analyst in consultation with either the Developer who would do the work, or if they were a junior Developer by a more senior Developer. The goal was to have it to a level that we could easily reassign the story to any developer, and there was sufficient detail for them to code to. We had a quality gate, where all solution designs were reviewed by a senior architect, to ensure quality and consistency of solution.

    Although that may seem like a lot of overhead, and not particularly agile, it meant that a high quality solution was maintained, even when the coding was done by remote/distributed developers. And it gave a lot of flexibility to balance workload when the heat was on - stories could be reassigned to wherever there was capacity, even if the developer wasn't involved earlier on in the requirements phase.

    If the team had been co-located in one location, I wouldn't have driven as much detail in the solution design, but with a globally distributed team, it was the price I felt needed to be paid to still maintain quality. Otherwise a lot more issues would have been found in SIT/UAT, resulting in more defects/rework.

    ReplyDelete
  5. Cliff, thanks for your detailed analysis. I find a little too prescriptive (specially because agile is anti-prescriptive by nature), but i take that as a sign of conviction, a good thing to find.

    The reason i decided to comment on your article is because when you point out the problems of "cultural dissonance" as something to careful look for in remote teams, you implicitly fall in the fallacy of assuming that such problem is intrinsically related to remoteness.

    In fact, it is not. In these times of intentional uber-integration of the workplace, with technical jobs *presumably* being in the forefront at handling and even leveraging differences (cultural yes, but also gender-based, character-based, physical and others), data indicates that the reality is very different. Cultural differences are dealt by obliteration, shy workers are dealt with by ignoring them, older workers... well older workers don't have work for long, gender differences are in the best case ostracized, and so on). From meetings dominated by "social extroverts", to stereotypes used to communicated with transexuals, to application forms for new jobs that blatantly violate existing law by asking age and other questions, to cultural differences being ignored and transgressed, the problems of diversity are everywhere, and running rampant.
    Thus, when you assign them to "remote teams" you take a privileged POV, because you implicitly describe collocation as an environment where those things don't happen.
    Whether co-located or remote, cultural and many other differences are to be vitally integrated into team mechanics and further yet, leveraged as a source of strength and celebrated. Agile is ALL about smooth, friction-free, unbiased communication, rich debate and personal respect. In that context, it is the role of the whole team (and specially the coach) to make sure that differences are *leveraged* and *enthusiastically embraced* as a source of business advantage.
    Thanks again

    ReplyDelete
  6. Excellent points Carlos. Indeed, I sometimes find it ironic when I encounter an Agile coach who stridently advocates for a particular paradigm for how team members should work, e.g., by all pairing all the time - to me, insisting on a particular practice is antithetical to Agile, which promotes the idea of people deciding for themselves how best to work - and people differ greatly.

    ReplyDelete