Agile pilots are crucial for discovering unanticipated problems before agile is used more broadly across the organization. However, pilots can also be very misleading. Pilots are often excused from the normal rules that other projects have to follow, and it is then no surprise that the pilot goes well: it does not have the same challenges. The greatest challenges are often related to all the rules - all the governance requirements, the official testing, the approval of enterprise architecture, the quality assurance processes, and so on. Those are the greatest challenges. Simply running an agile team and delivering some code is not a challenge by comparison.
For this reason, it is important that pilots be focused and realistic. In the words of Dr. Reg Butterfield, Fellow of the UK Chartered Management Institute,
"[A pilot should be] one that simulates the reality that you want in the organization and includes all aspects of the environment that it will be expected to operate in. It must link into and involve the relevant management and staff who will be expected to operate in this new way if it is shown to be appropriate."
A pilot should have a clear objective that addresses one or more real world challenges. For example, one might pilot the use of a new approach for security review, in which an agile team works collaboratively with security experts at regular intervals throughout a projects. That way, the actual process that will be used is tried and kinks can be worked out. In a similar vein, one could pilot the use of a risk based approach to quality assurance, in which QA analysts assess the agile team's automated test suite coverage rather than running QA's own tests during a testing phase. One should always pilot a real process - not a fantasy process.
Pilots serve a great function. They can enable the organization to identify and solve many problems at a small scale, and they can create momentum. Even so, pilots only tell you so much. When a team has to compete with other teams for the time of specialists such as security staff and QA staff, or data center resources, things change. When there is scrutiny at a portfolio level, the bar for demonstrating value is higher. Pilots are important, but you don't really know how things will work until you start to use the piloted processes more broadly, and that is why agile should be scaled in a gradual manner, using a process of continual reflection and adjustment.