Agility requires control

In the early days of the Agile method of software development, some believed incorrectly that Agile was the rejection of disciplined software development. A better perspective is that Agile was a rejection of techniques, such as the waterfall lifecycle, that are ill-suited to the dynamics of software. In the early days, before Agile, project managers were told to ‘plan their work and work their plan.’ Thus, measurement systems, such as Earned Value Management (EVM), were designed to measure the extent of adherence to the plan.

Consider that agility is the ability to respond rapidly to the environment. A jet fighter is very agile, but it does not plan its route. Rather, it responds very rapidly to sudden changes in its environment, such as threats or its mission. It accomplishes this agility with very tight control loops, constantly measuring and reacting to its environment and its operation.

Agile software teams also must constantly respond to changes, such as:

• New or clarified requirements,

• Missed dependencies,

• Failed tests,

• Work taking longer than expected

This applies to software teams as well as to any level of the organization that relies on software development. Development organizations must respond to changing operational challenges, such as unpredictable team performance, disruptive technologies, labor arbitrage, and uncontrolled operations. In turn, the enterprise must respond to all of the above, along with evolving IT needs and/or market sentiment.

Measures have cost

While control requires measures, they do not come without cost. First, significant manual effort may be required to collect the measures themselves, or some investment in automated data collection may be necessary. Even worse, collecting and driving the organization to the wrong analytics can lead to negative consequences. Such consequences can only be avoided by adopting analytics as part of a control loop that there is every intention of managing.

In brief,

“You can’t control what you don’t measure, and you shouldn’t measure what you don’t intend to manage.”

The right set of measures, chosen as part of the right control loop, promotes trust between the leadership and the staff. The wrong measures breed mistrust, waste time and resources, and drive undesired behavior. For example, some firms might measure the number of bugs removed. That measure might be part of the wrong control loop. That measure, along with productivity measures, could inadvertently reinforce sloppy coding. Improving quality takes a more subtle choice of measures.

Designing the measurement solution

For an organization to measure in a purposeful way, it must:

• Specify the goals for itself and its projects
• Trace the goals to the data that are required to track and confirm the attainment of the goals
• Provide a framework for interpreting the data with respect to the stated goals in order to take action.

The goal, question, metric (GQM) method is a design method for analytic solutions, much like noun-verb analysis in software design. It is a method to derive the metric’s solution from the organization’s goals.

Here is a summary of the steps in the GQM method:

1. Specify a set of corporate, division, and project business goals and associated measurement goals.
2. Specify a high-level model of the process and/or key artifacts identified in the goal; if necessary, decompose the qualitative goal to one or more quantitative goals.
3. Generate questions based on the lifecycle model, for example: How would I know if improvement is attained? What are the measures of the business process (steps)? What do I need to know about the context to understand process execution, e.g., external sources of work?
4. Study the data to specify the datasets, statistics, and analytics that must be collected to answer those questions and track progress toward achieving the goals.
5. If the data are not available, specify the data that must be collected.
6. Develop automated mechanisms for data collection and analytics.
7. Collect, validate, and analyze the data to identify patterns that diagnose the operational situation with respect to the goal and possibly provide suggestions for corrective actions.
8. Review the solution with stakeholders to make recommendations

Organizational Level

As depicted in Figure 2, different levels of a development organization have different goals. For the organization to work for the enterprise, the goals at one level must support the goals of the level above. Generally, goals flow down the organization. Another perspective is that, for one level of the organization to meet its commitments, the lower levels must meet their commitments.

From above, different goals entail different control loops that, in turn, entail different analytics. However, one level’s operation depends on the operation of the lower levels. This is very reminiscent of ‘black box, white box’ reasoning in system architecture. At each level, the blackbox measures reflect the performance of the encapsulated level as seen from the outside. The white box measures are the internal measures of the level required to meet the black box measures.

Since a level depends on the lower level, it is essential that the lower level’s blackbox measures be fully visible to the upper level and that those measures support the upper level’s measures.

To summarize, a full measurement architecture must reflect that

• Different levels have different control loops and, thus, different measures

• Blackbox measures at one level become the whitebox measures at the upper level.

It follows that the architecture of the measures should reflect both the organizational architecture and the goal architecture.

Figure 2 Analytics vary with the organization level

Type of development

Software development falls roughly into three classes (Figure 3):

• Maintenance and small change requests

These activities involve making small changes to an existing code base. In most cases, they are applied to deployed code. The dynamics of this activity involve receiving and servicing an unsteady flow of problem reports or small enhancement requests. Typically such requests are detailed and well understood, and the team is able to estimate the time and effort required with high confidence. About 70% of what the industry spends is on these activities. These teams are increasingly adopting DevOps and continuous delivery practices.

• New features on existing platforms

These efforts are to add significant new features to an existing application. For example, a bank may have a mobile app for account holders. The business analysts decide that the app should add an online check deposit feature. This is a major undertaking that involves several teams and several months to accomplish. Once the team receives the feature request from the business analyst, it estimates the time and effort required. Unlike change requests, the team has less confidence in the estimates of time and effort, and it may run some experiments to learn what is needed to have more confidence in making commitments to deliver the feature to the business.

• New platform

Occasionally, a development organization gets the opportunity to build an entirely new application, a so-called greenfield project. This may be a small mobile app requiring the so-called one-pizza team building something on the cloud or a big integrated system requiring hundreds of developers or more. The project is initiated with some business need or idea but without detailed requirements or product design. Hence, the team has little basis for making estimates, and whatever estimates they make will be low confidence. Lean startup methods are especially useful for such efforts.

Figure 3. The three kinds of development.

Many organizations have a mix of these types of efforts.

The different sorts of development require different sorts of analytics. The maintenance and small change efforts have information that is suitable to descriptive statistics. The new platform teams have little data, but they may have prior beliefs based on expert opinion. This situation calls for predictive approaches. New feature development calls for a mixture of both.

Simple reports

These consist mostly of data counts, such as the number of defects that are currently open on a given date or perhaps the trend of averages. An example , the number of defects that are currently open on a given date is shown in Figure 3.

Figure 3. Example of a simple report

Predictive Analytics

Going back to our control loop, once one has the information from the analytics to take action, there is still the problem of what actions to take. Before taking action, it would be good to know what is the probable outcome of the action. This is what is more generally called ‘what if’ analysis, a common use of spreadsheets.

Predictive analytics provide the means for describing what is the likely outcome of taking some action. As such, predictive analytics use the language of probability. A common predictive report is what is called a probability distribution (Figure 9).

Figure 9. Probability distribution

A probability distribution is much like a histogram, except that the total area under the curve is 1, representing the probability version of certainty. The vertical axis roughly corresponds to the probability of the event. The way to read a distribution is to treat the area under some region of the curve as the probability that one of those events will occur. For example, suppose the distribution represents when a project might ship. In this case, the events correspond to the days a product might ship, and, the higher the distribution, the more likely the product would ship on that day. In this case, we could determine, for example, that the probability of shipping between 10 and 20 is about 0.25 (Figure 10).

Figure 10. Area of the red zone is about 0.25

So, a predictive analytic returns a distribution that can be used to reason about the probability of events.

There is a variety of ways that a probability distribution can be computed. A powerful method uses what are called Bayesian nets, a technique based on Bayes theorem and conditional probability. These are built starting with a causal model of what is to be predicted. For example, suppose we want to predict the number of software defects that will be found in operation. Although we can’t predict with certainty, we might be able to find a distribution. We might build a causal network with Figure 11 that is shipped with the AgenaRisk tool. As can be seen from the model’s purple boxes, the number of defects found in the field depends on the complexity of the code, the quality of the design, and testing. Also, the more the code is used, the more likely defects will be found. These parameters can be used to determine the yellow boxes, i.e., the number of defects injected in the code and the number of defects discovered in testing. Other parameters could be added to the model, such as code size and coder quality.

Figure 11. Model for predicting defects found in the field

Which box affects which is designated by the arrows. The expert who builds the model specifies the nature of that influence using appropriate probability models.

Such a model can be used to build different scenarios. For example, suppose that we assume the team has medium skills in building a complex code with high operational usage (Figure 12). The output of this model is the distribution in the lower right (Figure 13). It tells us that it is very probable that the number of defects found in operation will be somewhere between 10 and 30.

Figure 12. Scenario 1, business as usual

Figure 13. Distribution of predicted defects found in the operation in scenario 1

Now, further assume that the business determines that addressing this number of defects in operations is very costly, not only to repair but also to the reputation of the company, which would lead to lower revenue. It has been determined that this level of defects is costing the company about $20M/yr. So, there is an economic incentive to reduce the number of defects to no more than 5/yr. That is their goal. The question is, “What is the best path to take?” The company considers four scenarios.

• Business as usual – no change

• Invest heavily in testing improvement

• Invest in an improved process, leading to less complex designs

• Investing in both.

They use the model to test scenario 2 (Figure 14) and find, unsurprisingly, the outcome of improved testing shows much improvement (Figure 15). The distribution shows that, under this scenario, they are around 90% likely to have 10/yr or fewer defects in operation. The goal of fewer than 5/yr is an even bet.

Figure 14. Scnario 2

Figure 15. Distribution of predicted defects found in operation in scenario 2

Then, the team ran scenario 3 with the improved design process, resulting in a more elegant, less complex design. The result of the improved design (Figure 17) had less impact than scenario 2.

Figure 16. Scenario 3, better design

Figure 17. Distribution of predicted defects found in operation with the improved design

Finally, the team ran scenario 4 (Figure 18) with the desired outcome (Figure 19).

Figure 18. Scenario 4, improving both testing and design

Figure 19. Distribution of defects found in operation with both improved testing and design

Table 1 provides a summary of the results.

Table 1

The way to read the table is that there is a better than even chance that investing in testing will meet the goal, but it is not certain. In fact, there is a 25% chance that there will be more than six defects. The only way to be close to certain that the goal will be met is to invest in both.

In the end, the business is faced with an economic decision. The scenario that should be implemented depends on the cost to the business and ultimately its return on investment (ROI) in terms of the savings in operational expense and avoidance of lost revenue over the costs of the improvements. This ROI involves uncertain quantities, so the use of predictive methods is required. It can be determined in a reasonably straightforward way using Monte Carlo simulation tools.