Instead of sponsoring a separate testing group recording defects, it is cost-effective to establish specialized software quality expertise on the development team preventing defects.
Conclusion: The separation of the quality function from the producer in the same organization is counter-productive, infects morale, and defeats software quality objectives.
NASA, and the U.S. Department of Defense have pioneered the use of an independent third party to corroborate software deliverables (Rakitin, 2001, p. 6). This affirmation process became known as Independent Verification and Validation (IV&V). In this government procurement paradigm, the IV&V contractor reports directly to the program manager, and has no organizational affiliation with the software suppler.
Commercial entities, searching desperately for a response to persistent software quality problems throughout the 1980's and 1990's, imitated the government's IV&V model. They attempted to simulate the independent testing approach by building organizational facades between development and testing inside of their organizations. Has this approach been successful? Empirical evidence points to the fact that the separation of the quality function from the producer in the same organization is counter-productive, infects morale, and defeats software quality objectives.
Numeric comparisons of organizational process metrics would be difficult to accurately represent due to disparate reporting procedures. An effective presentation of these metrics would require extensive surveys and data collection over time. That lengthy analysis is outside of the scope of this report.
Software Testing - Independent or Interdependent?
We don't trust you. That is the message conveyed to a software development team when a segregated testing group is imposed from inside the same organization. The noble concept of independent verification and validation works when, as NASA and the Department of Defense established--a truly separate entity to conduct it (Spector and Gifford, 1984). That is, a separate company. The economy version of the paradigm, the oxymoronic internal but independent verification and validation--has not proven to be an effective quality control for commercially produced software.
Opposite viewpoints of verification and validation are possible. One objective is to find errors in the software--a contradictory approach is to demonstrate that no errors are present. The primary motivators in this conflict are the organizational affiliations of the individuals involved. One view is destructive, another is constructive (Dorfman and Thayer, 1997, p. 277). In the governmental arena, this dissonance is obvious considering the government is the customer, purchasing software produced by various outside contractors. The government needs to ensure that the supplier who produces the software, does not verify the software--the logical genesis of IV&V.
Commercial project managers, trained in the techniques of government and defense contracting shops, have attempted to emulate the government IV&V paradigm. They have created separate testing groups inside their organizations--but placed an imaginary org-chart wall between software testing and software development. This contortion of independent is counter-productive and adversely impacts a team dynamic. In fact, it is destructive to the organization as a whole and subsequently--to overall quality objectives. The segregation is pure fantasy.
The following figure shows an organizational chart of a software development organization intentionally designed to produce conflict, under the guise of quality assurance.
The separation of the quality function from the producer in the same organization is counter-productive, infects morale, and defeats software quality objectives.
John R. Snyder, November 2003
In this relationship, the manager of the verification and validation team establishes formal procedures for receiving software releases from the development team; evaluation reports are forwarded to the director for analysis. The premise is to develop an independent assessment of the software quality and to "determine if the software satisfies critical system requirements" (Dorfman and Thayer, 1997, p. 224). Some would assert that this type of organizational conflict is a healthy system of checks and balances imposed by the opposite viewpoints.
My experiences with large (30 or more) and small (less than 30) development teams do not confirm this optimistic assessment. In fact, I have witnessed that this opposing viewpoint approach destroys the organization's morale from the inside out. Consider that one team has the goal of getting the software into production on time and under budget. The other team has the mandate to tear the guts out of the product and prove that it is a piece of junk (but be polite, please). For a dedicated software-testing group, there is no glory in defect-free software. Their only reward is proving to management their precarious existence is justified with a long list of defects--real or imagined. And it is those contrived defects are especially damaging to the internal culture.
Most segregated, internal testing groups are not staffed appropriately. They do not typically have training in software engineering and are usually inexperienced. The lack of expertise becomes a liability--sometimes a scapegoat. Project managers are told that testers on the other side of the wall could not possibly understand the complexities of software--their opinions are to be dismissed. Often, this is confirmed by testers who need to generate defect numbers. They create waste-of-time defects: the button is too big, the background is ugly. A software engineer has toiled 6-years in college, sweated through certification exams, and struggled up the ladder--only to have work second-guessed by someone who never studied engineering? Although these groups park in the same lot and in theory are working to the same goal--in reality they have conflicting agendas.
Perry and Rice (1997) confirm the pragmatic evidence with an analysis of the adversarial relationship between segregated testing and development teams. Their conclusions show manifestations of:
The common sense conclusion is that this approach breeds a culture built on negativity and distrust. But an even more sinister aspect is present, the shift of responsibility for quality.
The use of independent testing inside the organization transfers the ownership of quality away from the development team. Development is ultimately convinced that testing, and quality control--is not their responsibility. I can remember a developer telling me that he didn't test his code at all because "testing does that--it's their job". I can still hear the disdain reverberating from his voice, proclaiming "I am not a tester"! This adversarial relationship is a byproduct of the organizational tension introduced with the segregation of testing and development.
Social psychologists have studied group polarization dynamics and have concluded that in everyday life, "people associate mostly with others whose attitudes are similar to their own" (Myers, 1999, p. 312). Myers goes on to explain that in a group, discussion elicits a consolidation of ideas that will ultimately favor the dominant viewpoint. If that controlling viewpoint is that testing has a mission to make themselves look good at the expense of development--polarization will occur. Encouraged by support for their initial inclinations, developers will go even further with the adversarial view of a testing team--even to the point of sabotage.
I have witnessed, on more than one occasion and in several different employment situations, development teams withholding the product from testing on purpose. In each case the code was ready and at least part of it could have been passed over the wall to the testing group. Instead, various excuses were concocted so the software could be withheld until the very last possible minute. Most often, this would be late on a Friday night--right before a Monday release date, so those testing bastards will have to stay over the weekend if they want to invent defects in our software!
Remember the phrase heard often in the late 1970's and early 1980's, egoless programming? In that short-lived fad, "the goal of quality software was supposed to supersede any personal stake the programmers had in their work" (Pajerek, 2002). We don't hear this term much anymore because as Pajerek concluded, programmers do "take it personally", particularly when the inevitable defects are identified by someone outside of their immediate peer group; he summarized that "egoless programming flies in the face of human nature" (2002). The academics tell us that non-attribution will console the creator, "focus on the product, not the producer" (Rakitin, 2001, p.63). In the real world, we know that personalities get in the way of good theories.
Software quality control is essential--it must be rigorously applied to have any chance of delivering a decent product. Quality goals must be balanced against time to market--the principal driver now for most software systems (Sommerville, 2001). Adding layers of independent testers who begin their work when the software is mostly complete, assembled, and deployed into a testing environment does not add any value into that compromise. If defects are found at this point, there is usually no time to do anything about them! An alternate solution can be found by recognizing the interdependence of development and testing teams in your organization. Creating internal teams designed to attack each other is not the path to quality.
A philosophy of detection is based on the premise that defects will be identified--then corrected. A strategy of prevention is formed from the fact that defects can be avoided altogether through the application of various techniques. An analogy can be made to the destructive (find those defects) and constructive (already been removed) quality viewpoints. A segregated quality assurance group in your organization is not only a destructive philosophy, it is not cost effective. Several studies have proven the cost to find and fix defects increases significantly as software development and deployment proceeds. Boehm (1981) reported the cost to find and fix a defect found during the requirements definition phase increases by a factor of 50 if found in the IV&V phase and by a factor of 100 after the product ships. Boehm's study has been supported by more recent cost data analysis of Humphrey (1995). The following figure shows that cost relationship.
Copyright © Snyders.US. All rights reserved.
In this organization, quality goals are in alignment, as we have teams working under the same leadership. This dotted-line arrangement conveys the understanding that if obstacles to quality cannot be overcome without outside intervention, the testing team has the obligation to inform top-level management. With this understanding, it is doubtful testing will create trivial defects just to inflate defect counts, which would foster animosity with their teammates in development. Similarly, the development group will respond with respect for their partners on the testing team, working with them closely to ensure that schedules are met, and communicating the knowledge necessary to install and configure the software.
It used to be that a software build could be created and handed to another on a floppy-disk to test. Those days are gone forever. Today, creating a build is just one small step in the long line of elements that must be meticulously configured for software to function. The use of frameworks, middleware, application servers, and third-party components has complicated the process of rolling-out a new software version. The software build is usually only a small piece of the maze that must be navigated. The operating environment may consist of two, three, or even four different servers that must be configured with various software elements, configuration files, and off-the-shelf software. Without a comfortable working knowledge of the overall configuration, a testing effort is severely disadvantaged. Are defects occurring, or, is it that third-party software has not been configured properly? Becoming part of the development organization gives the testing team access to that knowledge. Development has incentive to assist them, viewpoints are in alignment--everyone is responsible for quality. It is not Independent Verification and Validation; it is Interdependent Verification and Validation.
Conflicting agendas of on-time delivery and impeccable quality have a much better chance to align through an affiliation of software development and software testing. The concept and application of IV&V is appropriate and effective, but only when it is performed by an entity external to the organization. Inside of a company, especially small ones, there can be only a perception of independence. Without tight controls and constant monitoring, the segregation of testing and development creates adversarial relationships that foster into a culture of negativity.
A change in viewpoint shifts the primary focus of quality resources into the development group and away from segregated testing. Defect prevention is cost-effective and can only be achieved before software is handed over to a segregated testing team. Therefore, it makes sense to assign professionals into the software development group who are dedicated to quality assurance and quality control activities. Programmers are the best people to evaluate and correct software errors as they have a vested interested and will learn the most from the process (Humphrey, 1995). Resources expended in the name of quality should be directed where it is the most effective--inside of the development group. You hired software engineers and assigned the task of creating quality applications. Does it make good business sense to hire another team just to record their mistakes? If you do not believe the engineers who work for you can deliver a high-quality product--fire them! Otherwise, bring software quality specialists into their team and trust them.
Boehm, B.W.(1981), Software Engineering Economics, NJ: Prentice-Hall.
Dorfman, M., Thayer, R. (Eds.) (1997). Software Engineering. CA: IEEE Computer Society Press.
Humphrey, W.S. (1995). A Discipline for Software Engineering. MA: Addison-Wesley. pp. 275-277.
Janis, I.L. (1972). Victims of Groupthink. A Psychological Study of Foreign Policy Decisions and Fiascos. MA: Houghton Mifflin. (Ch.22).
Myers, David G. (1999). Social Psychology (6th ed.). MA: McGraw-Hill
Pajerek, L, (2002). To Engineer Is Human , Draft paper submitted for publication in The Journal of the International Council on Systems Engineering, NY: John Wiley & Sons.
Perry, W. E., & Rice, R. W. (1997). Surviving the Top Ten Challenges of Software Testing. NY: Dorset House
Rakitin, Steven R. (2001). Software Verification and Validation for Practitioners and Managers (2nd ed.). Boston : Artech House.
Sommerville, I., (2001). Software Engineering (Sixth Edition). UK : Addison-Wesley.
Spector, A. & Gifford, D. (1984). Case Study: the space shuttle primary computer system. Commun. ACM Vol. 27 No. 9 pp. 874-900.
The evidence is overwhelming that it pays to eliminate defects as early as possible, but in practice--managers are just beginning to understand how to leverage quality assurance activities into the initial stages of the software life cycle. Artifact review and inspection techniques have gained in acceptance, and have shown to realize economic benefit (Humphrey, Rakitin, et al.). Typically, the oversight and management of these activities is left to the devices of the development team, and so the success or failure is a direct function of how well-trained and knowledgeable they are in the procedures. Software inspections are very different from meetings that software engineers usually participate in--training in the skill set is essential to success (Rakitin, 2001, p.64). Why leave such an important aspect of quality assurance to chance?
Instead of sponsoring a separate testing group to record defects, consider establishing specialized software quality expertise in the development team to prevent them. Depending on the size of the organization, this role could be filled by a developer on a part-time basis, by one or more dedicated quality assurance specialists. The basic responsibilities of this role would be to:
As one can see from the duties, this individual would need an aggregate skill set. The person would need the qualifications of a senior software engineer, in conjunction with experience and training in specific software quality assurance and control techniques. Certifications such as the Certified Software Quality Engineer (CSQE) from the American Society of Quality or the Certified Software Development Professional (CSDP) offered by the IEEE should be a requirement. This role needs to be able to counsel individual developers in how to implement best practices and prototype new ones.
Aside from the cost benefit, an implicit advantage of moving the primary quality control function into the development area is the creation of a group quality standard. Instead of being imposed from outside the team, standards are established through group consensus--and therefore are more likely to be observed. Credentials of those assigned to the quality role are no longer something to laugh about. Compliance can be monitored closely, at the source, and earlier in the life cycle minimizing costs. The creation of well-documented standards allows for continuity as team members move into and out of the group. Most importantly, the development team will understand that quality is everyone's mission--not something to push onto another department.
Skeptics will remind the concept of independent assessment, IV&V, was created to ensure an objective evaluation of the work products. Would you let a fox watch the henhouse? Of course not, that is why it is important to have a dedicated quality specialist as a member of the software development team--to guard against groupthink. Janis (1972) defined groupthink as the name given to the situation where the critical abilities of group members are eroded by group loyalties. For a software development team, the negative aspects of groupthink manifest as a dismissal of the customer's requirements. Removed from the customer and sent into a room to create a product on their own, software development teams have a tendency to create, and follow--their own agenda. Having a trained, professional quality specialist on the team will serve as the constant voice of the customer, guarding against groupthink by playing devil's advocate, forcing group members to think about and evaluate their activities from the customer's viewpoint (Sommerville, 2001).
For those still convinced that quality control must be in the hands of a team separated from the development group, consider an interdependent arrangement as illustrated in the following figure: