Test Strategy

Test Strategy

In continuation to my previous post on Test Strategy, I'm here describing it in more details.
Test strategy can be defined as a high level management method that confirms adequate confidence in the software product being tested, while ensuring that the cost efforts and timelines are all within acceptable limits.
Test strategy can be of different levels:Enterprise-wide test strategy to set up a test group for the entire organization.Application or the product test strategy to set up the test requirements of a software product for its entire lifeProject level test strategy to set up test plans for a single project life cycle.A good test strategy should:Define the objectives, timelines and approach for the testing effortList various testing activities and define roles & responsibilitiesIdentify and coordinate the test environment and data requirements before the starting of testing phaseCommunicate the test plans to stakeholders and obtain buy-in from business clientsProactively minimize fires during the testing phaseDecisions to be made in test strategy:When should the testing be stopped?What should be testing?What can remain untested?Further, the following three types of risks can be considered while making a test strategy:
1. Development related risks include:Inefficiently controlled project timelines.Complexity of the codeLess skilled programmersDefects in existing codeProblems in team co-ordinationLack of reviews and configuration controlLack of specifications2. Testing related risks include:Lack of domain knowledgeLack of testing and platform skillsLack of test bed and test dataLack of sufficient time3. Production related risks include:Dynamic frequency of usageComplexity of user interfaceHigh business impact of the functionTest strategies generally acknowledge thatThe earlier a bug is detected, the cheaper it is to fix.Splitting the testing into smaller parts and then aggregating, ensures a quicker debug and fix cycleBugs found in critical functions mean more time to fixSix steps for developing a test strategy:Determine the objective and scope of the testingIdentify the types of tests requiredBased on the external and internal risks, add, modify or delete processes.Plan the environment, test bed, test data and other infrastructurePlan strategy for managing changes, defects and timeline variations.Decide both the in-process and the post process metrics to match the objectives.While creating test strategy for maintenance projects, following aspects needs to be considered:How much longer is the software being supported and is it worth-while strategizing to improve the testing of this software?Is it worthwhile to incrementally automate the testing of the existing code and functionality?How much of the old functionality should we test to give us the confidence that the new code has not corrupted the old code?Should we have same set of people support the testing of multiple software maintenance projects?What analysis can be applied on the current defect database that will help us improve the development itself?However, for creating the test strategy of product development, following aspects needs to be considered:How much longer is the product going to last? This includes multiple versions etc. since test case and artifacts can continue to be used across versions.Would automation be worth considering?How much testing should we do per release (minor, patch, major, etc.)Do we have a risk based test plan that will allow releases to be made earlier than planned?Is the development cycle iterative? Should the test cycle follow that?As per IEEE software engineering standards, the Test Strategy document should answer the following aspects:
1. Objective and scope of testingWhat is the business objective of testing?What are the quality goals to be met by the testing effort?To what extent will each application be tested?What external systems will not be tested?What systems and components need to be tested?2. Types of testingDifferent phases of the testing that are required.Different types of testingTest Coverage3. Testing approachDefinition of testing process life cycleCreation of testing related templates, checklists and guidelinesMethodology for test development and executionSpecification of test environment setupPlanning for test execution cycles4. Test Environment specificationHardware and software requirementsTest data creation, database requirements and setupConfiguration management, maintenance of test bed and build management5. Test AutomationCriteria and feasibility of test automationTest tool identificationTest automation strategy (effort, timelines etc.)6. Roles and Responsibilities / Escalation mechanismTesting team organization and reporting structureDifferent roles as a part of testing activities and their corresponding responsibilitiesWho to escalate to and when?7. Defect ManagementCategorization of defects based on criticality and priorityDefinition of a workflow or the disposition of defectsTechniques and tools for tracking of defects.8. Communication and status reportingStatus meetings for communication of testing statusFormat and content of different status reportsPeriodicity of each status reportDistribution list for each report9. Risks and mitigation plansIdentification of all testing related risks, their impact and exposurePlan for mitigation and managing these risks10. Configuration managementList of testing artefacts under version controlTools and techniques for configuration management11. Change managementPlan for managing requirement changesModels for assessing impact of changes on testingProcess for keeping test artifacts in sync with development artifacts12. Testing metricsWhat metrics are to be collected? Do they match the strategic objectives?What will be the techniques for collection of metrics?What tools will be employed to gather and analyze metrics?What process improvements are planned based on these metrics?