Software Testing Principles

SoftwareTesting Principles

Below are some basic Software Testing Principles:
- A necessary part of a test case is a definition of the expected output or result. 
- A programmer should avoid attempting to test his or her own program. 
- A programming organization should not test its own programs. 
- Thoroughly inspect the results of each test. 
- Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected.
- Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do.
- Avoid throwaway test cases unless the program is truly a throwaway program.
- Do not plan a testing effort under the tacit assumption that no errors will be found.
- The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section.
- Software Testing is an extremely creative and intellectually challenging task.

Test Planning

Test Planning

The quality of Software Testing effort depends on the quality of quality of Software Testing Planning. Software Testing Planning is very critical and important part of Software Testing Process.

Below are some questions and suggestions for Software Test Planning:

- Have you planned for an overall testing schedule and the personnel required, and associated training requirements?

- Have the test team members been given assignments?

- Have you established test plans and test procedures for

1. Module testing
2. Integration testing
3. System testing
4. Acceptance testing

- Have you designed at least one black-box test case for each system function?

- Have you designed test cases for verifying quality objectives/factors (e.g. reliability, maintainability, etc.)?

- Have you designed test cases for verifying resource objectives?

- Have you defined test cases for performance tests, boundary tests, and usability tests?

- Have you designed test cases for stress tests (intentional attempts to break system)?

- Have you designed test cases with special input values (e.g. empty files)?

- Have you designed test cases with default input values?

- Have you described how traceability of testing to requirements is to be demonstrated (e.g. references to the specified functions and requirements)?

- Do all test cases agree with the specification of the function or requirement to be tested?

- Have you sufficiently considered error cases? Have you designed test cases for invalid and unexpected input conditions as well as valid conditions?

- Have you defined test cases for white-box-testing (structural tests)?

- Have you stated the level of coverage to be achieved by structural tests?

- Have you unambiguously provided test input data and expected test results or expected messages for each test case?

- Have you documented the purpose of and the capability demonstrated by each test case?

- Is it possible to meet and to measure all test objectives defined (e.g. test coverage)?

- Have you defined the test environment and tools needed for executing the software test?

- Have you described the hardware configuration an resources needed to implement the designed test cases?

- Have you described the software configuration needed to implement the designed test cases?

- Have you described the way in which tests are to be recorded?

- Have you defined criteria for evaluating the test results?

- Have you determined the criteria on which the completion of the test will be judged?

- Have you considered requirements for regression testing?

Test Specification

Test Specification

Test Specification – It is a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Trying to include all Editor Features or all Window Management Features into one Test Specification would make it too large to effectively read.

However, a Test Plan is a collection of all test specifications for a given area. The Test Plan contains a high-level overview of what is tested for the given feature area.

Contents of a Test Specification:

Revision History - This section contain information like Who created the test specification? When was it created? When was the last time it was updated?

Feature Description – A brief description of what area is being tested.

What is tested? – An overview of what scenarios are tested.

What is not tested? - Are there any areas that are not being tested. There can be several reasons like... being covered by different people or any test limitations etc. If so, include this information as well.

Nightly Test Cases – A list of the test cases and high-level description of what is tested whenever a new build becomes available.

Breakout of Major Test Areas - It is the most interesting part of the test specification where testers arrange test cases according to what they are testing.

Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.

Security tests – Any tests that are related to security.

Accessibility Tests – Any tests that are related to accessibility.

Performance Tests - This section includes verifying any performance requirements for your feature.

Localization / Globalization - tests to ensure you’re meeting your product’s Local and International requirements.

Please note that your Test Specification document should be in such a manner that should prioritize the test case easily like nightly test cases, weekly test cases and full test pass etc:

• Nightly - Must run whenever a new build is available.
• Weekly - Other major functionality tests run once every three or four builds.
• Lower priority - Run once every major coding milestone

Software Testing Estimation

Software Testing Estimation

Software Testing estimation process is one of the most difficult and critical activity. When say that one project will be completed in a particluar time at a particular cost, then it must happen. If it does not happen, it may result in peer's comments and senior management’s warnings to being fired depending on the reasons and seriousness of the failure.

Here are a few rules for effective software testing estimation:

- Estimation must be based on previous projects: All estimation should be based on previous projects.

- Estimation must be recorded: All decisions should be recorded. It is very important because if requirements change for any reason, the records would help the testing team to estimate again.

- Estimation shall be always based on the software requirements: All estimation should be based on what would be tested. The software requirements shall be read and understood by the testing team as well as development team. Without the testing participation, no serious estimation can be considered.

- Estimation must be verified. All estimation should be verified: Two spreadsheets can be created for recording the estimations. At the end, compare both the estimations. If the estimation has any deviation from the recorded ones, then a re-estimation should be made.

- Estimation must be supported by tools: Tools such as spreadsheet containing metrics calculates automatically the costs and duration for each testing phase. Also, a document containing sections such as: cost table, risks, and free notes should be created. Showing this document to customer can help the customer to decide which kind of test he needs.

- Estimation shall be based on expert judgment: The experinced resources can easily make estimate that how long it would take for testing.

Classifiy the requirements in the following categories:

Critical: The development team has little knowledge in how to implement it.

High: The development team has good knowledge in how to implement it but it is not an easy task.

Normal: The development team has good knowledge in how to implement

Test Strategy

Test Strategy

In continuation to my previous post on Test Strategy, I'm here describing it in more details.
Test strategy can be defined as a high level management method that confirms adequate confidence in the software product being tested, while ensuring that the cost efforts and timelines are all within acceptable limits.
Test strategy can be of different levels:Enterprise-wide test strategy to set up a test group for the entire organization.Application or the product test strategy to set up the test requirements of a software product for its entire lifeProject level test strategy to set up test plans for a single project life cycle.A good test strategy should:Define the objectives, timelines and approach for the testing effortList various testing activities and define roles & responsibilitiesIdentify and coordinate the test environment and data requirements before the starting of testing phaseCommunicate the test plans to stakeholders and obtain buy-in from business clientsProactively minimize fires during the testing phaseDecisions to be made in test strategy:When should the testing be stopped?What should be testing?What can remain untested?Further, the following three types of risks can be considered while making a test strategy:
1. Development related risks include:Inefficiently controlled project timelines.Complexity of the codeLess skilled programmersDefects in existing codeProblems in team co-ordinationLack of reviews and configuration controlLack of specifications2. Testing related risks include:Lack of domain knowledgeLack of testing and platform skillsLack of test bed and test dataLack of sufficient time3. Production related risks include:Dynamic frequency of usageComplexity of user interfaceHigh business impact of the functionTest strategies generally acknowledge thatThe earlier a bug is detected, the cheaper it is to fix.Splitting the testing into smaller parts and then aggregating, ensures a quicker debug and fix cycleBugs found in critical functions mean more time to fixSix steps for developing a test strategy:Determine the objective and scope of the testingIdentify the types of tests requiredBased on the external and internal risks, add, modify or delete processes.Plan the environment, test bed, test data and other infrastructurePlan strategy for managing changes, defects and timeline variations.Decide both the in-process and the post process metrics to match the objectives.While creating test strategy for maintenance projects, following aspects needs to be considered:How much longer is the software being supported and is it worth-while strategizing to improve the testing of this software?Is it worthwhile to incrementally automate the testing of the existing code and functionality?How much of the old functionality should we test to give us the confidence that the new code has not corrupted the old code?Should we have same set of people support the testing of multiple software maintenance projects?What analysis can be applied on the current defect database that will help us improve the development itself?However, for creating the test strategy of product development, following aspects needs to be considered:How much longer is the product going to last? This includes multiple versions etc. since test case and artifacts can continue to be used across versions.Would automation be worth considering?How much testing should we do per release (minor, patch, major, etc.)Do we have a risk based test plan that will allow releases to be made earlier than planned?Is the development cycle iterative? Should the test cycle follow that?As per IEEE software engineering standards, the Test Strategy document should answer the following aspects:
1. Objective and scope of testingWhat is the business objective of testing?What are the quality goals to be met by the testing effort?To what extent will each application be tested?What external systems will not be tested?What systems and components need to be tested?2. Types of testingDifferent phases of the testing that are required.Different types of testingTest Coverage3. Testing approachDefinition of testing process life cycleCreation of testing related templates, checklists and guidelinesMethodology for test development and executionSpecification of test environment setupPlanning for test execution cycles4. Test Environment specificationHardware and software requirementsTest data creation, database requirements and setupConfiguration management, maintenance of test bed and build management5. Test AutomationCriteria and feasibility of test automationTest tool identificationTest automation strategy (effort, timelines etc.)6. Roles and Responsibilities / Escalation mechanismTesting team organization and reporting structureDifferent roles as a part of testing activities and their corresponding responsibilitiesWho to escalate to and when?7. Defect ManagementCategorization of defects based on criticality and priorityDefinition of a workflow or the disposition of defectsTechniques and tools for tracking of defects.8. Communication and status reportingStatus meetings for communication of testing statusFormat and content of different status reportsPeriodicity of each status reportDistribution list for each report9. Risks and mitigation plansIdentification of all testing related risks, their impact and exposurePlan for mitigation and managing these risks10. Configuration managementList of testing artefacts under version controlTools and techniques for configuration management11. Change managementPlan for managing requirement changesModels for assessing impact of changes on testingProcess for keeping test artifacts in sync with development artifacts12. Testing metricsWhat metrics are to be collected? Do they match the strategic objectives?What will be the techniques for collection of metrics?What tools will be employed to gather and analyze metrics?What process improvements are planned based on these metrics?

How to improve Software testing process

How to improve Software testing process

Improving the testing process is not the responsibility of test team only. It is a joint effort of Development & Testing team and Management to understand the health of existing testing process and identify the necessary measures to improve it.

Points for Leads, Test Managers, Project Managers and Delivery Heads:

1. Identify all the platforms on which application will be run.
2. Improve requirements management process – business & functional requirements should be well documented. Involve testing team in requirement gathering phase.
3. Create a document / list of all possible scenarios before writing test cases. Include it into test planning.
4. Measure testing effort on periodical basis – New Bug Rate, Effort Variance, Schedule Variance, Test Case Effectiveness, Residual defect density etc.
5. Keep developers away from test environments.
6. Post mortem meetings must be planned after every release. Both testing & development teams should participate in these meetings.
7. Keep track of bug fixing time taken by development team. Keep informed about the time remaining for regression testing to relevant stack holders. Tracking of this point by relevant stack holders is very critical as it may result in low quality releases.

Points for Software Testers:

1. Try to understand the logic behind the screen and try to break that logic. Understand the internal workings of code from developers during lunch time or tea breaks.
2. Analyse test results thoroughly. Try to identify root cause from functional perspective.
3. Break the application into smaller functional modules.
4. First write test cases for valid conditions, then cover invalid conditions.
5. While writing test cases, refer design documents as well.
6. As soon as you complete the test case writing, share test case with development team. It could result in time saving.
7. During test case writing phase, group test cases using impact analysis. It will help in effective regression testing in less time.
8. If you are new tester for a old development team, have a look on old bug reports of modules / project where development worked previously. Generally developers, repeat similar mistakes.
9. Test the application for both implicit as well as explicit requirements.
10. Never communicate bugs verbally. For any critical / show stopper bugs, have an immediate discussion & then document via mail and share it to relevant stack holders.

Challenges in Software Testing

Challenges in Software Testing

All software engineering areas face lot of challenges during execution. So, I would say, as a tester, never ever get surprised when you face challenges in software testing. But, it is also a hard fact for tester community that most of the companies are not testing oriented. Management always do a good appraisal of development teams. Sometimes, appreciation goes to development teams only.

Few points that I want highlight to management / project managers & development teams:

1. If development teams are so good / intelligent, then why you need testers to test the application.
2. Remember - by testing a build / release, testers evaluate the Quality of work done by developers.
3. During every regression testing, lot of bugs got re-opened (what quality of work was done by developers)
4. Sometimes, a customer release requires 5-7 regression testing cycles. Think Why?
5. During these 5 to 7 regression cycles, developers made so many mistakes. Testers help to prevent the defect leakage to customer. But, at the end, if even if 2-3 minor bugs goes to customer, everybody from top to bottom will catch testers. Just imagine, with one mistake of a tester, everybody will catch testers from the neck. Are testers bound to make not even a single mistake? But, developers can repeat the same mistakes multiple times.
6. Most of the times, developers eat tester's time.
7. I never understand why management always keep a difference between developers and testers. You might be having answer to this question, but, I'm not.

Delivering a Quality product is a joint responsibility of developers, testers & other relevant stack holders. If some bugs, goes to customer, remember:

1. Developers also did a unit testing
2. An expert solution architect designed the system.
3. Some senior developer also reviewed the code / database
4. Customer also did a UAT - then, application goes live & bug appears
5. Product managers / business analysts review test cases (and test cases are executed by testers)

Now, apart from testing done by testers, these five steps were also performed which were not able to detect that bug. Then, why all blame came to testers.

The major root cause of all the above mentioned problems is only one challenge - "Lack of testing culture in the organization". To build a testing culture, management need to play an important role.

Second major challenge in the testing is the lack of skilled testers. Again, the major root cause for this is "Wrong Management Decision" at the time of selection of testers. As usual management do not want to invest in testers. This results into incomplete, insufficient and ad-hoc testing throughout the project life cycle. It is also true that sometimes, testers may add complications in the testing of a project due to their unskilled way of working.

There are few other challenges:

• Developers did not communicate what the build is about. Wake up testers - If developers didn't communicate about what this build is about, ask them before testing.
• Remember, complete Testing is never possible. Testing is always done on sampling basis. Selection of test data needs good proficiency in test data selection methods.
• Sometimes, testers and developers fight with each other on bugs and take the things personally rather than professionally. It becomes a major challenge for project stake holders.
• As project keep on growing, regression testing becomes challenge. For these kind of situations, both the testing & development teams need to work together to do a proper impact analysis.
• In most of the organizations, at the time of effort estimation, testing team is not involved. Testing team is always asked to test the build in a particular time. If you are facing this challenge, raise your voice to relevant stack holders.
• If requirements are not properly communicated to you, ask for detailed requirements. Never ever assume anything.
• Management always expects high return on investment for automation. Do a proper study for "whether to go for test automation or not". Communicate timely to all stack holders.