Wednesday, 1 July 2015

Risk management for testing...


Types of Risks in Software Projects
As testing is the last part of the project, it’s always under pressure and time constraint. To save time and money you should be able to prioritize your testing work. How will prioritize testing work? For this you should be able to judge more important and less important testing work. How will you decide which work is more or less important? Here comes need of risk-based testing.

What is Risk?

“Risk are future uncertain events with a probability of occurrence and a potential for loss”

Risk identification and management are the main concerns in every software project. Effective analysis of software risks will help to effective planning and assignments of work.

Risks are identified, classified and managed before actual execution of program. These risks are classified in different categories.

Categories of risks:

Schedule Risk: Project schedule get slip when project tasks and schedule release risks are not addressed properly. Schedule risks mainly affect on project and finally on company economy and may lead to project failure. Schedules often slip due to following reasons:
Wrong time estimation

Resources are not tracked properly. All resources like staff, systems, skills of individuals etc.

Failure to identify complex functionalities and time required to develop those functionalities.

Unexpected project scope expansions.

Budget Risk:

Wrong budget estimation.

Cost overruns

Project scope expansion

Operational Risks:

Risks of loss due to improper process implementation, failed system or some external events risks.




Causes of Operational risks:




Failure to address priority conflicts

Failure to resolve the responsibilities

Insufficient resources

No proper subject training

No resource planning

No communication in team.

Technical risks:

Technical risks generally leads to failure of functionality and performance.

Causes of technical risks are:




Continuous changing requirements




No advanced technology available or the existing technology is in initial stages.

Product is complex to implement.

Difficult project modules integration.

Programmatic Risks:

These are the external risks beyond the operational limits. These are all uncertain risks are outside the control of the program.




These external events can be:




Running out of fund.

Market development

Changing customer product strategy and priority

Government rule changes.




Risk Identification

Risks are identified within the scope of the project. Risks can be identified using a number of resources e.g. project objectives, risk lists of past projects, prior system knowledge, understanding of system usage, understanding of system architecture/ design, prior customer bug reports/ complaints, project stakeholders and industry practices. For example, if certain areas of the system are unstable and those areas are being developed further in the current project, it should be listed as a risk.




It is good to document the identified risks in detail so that it stays in project memory and can be clearly communicated to project stakeholders. Usually risk identification is an iterative process. It is important to re-visit the risk list whenever the project objectives change or new business scenarios are identified. As the project proceeds, some new risks appear and some old risks disappear.




Risk Prioritization

It is simpler to prioritize a risk if the risk is understood accurately. Two measures, Risk Impact and Risk Probability, are applied to each risk. Risk Impact is estimated in tangible terms (e.g. dollar value) or on a scale (e.g. 10 to 1 or High to Low). Risk Probability is estimated somewhere between 0 (no probability of occurrence) and 1 (certain to occur) or on a scale (10 to 1 or High to Low). For each risk, the product of Risk Impact and Risk Probability gives the Risk Magnitude. Sorting the Risk Magnitude in descending order gives a list in which the risks at the top are the more serious risks and need to be managed closely.

Adding all the Risk Magnitudes gives an overall Risk Index of the project. If the same Risk Prioritization scale is used across projects, it is possible to identify the riskier projects by comparing the Risk Magnitudes.







Risk Treatment

Each risk in the risk list is subject to one or more of the following Risk Treatments.

a. Risk Avoidance: For example, if there is a risk related to a new component, it is possible to postpone this component to a later release. Risk Avoidance is uncommon because it impacts the project objectives e.g. delivery of new features.

b. Risk Transfer: For example, if the risk is insufficient security testing of the system, it may be possible to hire a specialized company to perform the security testing. Risk Transfer takes place when this vendor is held accountable for ample security testing of the system. Risk Transfer increases the project cost.

c. Risk Mitigation: This is a common risk treatment. The objective of Risk Mitigation is to reduce the Risk Impact or Risk Probability or both. For example, if the testing team is new and does not have prior system knowledge, a risk mitigation treatment may be to have a knowledgeable team member join the team to train others on-the-fly. Risk Mitigation also increases the project cost.

d. Risk Acceptance: Any risk not treated by any prior treatments has to be accepted. This happens when there is no viable mitigation available due to reasons such as cost. For example, if the test environment has only one server, risk acceptance means not building another server. If the existing server crashes, there will be down-time and it will be a real issue in the project.







Few other points are:

1. Risk management brings clarity and focus to the team and other stakeholders. Though the team should avoid burning more time on risk management if it is not providing more value.

2. The risk list should be a live document, consisting of current risks, their prioritization and treatment plans. The test approach and test plan should be synched with the risk list whenever the latter is updated.

3. Bigger projects commonly involve more stakeholders and have more formal risk management process

Defect lifecycle...


NEW: Tester finds a ‘bug’ and posts it with the status NEW. This bug is yet to be studied/approved. The fate of a NEW bug is one of ASSIGNED, DROPPED and DEFERRED.

ASSIGNED / OPEN: Test / Development / Project lead studies the NEW bug and if it is found to be valid it is assigned to a member of the Development Team. The assigned Developer’s responsibility is now to fix the bug and have it COMPLETED. Sometimes, ASSIGNED and OPEN can be different statuses. In that case, a bug can be open yet unassigned.

DEFERRED: If a valid NEW or ASSIGNED bug is decided to be fixed in upcoming releases instead of the current release it is DEFERRED. This bug is ASSIGNED when the time comes.

DROPPED / REJECTED: Test / Development/ Project lead studies the NEW bug and if it is found to be invalid, it is DROPPED / REJECTED. Note that the specific reason for this action needs to be given.

COMPLETED / FIXED / RESOLVED / TEST: Developer ‘fixes’ the bug that is ASSIGNED to him or her. Now, the ‘fixed’ bug needs to be verified by the Test Team and the Development Team ‘assigns’ the bug back to the Test Team. A COMPLETED bug is either CLOSED, if fine, or REASSIGNED, if still not fine.

If a Developer cannot fix a bug, some organizations may offer the following statuses:

Won’t Fix / Can’t Fix: The Developer will not or cannot fix the bug due to some reason.

Can’t Reproduce: The Developer is unable to reproduce the bug.

Need More Information: The Developer needs more information on the bug from the Tester.

REASSIGNED / REOPENED: If the Tester finds that the ‘fixed’ bug is in fact not fixed or only partially fixed, it is reassigned to the Developer who ‘fixed’ it. A REASSIGNED bug needs to be COMPLETED again.

CLOSED / VERIFIED: If the Tester / Test Lead finds that the bug is indeed fixed and is no more of any concern, it is CLOSED / VERIFIED. This is the happy ending.

Bug Life Cycle Implementation Guidelines

Make sure the entire team understands what each bug status exactly means. Also, make sure the bug life cycle is documented.
Ensure that each individual clearly understands his/her responsibility as regards each bug.
Ensure that enough detail is entered in each status change. For example, do not simply DROP a bug but provide a reason for doing so.
If a bug tracking tool is being used, avoid entertaining any ‘bug related requests’ without an appropriate change in the status of the bug in the tool. Do not let anybody take shortcuts. Or else, you will never be able to get up-to-date bug metrics for analysis.

Test preparation process...

Baseline Documents
Construction of an application and testing are done using certain documents. These documents are written in sequence, each of it derived from the previous document.

Business Requirement
This document describes users needs for the application. This is done over a period of time, and going through various levels of requirements. This should also portrays functionalities that are technically feasible within the stipulated times frames for delivery of the application. As this contains user perspective requirements. User Acceptance Test is based on this document.

How to read a Business Requirement?
In case of the Integrated Test Process, this document is used to understand the user requirements and find the gaps between the User Requirement and Functional Specification. User Acceptance Test team should break the business requirement document into modules depending on how the user will use the application. While reading the document, test team should put themselves as end users of the application. This document would serve as a base for UAT test preparation.

Functional Specification
This document describes the functional needs, design of the flow and user maintained parameters. These are primarily derived from Business Requirement document, which specifies the client’s business needs.
The proposed application should adhere to the specifications specified in this document. This is used henceforth to develop further documents for software construction and validation and verification of the software. In order to achieve synchronization between the software construction and testing process. Functional Specification (FS) serves as the Base document.

How to read a Functional Specification?
The testing process begins by first understanding the functional specifications. The FS is normally divided into modules. The tester should understand the entire functionality that is proposed in the document by reading it thoroughly. It is natural for a tester at this point to get confused on the total flow and functionality. In order to overcome these, it is advisable for the tester to read the document multiple times, seeking clarifications then and there until clarity is achieved. Testers are then given a module or multiple modules for validation and verification. These modules then become the tester’s responsibility.
The Tester should then begin to acquire an in-depth knowledge of their respective modules. In the process, these modules should be split into segments like field level validations, module rules, business rules etc. In order to do the same modules precisely the tester should interpret importance and its role within the application. A high level understanding of the data requirements for respective modules is also expected from the tester at this point. Interaction with test lead at this juncture is crucial to draw a testing approach, like an end-to-end test coverage or individual test. (Explained later in the document)

Tester’s Reading Perspective
Functional specification is sometimes written assuming some level of knowledge of the Testers and constructors. We can categorize the explanations by

Explicit Rules: Functionality expressed as conditions clearly in writing, in the document.

Implicit Rules: Functionality that is implied based on what is expressed as a specification/condition or requirement of a user.

The tester must also bear in mind, the test type i.e. Integrated System Testing (IST) or User Acceptance Testing (UAT). Based on this, he should orient his testing approach.

Design Specification 
This document is prepared based on the functional specification. It contains the system architecture, table structures and program specifications. This is ideally prepared and used by the construction team. The Test Team should also have a detailed understanding of the design specification in order to understand the system architecture.

System Specification 
This document is a combination of functional specification and design specification. This is used in case of small applications or an enhancement to an application. Under such situations it may not be advisable make two documents.

Prototype
This is look and feel representation of the application that is proposed. This basically shows the placement of the fields, modules and generic flow of the application. The main objective of the prototype is to demonstrate the understanding of the application to the users and obtain their buy-in before actual design and construction begins.
The development team also uses the prototype as a guide to build the application. This is usually done using HTML or MS PowerPoint with user interaction facility.

Scenarios in Prototype
The flow and positioning of the fields and modules are projected using several possible business scenarios derived from the application functionality.
Testers should not expect all possible scenarios to be covered in the prototype.

Flow of Prototype
The flow and positioning are derived from initial documentation off the project. A project is normally dynamic during initial stages, and hence tester should bear in mind the changes to the specification, if any, while using the prototype to develop test conditions.
It is a value addition to the project when tester can identify mismatches between the specifications and prototype, as the application can be rectified in the initial stages itself.

Test estimation for testing process


Introduction:


In my opinion, one of the most difficult and critical activities in IT is the estimation process. I believe that it occurs because when we say that one project will be accomplished in such time by at such cost, it must happen.

The testing estimation process in place was quite simple. The inputs for the process, provided by the development team, were: the size of the development team and the number of working days needed for building a solution before starting systems tests.

The testing estimation process said that the number of testing engineers would be half of the number of development engineers and one third of the number of development working days.

1st Rule: Estimation shall be always based on the software requirements

All estimation should be based on what would be tested, i.e., the software requirements.

Normally, the software requirements were only established by the development team without any or just a little participation from the testing team. After the specification have been established and the project costs and duration have been estimated, the development team asks how long would take for testing the solution. The answer should be said almost right away. Then, the software requirements shall be read and understood by the testing team, too. Without the testing participation, no serious estimation can be considered.

2nd Rule: Estimation shall be based on expert judgment

Before estimating, the testing team classifies the requirements in the following categories:

" Critical: The development team has little knowledge in how to implement it;
" High: The development team has good knowledge in how to implement it but it is not an easy task;
" Normal: The development team has good knowledge in how to implement.

The experts in each requirement should say how long it would take for testing them. The categories would help the experts in estimating the effort for testing the requirements.

3rd Rule: Estimation shall be based on previous projects
All estimation should be based on previous projects. If a new project has similar requirements from a previous one, the estimation is based on that project.
4th Rule: Estimation shall be based on metrics
My organization has created an OPD, Organization Process Database, where the project metrics are recorded. We have recorded metrics from three years ago obtained from dozens of projects.
The number of requirements is the basic information for estimating a testing project. From it, my organization has metrics that guide us to estimate a testing project. The table below shows the metrics used to estimate a testing project. The team size is 01 testing engineer.

Metric Value

1.Number of test cases created for each requirement 4,53
2.Number of test cases developed by Working day 14,47
3.Number of test cases executed by Working day 10,20
4.Number of ARs for testcase 0,77
5.Number of ARs verified by Working day 24,64
For instance, if we have a project with 70 functional requirements and a testing team size of 2 engineers, we reach the following estimates:

Metric Value

Number of testcases - based on metric 1 317,10
Preparation phase - based on metric 2 11 working days
Execution phase - based on metric 3 16 working days
Number of ARs - based on metric 4 244 ARs
Regression phase - based on metric 5 6 working days

The testing duration is estimated in 22 (16+6) working days. Plus, 11 working days for preparing it.
A spreadsheet was created in order to find out the estimation and calculate the duration of tests and testing costs. They are based on the following formulas:
Testing working days = (Development working days) / 3.

Testing engineers = (Development engineers) / 2.

Testing costs = Testing working days * Testing engineers * person daily costs.
As the process was only playing with numbers, it was not necessary to register anywhere how the estimation was obtained.

To exemplify how the process worked, if one development team said that to deliver a solution for systems testing it would need 4 engineers and 66 working days then, the systems test would need 2 engineers (half) and 21 working days (one third). So, the solution would be ready for delivery to the customer after 87 (66+21) working days.

Just to be clear, in testing time, it was not included the time for developing the testcases and preparing the testing environment. Normally, it would need an extra 10 days for the testing team.

5th Rule: Estimation shall never forget the past

I have not sent away the past. The testing team continues using the old process and the spreadsheet. After the estimation is done following the new rules, the testing team estimates again using the old process in order to compare both results.

Normally, the results from the new estimate process are cheaper and faster than the old one in about 20 to 25%. If the testing team gets a different percentage, the testing team returns to the process in order to understand if something was missed.

6th Rule: Estimation shall be recorded

All decisions should be recorded. It is very important because if requirements change for any reason, the records would help the testing team to estimate again. The testing team would not need to return for all steps and take the same decisions again. Sometimes, it is an opportunity to adjust the estimation made earlier.

7th Rule: Estimation shall be supported by tools

A new spreadsheet has been created containing metrics that help to reach the estimation quickly. The spreadsheet calculates automatically the costs and duration for each testing phase.
There is also a letter template that contains some sections such as: cost table, risks, and free notes to be filled out. This letter is sent to the customer. It also shows the different options for testing that can help the customer decides which kind of test he needs.

8th Rule: Estimation shall always be verified

Finally, All estimation should be verified. I've created another spreadsheet for recording the estimations. The estimation is compared to the previous ones recorded in a spreadsheet to see if they have similar trend. If the estimation has any deviation from the recorded ones, then a re-estimation should be made.