(DV-2111) Test Management

You can start the Test Management directly from the main Validate cockpit.

_{Validate Dashboard}

Test Management is divided into two tabs:

1. Test Cases
2. Test Plans

_{Test Cases}

It is good to understand the differences between Test Cases and Test Plans. Test Plan is a basic organizational entity that allows combining different types and any number of Test Cases together. There are several specific actions that can be done on top of the Test Plan reminded further in the text. Definition of directories may seem similar to Test Plan definition but they have fewer/different options and are more folder-like. Any Test Case cannot be executed without Test Plan. Test Plan serves as a frame for its Test Cases. Test Plan holds the further definition of execution, settings, testers, statuses, documents, and others. Test Case carries basic information about the type (a type of tested subject) and variants (particular details about tested objects).

Test Cases Tab

Here you can do the following:

• New Test Case (Shift + F1)
• Create a new directory (Shift + F2)
• Delete selected Test Case/Directory (Shift + F4)
• Find (Shift + F7)
• Find next (Shift + F8)
• Test Management Settings (CTRL + F9)
• Variant Editor (F9)
• Go to Variant Editor (this option is only available if Backend Test Cases, like Query Test Case, for example, is enabled in Validate Setup /DVD/VALIDATE_SETUP)

New Test Case

One of the first steps while using Validate benefits is Test Case creation. 

There are 3 ways to create a new Test Case,

• New Test Case button,
• Choosing the directory from the context menu and selecting Add Test Case.
• Hotkey Shift + F1

_{Creating a new Test Case}

The technical name for the Test Case (e.g. TST_CASE) has a Description field and a Directory field that can be selected to be pre-populated with the current directory, if not selected you can manually add the directory. Note: This field is mandatory and you cannot create a Test Case outside of the directory.

_{New Test Case window}

The Fields marked with a tick box are mandatory. 
The Test Case Type is a drop-down menu, which contains the different types of Test Cases. Test Case Type 'Manual' is always preselected by default.

Depending on your Validate setup, only certain Test Case Types may be visible. 
For example, if the Validate setup were to also contain BW Test Case Types, you will see an additional button at the bottom of the 'New Test Case window', where you can define additional information, based on these Test Case Types. 

_{Variants Management for Validate BW Test Cases}

Create a new directory

By pressing the Create new directory, (Shift + F2) a new directory is created and you can add subdirectories by selecting in the context menu of the existing directory and choosing Add Subdirectory. 

_{Creating new directory}

A New Directory window looks like this:

_{New Directory window}

Dir. Technical Name contains the technical name (e.g. FIELD_TST_DIR), whereas the Dir. Description is what is displayed in the navigation tree panel.
The Parent Directory field is pre-populated with the current directory, if this option is not selected the Parent directory needs to fill in manually.

Delete directory and Test Cases

If Test Cases and/or directories are no longer necessary these can be deleted:

• By selecting in the context menu of an existing Test Case/directory and choosing delete.
• By Selecting Test Case/Directory and pressing the delete button, (Shift + F4)

_{Deleting items}

Transport Test Cases

Via the use of transports Test Cases and their content can be transported from the system to the system. Therefore you can define Test Cases in one system and then reuse the same Test Case in another system. If there already exists the same Test Cases or directories, these will be overwritten when transported into the system.

_{Transport Test Cases for a given directory}

Find and Find next

If you have many Test Cases and/or directories you can use the Find and Find next search features, to do so you can use the Find button or (Shift + F7).

_{Search window}

In the tree structure, you can search for elements individually or jointly by using the  asterisk for replacing a character or character sequence. If it's a broad search and has a high number of hits, you can search through these by using Find next (Shift + F8).

Where used for Test Cases

If you want to check all Test Plans where a particular Test Case is used, use the "Where Used" feature found in the context menu displayed by selecting the context menu of the Test Case. The Result table contains all Test Plan information where this variant is used.

_{Where-Used for given Test Case}

_{Where-Used results list}

Test Plans Tab

This part of the Test Management is dedicated to test planning and test monitoring.

_{Test Plans}

In the Test Plans tab you can find these buttons:

• New Test Plan (Shift + F1) / Edit Test Plan (Double-click on existing plan)
• Manage Test Plan Scope
• Assign testers (Shift + F2)
• Set Status (Shift + F4)
• Worklist view (Shift + F5)
• Filter (Shift + F6)
• Refresh (F5)
• Find (Shift + F7)
• Find next (Shift + F8)
• Go to Backend Variant Editor (F9)

New Test Plan / Edit Test Plan

Test Plan creation is another step to take when you want to order your Test Cases and explore other possible operations with Test Cases.

A new Test Plan is created by pressing the New Test Plan button (Shift + F1).

_{Creating Test Plan}

_{New Test Plan window}

Mandatory fields are checked with a checkbox, the Dialog box now contains a new button, Scope, and allows you to edit the content of the Test Plan immediately. Test Cases can be also assigned/edited afterward via the Edit command or by selecting in the context menu of on the created plan and choosing Edit. 

In the Edit Test Plan dialog box both Technical name and Description must be filled out.

_{Edit Test Plan window}

Manage Test Plan Scope

From this dialog window, you can edit existing Test Plans and create new Test Plans.

_{Manage Test Plan Scope window}

This dialog window contains two main parts:

• The left part containing a tree of Existing Test Cases
• The right part containing your currently selected Test Plan tree

You can add new items to the Test Plan either by drag and drop or double-clicking on the desired item. 
Please be aware there are some actions, which are invalid:

• Deleting Test Plan itself,
• Dragging items from Test Plan tree to Test Cases tree,
• Dragging parents onto their children,
• Dragging item outside of the plan,
• Dragging already incorporated test case/directory again
• Dragging or deleting variants of Test Case.

Except for above mentioned, the items are fully moveable. You can add new items/directories, change their order or parent, delete undesired items/directories, or select variants that are to be used in the testing. However, you are unable to create a new folder in the Test Plan or rename any of the items.

Assign testers

You can assign existing Test Plans to a specific person or to a list of people responsible for the testing. By clicking on the Assign testers icon or selecting in the context menu of the plan and choosing Assign testers (Shift + F2).

_{Assign testers window}

The User Name is the SAP login for the tester and here you can use the SAP F4 help. You can assign one or more testers to a Test Plan, a Test Case directory, and/or individual Test Cases.

Test plan statistics

Test plan statistics show state, status, and results of test plans, test cases, and test case variants.

The 'U|P|F' column represents the state of test plans, test cases, test case variants. 

• U - Stands for the number of untested test plans, test cases, test case variants
• P - Stands for the number of test plans, test cases, test case variants which are in process state
• F - Stands for the number of finished test plans, test cases, test case variants

Set Status

When setting the status of a Test Plan, it is necessary to have the status on Active so the testers can see this in their Worklist. 
Test Plans can have the following statuses:

• Active and Closed.

Test Cases can have the following statuses:

• Untested, In process, Failed, Successful.

_{Set Status window}

Test Cases and Test Plans can have a Status Document, here you can add a more detailed description of the testing results. To create a status document selecting in the context menu of a Test Case/Test Plan and select the „Create Status Document" option.

_{Status Document}

Worklist view

In the worklist view, you can display all the Test Cases assigned to a given user, and are visible in their worklist. You can navigate via the worklist view button (Shift + F5).

_{Worklist window}

Simply enter the user's SAP login and press Confirm if you want to reset the view, leave the User name field blank and press confirm.

Filter

You can filter the Test Cases by pressing Filter icon (Shift + F6). Test Cases or Test Plans can both be filtered by their Technical name, and Test Cases can be filtered according to their status. It is also possible to use an  asterisk to broaden the filter.

_{Filter window}

Find and Find next

When you have many Test Cases and/or directories, use the Find and Find next and search by pressing the Find button (Shift + F7).

_{Find window}

Test Plan Documentation

You can generate documentation from the Test Plan in the form of an MS Word document or HTML document. This documentation contains a list of all the Test Cases, including their statuses, Status Documents, and additional information depending on the type of the Test Case.

_{Generate Test Plan Documentation}

Before actual generation, you can select Test Cases to be part of Test Plan documentation, type of generated document for Test Plan Documentation, and what kind of information to be generated.

For generating the report as 'Microsoft Word document' or 'HTML document (*.html)' there is a possibility to edit execution options.

For generating the report as 'Microsoft Document (*.docx)' execution options are handled via Template, e.g. for skipping variant details you will want to select 'Test Plan w/o details Template'. 

_{Test Plan Documentation Options}

1. Microsoft Word document - will generate documentation with a fixed layout for an opened Word document, therefore it requires installed MS Office to be installed on local PC. 

Test Plan documentation generated to MS Word document 

2. Microsoft Document - will generate a .docx file based on the loaded template from Validate's repository or from local PC.  Two basic templates are delivered in Validate repository. The template has a layout free to change. Placeholders are used for the biding information in the template. For more information on how to create your own template, see in separate section Docx templates. You can select a different file path to save the generated document. Documentation does not contain Status Document (for now).

Template with placeholders to generate Microsoft Document 

3. An HTML document is more suitable for documentation with large volumes of data, e.g. variant details. Default directory for HTML document is Desktop. You can select a different directory for saving HTML outputs. HTML document does not contain Status Document.

_{HTML Test Plan documentation}

Test Plan Scheduling

You can schedule Test Plans to run on a regular basis and send notifications about the result.

Prerequisites

For Scheduling to take place, you need to start Scheduling Checker in the Test Management Settings.

_{Test Management Settings icon}

The Scheduling Checker is reoccurring background job that checks for all the scheduled Test Plans every 15 minutes (i.e. if Scheduling Checker is executed on 17:40 it will check for Test Plans that are scheduled between 17:25 and 17:40).
You can set the Checking Period, which defines how often the Scheduling Checker checks for the scheduled Test Plans.
The actual precision of the scheduled time is dependent on this Checking Period – the smaller the number, the better the precision. It is recommended to set a limit of up to 10 minutes. For a Checking Period of 10 minutes, the deviation is +10 minutes (so a Test Plan executed for 16:30 can be actually executed between 16:30 and 16:40).
You can also set the number of background jobs that are to be executed within a given Test Case.
Each run of the Scheduling Checker is logged and these logs are available from the Test Management Settings.

_{Test Management Settings}

In order for the notification functionality to be working properly, the mail server has to be functioning on the system on which Validate is installed on. To test whether Validate can send emails correctly, go into the Test Management Settings, and in the 'Notification' section, type your email address. If a test email arrives, Validate can then send notification emails as expected.
Validate uses Business Communication Service (BCS) for sending emails.

Scheduling Overview

To see only the Test Plans that have been scheduled, you can change the view by customizing the filter.

_{Filter only scheduled Test Plans}

Test Plans with this filter enabled have additional columns that are relevant to scheduling, such as logs, execution time, email address, etc.

_{Overview of scheduled Test Plans (with the filter enabled)}

How to Schedule Test Plan

To schedule a given Test Plan, choose the Test Plan from the context menu and select 'Schedule Test Plan'.

_{Creating new Scheduling for the Test Plan}

You can schedule a Test Plan to be run just once, daily, or weekly. There is also the option to execute a given Test Plan up to 3 times per day.

_{New Test Plan Scheduling window}

For each Test Plan, you can define a scope - select which Test Cases have to be executed in the scheduled Test Plan.

^{Selection of Test Cases}

For each Test Case type, execution steps are available and therefore a specific process of execution to each Test Case type can be defined.

^{Selection of Test Steps}

Test Plans that are scheduled, have the icon of the calendar with a clock, near the Test Plan.

_{Scheduled Test Plans with the icon}

Once the Test Plan has been scheduled and executed, you can check the logs by selecting them in the context menu of the Test Plan.

_{Show Scheduling Logs}

_{Scheduling Logs for given Test Plan}

If the email address has been set, a notification about the results will be sent. 

_{Notification email sent after finished Test Plan}

Scheduling Checker Algorithm

When Scheduling Test Plans, it is useful to know the algorithm for the Scheduling Checker to better understand its behavior.
Scheduling Checker is scheduled as a background job every X minutes (X is Checking Period in Test Management settings). During the Scheduling Checker execution, two actions are being performed:

1. Scheduling Checker will execute in the background all Test Plans that are scheduled within a given date range and within 15 minutes intervals (i.e. if Scheduling Checker is executing at 15:45, it will execute all Test Plans that are scheduled between 15:30 and 15:45).
2. Scheduling Checker will send email notifications for finished Test Plans where an email address is set.

Test Plan Snapshot

You can create a historical snapshot of Test Plan backend Test Cases. This snapshot will save information about the selected Test Plan and the actual execution status. To create the snapshot, in the context menu of the desired Test Plan select the 'Variants Overview' -> 'Create New Snapshot' option from the context menu. You can then select which Test Cases should be included in the snapshot.

_{Creating New History Snapshot for Test Plan}

It is possible to create snapshots automatically after each scheduled Test Plan execution by selecting the 'Create Snapshot After Execution' option within the scheduling options of the Test Plan. The Snapshot will be created only for Test Cases executed in scheduled execution.

_{Automatic Snapshot Creation for Scheduling}

In the current version of Validate snapshots are created only for backend Test Cases Types:

• ListCube
• Table
• Query

To review historical snapshots that were created, in the context menu of desired Test Plan select 'Variants Overview'->'Show Variants Overview' option from the context menu. A new screen will be displayed with the already created snapshots.

_{History Overview of Test Plan}

At the top of the screen, you can select a period for which you want to display created snapshots as you can display multiple Test Plan history snapshots at once. By using F4 help you can change the date range and then by using the F8 refresh button you will load the data of appropriate snapshots. The History overview screen contains 5 main tabs:

1. Test Case Overview
2. Variant Overview
3. ListCube Variants Details [optional]
4. Table Variants Details [optional]
5. Query Variants Details [optional]

Test Case Overview Tab

In this tab, you can see one record for each Snapshot/Test Case pair. It contains information about the total number of variants present in the Test Case when a snapshot was created, the overall status of the Test Case, untested/successful/failed variant counts, and rows information.

_{Test Case Overview Tab}

In the 'User Comment' column you can provide any custom comment that will be saved there for later review.

Variant Overview Tab

Here is you will see an overview of each Snapshot/Test Case/Variant combination. The information displayed includes the variant description, compare/reporting status, tested/error rows count, and reporting status text if any.

_{Variant Overview Tab}

ListCube Variants Details Tab

This tab is only displayed if any of the selected snapshots actually contain one or more ListCube Test Cases. For each Snapshot/Test Case/Variant combination you can see details of its variant settings and execution status of all steps.

Table Variants Details Tab

This tab is only displayed if any of the selected snapshots actually contain one or more Table Test Cases. For each Snapshot/Test Case/Variant combination you can see details of its variant settings and execution status of all steps.

Query Variants Details Tab

This tab is only displayed if any of the selected snapshots actually contain one or more Query Test Cases. For each Snapshot/Test Case/Variant combination you can see details of its variant settings and execution status of all steps.

Test Plan Replication to HP ALM

When replication to the HP ALM server is correctly set up in the Datavard Validate settings, you can replicate Datavard Validate Test Plans in the HP ALM server. To replicate a Test Plan, in the context menu select 'Others' -> 'Replicate to HP ALM'. To replicate multiple Test Plans, in the context menu of a Test Plan Folder select 'Replicate to HP ALM'.

_{Replicating Test Plans to HP ALM server}

Choose a template from Listbox to fill the set of settings needed for replication. For immediate customization of settings click on the 'Edit' button. These changes will not affect any template. For replication click on the 'Continue' button. The last used template is saved for each replicated Test Plan and it will be preselected during the next replication.

_{Selecting template before replication of Test Plans to HP ALM server}

The following Validate objects are replicated:

• Test Case
  • generates Test object in HP ALM Test Plan area
• Test Plan
  • generates Test Set object in HP ALM Test lab area
• Test Case in the scope of Test Plan
  • generates Test Instance under Test Set
• Test Case status in Test Plan
  • generates HP ALM Run

The application log is generated during the replication process. It contains information about the replication status and possible errors that occurred during the replication.

_{Replication log}

The replication works only from Validate to HP ALM. There is no transfer of data or synchronization from HP ALM back to Validate.

Worklist

This part of the documentation explains the basic navigation and functions of the Worklist section.

_{Worklist in Validate Dashboard}

This area within Validate is focused on the testers, and those who see their Test Cases/Test Plans that are assigned to them.

_Worklist

Test Plans with the status New are not displayed here, in order to make New Test Plans visible their status needs to be changed to Active. You can do this through the Test Management -> Test Plans tab. Users will only see their assigned Test Cases. Also, directories are not displayed here.
To indicate which phase of testing the Test Cases are in, you can change their status. To change the status, click on the Set Status icon (Shift + F4), or select it in the context menu of the Test Case. In the following window, you can add a description for the Test Cases and set the status to Untested, In process, Failed or Successful.

_{Set Status window}

Test Case Mass Execution

In the Test Plans tab, there is an option to execute all Test Cases in the given Test Plan. For each Test Case Type, execution options are given and you can select which parts to execute and which do not (i.e. execute only Before Image for all ListCube Test Cases). 

_{Mass Execution in Test Plans tab}

For each Test Case type, execution options are available and therefore specific execution to each Test Case type can be defined.

_{Mass Execution – Main Screen}

After clicking on the "Change Selection" button, the scope of Mass Execution can be defined. Functionality is supported for all Test Cases that can be selected (i.e. Manual Test Case type cannot be selected, because it does not support the Mass Execution option).

_{Mass Execution – Selection of Test Cases} 

Backend Test Cases Mass Execution

When backend Test Cases are selected for mass execution you can define which test steps should be executed from each Test Case type. The list of test steps depends on the chosen backend scenarios. Please refer to chapter 8 for details of test steps for each type. Setting 'Number of jobs per Test Case', defines how many background jobs for each chosen step is going to be executed. 
Each backend Test Case type contains the option 'Run Sequentially'. If this setting is set all chosen test steps will be executed in the order in which they are defined within the list. The execution of a step will be started once the preceding step is finished. There is a maximum time setting that once it is reached while waiting for the preceding step to finish will end the execution of the Test Case. Refer to the Settings chapter for details. If the 'Run Sequentially' option is not used all selected test steps are executed in parallel each using the specified number of jobs. No wait is used for these cases. 

_{Mass Execution of backend Test Case}