Lab Reduction for reqreduction
v102103 reqreduction.txt s_reqreduction.html ../logev

Lab Reduction is a companion web based tool to Data Reduction except that it operates on real test data rather than simulated test data. The data reduction is performed in real time as the test executes. Anyone on the network can view the test results in real time during the testing.

Running Real Time Test Results Script

This tool is started as a DOS application in a directory which contains the test data. The syntax is:

perl labreduction.pl [filename.txt] [delayinsecs] [TocCor] [TocCov] [LabData] [LeFind] [LeNoFind] [Results] [CPC]

where all [fields] are optional and [blank] or [-] is default or are set as follows:

filename.txt Is the test file name, the default is testlog.txt. The report name is derived from this file name. The default report name is testlog.html. When entering a file name the .txt extension must be used (e.g. AAAa.txt, AAAb.txt).

delayinsecs The script executes 100 times before ending, this is the delay between executions, the default is 20 seconds.

TocCor If set to [y] correlates the log events to requirements.

TocCov If set to [y] shows which requirements were addressed and which requirements were not addressed by the test results.

LabData If set to [y] shows the raw test data.

LeFind If set to [y] shows the log events that were found. The default is [y].

LeNoFind If set to [y] shows the log events NOT found in these test results.

Results If set to [y] shows the test results correlated to the original log events. The deafult is [y].

CPC If set to [all], [red], or [a cpc] shows req's satisfied and NOT satisfied by these test results.

Filter If set to [y], filters items containing certain words from the reports. The deafult is [y].

The file containing the test results is automatically updated by the BOOT application for some test cases. In all other BOOT and RED test cases the file is manually updated by the testers by either copying log events from the emulator window or entering log events based on emulator profile points. Using appropriate file naming conventions, parallel testing can be performed.

This application requires the inclusion of a special file (translate.dat) that is created by the logevent application and stored with the instrumentation analysis results. This file translates the log event codes to the original english equivalents. It also requires a version of the TOC (s_toc.dat) to correlate the TOC requirements with the lab test results.

Analyzing The Test Results

The test results analysis begins during informal testing by examining the real time test results.

Are the start and stop log events as expected
Are any log events grossly out of place
Do the log events and sequence look resonable
Do the number of log events match previous test runs
Do the source files and function names in the log events look resonable
Does the layout of the log event sequence appear to be similar to previous test runs

If the results appear to be reasonable, copy and paste them into the test procedures. They will form the factory template that will be used during formal testing.

As part of an offline analysis, use the IAT reports and software source to locate the test result log events. Examine the software and look for potential missing log events or wrong long events. This is a final examination to confirm that the test results are as expected.

For dry run, create a single complete test results report following the sequence that is expected during formal test. Use these dry run test results to support the formal test.

At the conclusion of all the DRY RUN tests enable the log events found, log events not found, requirements correlation, and requirements coverage reports. Look for log events and requirements that were not captured during the all the test activity. If needed, create new tests to capture the missing requirements.

Analysis Reports	Script Option	Comment
Lab Data	LabData	This report shows the raw numerical log events captured during lag testing or during a simulation run of a software collection in analysis
Test Comments	Comments	Summary of the comments entered by testers into the test file.
PUIs Verified	Comments	The list of PUIs verified in this test sequence. Use it in the filter service to compared different test sequences.
Req Coverage	Comments	This is a list of PUIs and the number of times the PUI was encountered in a log event with this test sequence. Use this to gain confidence in the test coverage. For example, a PUI that has been hit 50 times was probably the result of slightly different code paths.
Test Results	Results	This is the test results in log event format. This analysis connects the raw numerical log events captured during lab testing with the original source code comment statements, source file, and C-function.
Missing Req's	Results	This is a list of req's that should have been in this test but are missing. It is based on the IAT filter service where one set of reqs's are subtracted from another set of req's.
Log Events Found	LeFind	This analysis lists the log events found in these test results.
Log Events NOT Found Current Future Negative De-Instrumented User Defined	LeNoFind	This analysis lists the log events NOT found in these test results.
PUIs	TocCor TocCov	This analysis extracts the TOC numbers from the log events captured during lab testing.
TOC Text Correlation	TocCor	This analysis correlates the log events captured during lab testing with the baseline TOC. If multiple events are associated with the same req, that req is displayed multiple times. The sequence matches the original test events sequence.
Not in TOC Text	TocCor	This analysis shows the log events that are NOT in the current TOC baseline. For example, if a TOC req is deleted after the software is coded or if the implementer enters the wrong TOC number in the comment, this report will show those events.
In This CPC Collection	TocCov	The TOC reqs are allocated to CPCs. This analysis shows the TOC reqs that are in this CPC collection. All the reqs in this report area should contain the CPC or CPCs that represent this analysis run. If there are only foreign CPCs, then this software collection has implemented reqs that are implemented elsewhere, or the DOORS data base is in error. If this analysis represents all the software then this report is for the current software baseline.
Not In This CPC Collection - PWR LOG STP CA CPP DLP DRPS FIP FPCP GPS MCC MSAW RDP RPP SIM SYS WXP Current Future Negative De-Instrumented User Defined	TocCov	This analysis shows the TOC reqs that should be in the analyzed software but are NOT in this CPC collection. It may be that only a portion of a CPC was analyzed and so there are outstanding reqs. It may be that reqs are not properly allocated in DOORS. It may be that the implementation missed reqs and the software needs to be updated. This report only applies when CPCs are analyzed.
Not in Software Baseline	None	This analysis shows the TOC requirements that are NOT in the current analysis baseline. When the entire software set is analyzed, this report area should only contain paragraph headings, information text, and NON software reqs.

NOTE: These reports are option sensitive. If an option was not selected, the report link is not present.

Other Analysis Reports . TOC req's . simulated logevents . translate file . Error_Codes.txt

Settings

Paths
Main Path
Simulation Directory
Analysis Directory
Analysis Report Name
Test File Name

Settings case sensitive
Script Rerun Delay (sec)
CPC, Release, or Test Case Considered
Trace Events

Requirements Accounting - Fixed case sensitive
Future Events
De-Instrumented
Negative Paths

Requirements Accounting - User Defined case sensitive Categories
Name Description PUI's

Display Filters - flaten display case insensitive
Raw LCD Log Events
Raw PAD Log Events
Raw PIN Log Events
Raw LOC Log Events
Raw Any Log Events
Text Results

Ignore Data - for very fast bypassing of noise case insensitive
Bypass Noise Raw Events
Bypass Noise Text Results

Keywords copy paste REQs from word table column get converted to OR format - case insensitive
Red
Blue
Navy
Purple
Green
Maroon
case sensitive
Orange

Expected Req's From Test Procedure - copy & paste from doc case insensitive
Orange LE SV

LabData	Test Comments	Test Results	LE Found	LE NOT Found	Show Only Keyword Events
TOC Correlation	TOC Coverage	Not in Software Baseline	Missed Req's	Func Call Seq	Show Only Expected Req's
Filter Classified	Server Config

12/14/2003 08:45:57

Data Reduction Reports for reqreduction

lab data 13
0000 time I This file was automatically generated by reqreduction.pl Total Log Events 40 Unique Log Events 5 Note the EEEEEE field (100000) now represents the number of times a log event was encountered in the entire test collection 1111 1026 I 100008 1111 1027 I 100008 1111 1028 I 100006 1111 1029 I 100005 1111 1030 I 100010 1111 1031 I 100004

Test Comments Summary Comments entered by testers into test data file

Log Events Data
This file was automatically generated by reqreduction.pl
Total Log Events 40
Unique Log Events 5
Note the EEEEEE field (100000) now represents the number of times a log event was encountered in the entire test collection

Test Results Lab Data Connected To File Name, C-function, and Comment Statements 12 reqreduction.txt 12/14/2003 08:45:57
1 0000 time I Translate File Created 12/09/2003 13:52:41
This file was automatically generated by reqreduction.pl
Total Log Events 40
Unique Log Events 5
Note the EEEEEE field (100000) now represents the number of times a log event was encountered in the entire test collection

1 1111 1026 I badcoding.c function_first LE SV TOC-001 this is a real rocket scientist - will not compile here
2 1111 1027 I badcoding.c function_a LE SV TOC-001 this is a real rocket scientist - will not compile here
3 1111 1028 I badcoding.c function_b LE SV TOC-008 split the if block
4 1111 1029 I badcoding.c function_b LE SV TOC-002 this is paraphrase of req 2
5 1111 1030 I badcoding.c function_b LE SV TOC-003 this is paraphrase of req 3
6 1111 1031 I badcoding.c function_c LE SV TOC-007 req 7 about item 7 stuff AUTO COMMENT // too lazy to paraphrase req

99XX series reserved for IAT self tests

PUIs Verified - All in this thread use this in the filter service to determine test run differences

s_reqreduction.html |TOC-001 |TOC-002 |TOC-003 |TOC-007 |TOC-008 |done

Total Verified Req's 5

Req Coverage - All in this thread

TOC-001

TOC-002

TOC-003

TOC-007

TOC-008

Total Verified Req's 5

Req's Missed In This Test Run This report with the default settings shows the req's that were MISSED in this test. It is based on the filter tool where an initial set of items are found and a second set is subtracted from the intial set. In this case the initial set are the expected req's as defined in the test procedure, and the second set are the req's found in the actual test data. Req's are Marked if they are not in the software. Req's are also Marked if they are turned off in the debugger.

Don't Show Located Items Show Removed Items Flip Locate & Remove

Locate: nnoonnee

Remove: s_reqreduction.html |TOC-001 |TOC-002 |TOC-003 |TOC-007 |TOC-008 |done

Marked: Bx_xx|TOC-009 |TOC-018 |done

Check if it was Manually verified

Reqs Verification Summary . . . Missed 0 . . . Marked 0 . . . DeInst 0 . . . Manual 0 . . . Found 0

Log Events Found in these test results 7 out of a total of 46

1 time, Translate File Created 12/09/2003 13:52:41
2 1026, badcoding.c function_first LE SV TOC-001 this is a real rocket scientist - will not compile here
3 1027, badcoding.c function_a LE SV TOC-001 this is a real rocket scientist - will not compile here
4 1028, badcoding.c function_b LE SV TOC-008 split the if block
5 1029, badcoding.c function_b LE SV TOC-002 this is paraphrase of req 2
6 1030, badcoding.c function_b LE SV TOC-003 this is paraphrase of req 3
7 1031, badcoding.c function_c LE SV TOC-007 req 7 about item 7 stuff AUTO COMMENT // too lazy to paraphrase req

Log Events NOT Found in these test results 39 out of a total of 46

Current Positive paths that were not logged during testing and are not accounted for in the following categories.

TRACE Paths These are not req's verification log events. They are trace paths to facilitate understanding the operation of the software. Missing them is not important.

Future

Negative

Special De-Instrumented

User Defined

All User Defined
Analysis Verify by analysis.
Missing Log Events The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events.
Inspections and Analysis These reqs will be verified using manual inspection and analysis techniques

Current Positive Paths These are requirements that should not be missed. If they are here either there is dead code or testing is not inclusive of the path.

1 1001, anylogevents.c function_lefirst LE SV TOC-001 completed the orange portion of req 1
2 1002, anylogevents.c function_lea LE SV TOC-001 completed the blue portion of req 1
3 1003, anylogevents.c function_lea LE SV TOC-002 completed req 2
4 1004, anylogevents.c function_leb LE SV TOC-003 starting req 3 - this is a huge loop 3
5 1005, anylogevents.c function_leb LE SV TOC-003 req 3 - but you can log in loop and filter during test
6 1006, anylogevents.c function_leb LE SV TOC-003 finished req 3 - so its ok to bracket huge loops
7 1007, anylogevents.c function_leb LE SV TOC-004 starting req 4 - this is a realtime loop
8 1008, anylogevents.c function_leb LE SV TOC-004 req - but you can log in loop and filter during test 3
9 1009, anylogevents.c function_leb LE SV TOC-004 finished req 4 - so its ok to bracket real time loops
10 1010, anylogevents.c function_leb LE SV TOC-004 completed req 4 its ok to no place a req in the exact spot where its completed
11 1011, anylogevents.c function_leb LE SV TOC-005 completed req 5 stuff
12 1012, anylogevents.c function_leb LE SV TOC-006 completed req 6
13 1013, anylogevents.c function_leb LE SV TOC-007 completed req 7
14 1014, anylogevents.c function_leb LE SV TOC-008 completed req 8
15 1015, anylogevents.c function_leb LE SV TOC-010 this is divide by zero error
16 1016, anylogevents.c function_led LE SV TOC-011 completed req 11 stuff
17 1017, anylogevents.c function_led LE SV TOC-012 completed req 12 stuff
18 1018, anylogevents.c function_led LE SV TOC-013 completed req 13 stuff
19 1019, anylogevents.c function_led LE SV TOC-014 completed req 14 stuff
20 1020, anylogevents.c function_lee LE SV TOC-015 started req 15 stuff - pick start or end but not both
21 1021, anylogevents.c function_lee LE SV TOC-016 started req 16 stuff - pick start or end but not both
23 1023, anylogevents.c function_lee LE SV TOC-015 completed req 15 stuff
24 1024, anylogevents.c function_lee LE SV TOC-016 completed req 16 stuff
26 1032, badcoding.c function_c LE SV TOC-008 we really should log all error calls but NOT HERE
27 1033, badcoding.c function_c LE SV TOC-008 we really should log all error calls
28 1034, badcoding.c function_c LE SV TOC-008 we really should log all error calls
29 1035, badcoding.c function_c LE SV TOC-008 we really should log all error calls
Sub Total 27

Current Negative PathsThese requirements are the result of negative path in the code. They can only be surfaced with inserted errors via the emulator. nnoonnee

Sub Total

Future These are future requirements and should not be part of the test results. \$1\.5

Sub Total

Special De-Instrumented PathsThese requirements are in a real time path and have not been instrumented. TOC-017
22 1022, anylogevents.c function_lee LE SV TOC-017 started req 17 stuff - pick start or end but not both DE-INST PATH
25 1025, anylogevents.c function_lee LE SV TOC-017 completed req 17 stuff DE-INST PATH

Sub Total 2

User Defined These categories are defined by the user. TOC-xxxx|Bx_xx|TOC-009 |TOC-018 |done |TOC-011 |TOC-012

User Defined - Analysis Verify by analysis. TOC-xxxx

Sub Total 0

User Defined - Missing Log Events The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events. Bx_xx|TOC-009 |TOC-018 |done

Sub Total 0

User Defined - Inspections and Analysis These reqs will be verified using manual inspection and analysis techniques TOC-011 |TOC-012

Sub Total 0

TRACE Paths These are not req's verification log events. They are trace paths to facilitate understanding the operation of the software. Missing them is not important.

30 5001, anylogevents.c function_lee TRACE PURGE_BUFFER
31 5002, anylogevents.c function_lee TRACE PURGE_BUFFER
32 7001, anylogevents.c function_lee LOC log = any + loc(aslong_as_there_is_semicolon);
33 7002, anylogevents.c function_lee LOC log = any + loc;
34 7003, anylogevents.c function_lee LOC log = any+loc;
35 7004, anylogevents.c function_lee LOC log = any-loc*(some+more+stuff);
36 7005, anylogevents.c function_lee LOC log = any*loc;
37 7006, anylogevents.c function_lee LOC log = any/loc(nice_broad_pattern);
38 8001, HMI HMI_CALL (Happy Face);
39 9998, TOC-9998 Tool self test - missing toc text test

Sub Total 10

Total 39

PUIs 12
TOC-001 TOC-001 TOC-001 TOC-001 TOC-008 TOC-008 TOC-002 TOC-002 TOC-003 TOC-003 TOC-007 TOC-007

TOC Text Correlation 12
1 TOC-001 - $req 1 is about item 1 stuff $PWR $TST-1 Power Up $1.0

2 TOC-001 - $req 1 is about item 1 stuff $PWR $TST-1 Power Up $1.0

3 TOC-001 - $req 1 is about item 1 stuff $PWR $TST-1 Power Up $1.0

4 TOC-001 - $req 1 is about item 1 stuff $PWR $TST-1 Power Up $1.0

5 TOC-008 - $req 8 is about item 8 stuff $LOG $TST-2 Logon $1.0

6 TOC-008 - $req 8 is about item 8 stuff $LOG $TST-2 Logon $1.0

7 TOC-002 - $req 2 is about item 2 stuff $PWR LOG STP $TST-1 Power Up $1.0

8 TOC-002 - $req 2 is about item 2 stuff $PWR LOG STP $TST-1 Power Up $1.0

9 TOC-003 - $req 3 is about item 3 stuff $PWR $TST-1 Power Up $1.0

10 TOC-003 - $req 3 is about item 3 stuff $PWR $TST-1 Power Up $1.0

11 TOC-007 - $req 7 is about item 7 stuff $LOG $TST-2 Logon $1.0

12 TOC-007 - $req 7 is about item 7 stuff $LOG $TST-2 Logon $1.0

Not in TOC Text

Total 0

In Software Baseline 5 out of a total of 17 PWR|LOG|STP|CA|CPP|DLP|DRPS|FIP|FPCP|GPS|MCC|MSAW|RDP|RPP|SIM|SYS|WXP

3 - TOC-001 $req 1 is about item 1 stuff --- $PWR $TST-1 Power Up $1.0

4 - TOC-002 $req 2 is about item 2 stuff --- $PWR LOG STP $TST-1 Power Up $1.0

5 - TOC-003 $req 3 is about item 3 stuff --- $PWR $TST-1 Power Up $1.0

9 - TOC-007 $req 7 is about item 7 stuff --- $LOG $TST-2 Logon $1.0

10 - TOC-008 $req 8 is about item 8 stuff --- $LOG $TST-2 Logon $1.0

Not in this CPC Collection 12 out of a total of 17 PWR|LOG|STP|CA|CPP|DLP|DRPS|FIP|FPCP|GPS|MCC|MSAW|RDP|RPP|SIM|SYS|WXP

Different Buckets of Missing Log Events
Current Positive paths that were not logged during testing and are not accounted for in the following categories.

Future

Negative

Special De-Instrumented

User Defined

Note: All User Defined
Analysis Verify by analysis.
Missing Log Events The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events.
Inspections and Analysis These reqs will be verified using manual inspection and analysis techniques
All patterns for placing items in these buckets are case sensitive.

**Different Buckets of Missing Log Events**
Current	Positive paths that were not logged during testing and are not accounted for in the following categories.
Future
Negative
Special De-Instrumented
User Defined
Note: All User Defined
Analysis	Verify by analysis.
Missing Log Events	The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events.
Inspections and Analysis	These reqs will be verified using manual inspection and analysis techniques
	All patterns for placing items in these buckets are case sensitive.

Current Positive Paths These are requirements that should not be missed. If they are here either there is dead code or testing is not inclusive of the path.

6 TOC-004 $req 4 is about item 4 stuff --- $PWR $TST-1 Power Up $1.0

8 TOC-006 $req 6 is about item 6 stuff --- $LOG $TST-2 Logon $1.0

15 TOC-013 $req 13 is about item 13 stuff --- $LOG $TST-3 Logoff $1.0

16 TOC-014 $req 14 is about item 14 stuff --- $LOG $TST-3 Logoff $1.0

18 TOC-016 $req 16 is about item 16 stuff --- $STP $TST-4 Self Test $1.0

Sub Total 5

Current Negative PathsThese requirements are the result of negative path in the code. They can only be surfaced with inserted errors via the emulator. nnoonnee

Sub Total 0

Future These are future requirements and should not be part of the test results. \$1\.5

21 TOC-019 $req 19 is about item 19 stuff --- $STP $TST-4 Self Test $1.5 FUTURE

22 TOC-020 $req 20 is about item 20 stuff --- $STP $TST-4 Self Test $1.5 FUTURE

Sub Total 2

Special De-Instrumented PathsThese requirements are in a real time path and have not been instrumented. TOC-017

19 TOC-017 $req 17 is about item 17 stuff --- $STP $TST-4 Self Test $1.0 DE-INST PATH

Sub Total 1

User Defined These categories are defined by the user.

User Defined - Analysis Verify by analysis. TOC-xxxx

Sub Total 0

User Defined - Missing Log Events The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events. Bx_xx|TOC-009 |TOC-018 |done

11 TOC-009 $req 9 is about item 9 stuff --- $LOG $TST-2 Logon $1.0 MISSING LOG EVENTS

20 TOC-018 $req 18 is about item 18 stuff --- $STP $TST-4 Self Test $1.0 MISSING LOG EVENTS

Sub Total 2

User Defined - Inspections and Analysis These reqs will be verified using manual inspection and analysis techniques TOC-011 |TOC-012

13 TOC-011 $req 11 is about item 11 stuff --- $LOG $TST-3 Logoff $1.0 INSPECTIONS AND ANALYSIS

14 TOC-012 $req 12 is about item 12 stuff --- $LOG $TST-3 Logoff $1.0 INSPECTIONS AND ANALYSIS

Sub Total 2

Total 12

Not in Software Baseline 6

1 UNCLASSIFIED

2 TOC-000 $Power Up boiler plate text $ --- $ $

7 TOC-005 $Log on boiler plate text $ --- $ $

12 TOC-010 $Log off boiler plate text $ --- $ $

17 TOC-015 $Self test boiler plate text $ --- $ $

23 UNCLASSIFIED

12/14/2003 08:45:57 start
12/14/2003 08:45:57 end
0 secs

done

filename.txt	Is the test file name, the default is testlog.txt. The report name is derived from this file name. The default report name is testlog.html. When entering a file name the .txt extension must be used (e.g. AAAa.txt, AAAb.txt).
delayinsecs	The script executes 100 times before ending, this is the delay between executions, the default is 20 seconds.
TocCor	If set to [y] correlates the log events to requirements.
TocCov	If set to [y] shows which requirements were addressed and which requirements were not addressed by the test results.
LabData	If set to [y] shows the raw test data.
LeFind	If set to [y] shows the log events that were found. The default is [y].
LeNoFind	If set to [y] shows the log events NOT found in these test results.
Results	If set to [y] shows the test results correlated to the original log events. The deafult is [y].
CPC	If set to [all], [red], or [a cpc] shows req's satisfied and NOT satisfied by these test results.
Filter	If set to [y], filters items containing certain words from the reports. The deafult is [y].

Current	Positive paths that were not logged during testing and are not accounted for in the following categories.
TRACE Paths	These are not req's verification log events. They are trace paths to facilitate understanding the operation of the software. Missing them is not important.
Future
Negative
Special De-Instrumented
User Defined
All User Defined
Analysis	Verify by analysis.
Missing Log Events	The instrumentation analysis report, s_datareduction.html, indicates that these reqs never made it into the SW as log events.
Inspections and Analysis	These reqs will be verified using manual inspection and analysis techniques