Lab Reduction for z-3-03-logonoff
v102103 z-3-03-logonoff.txt s_z-3-03-logonoff.html ../logev

Lab Reduction is a companion web based tool to Data Reduction except that it operates on real test data rather than simulated test data. The data reduction is performed in real time as the test executes. Anyone on the network can view the test results in real time during the testing.

Running Real Time Test Results Script

This tool is started as a DOS application in a directory which contains the test data. The syntax is:

perl labreduction.pl [filename.txt] [delayinsecs] [TocCor] [TocCov] [LabData] [LeFind] [LeNoFind] [Results] [CPC]

where all [fields] are optional and [blank] or [-] is default or are set as follows:
filename.txtIs the test file name, the default is testlog.txt. The report name is derived from this file name. The default report name is testlog.html. When entering a file name the .txt extension must be used (e.g. AAAa.txt, AAAb.txt).
delayinsecsThe script executes 100 times before ending, this is the delay between executions, the default is 20 seconds.
TocCorIf set to [y] correlates the log events to requirements.
TocCovIf set to [y] shows which requirements were addressed and which requirements were not addressed by the test results.
LabDataIf set to [y] shows the raw test data.
LeFindIf set to [y] shows the log events that were found. The default is [y].
LeNoFindIf set to [y] shows the log events NOT found in these test results.
ResultsIf set to [y] shows the test results correlated to the original log events. The deafult is [y].
CPCIf set to [all], [red], or [a cpc] shows req's satisfied and NOT satisfied by these test results.
FilterIf set to [y], filters items containing certain words from the reports. The deafult is [y].

The file containing the test results is automatically updated by the BOOT application for some test cases. In all other BOOT and RED test cases the file is manually updated by the testers by either copying log events from the emulator window or entering log events based on emulator profile points. Using appropriate file naming conventions, parallel testing can be performed.

This application requires the inclusion of a special file (translate.dat) that is created by the logevent application and stored with the instrumentation analysis results. This file translates the log event codes to the original english equivalents. It also requires a version of the TOC (s_toc.dat) to correlate the TOC requirements with the lab test results.

Analyzing The Test Results

The test results analysis begins during informal testing by examining the real time test results.

  1. Are the start and stop log events as expected
  2. Are any log events grossly out of place
  3. Do the log events and sequence look resonable
  4. Do the number of log events match previous test runs
  5. Do the source files and function names in the log events look resonable
  6. Does the layout of the log event sequence appear to be similar to previous test runs

If the results appear to be reasonable, copy and paste them into the test procedures. They will form the factory template that will be used during formal testing.

As part of an offline analysis, use the IAT reports and software source to locate the test result log events. Examine the software and look for potential missing log events or wrong long events. This is a final examination to confirm that the test results are as expected.

For dry run, create a single complete test results report following the sequence that is expected during formal test. Use these dry run test results to support the formal test.

At the conclusion of all the DRY RUN tests enable the log events found, log events not found, requirements correlation, and requirements coverage reports. Look for log events and requirements that were not captured during the all the test activity. If needed, create new tests to capture the missing requirements.

Analysis Reports Script Option Comment
Test CommentsComments Summary of the comments entered by testers into the test file.
PUIs VerifiedComments The list of PUIs verified in this test sequence. Use it in the filter service to compared different test sequences.
Req CoverageComments This is a list of PUIs and the number of times the PUI was encountered in a log event with this test sequence. Use this to gain confidence in the test coverage. For example, a PUI that has been hit 50 times was probably the result of slightly different code paths.
Test ResultsResults This is the test results in log event format. This analysis connects the raw numerical log events captured during lab testing with the original source code comment statements, source file, and C-function.
Missing Req'sResults This is a list of req's that should have been in this test but are missing. It is based on the IAT filter service where one set of reqs's are subtracted from another set of req's.

NOTE: These reports are option sensitive. If an option was not selected, the report link is not present.

Other Analysis Reports . TOC req's . simulated logevents . translate file . Error_Codes.txt

Settings

Paths
Main Path
Simulation Directory
Analysis Directory
Analysis Report Name
Test File Name

Settings case sensitive
Script Rerun Delay (sec)
CPC, Release, or Test Case Considered
Trace Events

Requirements Accounting - Fixed case sensitive
Future Events
De-Instrumented
Negative Paths

Requirements Accounting - User Defined case sensitive Categories
Name             Description                                         PUI's








Display Filters - flaten display case insensitive
Raw LCD Log Events
Raw PAD Log Events
Raw PIN Log Events
Raw LOC Log Events
Raw Any Log Events
Text Results

Ignore Data - for very fast bypassing of noise case insensitive
Bypass Noise Raw Events
Bypass Noise Text Results

Keywords copy paste REQs from word table column get converted to OR format - case insensitive
Red
Blue
Navy
Purple
Green
Maroon
case sensitive
Orange

Expected Req's From Test Procedure - copy & paste from doc case insensitive
Orange LE SV

LabData Test Comments Test Results LE Found LE NOT Found Show Only Keyword Events
TOC Correlation TOC Coverage Not in Software Baseline Missed Req's Func Call Seq Show Only Expected Req's
Filter Classified Server Config

11/23/2003 19:19:20

Data Reduction Reports for z-3-03-logonoff

Test Comments Summary Comments entered by testers into test data file

Log Events Data

3-3 Log On Off B1.0

Tester: ws

1. Log On
. . . HMI-30300-5
LOG-30300-800
. . . HMI-30300-3
1. Done

2. Log Off
. . . HMI-30300-5
LOG-30300-50
. . . HMI-30300-3
2. Done

      

Test Results Lab Data Connected To File Name, C-function, and Comment Statements 72 z-3-03-logonoff.txt 11/23/2003 19:19:20
3-3 Log On Off B1.0

Tester: ws

1. Log On
HMI-30300-5
LOG-30300-800
1 6 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
4 9 1515 I log-01.c log.1.1 LE SV TOC-2001 this is a log-01.1.6 req to TRANSFORM Expected LE
41 46 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
45 50 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
64 69 1523 I log-01.c log.1.2 LE SV TOC-2009 this is a log-01.2.16 req to TRANSFORM Expected LE
69 74 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
75 80 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
76 81 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
86 91 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
103 108 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
126 131 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
142 147 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
150 155 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
155 160 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
159 164 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
167 172 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
173 178 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
183 188 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
201 206 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
213 218 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
217 222 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
225 230 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
229 234 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
236 241 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
249 254 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
268 273 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
276 281 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
288 293 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
292 297 1519 I log-01.c log.1.1 LE SV TOC-2005 this is a log-01.1.47 req to INHIBIT Expected LE
297 302 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
328 333 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
355 360 1519 I log-01.c log.1.1 LE SV TOC-2005 this is a log-01.1.47 req to INHIBIT Expected LE
376 381 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
386 391 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
404 409 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
453 458 1523 I log-01.c log.1.2 LE SV TOC-2009 this is a log-01.2.16 req to TRANSFORM Expected LE
463 468 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
481 486 1521 I log-01.c log.1.2 LE SV TOC-2007 this is a log-01.2.9 req to TRANSFORM Expected LE
494 499 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
518 523 1515 I log-01.c log.1.1 LE SV TOC-2001 this is a log-01.1.6 req to TRANSFORM Expected LE
541 546 1519 I log-01.c log.1.1 LE SV TOC-2005 this is a log-01.1.47 req to INHIBIT Expected LE
604 609 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
607 612 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
620 625 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
638 643 1519 I log-01.c log.1.1 LE SV TOC-2005 this is a log-01.1.47 req to INHIBIT Expected LE
647 652 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
685 690 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
711 716 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
737 742 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
740 745 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
742 747 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
771 776 1516 I log-01.c log.1.1 LE SV TOC-2002 this is a log-01.1.12 req to set RTC Expected LE
800 805 1522 I log-01.c log.1.2 LE SV TOC-2008 this is a log-01.2.10 req to RECOVER Expected LE
HMI-30300-3
1. Done

2. Log Off
HMI-30300-5
LOG-30300-50
1 814 1518 I log-01.c log.1.1 LE SV TOC-2004 this is a log-01.1.38 req to COMPARE Expected LE
4 817 1515 I log-01.c log.1.1 LE SV TOC-2001 this is a log-01.1.6 req to TRANSFORM Expected LE
41 854 1520 I log-01.c log.1.2 LE SV TOC-2006 this is a log-01.2.2 req to HALT Expected LE
45 858 1517 I log-01.c log.1.1 LE SV TOC-2003 this is a log-01.1.20 req to CONVERT Expected LE
HMI-30300-3
2. Done

99XX series reserved for IAT self tests

PUIs Verified - All in this thread use this in the filter service to determine test run differences

s_z-3-03-logonoff.html |TOC-1801 |TOC-1803 |TOC-1811 |TOC-1837 |TOC-1906 |TOC-2001 |TOC-2002 |TOC-2003 |TOC-2004 |TOC-2005 |TOC-2006 |TOC-2007 |TOC-2008 |TOC-2009 |TOC-2010 |TOC-2011 |TOC-2012 |TOC-2013 |TOC-2014 |TOC-2015 |TOC-2016 |TOC-2017 |TOC-2018 |TOC-2019 |TOC-2020 |TOC-2021 |TOC-2022 |TOC-2023 |TOC-2024 |TOC-2025 |TOC-2026 |TOC-2027 |TOC-2028 |TOC-2029 |TOC-2030 |TOC-2031 |TOC-2032 |TOC-2033 |TOC-2034 |TOC-2035 |TOC-2036 |TOC-2037 |TOC-2038 |TOC-2039 |TOC-2040 |TOC-2041 |TOC-2042 |TOC-2043 |TOC-2044 |TOC-2045 |TOC-2046 |TOC-2047 |TOC-2048 |TOC-2049 |TOC-2051 |TOC-2052 |TOC-2053 |TOC-2054 |TOC-2055 |TOC-2056 |TOC-2057 |TOC-2058 |TOC-2059 |TOC-2060 |TOC-2061 |TOC-2062 |TOC-2063 |TOC-2064 |TOC-2065 |TOC-2066 |TOC-2067 |TOC-2068 |TOC-2069 |TOC-2070 |TOC-2071 |TOC-2072 |TOC-2073 |TOC-2074 |TOC-2075 |TOC-2076 |TOC-2077 |TOC-2078 |TOC-2079 |TOC-2080 |TOC-2081 |TOC-2082 |TOC-2083 |TOC-2084 |TOC-2085 |TOC-2086 |TOC-2087 |TOC-2088 |TOC-2089 |TOC-2090 |TOC-2091 |TOC-2092 |TOC-2093 |TOC-2094 |TOC-2095 |TOC-2096 |TOC-2097 |TOC-2098 |TOC-2099 |TOC-2100 |TOC-2101 |TOC-2102 |TOC-2103 |TOC-2104 |TOC-2105 |TOC-2106 |TOC-2107 |TOC-2108 |TOC-2109 |TOC-2110 |TOC-2111 |TOC-2112 |TOC-2113 |TOC-2114 |TOC-2115 |TOC-2116 |TOC-2117 |TOC-2118 |TOC-2119 |TOC-2120 |TOC-2121 |TOC-2122 |TOC-2123 |TOC-2124 |TOC-2125 |TOC-2126 |TOC-2127 |TOC-2128 |TOC-2129 |TOC-2130 |TOC-2131 |TOC-2132 |TOC-2133 |TOC-2134 |TOC-2135 |TOC-2136 |TOC-2137 |TOC-2138 |TOC-2139 |TOC-2140 |TOC-2141 |TOC-2142 |TOC-2143 |TOC-2144 |TOC-2145 |TOC-2146 |TOC-2147 |TOC-2148 |TOC-2149 |TOC-2150 |done

Total Verified Req's 154

Req Coverage - All in this thread

18 TOC-2051 13 TOC-2072 11 TOC-2139 11 TOC-2014 11 TOC-2087 11 TOC-2147 11 TOC-2004 10 TOC-2025 10 TOC-2140
10 TOC-2043 10 TOC-2134 9 TOC-2008 9 TOC-2149 9 TOC-2055 9 TOC-2036 9 TOC-2023 9 TOC-2113 9 TOC-2044
8 TOC-2074 8 TOC-2013 8 TOC-2029 8 TOC-2028 8 TOC-2122 8 TOC-2138 8 TOC-2038 8 TOC-2105 8 TOC-2096
8 TOC-2103 8 TOC-2136 8 TOC-2002 8 TOC-2062 8 TOC-2106 8 TOC-2119 7 TOC-2006 7 TOC-2081 7 TOC-2131
7 TOC-2089 7 TOC-2010 7 TOC-2109 7 TOC-2018 7 TOC-2092 7 TOC-2095 7 TOC-2046 7 TOC-2100 7 TOC-2041
7 TOC-2102 7 TOC-2032 7 TOC-2118 7 TOC-2133 7 TOC-2135 7 TOC-2070 7 TOC-2145 7 TOC-2076 7 TOC-2079
7 TOC-2003 6 TOC-2123 6 TOC-2114 6 TOC-2068 6 TOC-2061 6 TOC-2037 6 TOC-2071 6 TOC-2075 6 TOC-2128
6 TOC-2077 6 TOC-2121 6 TOC-2084 6 TOC-2047 6 TOC-2007 5 TOC-2065 5 TOC-2093 5 TOC-2066 5 TOC-2067
5 TOC-2069 5 TOC-2142 5 TOC-2030 5 TOC-2115 5 TOC-2019 5 TOC-2033 5 TOC-2073 5 TOC-2016 5 TOC-2012
5 TOC-2132 5 TOC-2026 5 TOC-2031 5 TOC-2111 5 TOC-2120 5 TOC-2130 5 TOC-2110 5 TOC-2098 5 TOC-2064
5 TOC-2137 5 TOC-2083 4 TOC-2020 4 TOC-2021 4 TOC-2022 4 TOC-2024 4 TOC-2027 4 TOC-1803 4 TOC-2101
4 TOC-2107 4 TOC-2108 4 TOC-2039 4 TOC-2141 4 TOC-1811 4 TOC-2112 4 TOC-2143 4 TOC-2144 4 TOC-2042
4 TOC-2117 4 TOC-2048 4 TOC-2125 4 TOC-2146 4 TOC-2126 4 TOC-2056 4 TOC-2129 4 TOC-2057 4 TOC-1837
4 TOC-2080 4 TOC-2078 4 TOC-2011 4 TOC-2060 4 TOC-2017 4 TOC-2090 4 TOC-2005 4 TOC-2094 4 TOC-2097
4 TOC-2099 4 TOC-2150 3 TOC-2116 3 TOC-2063 3 TOC-2040 3 TOC-2001 3 TOC-2124 3 TOC-2015 3 TOC-2052
3 TOC-2035 3 TOC-2045 3 TOC-2058 3 TOC-2059 3 TOC-2034 2 TOC-2127 2 TOC-2088 2 TOC-2082 2 TOC-2104
2 TOC-2049 2 TOC-1801 2 TOC-2148 2 TOC-1906 2 TOC-2054 2 TOC-2009 2 TOC-2085 1 TOC-2053 1 TOC-2086
1 TOC-2091

Total Verified Req's 154

Req's Missed In This Test Run This report with the default settings shows the req's that were MISSED in this test. It is based on the filter tool where an initial set of items are found and a second set is subtracted from the intial set. In this case the initial set are the expected req's as defined in the test procedure, and the second set are the req's found in the actual test data. Req's are Marked if they are not in the software. Req's are also Marked if they are turned off in the debugger.

Don't Show Located Items Show Removed Items Flip Locate & Remove

Locate: TOC-200[123456789]

Remove: s_z-3-03-logonoff.html |TOC-1801 |TOC-1803 |TOC-1811 |TOC-1837 |TOC-1906 |TOC-2001 |TOC-2002 |TOC-2003 |TOC-2004 |TOC-2005 |TOC-2006 |TOC-2007 |TOC-2008 |TOC-2009 |TOC-2010 |TOC-2011 |TOC-2012 |TOC-2013 |TOC-2014 |TOC-2015 |TOC-2016 |TOC-2017 |TOC-2018 |TOC-2019 |TOC-2020 |TOC-2021 |TOC-2022 |TOC-2023 |TOC-2024 |TOC-2025 |TOC-2026 |TOC-2027 |TOC-2028 |TOC-2029 |TOC-2030 |TOC-2031 |TOC-2032 |TOC-2033 |TOC-2034 |TOC-2035 |TOC-2036 |TOC-2037 |TOC-2038 |TOC-2039 |TOC-2040 |TOC-2041 |TOC-2042 |TOC-2043 |TOC-2044 |TOC-2045 |TOC-2046 |TOC-2047 |TOC-2048 |TOC-2049 |TOC-2051 |TOC-2052 |TOC-2053 |TOC-2054 |TOC-2055 |TOC-2056 |TOC-2057 |TOC-2058 |TOC-2059 |TOC-2060 |TOC-2061 |TOC-2062 |TOC-2063 |TOC-2064 |TOC-2065 |TOC-2066 |TOC-2067 |TOC-2068 |TOC-2069 |TOC-2070 |TOC-2071 |TOC-2072 |TOC-2073 |TOC-2074 |TOC-2075 |TOC-2076 |TOC-2077 |TOC-2078 |TOC-2079 |TOC-2080 |TOC-2081 |TOC-2082 |TOC-2083 |TOC-2084 |TOC-2085 |TOC-2086 |TOC-2087 |TOC-2088 |TOC-2089 |TOC-2090 |TOC-2091 |TOC-2092 |TOC-2093 |TOC-2094 |TOC-2095 |TOC-2096 |TOC-2097 |TOC-2098 |TOC-2099 |TOC-2100 |TOC-2101 |TOC-2102 |TOC-2103 |TOC-2104 |TOC-2105 |TOC-2106 |TOC-2107 |TOC-2108 |TOC-2109 |TOC-2110 |TOC-2111 |TOC-2112 |TOC-2113 |TOC-2114 |TOC-2115 |TOC-2116 |TOC-2117 |TOC-2118 |TOC-2119 |TOC-2120 |TOC-2121 |TOC-2122 |TOC-2123 |TOC-2124 |TOC-2125 |TOC-2126 |TOC-2127 |TOC-2128 |TOC-2129 |TOC-2130 |TOC-2131 |TOC-2132 |TOC-2133 |TOC-2134 |TOC-2135 |TOC-2136 |TOC-2137 |TOC-2138 |TOC-2139 |TOC-2140 |TOC-2141 |TOC-2142 |TOC-2143 |TOC-2144 |TOC-2145 |TOC-2146 |TOC-2147 |TOC-2148 |TOC-2149 |TOC-2150 |done

Marked: Bx_xxTOC-1001|done

Check if it was Manually verified

Reqs Verification Summary . . . Missed 0 . . . Marked 0 . . . DeInst 0 . . . Manual 0 . . . Found 9

11/23/2003 19:19:20 start
11/23/2003 19:19:26 end
6 secs
done