Lab Reduction for z-3-01-poweronoff
v102103 z-3-01-poweronoff.txt s_z-3-01-poweronoff.html ../logev

Lab Reduction is a companion web based tool to Data Reduction except that it operates on real test data rather than simulated test data. The data reduction is performed in real time as the test executes. Anyone on the network can view the test results in real time during the testing.

Running Real Time Test Results Script

This tool is started as a DOS application in a directory which contains the test data. The syntax is:

perl labreduction.pl [filename.txt] [delayinsecs] [TocCor] [TocCov] [LabData] [LeFind] [LeNoFind] [Results] [CPC]

where all [fields] are optional and [blank] or [-] is default or are set as follows:
filename.txtIs the test file name, the default is testlog.txt. The report name is derived from this file name. The default report name is testlog.html. When entering a file name the .txt extension must be used (e.g. AAAa.txt, AAAb.txt).
delayinsecsThe script executes 100 times before ending, this is the delay between executions, the default is 20 seconds.
TocCorIf set to [y] correlates the log events to requirements.
TocCovIf set to [y] shows which requirements were addressed and which requirements were not addressed by the test results.
LabDataIf set to [y] shows the raw test data.
LeFindIf set to [y] shows the log events that were found. The default is [y].
LeNoFindIf set to [y] shows the log events NOT found in these test results.
ResultsIf set to [y] shows the test results correlated to the original log events. The deafult is [y].
CPCIf set to [all], [red], or [a cpc] shows req's satisfied and NOT satisfied by these test results.
FilterIf set to [y], filters items containing certain words from the reports. The deafult is [y].

The file containing the test results is automatically updated by the BOOT application for some test cases. In all other BOOT and RED test cases the file is manually updated by the testers by either copying log events from the emulator window or entering log events based on emulator profile points. Using appropriate file naming conventions, parallel testing can be performed.

This application requires the inclusion of a special file (translate.dat) that is created by the logevent application and stored with the instrumentation analysis results. This file translates the log event codes to the original english equivalents. It also requires a version of the TOC (s_toc.dat) to correlate the TOC requirements with the lab test results.

Analyzing The Test Results

The test results analysis begins during informal testing by examining the real time test results.

  1. Are the start and stop log events as expected
  2. Are any log events grossly out of place
  3. Do the log events and sequence look resonable
  4. Do the number of log events match previous test runs
  5. Do the source files and function names in the log events look resonable
  6. Does the layout of the log event sequence appear to be similar to previous test runs

If the results appear to be reasonable, copy and paste them into the test procedures. They will form the factory template that will be used during formal testing.

As part of an offline analysis, use the IAT reports and software source to locate the test result log events. Examine the software and look for potential missing log events or wrong long events. This is a final examination to confirm that the test results are as expected.

For dry run, create a single complete test results report following the sequence that is expected during formal test. Use these dry run test results to support the formal test.

At the conclusion of all the DRY RUN tests enable the log events found, log events not found, requirements correlation, and requirements coverage reports. Look for log events and requirements that were not captured during the all the test activity. If needed, create new tests to capture the missing requirements.

Analysis Reports Script Option Comment
Test CommentsComments Summary of the comments entered by testers into the test file.
PUIs VerifiedComments The list of PUIs verified in this test sequence. Use it in the filter service to compared different test sequences.
Req CoverageComments This is a list of PUIs and the number of times the PUI was encountered in a log event with this test sequence. Use this to gain confidence in the test coverage. For example, a PUI that has been hit 50 times was probably the result of slightly different code paths.
Test ResultsResults This is the test results in log event format. This analysis connects the raw numerical log events captured during lab testing with the original source code comment statements, source file, and C-function.
Missing Req'sResults This is a list of req's that should have been in this test but are missing. It is based on the IAT filter service where one set of reqs's are subtracted from another set of req's.

NOTE: These reports are option sensitive. If an option was not selected, the report link is not present.

Other Analysis Reports . TOC req's . simulated logevents . translate file . Error_Codes.txt


Settings

Paths
Main Path
Simulation Directory
Analysis Directory
Analysis Report Name
Test File Name

Settings case sensitive
Script Rerun Delay (sec)
CPC, Release, or Test Case Considered
Trace Events

Requirements Accounting - Fixed case sensitive
Future Events
De-Instrumented
Negative Paths

Requirements Accounting - User Defined case sensitive Categories
Name             Description                                         PUI's








Display Filters - flaten display case insensitive
Raw LCD Log Events
Raw PAD Log Events
Raw PIN Log Events
Raw LOC Log Events
Raw Any Log Events
Text Results

Ignore Data - for very fast bypassing of noise case insensitive
Bypass Noise Raw Events
Bypass Noise Text Results

Keywords copy paste REQs from word table column get converted to OR format - case insensitive
Red
Blue
Navy
Purple
Green
Maroon
case sensitive
Orange

Expected Req's From Test Procedure - copy & paste from doc case insensitive
Orange LE SV

LabData Test Comments Test Results LE Found LE NOT Found Show Only Keyword Events
TOC Correlation TOC Coverage Not in Software Baseline Missed Req's Func Call Seq Show Only Expected Req's
Filter Classified Server Config

11/22/2003 23:17:18


Data Reduction Reports for z-3-01-poweronoff


Test Comments Summary Comments entered by testers into test data file

Log Events Data

3-1 Power On Off zz-iatdemo

Tester: ws

1. Power On
PWR-30100-800
. . . HMI-30100-15
1. Done

2. Power Off
PWR-30100-150
2. Done

     


Test Results Lab Data Connected To File Name, C-function, and Comment Statements 33 z-3-01-poweronoff.txt 11/22/2003 23:17:18
3-1 Power On Off zz-iatdemo

Tester: ws

1. Power On
PWR-30100-800
25 25 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
63 63 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
64 64 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
124 124 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
127 127 1830 I pwr-05.c pwr.5.5 LE SV TOC-2504 this is a pwr-05.5.3 req to check PSOS Expected LE
151 151 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
280 280 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
300 300 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
325 325 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
329 329 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
353 353 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
394 394 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
536 536 1830 I pwr-05.c pwr.5.5 LE SV TOC-2504 this is a pwr-05.5.3 req to check PSOS Expected LE
632 632 1830 I pwr-05.c pwr.5.5 LE SV TOC-2504 this is a pwr-05.5.3 req to check PSOS Expected LE
694 694 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
730 730 1830 I pwr-05.c pwr.5.5 LE SV TOC-2504 this is a pwr-05.5.3 req to check PSOS Expected LE
HMI-30100-15
1. Done

2. Power Off
PWR-30100-150
25 840 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
63 878 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
64 879 1831 I pwr-05.c pwr.5.5 LE SV TOC-2505 this is a pwr-05.5.27 req to HALT Expected LE
124 939 1832 I pwr-05.c pwr.5.5 LE SV TOC-2506 this is a pwr-05.5.46 req to check unix Expected LE
127 942 1830 I pwr-05.c pwr.5.5 LE SV TOC-2504 this is a pwr-05.5.3 req to check PSOS Expected LE
2. Done

99XX series reserved for IAT self tests


PUIs Verified - All in this thread use this in the filter service to determine test run differences

s_z-3-01-poweronoff.html |TOC-008 |TOC-1475 |TOC-1478 |TOC-1602 |TOC-1613 |TOC-1615 |TOC-1804 |TOC-1806 |TOC-1808 |TOC-1812 |TOC-1817 |TOC-1828 |TOC-1836 |TOC-1840 |TOC-1897 |TOC-2432 |TOC-2433 |TOC-2434 |TOC-2435 |TOC-2436 |TOC-2438 |TOC-2439 |TOC-2440 |TOC-2441 |TOC-2442 |TOC-2444 |TOC-2445 |TOC-2446 |TOC-2447 |TOC-2448 |TOC-2449 |TOC-2450 |TOC-2451 |TOC-2452 |TOC-2453 |TOC-2455 |TOC-2456 |TOC-2457 |TOC-2458 |TOC-2459 |TOC-2460 |TOC-2461 |TOC-2462 |TOC-2463 |TOC-2464 |TOC-2465 |TOC-2466 |TOC-2467 |TOC-2468 |TOC-2469 |TOC-2470 |TOC-2471 |TOC-2472 |TOC-2473 |TOC-2474 |TOC-2475 |TOC-2476 |TOC-2477 |TOC-2479 |TOC-2480 |TOC-2481 |TOC-2483 |TOC-2484 |TOC-2485 |TOC-2486 |TOC-2487 |TOC-2488 |TOC-2489 |TOC-2490 |TOC-2491 |TOC-2492 |TOC-2493 |TOC-2494 |TOC-2495 |TOC-2496 |TOC-2497 |TOC-2498 |TOC-2499 |TOC-2500 |TOC-2501 |TOC-2502 |TOC-2503 |TOC-2504 |TOC-2505 |TOC-2506 |TOC-2507 |TOC-2508 |TOC-2509 |TOC-2510 |TOC-2511 |TOC-2512 |TOC-2513 |TOC-2514 |TOC-2515 |TOC-2516 |TOC-2517 |TOC-2518 |TOC-2519 |TOC-2520 |TOC-2521 |TOC-2522 |TOC-2523 |TOC-2524 |TOC-2525 |TOC-2526 |TOC-2527 |TOC-2528 |TOC-2529 |TOC-2530 |TOC-2531 |TOC-2532 |TOC-2533 |TOC-2534 |TOC-2535 |TOC-2536 |TOC-2537 |TOC-2538 |TOC-2539 |TOC-2540 |TOC-2541 |TOC-2542 |TOC-2544 |TOC-2545 |TOC-2546 |TOC-2547 |TOC-2604 |TOC-2605 |TOC-2606 |TOC-2607 |TOC-2608 |TOC-2609 |TOC-2610 |TOC-2611 |TOC-2612 |TOC-2613 |done

Total Verified Req's 135


Req Coverage - All in this thread

43 TOC-008 16 TOC-2508 16 TOC-2470 14 TOC-2604 14 TOC-2612 14 TOC-2436 14 TOC-2496 13 TOC-2446 13 TOC-2434
13 TOC-2481 13 TOC-2495 12 TOC-2455 12 TOC-2501 12 TOC-2613 11 TOC-2474 11 TOC-2512 11 TOC-2475 11 TOC-2484
10 TOC-2609 10 TOC-2472 10 TOC-2541 10 TOC-2465 10 TOC-2523 10 TOC-2439 10 TOC-2432 10 TOC-2510 10 TOC-2488
10 TOC-2499 10 TOC-2485 10 TOC-2535 10 TOC-2531 10 TOC-2463 10 TOC-2492 9 TOC-2500 9 TOC-2529 9 TOC-2610
9 TOC-2479 9 TOC-2607 9 TOC-2483 9 TOC-2540 9 TOC-2527 9 TOC-2486 9 TOC-2511 9 TOC-2491 8 TOC-2502
8 TOC-2505 8 TOC-2506 8 TOC-2507 8 TOC-2457 8 TOC-2515 8 TOC-2458 8 TOC-2466 8 TOC-2469 8 TOC-2611
8 TOC-2445 8 TOC-2471 8 TOC-2545 8 TOC-2473 8 TOC-2449 8 TOC-2490 7 TOC-2525 7 TOC-2468 7 TOC-2608
7 TOC-2441 7 TOC-2489 7 TOC-2544 7 TOC-2444 7 TOC-2517 7 TOC-2464 7 TOC-2519 7 TOC-2433 7 TOC-2530
7 TOC-2460 7 TOC-2452 7 TOC-2476 7 TOC-2450 6 TOC-2537 6 TOC-2526 6 TOC-2514 6 TOC-2453 6 TOC-2539
6 TOC-2547 6 TOC-2435 6 TOC-2462 6 TOC-2534 6 TOC-2606 6 TOC-2497 6 TOC-2451 6 TOC-2538 5 TOC-2498
5 TOC-2494 5 TOC-2442 5 TOC-2477 5 TOC-2518 5 TOC-2542 5 TOC-2448 5 TOC-2504 5 TOC-2528 5 TOC-2456
5 TOC-2467 5 TOC-2459 4 TOC-2440 4 TOC-2524 4 TOC-2493 4 TOC-2447 4 TOC-2532 4 TOC-2521 4 TOC-2522
4 TOC-2487 4 TOC-2480 3 TOC-2513 3 TOC-2546 3 TOC-2520 3 TOC-2605 3 TOC-2509 2 TOC-1808 2 TOC-2516
2 TOC-2533 2 TOC-2503 1 TOC-2461 1 TOC-1478 1 TOC-1806 1 TOC-1812 1 TOC-1475 1 TOC-1804 1 TOC-1817
1 TOC-1602 1 TOC-1828 1 TOC-1897 1 TOC-1615 1 TOC-1613 1 TOC-2536 1 TOC-1840 1 TOC-2438 1 TOC-1836

Total Verified Req's 135


Req's Missed In This Test Run This report with the default settings shows the req's that were MISSED in this test. It is based on the filter tool where an initial set of items are found and a second set is subtracted from the intial set. In this case the initial set are the expected req's as defined in the test procedure, and the second set are the req's found in the actual test data. Req's are Marked if they are not in the software. Req's are also Marked if they are turned off in the debugger.

Don't Show Located Items Show Removed Items Flip Locate & Remove

Locate: TOC-2504|TOC-2505|TOC-2506|TOC-1870|TOC-1871

Remove: s_z-3-01-poweronoff.html |TOC-008 |TOC-1475 |TOC-1478 |TOC-1602 |TOC-1613 |TOC-1615 |TOC-1804 |TOC-1806 |TOC-1808 |TOC-1812 |TOC-1817 |TOC-1828 |TOC-1836 |TOC-1840 |TOC-1897 |TOC-2432 |TOC-2433 |TOC-2434 |TOC-2435 |TOC-2436 |TOC-2438 |TOC-2439 |TOC-2440 |TOC-2441 |TOC-2442 |TOC-2444 |TOC-2445 |TOC-2446 |TOC-2447 |TOC-2448 |TOC-2449 |TOC-2450 |TOC-2451 |TOC-2452 |TOC-2453 |TOC-2455 |TOC-2456 |TOC-2457 |TOC-2458 |TOC-2459 |TOC-2460 |TOC-2461 |TOC-2462 |TOC-2463 |TOC-2464 |TOC-2465 |TOC-2466 |TOC-2467 |TOC-2468 |TOC-2469 |TOC-2470 |TOC-2471 |TOC-2472 |TOC-2473 |TOC-2474 |TOC-2475 |TOC-2476 |TOC-2477 |TOC-2479 |TOC-2480 |TOC-2481 |TOC-2483 |TOC-2484 |TOC-2485 |TOC-2486 |TOC-2487 |TOC-2488 |TOC-2489 |TOC-2490 |TOC-2491 |TOC-2492 |TOC-2493 |TOC-2494 |TOC-2495 |TOC-2496 |TOC-2497 |TOC-2498 |TOC-2499 |TOC-2500 |TOC-2501 |TOC-2502 |TOC-2503 |TOC-2504 |TOC-2505 |TOC-2506 |TOC-2507 |TOC-2508 |TOC-2509 |TOC-2510 |TOC-2511 |TOC-2512 |TOC-2513 |TOC-2514 |TOC-2515 |TOC-2516 |TOC-2517 |TOC-2518 |TOC-2519 |TOC-2520 |TOC-2521 |TOC-2522 |TOC-2523 |TOC-2524 |TOC-2525 |TOC-2526 |TOC-2527 |TOC-2528 |TOC-2529 |TOC-2530 |TOC-2531 |TOC-2532 |TOC-2533 |TOC-2534 |TOC-2535 |TOC-2536 |TOC-2537 |TOC-2538 |TOC-2539 |TOC-2540 |TOC-2541 |TOC-2542 |TOC-2544 |TOC-2545 |TOC-2546 |TOC-2547 |TOC-2604 |TOC-2605 |TOC-2606 |TOC-2607 |TOC-2608 |TOC-2609 |TOC-2610 |TOC-2611 |TOC-2612 |TOC-2613 |done

Marked: Bx_xxTOC-1001|done

Check if it was Manually verified
TOC-1870 This req is about hmi-04.5.32 the system shall set RTC HMI 1.0

TOC-1871 This req is about hmi-05.4.3 the system shall record HMI 1.0

Reqs Verification Summary . . . Missed 1 . . . Marked 0 . . . DeInst 0 . . . Manual 1 . . . Found 3


11/22/2003 23:17:18 start
11/22/2003 23:17:27 end
9 secs
done