The IAT was created on the Project-O program. It is a tool platform that comes with an engineering process. The process is documented and available for consumption. It includes when and how IAT is used during various phases of the program. Project-N is the first program to attempt to adopt the IAT. IProject-N is choosing to modify the process established on Project-O. The following tables summarize the changes being made to the IAT and its Project-O processes. The Project-O IAT engineering process is summarized at Process and Traceability and can be found by accessing the project documentation such as plans and related memos. Tailoring FAQ and Metrics FAQ

IAT process and metrics across multiple projects is summarized in the following tables. Project metrics . IAT Application . Automated Code Analysis Detectors . Reports . IAT Web Enabled Services . Possible Future Enhancements

Process and Metrics

Metrics

The IAT produces metrics reports automatically and transparently as its services are provide to support peer reviews, code walk throughs, and instrumentation. The reports include automatic comparisons to a previous baseline and to a selected "gold standard" baseline. This allows an analyst to quickly determine if things are getting better, worse, or staying the same. There is a great deal of data that is collected and categorized and visually organized so that a user can quickly spot potential issues. There is also a big picture summary at the end of the detailed metrics areas which offers an "IAT judgment" on the collection of code. The IAT judgment call is marked as WORSE, BETTER, or SAME.

zz-iatdemo-B1.0
ft-app-B1.0
ft-app-B1.1

Analysis Group Stats This area gathers traditional metrics such as LOC and number of Req's. But it provides different views of those fundamental metrics such as ratio of "comments" to LOC.

Possible Problems This area is a remnant from the stats-problems report, but it provides a description of each metric item in the graphical sections of the report.

Metrics - Percents Within Collections This area provides metrics in terms of percent of the code analyzed. This is a great way to compare peer reviews (small collections of code) against a previous baseline release (the whole thing). It is also a great way to compare different projects which follow similar, but not exact, coding standards. For Example, assume version A has 10 items flagged and is 10,000 LOC while version B has 20 items flagged but is 50,000 LOC. Even though version B has more items flagged one could argue that it is better than Version A since as a percentage, the flagged items are lower.

Metrics - Counts Within Collections This area provides metrics in terms of absolute counts. It is used to compare version releases on the same project and to get an assessment of the exact number of problems.

In all cases the bar charts represent a deviation from the norm. The norm is the green area. In general, for the detailed problems report area anything to the left on the bar chart is better and anything to the right is worse. A pegged needle is red. The bar charts at the top of the report are not in terms of good or bad, just different. The up down triangles are another indication of how the software is progressing. Green tends to indicate progress and red tends to indicate entropy, hence the names of the reports, entropy.html :)

The following is a brief description of the Analysis Group Stats metric items.

1. Total Source Lines This is a count of all the lines.
2. Total Non Blank Lines This is a count of all the comments, headers, and code.
3. Total Logical This is a subset of the total LOC.
4. Total SemiColons  This is a subset of the total LOC.
5. Total LOC This is the sum of the total logical and semi colon count.
6. All Files Count of all the files analyzed.
7. Files - C Count of all the C files analyzed.
8. Files - H Count of all the H files analyzed.
9. Files - ASM Count of all the ASM files analyzed.
10. Functions - C Count of all the functions located by the IAT. The IAT uses the function headers to locate functions.
11. Log Events in Code Count of the log events.
12. TOC Reqs in Code Count of all the TOC and PS req's located in the files analyzed.
13. TOC Reqs in SRDB Count of the req's in the SRDB file fed to the IAT. Rules are used to separate a requirement from a description.
14. Debug Events Count of the debug log events.
15. Code Req per SRDB Req The number of req's located in the code divided by the number of req's located in the SRDB.
16. LOC per File Average Lines of code per file.
17. LOC per Function Average Lines of code per function.
18. Functions per File - C Average number of functions per file.
19. Non Blank Lines Per LOC If headers are considered comments, this is a comment to LOC ratio.
20. LOC Per Logical Indicates LOC for each decision.
21. C Per H Ratio of C to H files. It should be greater than 1.
22. LOC per Req Ratio of the lines of code to req's found in the code.
23. Functions per Req Ratio of functions  to req's found in the code.
24. Logevents per Req Ratio of log events to req's found in the code.

The following is a brief description of the Analysis Summary metric items. These items are reported in terms of flagged LOC, files, and/or functions.

1. Fatal Errors These are artifacts of instrumentation that will lead to compile errors most of the time, such as printf's, breaking if-else sequences, splitting declaration areas, and instrumenting previously instrumented code.
2. C Header The .c .h and assembly headers are separately checked for the appropriate fields. Missing fields, wrong sequence of the fields, or missing headers are detected. Extra fields are ignored. Valid format:
3. H Header The .c .h and assembly headers are separately checked for the appropriate fields. Missing fields, wrong sequence of the fields, or missing headers are detected. Extra fields are ignored. Thread ID is optional and will only appear in the extra fields note if there is another header problem. Valid format:
4. ASM Header The .c .h and assembly headers are separately checked for the appropriate fields. Missing fields, wrong sequence of the fields, or missing headers are detected. Extra fields are ignored. Valid format:
5. Must Fix Classification The header classification field must be marked, no other markings are permitted. The file naming convention is checked against the header classification marking. Keywords that may trigger a higher classification are compared against the file naming convention. Missing headers against classified file names are detected.
6. Possible Classification Issues The header classification field must be marked, no other markings are permitted. The file naming convention is checked against the header classification marking. Keywords that may trigger a higher classification are compared against the file naming convention. Missing headers against classified file names are detected.
7. SV Marking The SV field is checked for YES or NO marking. Blank or other markings are detected and noted in the detailed message. The SV header field must be present for this check to detect an error. If the SV header field is missing, that is noted in the bad header area.
8. CV Marking The CV field is checked for YES or NO marking. Blank or other markings are detected and noted in the detailed message. The CV header field must be present for this check to detect an error. If the CV header field is missing, that is noted in the bad header area.
9. Fixed Keywords Code Although there is a separate keywords report, there are several keywords that must be explained or removed. The bad fixed keywords form this set. Examples include tbd, tbs, tbsl, demo, ?'s etc.
10. Fixed Keywords Prologue Although there is a separate keywords report, there are several keywords that must be explained or removed. The bad fixed keywords form this set. Examples include tbd, tbs, tbsl, demo, ?'s etc.
11. Conditional Compiles ifdefs Conditional compiles can become a problem if they are not properly managed.
12. Missing Curly Braces If and Else statements should enclose their operations with curly braces. Missing curly braces can lead to potential problems as the code changes and if or else blocks are potentially modified to include more than one code statement.
13. Switch Default balance green is good red is bad All switch statements must be paired with a default statement. Missing default statements are detected exclusive of comments.
14. Default gsiSwError balance green is good red is bad All default statements must call gsiSwError. Missing gsiSwError calls and calls that are commented out are detected.
15. Case Break balance All case statements with code should be paired with a break or return statement. Missing break statements are detected exclusive of comments. Without a break the code is less robust and prone to disintegration when modified.
16. Nested Switches Some case statements have a nested switch sequence.
17. Stacked Case Statements Some case statements are stacked.
18. Calling Rules The software is checked to determine if calling rules are violated.
19. File with: No Error Exits Most of the software should include an error exit. This is not a hard and fast rule, but may need to be justified.
20. Files with: 15 or more Functions Good practices limit the number of functions to in any given source file.
21. Files with: 500 or more than LOC Good practices limit the number of LOC to in any given source file.
22. Functions with: 100 more than LOC For proper analysis there must be a valid header termination before the start of a function. Good practices limit the LOC in any given function.
23. C Functions with: less than 5 LOC For proper analysis there must be a valid header termination before the start of a function. Good practices limit the LOC in any given function.
24. Dead Code This is a very simple detector looking for single line commented out code. Code that is block commented out is not detected. Dead code that is the result of logical errors is not detected.
25. Log Event Logevents are subjected to several checks to determine if there may be a potential problem. The checks include redundant text, consecutive events without program code in between events, encoding problems (LE SV not LE, not SV, not TE, etc), potentially placing a logevent at the start of a procedure rather than at the conclusion, potentially placing a logevent within a loop.
26. Line Length Report lines greater than characters where characters are CR/LF. Ignore lines generated by clear case, they exceed .
27. do Loops do loops have long been considered a bad practice since they tend to invite endless loops into into the code.
28. goto Statements goto statements have long been considered a bad practice and result in unstructured code.
29. ?: Operator This operator leads to confusion many times and should be avoided.
30. Storage malloc free Requesting storage using malloc, alloc, calloc should be minimized and include a symetrical free cfree sequence. This can lead to memory leaks if not properly freed. There are also SV related issues.
31. Type ReCasting Recasting variables should not be practiced since it can lead to truncation of data.

The following is a brief description of the Analysis Summary metric items at the end of the metric details. They represent a total of the metrics and include the following items.

1. LOC Total number of LOC flagged from the individual metric item areas.
2. Functions Total number of functions flagged from the individual metric item areas.
3. Files Total number of files flagged from the individual metric item areas.
4. Total Up Flags Total number of up tic's. As the up tic's increase, things get worse.
5. Total Down Flags Total number of down tic's. As the  down tic's decrease, things get better.
6. Total Same Flags Total number of neutral tic's. As the neutral tic's increase or decrease, examine the up and down tic's to get a feel for what happened.
7. Final Tally of Flags This is the IAT judgment call indicating if this release is better or worse than the baseline. This result appears both in the percent and count areas. Remember that the percent area is an attempt to normalize against the size of the analysis package. If both areas agree then IAT is firm on its judgment. If there is a discrepancy between the 2 areas, then using the normalized result in the percent area is the more prudent answer.

PR - Peer Review
FT - Final Test / Release

* item has been incorporated into IAT

Tailoring Frequently Asked Questions

1. Why do I need to add descriptive text to my log events why can't the IAT attach the actual req text based on the PUI number?
2. What does IAT do and does the IAT do anything positive for a program?
3. I don't have time to place the log events into the code and it is a huge burden and overhead to the software staff?
4. I only want a specific set of things from the IAT during code analysis so why are we talking about other things associated with the code?
5. Why do I need PUIs in the DOORS database?
6. Why do I need to run the IAT to support my internal peer reviews?
7. What is CM and why do I need to understand it?
8. Where did the IAT come from?
9. Where is IAT Going?
10. How do I know I should use the IAT on my program?
11. How was the IAT validated and can I trust its outputs?
12. This is really hard and my head is going to explode so why can't software just focus on the functionality of the box?
13. Ok I am sold, but if I do this how does the data come out of the device during test?
14. Wooow, wait a minute, this is not my job and this is going to cost extra money and time, we are not doing this?
15. You know, I did all this and I am getting 140 log events per second coming out of my box?

1. Why do I need to add descriptive text to my log events why can't the IAT attach the actual req text based on the PUI number?

Yes the IAT can apply the original req text in its reports. In fact one of the existing reports does exactly that function. However, experience has shown that req's are satisfied in multiple locations and so a log event should provide some indication of the portion of the req satisfied by the previous code. Further, when someone attempts to look at the listings, the log events provide no information without the descriptive text. Some folks still rely on examining the listing to understand and solve their particular problems (e.g. BAH).

The testers rely heavily on the descriptive text. It is succinct and supports their needs during the test and during initial phases of analysis without being drowned in a sea of words and meaningless statements. In fact the original approach was to provide the req text to the testers when the original data reduction report was created. It was actually easier to implement the PERL for this approach. It was a bad solution. Although the data reduction report still contains that switch option, it is never used.

Finally, some folks have been wondering what it is specifically that the IAT does to help increase the quality of the product. Some folks think that the process of thinking about a req and what it means so that it can be placed properly by the developer provides insights into the design that may have been previously missed. A light bulb goes off in the developers heads as they finally reach a resolution to each of the 1% of the log events that are difficult to place and "paraphrase".

2. What does IAT do and does the IAT do anything positive for a program?

As far as what IAT does, it:

• Loads in the real software source code
• Loads in the real reqs from the SRDB (DOORS report)
• Instruments the source code thus reducing emulator break points
• Looks at the source code and provides reports for peer reviews and code walk throughs
• Correlates raw test data with log event comment statements in the source code
• Correlates raw test data with the SRDB req's
• Provides very specialized and unique searches, compares, and filtering of software, req's, and test results

The Project-O customer has stated that they think IAT was one of the factors in why they received a superior product. Some folks think that the process of thinking about a req and what it means so that it can be placed properly by the developer provides insights into the design that may have been previously missed. This results in a better product as the realization sets in that a log event does not fit someplace.

Other folks think that it is a forcing function. In the process of running IAT certain things that may be ignored in a program till it is too late, surface as part of the process of running IAT.

Finally, some folks think the IAT puts fear into the developers hearts and they are coerced into jumping through the hoops.

3. I don't have time to place the log events into the code and it is a huge burden and overhead to the software staff?

A funny thing happened on the way to this process, a 1980's thing, but not exactly. This thing actually makes sense. Work previously done by a chief programmer type and a test engineer has been pushed to the developers, --- but not really.

In the past the chief programmer would annotate the code in anticipation of a formal code walk through with the government. This would include going back to the source (original developer) to determine or validate where a requirement was satisfied. This function is now performed once and it is done by the most knowledgeable member of the program, the original developer.

Also, as part of the old test process, emulator break points would need to be identified to support SV testing. In the hunt for the elusive breakpoint, which would work given all the peculiar characteristics of an emulator, the original software developer again would be called in to identify many if not all of the points and emulator settings. Introducing log events reduces emulator breakpoints in the final test phase freeing the software developers to work future functionality or correct deficiencies.

Upon further reflection, the work was not really pushed to the software developers. In the past the work was just repeated - do it in the beginning, in the middle, and in the end. The work would be capture in various disjoint information products such as SRDBs (DOORS), SDFs, memos, etc. In this case, with the IAT, the information is not duplicated (or lost) but capture in the final product, the source code.

This process adds some rigor to the original work performed by the software staff so that it is effectively captured and not revisited during different phases of the program. The rigor is the process of placing the log event in the appropriate location in the code.

If you still think this is a huge burden, some stats on Project-O provide further insight into the overhead issue. There are approximately 128,000 lines of stuff where 28,000 is executable code for 1052 log events. That is 0.8% or 3.8% depending on how you look at it. For me the answer is 0.8% because someone needs to enter all that stuff into the code which is not executable.

4. I only want a specific set of things from the IAT during code analysis so why are we talking about other things associated with the code?

The IAT is not a replacement for the human. In the end it is the analysts that determine whether there is a problem with the software that should be addressed. So the IAT is a tool to help crunch through the stack of paper. It does this by providing different isolated views so that the analyst can focus on specific areas. For example, all decisions (if, else, switch, case, etc) are pulled out and color coded in a report so that an analyst can determine if there are special needs unique to this domain. So, if in the process the analyst sees that there are 4 lines of code on the same line (and the coding standards are silent) the analyst should not remain mum. The analyst needs to raise this issue at the review. Anything else would subvert the purpose and spirit of the IAT and it process.

Further, the detectors and reports are in a constant state of evolution. This evolution is based on the fact that the analysts constantly need to respond to new questions as they arise on a program and the IAT is a tool to help them answer those questions. For example, if it is determined that a type recast has caused a problem, the analyst has 3 choices: (1) do nothing, (2) print out the hardcopy and look for type recasts in all the source code using a yellow highlighter, or (3) spend 15 minutes and add a new IAT service to look for type recasts. Many folks do not understand this simple concept and they resist adding to the IAT because it is viewed as adding more work to the software effort, which by the way has nothing to do with getting basic functionality out the door. However, from a program perspective there is only one right answer, anything else would leave the program open to risk. One could even argue that the DR should not be closed until all type recasts are checked.

5. Why do I need PUIs in the DOORS database?

Project Unique Identifiers exist so that keyword searches return expected results. The IAT reads in the DOORS data base. If PUIs are not established then there is the risk of req text being confused with attributes as the IAT performs its various automated correlations. In the end you won't be able to track the progress of your req's being implemented with each review or release.

6. Why do I need to run the IAT to support my internal peer reviews?

Because you won't have time to fix all the things that will be surfaced to support either a formal code walk through or test event.

This is independent of IAT. It is a fundemental principal of software engineering to catch things as early as possible and avoid them as future development continues. So why would anyone want to remove critical information at that stage, wheither it comes from a person or the IAT or a person who uses and understands the IAT?

7. What is CM and why do I need to understand it?

The IAT fundemental input is software source code and req's from the SRBD (System Reqs Data Base). If you don't know where to find the software to support each activity, you can't provide the service. The activities are peer reviews, code walk throughs, and test / releases.

8. Where did the IAT come from?

The original IAT was focused only on instrumentation. A prototype was developed in a few days which instrumented some code, simulated the results, and performed data reduction.

Since IAT examines each line of code, it became apparent that it might be useful in helping to support reviews. The idea was to use the Internet approach to search for things but rather than return web pages, return lines of stuff from one or more files. The original code analysis extracted source code instructions and removed the comment and header information. The goal was to shrink the size of the paper stack. This then evolved to create reports where only comments, or abstracts were provided in addition to just the source code instructions report. These reports are still present, but rarely used. The code reports eventually evolved into the details area of the new reports. Eventually a report was created based on searches for multiple keywords. The keywords search eventually evolved into automatic detectors that are now part of the stats-problem report and three new reports: hamming distances, error handling, and watchdog timer. Early in the activity the reports were stand alone with no hyper links. Eventually the reports evolved to include hyper links across the reports, source code, and specifications.

The IAT was eventually fitted with services to support the back end test portion of a program. The original data reduction capability was present, but rather than have it process through instrumented code it was modified to process through live log events from a target test configuration. A regression analysis capability was added which performs a bi-directional DIFF and lists new, modified and deleted files. Eventually web enabled services were added to search through the software repository, filter the req's, and compare test data from different test runs.

The final stage of the IAT was the creation of a web portal with links to its services and to IAT project areas. The portal exists for each project and for this generic space. So the IAT  has evolved into a tool platform that provides services that tie many things together on a program which previously were not possible. Further it does many of the manual tasks that no one really wanted to do in the past, like pour through stacks of source code listings or find workable emulator break points.

9. Where is IAT Going?

There were 3 alternatives presented:

1. Move it out and contact the incubator across the street.
2. Shrink wrap it into a product and or service and offer it to other groups, DOD companies, Government, and eventually commercial enterprises.
3. Keep it in house and perhaps convert it to a best practices entity.

The current approach is (3), Keep it in house and perhaps convert it to a best practices entity. Note: The government has formally stated that they would like to place it on their internal shelf of tools at their site.

10. How do I know I should use the IAT on my program?

We could talk about mission critical, safety, and security but it is all irrelevant.

If you have taken the time to conclude that you need an SRDB and you have taken further time to actually populate that SRDB using a tool like DOORS then you should use IAT. Why? Because all the time you spent on the SRDB is a waste unless that SRDB is connected to the software. The IAT is the only thing that will connect all that work to your software in a precise unambiguous manner. Further, the connection is performed once as part of a process and activity that already exists when the software folks find each of their collection of req's for each of their pieces prior to implementation. There is no duplication of activity where systems and test revisit these allocations. In the past the connection, and I use that word loosely, was performed by testers as they allocated requirements to their test procedures. The link to the actual software was approximate via the test cloud and imprecise via word processing products.

If you do not plan to use the IAT then don't use DOORS. Use word processing tables and excel spreadsheets. The rigor available via DOORS is lost once views of the software as implemented are pulled. The original CPC allocations in DOORS do not match the final as implemented code.

11. How was the IAT validated and can I trust its outputs?

IAT evolved on the Project-O program over an 18 month period. It has approximately 500 hours of execution time that was used to support approximately 75 peer reviews and approximately 15 internal / external releases.

It started as just simple Internet based search queries based on regular expressions and grew to its current state. The search queries were validated by the analyst as the findings were understood by clicking to the actual source code or req's. The search queries stayed in place and were used to verify the more elaborate mechanisms such as the automated error detectors in the stats-problem report. So there is built in redundancy in the views of data where in the end the analyst can always go back to the source code or req text. That is how each detector and keyword search regular expression was matured.

The instrumentation was built with the same redundancy concepts. When instrumentation is performed a summary report is provided. Links to the instrumented and non instrumented code are easily accessible and bound the instrumentation results. Finally there is an automatic DIFF that is performed, which can not be bypassed, whose output uses color and layout to quickly ascertain if instrumentation performed as expected.

So the elements to the IAT that instill trust and validation of its products are:

1. Evolution from simple to more complex
2. Significant time in use on a real program
3. Quick HTML access to alternative views
4. Quick HTML access to the original source code and req's text
5. Automatic DIFFs where needed
6. Keyword backups to automated detectors

Can you trust the IAT? Only if you use its services. That means using the hyper links and thinking about the findings.

12. This is really hard and my head is going to explode so why can't software just focus on the functionality of the box?

If we just focus on the functionality of the box there is no need for our services. Anyone can then do this and offer the product for $19.95. This may sound like a funny statement, but there is a significant element of truth. A device is not just about functionality. It is also about performance. We happen to work in an area which requires a certain level of performance that few organizations are able to provide at this time. The IAT attempts to add automation to a process that is years in the making. IAT and its processes may not be perfect but do you have a better alternative? If so, go for it, that's fundamentally what we all do anyway, but watch out for exploding heads :)

13. Ok I am sold, but if I do this how does the data come out of the device during test?

The IAT converts the comment statements to a function call. This unique function is responsible for getting the data out to an external interface port. It does not matter what the data format is, however ASCII is strongly suggested. It does not matter what fields are included in the output, however an internal counter along with the log event is strongly suggested. Experience has shown that 4 decimal digits for the log events prefaced by a decimal counter is sufficient to do everything IAT needs for its test data analysis. The counter is used to determine if data is internally lost. This is accomplished with an IAT counter check during data reduction.

This unique function call is very important because you want to make sure you keep the overhead to a minimum. Most operating systems and executives like PSOS and TIs' DSP BIOS provide unique services to support this capability. You just need to search for keywords like "log, log_event, and event" in their documentation. It's usually less than 10 pages, but those 10 pages are extremely important in getting the data out of your box. Do not fall into the trap of using a printf or rolling your own routine until you have completely exhausted your search of the documentation on your system.

14. Wooow, wait a minute, this is not my job and this is going to cost extra money and time, we are not doing this?

Here is an old architecture concept from a few decades ago. When you build something it needs to be PRODUCED, MAINTAINED, and yes TESTED. Design for produceability, maintainability, and testability may not appear to fit with current philosophies of lets get lean and mean, but if you want a successful project you better expand your horizons and you need to work together with all the stake holders on a project.

So yes, there is work that needs to be done. The work should not be short changed. Even though the logging function is very small, it can have a big impact on the device from Hiezenbergs' point of view. Yes we are perturbing the device, and yes Mr Hiezenberg did say something like you can never measure a thing because by the very nature of looking at it you have modified its true nature. However, engineering is about tradeoffs not the ideal. It is about approaching reasonable ideals.

So, whoever is tasked with getting the test data out of the box needs to research the computer and support software. In the case of the TI DSP, which was the target platform for the initial IAT, the TI folks created a service in their "operating system". The TI BIOS has a service that allows test data to be sent out the JTAG port with minimal overhead. The service is coincidentally called "Log_Event". When a Log_Event call is made only 40 machine cycles are executed as opposed to a printf which would take 40,000+ machine cycles. Also, no additional memory is needed from an include perspective. In the case of a printf, the STDIO library needs to be included which adds to the memory.

So, yes someone needs to determine the most effective mechanism to get the data out of the box. The mechanism needs to be very simple and take advantage of any unique services available from the hardware and support software.

15. You know, I did all this and I am getting 140 log events  per second coming out of my box?

The IAT will filter the noise log events during data reduction, however, you messed up really bad. Remember you have to design for produceability, maintainability, and testability. Assuming you followed the log event placement rules and there are no log events in the real time path, there obviously are log events in some recurring path like a periodic self test path. Shame on you. You should have thought about this earlier and it should have been addressed in the design and definitely closed out properly during code reviews.

But here you are getting ready to test this beast and its bad. So what are the possible solutions?

1. IAT will allow you to de-instrument individual log events or whole groups of log events based on PUI or file names. The offending log events can be verified with special build(s) or with a different method. This will require 2 or more instrumented builds.
2. You can create your logging function with a bank of switches that can be enabled or disabled from the debugger / emulator or other source. During instrumentation, log events can be assigned groups of numbers like the 1000, 2000, 3000 series and the switch bank can enable disable log events in selected groups. So offending log events can be selectively turned on or off by the testers within the unique log event function during the test. Only 1 instrumented build is needed.
3. Your device can be given an operational capability to turn off certain periodic events like self tests. If you have a successful program, that probably will be eventually directed by the customer because such periodic processing also causes operational problems in complex systems (IMH experience).

Metrics Frequently Asked Questions

1. Why do we need metrics and how should we gather all this stuff?
2. Why not just use cost and schedule data - why gather all this other stuff?
3. The metrics the IAT gathers are trivial and useless for our real time systems so why do this to ourselves?
4. What data should we collect?
5. Ok, if I gather the metrics, who should look at these things?

1. Why do we need metrics and how should we gather all this stuff?

All organizations gather metrics, even if they are not written down. We gather metrics to try to figure out what is going on, if there are problems, and to try to make things easier / better. Why do I start off this way? Because many software organizations view metrics as an unnecessary burden taking away time from the "real work". These "evil" metrics provide no insight into the real problems and can be misused within an organization. So the answer is to avoid metrics and just code the software. It's important to understand this because any metric data that is gathered needs to be "correct" and common across "different" activities so that meaningful comparisons can be performed. In the ideal situation the metrics would be gathered transparently, in the background, with little or no human intervention. IAT does that for the collection of software and requirements that it is provided.

2. Why not just use cost and schedule data - why gather all this other stuff?

Bottom line is that cost and schedule are not good enough. More visibility is needed into an organization. Without this visibility the cost and schedule once blown is just that, blown. If finer grain data can be provided to show what may happen prior to losing the milestone, then corrective action can be taken sooner and less time and money might be wasted. A typical response is to develop plans with more milestones and shorter schedule cycles. This is called micro management and is usually a failure since its not possible to anticipate activities / tasks below a certain level. So the answer is not to develop "better micro" schedules! The answer is to gather metrics and look at the data.

3. The metrics the IAT gathers are trivial and useless for our real time systems so why do this to ourselves?

"The software folks are busy trying to get basic functionality into the box and chasing real time problems. We need better real time tools and this paper stuff will not help us."

These are the typical responses when the topic of software metrics surface. So what does gathering information on the number of missing curly braces have to do with the serious problems in the lab? Some folks think that if these basic kinds of software quality characteristics are missing, then its an indication of a stressed organization - an organization that is fighting with getting basic functionality into the box and getting it to work. The converse can be true where the team goes out of its way to make "pretty code" without any emphasis on having the box perform. However, if that approach is taken, testing and the metrics associated with testing should surface that "bad" approach. So its a balance. Don't go out of the way to stress that the code "be pretty" at the expense of working functionality. If this balance between the "software" and "test" metrics is maintained, then there is a reasonable set of software metrics. Remember that the test metrics counter balance the software metrics.

4. What data should we collect?

Tricky question. The problem is to minimize the noise so that the real information can be detected from the data. However, in one instance data considered noise becomes valuable information in another instance. So visualization is key to being able to reduce the noise level. What should be gathered? Anything that is easy to count on a consistent basis. The IAT has an initial set of metric item counters. If someone can identify an item to count, and the IAT can be easily modified, it should be added to the IAT.

5. Ok, if I gather the metrics , who should look at  these things?

You should, along with your lead, and the TD. In fact, the leads and TDs should use them to help understand where there are potential issues. The metrics should also percolate to the organizational level. The organizational level should use this information to gain a better insight into how things are progressing, given that cost and schedule indicators are usually too late, they are just a confirmation of a problem. Many folks think that everyone should have access to this information and be excited about being able to interpret the data and that a mature organization (level 3 to 5) has metrics charts in almost every office :)

sat@cassbeth.com