Science Data Systems Regression testing of CIAO
Lee, Nicholas P.
The Chandra Interactive Analysis of Observations (CIAO) system is a software suite developed by the CXC Science Data Systems (SDS) and Data Systems groups and is released on a number of platforms, including Linux, Solaris and Mac. The science unit and regression testing of CIAO is an integral part of each release in addition to the testing by the s/w developers.
Unit science tests are performed by SDS scientists to meet specified scientific requirements. The SDS commitment to support a release requires regression testing every CIAO module, supporting libraries, and contributed scripts on each platform to demonstrate new functionality and ensure that bugs are not introduced during the development process.
New and updated regression tests are created based on unit testing and added to the test suite, currently containing over 1000 individual tests. The results are analyzed and compared with previously saved multi-platform tests. The uncovered differences are confirmed as software enhancements and/or anticipated changes, or identified as issues, including cross-platform differences, to be addressed before the release.
The basic regression testing system has been in place ever since the Chandra launch, however, over the past few years, significant improvements have been made including revision and restructuring of the tests and the testing procedures, introduction of tolerances for automatic comparison, and moving the baseline sets and test results onto a nettapp server. This has decreased the number of flagged test failures during the automated result analysis process, requiring less manual checks. Additionally, significant manual analysis improvements have been made. These enhancements resulted in a significant decrease of run and analysis time, in spite of the significant increase of the number of tests over time. The results of the SDS unit and regression tests continue to significantly contribute to the verification, consistency and stability of the CIAO system.
Return to poster list