(note that this page requires updating since it has references to the Redmine wiki)

FOSSology_system_testing_2.6.2

The FOSSology team believes that quality must be "built-in" through good engineering practices, not "bolted-on" later through a lot of testing. To provide a product that not only meets its requirements but also is maintainable and supportable over time, the team pursues solutions that are engineered, not just programmed. The QA infrastructure provides a framework for engineered solutions to be built, for quality to be tested and validated at every step, and for the product to be able to be released as frequently as needed.

For more information on test automation, see FOSSology Automation.

Requirements for new or improved FOSSology features come from various stakeholders like internal HP groups (primarily the OSRB) and external users. The requirements are captured and tracked in the external wiki as part of the project "Roadmap/Backlog":http://www.fossology.org/projects/fossology/roadmap. This allows everyone to review and comment on the FOSSology requirements as they are documented and implemented.

Testing occurs not only to make sure the mainstream use cases function appropriately but also the corner cases (like invalid data) get handled appropriately. It ensures expected errors are trapped and the system produces expected behavior. (Click for more information re: Testing Basics)

The FOSSology team uses several types of tests to achieve its software quality goals. The continuous integration (CI) environment exercises all these tests, except manual system tests, on a regular basis.

key Areas of Focus

Important Areas of Future Focus

The FOSSology source code is organized so that each agent has all the needed components for its operation in its source tree. This includes the unit and functional tests for that agent and associated Makefiles to drive the necessary test creation and execution. See Module Structure for details.

Each module has a high-level Makefile and lower-level Makefiles to drive testing, whether it be unit or functional, with associated test targets. If both unit and functional testing is desired, the high-level Makefile is invoked from the module's root directory which in turn calls the lower-level Makefiles to run the tests. If only one type of testing is desired, the lower-level Makefile is invoked directly with its test target and only that test is executed. In this way, a developer has complete control over what gets run in his sandbox, and the Continuous Integration server has the necessary granularity to run the desired tests whenever it wants.

###1.4.3 Test Data

TBD

• A basic database that the tests can query with known-good data
• A common test repo that the tests can use

The test database/repo should not have any dependencies on any system value or configuration outside of the sandbox; in other words, all information to make the tests work should either be checked into the sandbox that gets created or created by the high-level test target as a matter of initiation. There shouldn’t be any dependencies on any other system file to run within the sandbox.

All tests produce output in junit format for test reporting; if one of the existing test frameworks are not used, then whatever is used must produce a xml report in junit format. The definition for the junit report format can be found on the fossology wiki at: http://fossology.org/test:junit_format.

It is strongly suggested that one of the existing test frameworks be used to create tests. If an existing test framework is not chosen, a suggested method to produce junit report formats is to wrap all test cases in PHUnit. By running PHPUnit with --log-junit flag, the results will be in junit format.

--log-junit <file>

All tests should be documented using doxygen. Test documentation is needed for a number of reasons.

• Tests are code, code should be documented. For example, what does the test test?
• Test documentation can serve as the test plan, so either a small test plan is needed or none at all depending on how well the test documentation is done.
• Tests can have an API, the API should be documented so other tests can use it.

The documentation for the tests can be found at "Test Documentation":http://fotest.fc.hp.com/~markd/testDocs/Docs/index.html. These documents are updated frequently as part of the continuous integration.

TBD - a new server/cluster that mimics production and allows testing and troubleshooting outside the production and sandbox environments.

• Testing-vm_machine_list

• "FOSSology Testing Dashboard":http://fotest.fc.hp.com:8080 (HP only)

To ensure an expected level of quality and to improve that level over time, there are specific metrics that are measured and tracked over time.

• number of tests
  • Unit
  • Functional
  • System
  • Stress
• Code Coverage
  • Lines of Code (LOC) covered
  • Functions covered
  • Branches covered

!Test_Metrics_Pictures.jpg! !Test_Metrics_Pictures_201306.jpg! !Test_Metrics_Numbers_201306.jpg!

• "Latest FOSSology Unit Test report":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Unit_Test_Results/Unit_Tests_Results.html (HP only)
• "Mimetype code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Mimetype_Coverage_Report/
• "Delagent code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Mimetype_Coverage_Report/
• "Pkgagent code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Pkgagent_Coverage_Report/
• "Ununpack code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Ununpack_Coverage_Report/
• "Copyright code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Copyright_Coverage_Report/
• "Wgetagent code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Wget_Agent_Coverage_Report/
• "Scheduler code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/scheduler_Coverage_Report/
• "Common php code coverage":http://fotest.fc.hp.com:8080/job/Fossology_Unit_Tests/Common_PHP_Lib_Coverage_Report/

• Automation_Testing_Unit_Tests

• "Latest FOSSology Functional Test report":http://fotest.fc.hp.com:8080/job/Fossology_Functional_Tests/Agent_Functional_Test_Results/Functional_Tests_Results.html (HP only)

• Automation_Testing_Functional_Tests

• FOSSology_system_testing_2.6.2

• FOSSology_system_testing_2.6.0

• FOSSology_system_testing_2.5.0

• FOSSology_system_testing_2.4.0

• FOSSology_system_testing_2.3.0

• FOSSology_system_testing_2.2.0

• FOSSology_system_testing_pre_2.2.0

• Automation_Testing_System_Tests

UI Tests

• How to Write PHP UI tests
• FOSSology Test Suite
• How to configure and run UI Tests