TestArchiver
Tools for serialising test results to SQL database
Install / Use
/learn @salabs/TestArchiverREADME
TestArchiver
TestArchiver is a tool for archiving your test results to a SQL database.
And Epimetheus is the tool for browsing the results you archived.
Testing framework support
| Framework | Status | Fixture test status | Parser option | | --------------- | --------------------------- | ------------------- | ------------- | | Robot Framework | Supported | Done | robot | | Mocha | Supported | Done | mocha-junit | | pytest | Supported | Done | pytest-junit | | PHPUnit | Supported | Done | php-junit | | JUnit | Experimental | Missing | junit | | xUnit | Experimental | Missing | xunit | | MSTest | Experimental | Missing | mstest |
Experimental status here means that there is a parser that can take in e.g. generic JUnit formatted output but there is no specific test set or any extensive testing or active development for the parser.
Contributions for output parsers or listeners for different testing frameworks are appreciated. Contributing simply a fixture test set (that can be used to generate output files for developing a specific parser) is extremely helpful for any new framework.
Installation
sudo -H python3 -m pip install testarchiver
Supported databases
SQLite
SQLite default database for the archiver and is mainly useful for testing and demo purposes. Sqlite3 driver is part of the python standard library so there are no additional dependencies for trying out the archiver.
PostgreSQL
PostgreSQL is the currently supported database for real projects. For example Epimetheus service uses a PosrgreSQL database. For accessing PostgreSQL databases the script uses psycopg2 module: pip install psycopg2-binary (comes with pip install)
Basic usage
The output files from different testing frameworks can be parsed into a database using test_archiver/output_parser.py script.
testarchiver --database test_archive.db output.xml
Assuming that output.xml is a output file generated by Robot Framework (the default parser option), this will create a SQLite database file named test_archive.db that contains the results.
The archiver script can also be invoked from the module:
python3 -m test_archiver.output_parser --database test_archive.db output.xml
For list of other options: testarchiver --help
positional arguments:
output_files list of test output files to parse in to the test
archive
optional arguments:
-h, --help show this help message and exit
--version, -v show program's version number and exit
--config CONFIG_FILE Path to JSON config file containing database
credentials and other configurations. Options given on
command line will override options set in a config
file.
--format {robot,robotframework,xunit,junit,mocha-junit,pytest-junit,mstest,php-junit}
output format (default: robotframework)
--repository REPOSITORY
The repository of the test cases. Used to
differentiate between test with same name in different
projects.
--team TEAM Team name for the test series
--series SERIES Name of the test series (and optionally build number
'SERIES_NAME#BUILD_NUM' or build id
'SERIES_NAME#BUILD_ID')
--metadata NAME:VALUE
Adds given metadata to the test run. Expected format:
'NAME:VALUE'
Database connection:
--dbengine DB_ENGINE Database engine, postgresql or sqlite (default)
--database DATABASE database name
--host HOST database host name
--user USER database user
--pw PASSWORD, --password PASSWORD
database password
--port PORT database port (default: 5432)
--dont-require-ssl Disable the default behavior to require ssl from the
target database.
Schema updates:
--allow-minor-schema-updates
Allow TestArchiver to perform MINOR (backwards
compatible) schema updates the test archive
--allow-major-schema-updates
Allow TestArchiver to perform MAJOR (backwards
incompatible) schema updates the test archive
Clean history:
If any of the following options are used the archiver will delete the oldest
result data. What data is kept can be controlled with these --keep-X options
that can also be mixed. If deletion targets are not specified with --clean-X
options all test run data is cleaned. --keep-X options should be used when
using --clean-X options otherwise entire history is cleared.
--keep-builds KEEP_BUILDS
Delete old result data but keep the data for at least
given number of builds in each series.
--keep-months KEEP_MONTHS
Delete old result data but keep data that was archived
more recently than given number of months ago
--keep-after KEEP_AFTER
Delete old result data but keep data that was archived
more recently than given date in ISO 8601 format
yyyy-mm-dd.
--clean-team CLEAN_TEAM
Delete results only archived under given team name. Use
this with --keep-X options. Otherwise will delete
entire history
--clean-logs Delete oldest log messages but not test results or runs.
Use this with --keep-X options. Otherwise will delete
entire log history
--clean-logs-below {TRACE,DEBUG,INFO,WARN}
Delete oldest log messages that are bellow given log
level but not test results or runs. Use this with
--keep-X options. Otherwise will delete entire log
history
--clean-keyword-stats
Delete oldest keyword statistics data but not test
results or runs. Use this with --keep-X options.
Otherwise will delete entire log history
Limit archived data:
--no-keywords Do not archive keyword data
--no-keyword-stats Do not archive keyword statistics
--ignore-logs-below {TRACE,DEBUG,INFO,WARN}
Sets a cut off level for archived log messages. By
default archives all available log messages.
--ignore-logs Do not archive any log messages
--max_log_message_length MAX_LOG_MESSAGE_LENGTH
Specify how many characters of the log message that is
archived. full: archives the complete log. positive
integers: archives number of characters from the
beginning. negative integers: archives number of
characters from the end.
Adjust timestamps:
--time-adjust-secs TIME_ADJUST_SECS
Adjust time in timestamps by given seconds. This can
be used to change time to utc before writing the
results to database, especially if the test system
uses local time, such as robot framework. For example
if test were run in Finland (GMT+3) in summer (+1hr),
calculate total hours by minutes and seconds and
invert to adjust in correct direction, i.e.
-(3+1)*60*60, so --time-adjust-secs -14400. This
option is useful if you are archiving in a different
location to where tests are run.If you are running
tests and archiving in same timezone, time-adjust-
with-system-timezone may be a better option. This
option may be used in conjunction with --time-adjust-
with-system-timezone if desired.
--time-adjust-with-system-timezone
Adjust the time in timestamps by the system timezone
(including daylight savings adjust). If you are
archiving tests in the same timezone as you are
running tests, setting this option will ensure time
written to the database is in UTC/GMT time. This
assumes that if multiple computers are used that their
timezone and daylight savings settings are identical.
Take care also that you do not run tests just before a
daylight savings time adjust and archive just after,
as times will be out by one hour. This could easily
happen if long running tests cross a timezone adjust
boundary. This option may be used in conjunction with
--time-adjust-secs.
ChangeEngine:
--change-engine-url CHANGE_ENGINE_URL
Starts a listener that feeds results to ChangeEngine
--execution-context EXECUTION_CONTEXT
To separate data from different build pipelines for
ChangeEngine prioritization. Example if same changes
or tests may be used to verify app in A
