| # Testing astcenc |
|
|
| The repository contains a small suite of tests which can be used to sanity |
| check source code changes to the compressor. It must be noted that this test |
| suite is relatively limited in scope and does not cover every feature or |
| bitrate of the standard. |
|
|
| # Required software |
|
|
| Running the tests requires Python 3.7 to be installed on the host machine, and |
| an `astcenc-avx2` release build to have been previously compiled and installed |
| into an directory called `astcenc` in the root of the git checkout. This |
| can be achieved by configuring the CMake build using the install prefix |
| `-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build |
| target. |
|
|
| # Running C++ unit tests |
|
|
| We support a small (but growing) number of C++ unit tests, which are written |
| using the `googletest` framework and integrated in the CMake "CTest" test |
| framework. |
|
|
| To build unit tests pull the `googletest` git submodule and add |
| `-DASTCENC_UNITTEST=ON` to the CMake command line when configuring. |
|
|
| To run unit tests use the CMake `ctest` utility from your build directory after |
| you have built the tests. |
|
|
| ```shell |
| cd build |
| ctest --verbose |
| ``` |
|
|
| # Running command line tests |
|
|
| To run the command line tests, which aim to get coverage of the command line |
| options and core codec stability without testing the compression quality |
| itself, run the command line: |
|
|
| python3 -m unittest discover -s Test -p astc_test*.py -v |
| |
| # Running image tests |
|
|
| To run the image test suite run the following command from the root directory |
| of the repository: |
|
|
| python3 ./Test/astc_test_image.py |
| |
| This will run though a series of image compression tests, comparing the image |
| PSNR against a set of reference results from the last stable baseline. The test |
| will fail if any reduction in PSNR above a set threshold is detected. Note that |
| performance information is reported, but regressions will not flag a failure. |
|
|
| For debug purposes, all decompressed test output images and result CSV files |
| are stored in the `TestOutput` directory, using the same test set structure as |
| the `Test/Images` folder. |
|
|
| ## Test selection |
|
|
| The runner supports a number of options to filter down what is run, enabling |
| developers to focus local testing on the parts of the code they are working on. |
|
|
| * `--encoder` selects which encoder to run. By default the `avx2` encoder is |
| selected. Note that some out-of-tree reference encoders (older encoders, and |
| some third-party encoders) are supported for comparison purposes. These will |
| not work without the binaries being manually provided; they are not |
| distributed here. |
| * `--test-set` selects which image set to run. By default the `Small` image |
| test set is selected, which aims to provide basic coverage of many different |
| color formats and color profiles. |
| * `--block-size` selects which block size to run. By default a range of |
| block sizes (2D and 3D) are used. |
| * `--color-profile` selects which color profiles from the standard should be |
| used (LDR, LDR sRGB, or HDR) to select images. By default all are selected. |
| * `--color-format` selects which color formats should be used (L, XY, RGB, |
| RGBA) to select images. By default all are selected. |
|
|
| ## Performance tests |
|
|
| To provide less noisy performance results the test suite supports compressing |
| each image multiple times and returning the best measured performance. To |
| enable this mode use the following options: |
|
|
| * `--repeats <M>` : Run M test compression passes which are timed. |
|
|
| **Note:** The reference CSV contains performance results measured on an Intel |
| Core i5 9600K running at 4.3GHz, running each test 5 times. |
|
|
| ## Updating reference data |
|
|
| The reference PSNR and performance scores are stored in CSVs committed to the |
| repository. This data is created by running the tests using the last stable |
| release on a standard test machine we use for performance testing builds. |
|
|
| It can be useful for developers to rebuild the reference results for their |
| local machine, in particular for measuring performance improvements. To build |
| new reference CSVs, download the current reference `astcenc` binary (1.7) from |
| GitHub for your host OS and place it in to the `./Binaries/1.7/` directory. |
| Once this is done, run the command: |
|
|
| python3 ./Test/astc_test_image.py --encoder 1.7 --test-set all --repeats 5 |
| |
| ... to regenerate the reference CSV files. |
|
|
| **WARNING:** This can take some hours to complete, and it is best done when the |
| test suite gets exclusive use of the machine to avoid other processing slowing |
| down the compression and disturbing the performance data. It is recommended to |
| shutdown or disable any background applications that are running. |
|
|
| ## Valgrind memcheck |
|
|
| It is always worth running the Valgrind memcheck tool to validate that we have |
| not introduced any obvious memory errors. Build a release build with symbols |
| information with `-DCMAKE_BUILD_TYPE=RelWithDebInfo` and then run: |
|
|
| valgrind --tool=memcheck --track-origins=yes <command> |
| |
| - - - |
|
|
| _Copyright © 2019-2022, Arm Limited and contributors. All rights reserved._ |
|
|
