Whisker

Whisker is an automated testing framework for Scratch projects.

Description

Block-based programming environments like Scratch foster engagement with computer programming and are used by millions of young learners. Scratch allows learners to quickly create entertaining programs and games, while eliminating syntactical program errors that could interfere with progress.

However, functional programming errors may still lead to incorrect programs, and learners and their teachers need to identify and understand these errors. This is currently an entirely manual process.

Whisker provides automated and property-based testing functionality for Scratch programs.

Building Whisker

Whisker requires Node.js v24.11.0 and is built using the yarn (classic) package manager.

After cloning the repository, install all dependencies using:

yarn install

Now you can build the application using:

yarn build

Running Whisker

After building Whisker, simply open 'whisker-web/dist/index.html' in your browser (e.g. Firefox)

firefox whisker-web/dist/index.html

The Servant is a wrapper around the web client, allowing to run tests automatically in a headless environment (via chrome-headless / puppeteer). It is called the Servant as in "Cats don't have owners they have servants".

To use Whisker on the command line, you can use the Servant node frontend as follows:

node servant run -s <Scratch project file> -t <test file>

An overview of options is provided using node servant help:

Usage: node servant [options] [command]

A Testing Utility for Scratch 3.0

Options:
  -h, --help                display help for command
  -V, --version             output the version number

Commands:
  open [options]            open the Whisker web interface
  dynamic [options]         dynamic test suites using Neuroevolution
  generate [options]        generate Whisker test suites
  help [options] [command]  display help for command
  model [options]           test with model
  run [options]             run Whisker tests
  witness [options]         generate and replay error witnesses

To show further help, additionally pass the name of the command you are interested in, e.g., the run command: node servant help run.

Usage: node servant run [options]

run Whisker tests

Options:
  -a, --acceleration <Integer>      acceleration factor (default: 1)
  -bm, --max-mutants <Integer>      upper bound of analysed mutants during mutation analysis
  -bt, --mutation-budget <Integer>  timeout for the mutation analysis
  -d, --headless                    run headless ("d" like in "decapitated") (default: false)
  -dm, --download-mutants           downloads the generated mutants
  -h, --help                        display help for command
  -j, --number-of-jobs <Integer>    number of jobs (Chromium tabs) for test execution (default: 1)
  -m, --mutators <String...>        mutation operators to apply
  -s, --scratch-path <Path>         path to file (".sb3") or folder with scratch application(s)
  -t, --test-path <Path>            path to Whisker tests to run (".js")
  -o, --output <Path>               create CSV file with results
  -z, --seed <String>               custom seed for Scratch-VM
  -ta, --trace-attributes           activates tracing of sprite attributes after every executed block (default: false)
  -v, --verbose                     Verbose mode. Prints debug messages. Multiple -v increase verbosity.

To run tests in accelerated mode, provide an acceleration factor using the option -a. We recommend using an acceleration factor of at most 10, as very low execution times may lead to non-deterministic program behaviour.

For example, the following command runs tests with a 10-fold speedup and two parallel executions in a headless chrome instance:

node servant run -s project.sb3 -t tests.js -a 10 -d -j 2

Using Apptainer (Headless Mode)

Alternatively, you can run Whisker in headless mode using Apptainer. This can be beneficial if you want to conduct large-scale experiments on a computing cluster. To this, build a SIF image of Whisker:

./build-whisker-container.sh

Prerequisites: You need to install Apptainer and Docker on the build machine. Now, you can run the containerized version of Whisker via

apptainer run <name-of-the-sif-file>.sif <Whisker args>

The main entry point to the container is the wrapper script whisker-container.sh, which calls Whisker's servant in headless mode (using the flags -d and -vv.) The <Whisier args> given by the user will be forwarded by the script to the servant.

Writing Tests

Details on writing Whisker tests in JavaScript can be found here.

Generating Tests Automatically

The web interface provides the possibility to automatically generate tests. In the web interface, choose an appropriate search configuration (examples can be found in the config directory), and click Test Generation. Warning: This may take a while! Once the search has completed, the generated tests are loaded into the editor window.

If you run test generation with Servant (command line option -g), at the end of the search a file called tests.js is created in the current directory which contains the tests. These can now be loaded into Whisker.

Contributors

Whisker is developed at the Chair of Software Engineering II of the University of Passau.

Contributors:

Adina Deiner
Patric Feldmeier
Christoph Frädrich
Gordon Fraser
Sophia Geserer
Katharina Götz
Eva Gründinger
Nina Körber
Marvin Kreis
Sebastian Schweikl
Andreas Stahlbauer
Siegfried Steckenbiller
Emma Wang
Phil Werli
Nik Zantner

Whisker is supported by the project FR 2955/3-1 funded by the "Deutsche Forschungsgemeinschaft".

References

@inproceedings{DBLP:conf/sigsoft/StahlbauerKF19,
  author    = {Andreas Stahlbauer and
               Marvin Kreis and
               Gordon Fraser},
  title     = {Testing scratch programs automatically},
  booktitle = {{ESEC/SIGSOFT} {FSE}},
  pages     = {165--175},
  publisher = {{ACM}},
  year      = {2019}
}

@inproceedings{DBLP:conf/ssbse/DeinerFFGZ20,
  author    = {Adina Deiner and Christoph Fr{\"{a}}drich and
               Gordon Fraser and Sophia Geserer and
               Niklas Zantner},
  title     = {Search-Based Testing for Scratch Programs},
  booktitle = {12th International Symposium on Search-Based Software Engineering},
  series    = {Lecture Notes in Computer Science},
  volume    = {12420},
  pages     = {58--72},
  publisher = {Springer},
  year      = {2020},
}

@InProceedings{gotz2022model,
  title 		= {Model-based Testing of Scratch Programs},
  author 		= {G{\"o}tz, Katharina and Feldmeier, Patric and Fraser, Gordon},
  booktitle 	= {Proceedings of the International Conference on Software Testing, Verification and Validation (ICST'22)},
  pages 		= {411--421},
  year 			= {2022},
  publsiher     = {IEEE},
  doi 			= {10.1109/ICST53961.2022.00047}
}

@misc{deiner2022automated,
      title         = {Automated Test Generation for Scratch Programs},
      author        = {Adina Deiner and Patric Feldmeier and Gordon Fraser and Sebastian Schweikl and Wengran Wang},
      year          = {2022},
      eprint        = {2202.06274},
      archivePrefix = {arXiv},
      primaryClass  = {cs.SE}
}

@InProceedings{feldmeier2022neuroevolution,
	author 		= {Feldmeier, Patric and Fraser, Gordon},
	booktitle 	= {Proceedings of the International Conference on Automated Software Engineering (ASE'22)},
	publisher 	= {ACM},
	title 		= {Neuroevolution-Based Generation of Tests and Oracles for Games},
	year 		= {2023},
	doi 		= {10.1145/3551349.3556939},
	pages 		= {1--13}
}

@article{feldmeier2023learning,
  title         = {Learning by Viewing: Generating Test Inputs for Games by Integrating Human Gameplay Traces in Neuroevolution},
  author        = {Feldmeier, Patric and Fraser, Gordon},
  journal       = {arXiv preprint arXiv:2304.06413},
  year          = {2023},
  doi           = {10.48550/arXiv.2304.06413}
}

@inproceedings{nuzzlebug24,
  author    = {Adina Deiner and Gordon Fraser},
  title     = {NuzzleBug: Debugging Block-Based Programs in Scratch },
  booktitle = {ACM/IEEE International Conference on Software Engineering (ICSE)},
  publisher = {{IEEE}},
  year      = {2024}
}