Beet

<img src="https://github.com/user-attachments/assets/7ef41204-2820-4409-b6bd-80d78d0399c5" alt="Beet logo" width="200">

Beet is a Daikon front-end (also known as instrumenter) that converts any OpenAPI specification and a collection of API requests (and responses) into a format processable by the Daikon invariant detector. Beet is one of the core components of AGORA+, an approach for the Automated Generation of test Oracles for REST APIs through the detection of invariants (properties of the output that should always hold).

AGORA+ aims to learn the expected behavior of an API by analyzing previous API requests and their corresponding responses. AGORA+ operates in black-box mode, making it applicable to any REST API and seamlessly integrable into existing API testing tools.

The test oracles generated by AGORA+ have revealed dozens of bugs in APIs with millions of users: Amadeus Hotel, Deutschebahn, FDIC, Foursquare, GitHub, GitLab, Marvel, NYTimesBooks, OMDb, RESTCountries and YouTube. Our reports have guided developers in improving their APIs, including bug fixes and documentation updates in GitHub.

Index

1. Wiki
2. AGORA+ Overview
3. Publication
4. Motivational example
5. Types of errors detected by AGORA+
6. Executing Beet
7. Executing modified version of Daikon

Wiki

In this page, you can find a brief description of Beet and AGORA+ workflows with an illustrating example. If you want to know more details about Beet instrumentation process, the types of invariants supported by our modified version of Daikon and the different configuration options, please refer to our Wiki.

AGORA+ Overview

The image below shows an overview of the complete workflow of AGORA+. At the core of the approach is Beet, that receives two inputs:

1. The OAS specification of the API under test.
2. A set of API requests with their corresponding responses.

As a result, Beet returns an instrumentation of the API requests consisting of a declaration file (describing the format of the API operations inputs and outputs) and a data trace file (specifying the values assigned to each input parameter and response field in each API call). This instrumentation is then processed by our customized version of Daikon, resulting in a set of likely invariants that can be potentially used as test oracles, once confirmed by the developer.

Additionally, the reported invariants can be converted into executable assertions using PostmanAssertify, a software tool that automatically generates a Postman collection containing a request for each response code of all the operations for which AGORA+ has detected invariants, each of them containing a test script with all the confirmed invariants implemented as assertions.

Publication

AGORA+ has been accepted for publication in ACM Transactions on Software Engineering and Methodology.

AGORA (the conference version of AGORA+) was accepted for publication in the technical track of the ACM SIGSOFT International Symposium on Software Testing and Analysis 2023 (ISSTA 2023) (pre-print available here), and received the ACM SIGSOFT Distinguished Artifact Award. A very preliminary version of AGORA (two-page abstract) obtained the first prize (graduate category) in the ACM Student Research Competition of ESEC/FSE 2022, and the second prize in the ACM Student Research Competition Grand Finals 2023.

If you want to cite Beet or AGORA+ in your research, please use the BibTeX entry below.

@article{Alonso2025AGORA_plus,
author = {Alonso, Juan C. and Ernst, Michael D. and Segura, Sergio and Ruiz-Cort\'{e}s, Antonio},
title = {Test Oracle Generation for REST APIs},
year = {2025},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {1049-331X},
url = {https://doi.org/10.1145/3726524},
doi = {10.1145/3726524},
abstract = {The number and complexity of test case generation tools for REST APIs have significantly increased in recent years. These tools excel in automating input generation but are limited by their test oracles, which can only detect crashes, regressions, and violations of API specifications or design best practices. This article introduces AGORA+, an approach for generating test oracles for REST APIs through the detection of invariants—output properties that should always hold. AGORA+ learns the expected behavior of an API by analyzing API requests and their corresponding responses. We enhanced the Daikon tool for dynamic detection of likely invariants, adding new invariant types and creating a front-end called Beet. Beet translates any OpenAPI specification and a set of API requests and responses into Daikon inputs. AGORA+ can detect 106 different types of invariants in REST APIs. We also developed PostmanAssertify, which converts the invariants identified by AGORA+ into executable JavaScript assertions. AGORA+ achieved a precision of 80\% on 25 operations from 20 industrial APIs. It also identified 48\% of errors systematically seeded in the outputs of the APIs under test. AGORA+ uncovered 32 bugs in popular APIs, including Amadeus, Deutschebahn, GitHub, Marvel, NYTimesBooks, and YouTube, leading to fixes and documentation updates.},
note = {Just Accepted},
journal = {ACM Trans. Softw. Eng. Methodol.},
month = mar,
keywords = {REST APIs, test oracle, invariant detection, automated testing}
}

@inproceedings{Alonso2023AGORA,
author = {Alonso, Juan C. and Segura, Sergio and Ruiz-Cort\'{e}s, Antonio},
title = {AGORA: Automated Generation of Test Oracles for REST APIs},
year = {2023},
isbn = {9798400702211},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3597926.3598114},
doi = {10.1145/3597926.3598114},
abstract = {Test case generation tools for REST APIs have grown in number and complexity in recent years. However, their advanced capabilities for automated input generation contrast with the simplicity of their test oracles, which limit the types of failures they can detect to crashes, regressions, and violations of the API specification or design best practices. In this paper, we present AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants—properties of the output that should always hold. In practice, AGORA aims to learn the expected behavior of an API by analyzing previous API requests and their corresponding responses. For this, we extended the Daikon tool for dynamic detection of likely invariants, including the definition of new types of invariants and the implementation of an instrumenter called Beet. Beet converts any OpenAPI specification and a collection of API requests and responses to a format processable by Daikon. As a result, AGORA currently supports the detection of up to 105 different types of invariants in REST APIs. AGORA achieved a total precision of 81.2\% when tested on a dataset of 11 operations from 7 industrial APIs. More importantly, the test oracles generated by AGORA detected 6 out of every 10 errors systematically seeded in the outputs of the APIs under test. Additionally, AGORA revealed 11 bugs in APIs with millions of users: Amadeus, GitHub, Marvel, OMDb and YouTube. Our reports have guided developers in improving their APIs, including bug fixes and documentation updates in GitHub. Since it operates in black-box mode, AGORA can be seamlessly integrated into existing API testing tools.},
booktitle = {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis},
pages = {1018–1030},
numpages = {13},
keywords = {automated testing, invariant detection, REST APIs, test oracle},
location = {Seattle, WA, USA},
series = {ISSTA 2023}
}

Motivational example

The image below depicts an excerpt of a simplified version of the getAlbumTracks operation of the Spotify API. This document describes the API functionality in terms of the input parameters (lines 5-20) and the possible response formats (lines 21-61). This operation receives as input an album id (id parameter), a country code (market) and the maximum number of songs to fetch (limit) and returns the list of tracks of the album.

The JSON below shows a response for the getAlbumTracks operation conforming to the provided specification. Existing approaches for automated testing of REST APIs would label this response as valid because the API is not returning a server error and the response is conformant to the API specification.

{
    "total": 14,
    "href": "https://api.spotify.com/albums/4Em5W5HgYEvhpc/tracks?limit=1&market=ES",
    "items": [
        {
            "artists": [
                {
                    "id": "2CvCyf1gEVhI0mX6aFXmVI",
                    "name": "Paul Simon"
                },
                {
                    "id": "70cRZdQywnSFp9pnc2WTCE",
                    "name": "Arthur Garfunkel"
                }
            ],
            "available_markets": [ "ES", "US", "JP" ],
            "id": "0gFvkiT2afIcJwNxXQ7W51",
            "name": "Mrs. Robinson",
            "explicit": false,
            "linked_from": {
                "id": "98cZPdKywnMGp8fnw2XTYU",