To assess the coverage of our mission specification patterns (a) we collected a set of mission requirements from the literature, and (b)~we assessed whether our patterns enabled expressing these requirements.

Dataset

We considered a benchmark of 21 requirements (see Validation column of Table 1), collected from papers from 2020 and 2021 by following the same methodology presented used to define the QUARTET patterns (see Section 4.1). We essentially followed a train-test split approach, popular in evaluation of machine learning and data science research, by considering collection of six years of requirements for the bottom-up pattern formulation, and subsequently evaluating coverage against requirements collected the last two years.

rq1-coverage.csv: “Year”, “Link”, “Conference”, “Title”, “Mission”, “Patterns”.

Methodology

We considered each of the 21 mission requirements of our dataset and proceeded as follows. Three of the authors analyzed the mission requirement and attempted to use the DSL of Figure 4 to express the mission requirement. If it was possible to write the mission requirement using the constructs provided by the DSL, patterns were deemed sufficiently expressive to capture the mission requirement. If it was not possible to completely express the mission requirement using the constructs provided by our DSL, we identified the portion of the requirement that could not be expressed.