Imagine you have a software-as-a-service product that includes multiple services that rely on each other. Each of the services is tested. Let's say for the sake of argument they have code coverage of between 85% and 95%, you may have some confidence that the services work.
What is your confidence that a specific end-to-end process that implicates several services works and has a low failure rate?
| Approach | Downsides |
|---|---|
| Browser-based end-to-end tests | 1. Slow 2. Cannot be run at a high rate (due to slowness) and thus cannot be used to ascertain failure rates 3. Easy to write bugs |
| API-based end-to-end tests using common general-purpose frameworks | 1. Easy to write bugs 2. Common test frameworks are not designed to re-run tests and calculate failure rates 3. Significant start-up time |
Agnostic API Reliability Testing Language (AARTL) is a platform-agnostic declarative domain-specific language for testing HTTP servers using the server’s API, it is implemented in TypeScript as a dependency-free high-performance Node.js application and can run on all major operating systems (Windows, macOS, Linux-based OSs and FreeBSD), it can also run on GraalVM, and can test servers irrespective of the platform used by the server. An AARTL test is a human-readable declaration of the expected response from a server endpoint given one or more requests. Its simple syntax offers fewer opportunities for writing bugs than a traditional test which may include any arbitrary code.
- To facilitate efficient testing of HTTP servers
- To offer data matching rules on par with procedural test frameworks
- To avoid procedural code in tests (as much as is practicable)
- To avoid unobvious syntax such as semi colons, brackets, braces, and significant whitespace
- To be easy to read for a person well-versed in HTTP and JSON irrespective of whether the person is a programmer
Test that it should save a post
Using values
@postText: Hello world
After HTTP request
method: post
url: http://localhost:3000/posts
body: @postText
Pass on "$..id" as _id
Expect HTTP request
headers:
"Accept-Encoding":"*/*"
method: get
url: http://localhost:3000/posts/_id
To respond with status code 200 /** OK **/
To match JSON rules
"$..id": _id
"$..text": @postText
A test consists of a name, optional constant values, optional “After” blocks, and a required "Expect" block. JSON data is referred to within AARTL using JSON paths. For more information about the JSON path standard you may refer to: https://support.smartbear.com/alertsite/docs/monitors/api/endpoint/jsonpath.html
An After block consists of:
- The data needed to make a request
- Optional statements about how to handle transitioning to the next request.
Data needed for an HTTP request is:
- headers Example: 'Accept-Encoding': '*/*'
- method Example: post
- url Example: http://example.org/things/123
- body Example: Hello World
- body can be loaded from a fixture file like so
body from fixture: myfixture - this will load the content of fixtures/myfixture.fixture and use that as the body of the request
- body can be loaded from a fixture file like so
Optional statements about how to handle transitioning to the next request are:
- Pass on – the Pass on statement can be used to pass on a value from a response to the next block. Example: Pass on “$..id” as _id, this means that we can refer to the id returned by this request in the next request using the _id symbol. The $..id is a JSON path.
- Wait – the Wait statement indicates that there will be a waiting period before the next request starts
An Expect block consists of:
- Data needed to make a request (same as an After block)
- Expectations for what the response should be, there are three types of expectations: - JSON data expectations - Header expectations - Status code expectation
JSON data expectations consist of:
- A JSON path and either a literal value or a matcher.
Examples of rules are (x, y, z in the rules refer to parameters):
| Matcher | What it means |
|---|---|
| is a number | checks if every value that matches the JSON path is a number |
| > x | checks if every value that matches the JSON path is a number greater than x |
| >= x | checks if every value that matches the JSON path is a number greater than or equal to x |
| < x | checks if every value that matches the JSON path is a number less than x |
| <= x | checks if every value that matches the JSON path is a number less than or equal to x |
| is text | checks if every value that matches the JSON path is text of length 1 or longer |
| is text containing x | checks if every value that matches the JSON path is text that contains x |
| is text not containing x | checks if every value that matches the JSON path is text that does not contain x |
| is any of x y z | checks if every value that matches the JSON path is one of the values |
| is not x | checks if every value that matches the JSON path is not x |
For an up to date list of rules consult the wiki page on rules
Header expectations consist of:
Header name followed by a colon and the expected value or a rule. There is one header rule, it is "must not be present". Examples:
"X-Powered-By":"Express"
"X-Powered-By": must not be present /* will fail if this header is sent */
Comments can be written within /** **/ and are ignored. Example:
/** This is a comment **/
The syntax is case sensitive, statements start with an upper-case letter, data items start with a lowercase letter.
- Few opportunities for writing bugs
- Runs fast
- Detailed logging
- Cross-platform
- Flexible matching rules
- 100% test coverage of all modules
- Human-readable reports
-
Unzip it
-
Open a terminal window
-
Run the example file
aartl.exe -f example.aartl --reporton Windows./aartl -f example.aartl --reporton Linux and macOS -
Observe the output and open the report file
-
Optionally install the Visual Studio Code extension
To see the CLI options run the program with no arguments, the key options are:
| Option | What it means |
|---|---|
| --hello | Print the name of the program before running tests |
| --xml | Output results as JUnit XML |
| --novalidation | Don't validate test file |
| --r | Randomize test order |
| -n N | Rerun the tests a number of times |
| -m N | Maximum concurrent tests. Default: no limit |
| --report | Output an html report with failure rates |
| --q | Don't output real-time test results |
| --log | Output request logs |
| --ff | Exit with error code 1 as as soon as one test fails |
The "runs fast" claim refers to the following:
- Checking if a response passes the rules is as fast as 0.1 milliseconds
- HTTP requests are handled by a very lightweight wrapper over the native HTTP APIs
- Tests are executed asynchronously so while one test is "waiting" for a response, other tests are being executed
- Can send more concurrent requests than an Express-based server can handle and thus has to be throttled
The bottleneck is likely to be how many requests the server being tested can handle not how fast the tests are executed.
MIT