https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Automatic Property-based Testing of GraphQL APIs
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.
ABB AB, Västerås, Sweden.ORCID iD: 0000-0001-8009-9052
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-5032-2310
2021 (English)In: Proceedings - 2021 IEEE/ACM International Conference on Automation of Software Test, AST 2021, 2021Conference paper, Published paper (Refereed)
Abstract [en]

In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs.

In this paper, we propose a method for automatically producing GraphQL queries to test GraphQL APIs. This is achieved using a property-based approach to create a generator for queries based on the GraphQL schema of the system under test.

Our evaluation on a real world software system shows that this approach is both effective, in terms of finding real bugs, and efficient, as a complete schema can be covered in seconds. In addition, we evaluate the fault finding capability of the method when seeding known faults. 73% of the seeded faults were found, with room for improvements with regards to domain-specific behavior, a common oracle challenge in automatic test generation.

Place, publisher, year, edition, pages
2021.
National Category
Software Engineering Computer Sciences
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-54267DOI: 10.1109/AST52587.2021.00009ISI: 000695707800001Scopus ID: 2-s2.0-85113790213OAI: oai:DiVA.org:mdh-54267DiVA, id: diva2:1557610
Conference
2nd IEEE/ACM International Conference on Automation of Software Test, AST 2021, Virtual, Online, 20 May 2021
Available from: 2021-05-26 Created: 2021-05-26 Last updated: 2024-05-31Bibliographically approved
In thesis
1. Towards Augmented Exploratory Testing
Open this publication in new window or tab >>Towards Augmented Exploratory Testing
2021 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software systems have an increasing presence in our society. With our infrastructure, such as food and water supply, controlled by complex software systems, it is essential to keep failures to a minimum. Exploratory testing has shown to be a good method of finding bugs that require more complex interactions, as well as to gain insights into how the system under test behaves. However, tool support in this area is lacking. Supporting exploratory testing with automation tools has the potential of freeing humans to spend their time where it is of the highest yield and also aid in covering the vast state space of testing a complex software system in a meaningful way. To be able to engineer such tools, targeting interaction models of contemporary systems, deeper knowledge is needed of the possibilities and limitations of applying available software testing methods. 

In this thesis we propose and evaluate several approaches that can be used to automatically perform black-box system-level testing, potentially augmenting humans in exploratory testing. We base these approaches on random-based methods, such as property-based testing, and present the results of evaluations on several industry-grade systems.

Our results show that the proposed and evaluated methods automatically can find faults in real-world software systems. In addition to fault finding, our methods can also find insights such as deviations between the specification of the system under test and the actual behavior.

The work in this thesis is a first step towards augmenting developers and testers in exploratory testing.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2021. p. 134
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 307
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-54269 (URN)978-91-7485-508-1 (ISBN)
Presentation
2021-06-22, Delta, Mälardalens högskola, Västerås, 14:30 (English)
Opponent
Supervisors
Available from: 2021-05-27 Created: 2021-05-26 Last updated: 2022-11-08Bibliographically approved
2. Exploring API Behaviours by Example Generation
Open this publication in new window or tab >>Exploring API Behaviours by Example Generation
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Understanding the behaviour of complex software-intensive systems is a hard task. For developers of such systems, understanding the actual behaviours is critical in order to successfully create, extend, and maintain them.

The goal of the work in this thesis is to support explorations of the behaviour of software systems through their APIs. We fulfil this goal by generating examples of behaviours the system exhibits. An example is expressed as a sequence of API operations---with parameters, if required---that conforms to a specific behaviour.

Examples of behaviours, such as sequences of operations performed on the system, have been shown to be a good way to further the understanding of software systems for both end users and developers. However, manually creating examples requires effort. In addition, manually created examples only contain what a human can imagine---which might miss important cases, such as unintended behaviours.

The main proposed approach in this thesis is to support users in exploring the behaviour of their software system by automatically generating examples of actual behaviour. By only interacting with the system by the exposed API, we assess the behaviours as exposed to an end user of the API. The input to the approach is a set of API operations and schema of operation parameters. Sequences of operations are generated containing these provided operations. The observed responses from executing the generated sequences are used to assess if the API show an example of a sought behaviour. Found examples go through a shrinking process---trying to find a more minimal sequence showing the same behaviour---and are then reported to the user of the approach. 

The approach is capable of both generating examples of faults in the system and of generating examples of general behaviours. We show evidence of this through multiple evaluations. We have evaluated the fault-finding capabilities by generating examples producing fault-indicating error codes and showing how the configuration of generators affects the interaction with the system. In addition, we evaluate the capability of the approach to generate relevant examples, both in the general API case and in the specific case of REST APIs. By conducting multiple focus group sessions, we conclude that the examples of behaviours produced by the approach indeed aid industry practitioners. The generated examples are deemed relevant for use cases such as testing, documenting, and understanding the behaviour of the system.

Place, publisher, year, edition, pages
Mälardalens universitet, 2024
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 412
National Category
Software Engineering
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-67064 (URN)978-91-7485-654-5 (ISBN)
Public defence
2024-09-10, Zeta, Mälardalens universitet, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2024-05-31 Created: 2024-05-31 Last updated: 2024-08-20Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopushttps://arxiv.org/pdf/2012.07380.pdf

Authority records

Karlsson, StefanCausevic, AdnanSundmark, Daniel

Search in DiVA

By author/editor
Karlsson, StefanCausevic, AdnanSundmark, Daniel
By organisation
Embedded Systems
Software EngineeringComputer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 128 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf