https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluating the Robustness of ML Models to Out-of-Distribution Data Through Similarity Analysis
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. Saab AB, Linköping, Sweden.
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.
Royal Institute of Technology, Stockholm, Sweden; Saab AB, Linköping, Sweden.
2023 (English)In: Commun. Comput. Info. Sci., Springer Science and Business Media Deutschland GmbH , 2023, p. 348-359Conference paper, Published paper (Refereed)
Abstract [en]

In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. We present a method for analyzing datasets from a use-case scenario perspective, detecting and quantifying out-of-distribution (OOD) data on dataset level. Our main contribution is the novel use of similarity metrics for the evaluation of the robustness of a model by introducing relative Fréchet Inception Distance (FID) and relative Kernel Inception Distance (KID) measures. These relative measures are relative to a baseline in-distribution dataset and are used to estimate how the model will perform on OOD data (i.e. estimate the model accuracy drop). We find a correlation between our proposed relative FID/relative KID measure and the drop in Average Precision (AP) accuracy on unseen data.

Place, publisher, year, edition, pages
Springer Science and Business Media Deutschland GmbH , 2023. p. 348-359
Keywords [en]
accuracy estimation, datasets, neural networks, similarity metrics, Learning systems, Dataset, Distance measure, Frechet, Machine learning systems, Modeling architecture, Neural-networks, Performance, Similarity analysis, Drops
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:mdh:diva-64446DOI: 10.1007/978-3-031-42941-5_30Scopus ID: 2-s2.0-85171979824ISBN: 9783031429408 (print)OAI: oai:DiVA.org:mdh-64446DiVA, id: diva2:1802686
Conference
Communications in Computer and Information Science
Available from: 2023-10-05 Created: 2023-10-05 Last updated: 2023-10-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Lindén, JoakimForsberg, HåkanDaneshtalab, Masoud

Search in DiVA

By author/editor
Lindén, JoakimForsberg, HåkanDaneshtalab, Masoud
By organisation
Embedded Systems
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 7 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf