https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Efficient On-device Transfer Learning using Activation Memory Reduction
Mälardalen University, School of Innovation, Design and Engineering. School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran.
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. School of Computer Systems, Tallinn University of Technology, Tallinn, Estonia.
School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran.
2023 (English)In: Int. Conf. Fog Mob. Edge Comput., FMEC, Institute of Electrical and Electronics Engineers Inc. , 2023, p. 210-215Conference paper, Published paper (Refereed)
Abstract [en]

On-device transfer learning suggests fine-tuning pretrained neural networks on new input data directly on edge devices. The memory limitation of edge devices necessitates using memory-efficient fine-tuning methods. Fine-tuning involves two primary phases: the forward-pass phase and the backwardpass phase. The forward-pass phase generates output activations, and the backward-pass phase computes gradients and updates the parameters accordingly. Although the forward-pass phase demands a temporary memory to store a layer’s input and output activations, the backward-pass phase may require storing the output activations from all layers to compute gradients. This fact introduces the memory cost of the backward-pass phase as the main contributor to the huge training memory demands of deep neural networks (DNNs), which has been the focus of many studies. However, little attention has been made to how the temporary activation memory involved in the forward-pass phase may also act as the memory bottleneck, which is the main focus of this paper. This paper aims to mitigate this memory bottleneck by pruning unimportant channels from layers that require significant temporary activation memory. Experimental results demonstrate how the proposed method effectively reduces peak activation memory and total memory costs of MobileNetV2 by 65% and 59%, respectively, at the cost of 3% accuracy drop.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc. , 2023. p. 210-215
Keywords [en]
activation memory, deep neural networks (DNNs), memory efficiency, On-device transfer learning, Chemical activation, Transfer learning, Tuning, Deep neural network, Device transfer, Fine tuning, Memory bottleneck, Memory cost, Temporary activation, Deep neural networks
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:mdh:diva-65147DOI: 10.1109/FMEC59375.2023.10306182ISI: 001103180200028Scopus ID: 2-s2.0-85179519155ISBN: 9798350316971 (print)OAI: oai:DiVA.org:mdh-65147DiVA, id: diva2:1821891
Conference
2023 8th International Conference on Fog and Mobile Edge Computing, FMEC 2023
Available from: 2023-12-21 Created: 2023-12-21 Last updated: 2024-01-18Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Amin, YoosefiMousavi, HamidDaneshtalab, Masoud

Search in DiVA

By author/editor
Amin, YoosefiMousavi, HamidDaneshtalab, Masoud
By organisation
School of Innovation, Design and EngineeringEmbedded Systems
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 23 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf