Awesome Adapter Resources
This repository collects important tools and papers related to adapter methods for recent large pre-trained neural networks.
Adapters (aka Parameter-Efficient Transfer Learning (PETL) or Parameter-Efficient Fine-Tuning (PEFT) methods) include various parameter-efficient approaches of adapting large pre-trained models to new tasks.
Content
- Why Adapters?
- Frameworks and Tools
- Surveys
- Natural Language Processing
- Computer Vision
- Audio Processing
- Multi-Modal
- Contributing
Why Adapters?
Large pre-trained (Transformer-based) models have become the foundation of various ML domains in recent years. While the most prevalent method of adapting these models to new tasks involves costly full fine-tuning of all model parameters, a series of parameter-efficient and lightweight alternatives, adapters, have been established in recent time.
Using adapters provides multiple benefits. They are …
- … parameter-efficient, i.e. they only update a very small subset (e.g. under 1%) of a model’s parameters.
- … modular, i.e. the updated parameters can be extracted and shared independently of the base model parameters
- … easy to share and easy to deploy at scale due to their small file sizes. E.g. requiring only ~3MB per task instead of ~500MB for sharing a full model.
- … often composable, i.e. can be stacked, fused or mixed to leverage their combined knowledge.
- … often on-par in terms of performance with full fine-tuning.
Frameworks and Tools
-
AdapterHub: A Framework for Adapting Transformers
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Andreas Rücklé, Clifton A. Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych (2020)
TLDR
AdaptersHub is proposed, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages that enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. -
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning
Conference on Empirical Methods in Natural Language Processing
Clifton A. Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Arne Engländer, Timo Imhof, Ivan Vuli’c, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer (2023)
TLDR
Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models and allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups, is introduced. -
OpenDelta
-
PEFT: State-of-the-art Parameter-Efficient Fine-Tuning
-
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
Conference on Empirical Methods in Natural Language Processing
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, R. Lee, Lidong Bing, Soujanya Poria (2023)
TLDR
LLM-Adapters is presented, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks, demonstrating that using adapter- based PEFT in smaller-scale LLMs with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs in zero-shot inference on both reasoning tasks. -
Alpaca-LoRA
Surveys
-
Modular Deep Learning
arXiv.org
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, E. Ponti (2023)
TLDR
A survey of modular architectures is offered, providing a unified view over several threads of research that evolved independently in the scientific literature, and various additional purposes of modularity are explored, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. -
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
arXiv.org
Vladislav Lialin, Vijeta Deshpande, Anna Rumshisky (2023)
TLDR
A taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency and fine-tuning multibillion-scale language models is provided. -
PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques
arXiv.org
Mohammed Sabry, Anya Belz (2023)
TLDR
A reference architecture is presented which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. -
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
arXiv.org
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, Sai Qian Zhang (2024)
TLDR
This survey presents comprehensive studies of various PEFT algorithms, examining their performance and computational overhead, and overview of applications developed using different PEFT algorithms and discusses common techniques employed to mitigate computation costs for PEFT.
Natural Language Processing
Methods
-
Parameter-Efficient Transfer Learning for NLP
International Conference on Machine Learning
N. Houlsby, A. Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, S. Gelly (2019)
TLDR
To demonstrate adapter's effectiveness, the recently proposed BERT Transformer model is transferred to 26 diverse text classification tasks, including the GLUE benchmark, and adapter attain near state-of-the-art performance, whilst adding only a few parameters per task. -
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Findings
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou (2020)
TLDR
K-Adapter is proposed, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion and captures richer factual and commonsense knowledge than RoBERTa. -
Parameter-Efficient Transfer Learning with Diff Pruning
Annual Meeting of the Association for Computational Linguistics
Demi Guo, Alexander M. Rush, Yoon Kim (2020)
TLDR
Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches. -
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Annual Meeting of the Association for Computational Linguistics
Xiang Lisa Li, Percy Liang (2021)
TLDR
Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix. -
The Power of Scale for Parameter-Efficient Prompt Tuning
Conference on Empirical Methods in Natural Language Processing
Brian Lester, Rami Al-Rfou, Noah Constant (2021)
TLDR
This work explores “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “Prompt ensembling.” -
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Neural Information Processing Systems
Joe Davison (2021)
TLDR
Compacter is proposed, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work, and accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. -
LoRA: Low-Rank Adaptation of Large Language Models
International Conference on Learning Representations
J. E. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu Chen (2021)
TLDR
Low-Rank Adaptation, or LoRA, is proposed, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. -
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Annual Meeting of the Association for Computational Linguistics
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, J. Henderson (2021)
TLDR
This paper shows that one can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. -
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer
Conference on Empirical Methods in Natural Language Processing
Alan Ansell, E. Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, A. Korhonen (2021)
TLDR
MAD-G is proposed, which contextually generates language adapters from language representations based on typological features and remains competitive with more expensive methods for language-specific adapter training across the board. -
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Annual Meeting of the Association for Computational Linguistics
Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg (2021)
TLDR
BitFit is introduced, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified, which shows that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. -
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Neural Information Processing Systems
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel (2022)
TLDR
This paper rigorously compares few-shot ICL and PEFT and demonstrates that the latter offers better accuracy as well as dramatically lower computational costs, and introduces a new PEFT method called (IA)$^3$ that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. -
AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning
arXiv.org
Han Zhou, Xingchen Wan, Ivan Vulic, A. Korhonen (2023)
TLDR
Inspired by advances in neural architecture search, AutoPEFT is proposed for automatic PEFT configuration selection and it is shown that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs. -
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao (2023)
TLDR
The proposed AdaLoRA adaptively allocates the parameter budget among weight matrices according to their importance score, which allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. -
QLoRA: Efficient Finetuning of Quantized LLMs
Neural Information Processing Systems
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023)
TLDR
QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA, and current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. -
Composable Sparse Fine-Tuning for Cross-Lingual Transfer
Annual Meeting of the Association for Computational Linguistics
Alan Ansell, E. Ponti, A. Korhonen, Ivan Vulic (2021)
TLDR
This work introduces a new fine-tuning method that outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. -
VeRA: Vector-based Random Matrix Adaptation
arXiv.org
Dawid Jan Kopiczko, Tijmen Blankevoort, Yuki Markus Asano (2023)
TLDR
Vector-based Random Matrix Adaptation (VeRA) is presented, which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. -
DoRA: Weight-Decomposed Low-Rank Adaptation
arXiv.org
Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen (2024)
TLDR
Weight-Decomposed LowRank Adaptation (DoRA) is proposed, which decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. -
ReFT: Representation Finetuning for Language Models
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Daniel Jurafsky, Christopher D. Manning, Christopher Potts (2024)
TLDR
A strong instance of the ReFT family is defined, Low-rank Linear Subspace ReFT (LoReFT), which is a drop-in replacement for existing PEFTs and learns interventions that are 10x-50x more parameter-efficient than prior state-of-the-art PEFTs.
^3-blue){kind=icon}
Composition Methods
-
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)
TLDR
MAD-X is proposed, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations and introduces a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language. -
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Conference of the European Chapter of the Association for Computational Linguistics
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych (2020)
TLDR
This work proposes AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks by separating the two stages, i.e., knowledge extraction and knowledge composition, so that the classifier can effectively exploit the representations learned frommultiple tasks in a non-destructive manner. -
Towards a Unified View of Parameter-Efficient Transfer Learning
International Conference on Learning Representations
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig (2021)
TLDR
This paper re-frame state-of-the-art parameter-efficient transfer learning methods as modifications to specific hidden states in pre-trained models, and defines a set of design dimensions along which different methods vary, achieving comparable results to fine-tuning all parameters on all four tasks. -
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Conference on Empirical Methods in Natural Language Processing
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Jianfeng Gao (2022)
TLDR
AdaMix is proposed as a general PEFT method that tunes a mixture of adaptation modules – given the underlyingPEFT method of choice – introduced in each Transformer layer while keeping most of the PLM weights frozen, and outperforms SOTA parameter-efficient fine-tuning and full model fine- Tuning for both NLU and NLG tasks. -
Composing Parameter-Efficient Modules with Arithmetic Operations
Neural Information Processing Systems
Jinghan Zhang, Shiqi Chen, Junteng Liu, Junxian He (2023)
TLDR
This paper proposes to compose parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities and extends this approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. -
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
Findings
Alexandra Chronopoulou, Matthew E. Peters, Alexander M. Fraser, Jesse Dodge (2023)
TLDR
This paper introduces AdapterSoup, an approach that performs weight-space averaging of adapters trained on different domains, and explores various approaches for choosing which adapters to combine, such as text clustering and semantic similarity. -
Combining Modular Skills in Multitask Learning
arXiv.org
E. Ponti, Alessandro Sordoni, Siva Reddy (2022)
TLDR
It is found that the modular design of a network significantly increases sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to baselines with fully shared, task-specific, or conditionally generated parameters where knowledge is entangled across tasks. -
Multi-Head Adapter Routing for Data-Efficient Fine-Tuning
arXiv.org
Lucas Caccia, E. Ponti, Lu Liu, Matheus Pereira, Nicolas Le Roux, Alessandro Sordoni (2022)
TLDR
This paper investigates to what extent the ability to control which adapters are active for each task leads to sample-efficient generalization and proposes less expressive variants where the authors perform weighted averaging of the adapters before few-shot adaptation (Poly - µ ) instead of learning a routing function. -
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
arXiv.org
Ted Zadouri, A. Ustun, Arash Ahmadian, Beyza Ermics, Acyr Locatelli, Sara Hooker (2023)
TLDR
This paper proposes extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts and is on par with full fine-tuning by only updating the lightweight experts -- less than 1% of an 11B parameters model.
Analysis and Evaluation
-
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers
Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out
Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, N. Rozanov, Goran Glavavs (2020)
TLDR
A deeper analysis reveals that the adapter-based models substantially outperform BERT on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and its corresponding Open Mind Common Sense corpus. -
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
Annual Meeting of the Association for Computational Linguistics
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si (2021)
TLDR
It is demonstrated that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates. -
Robust Transfer Learning with Pretrained Language Models through Adapters
Annual Meeting of the Association for Computational Linguistics
Wenjuan Han, Bo Pang, Y. Wu (2021)
TLDR
This work inserts small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, leading to improved stability and adversarial robustness in transfer learning to various downstream tasks. -
AdapterDrop: On the Efficiency of Adapters in Transformers
Conference on Empirical Methods in Natural Language Processing
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych (2020)
TLDR
This paper proposes AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions and can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. -
What to Pre-Train on? Efficient Intermediate Task Selection
Conference on Empirical Methods in Natural Language Processing
Clifton A. Poth, Jonas Pfeiffer, Andreas Ruckl’e, Iryna Gurevych (2021)
TLDR
This work provides a comprehensive comparison of different methods for efficiently identifying beneficial tasks for intermediate transfer learning, focusing on parameter and computationally efficient adapter settings, highlight different data-availability scenarios, and provide expense estimates for each method. -
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
arXiv.org
M. Vidoni, Ivan Vulic, Goran Glavas (2020)
TLDR
This work proposes orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer that are trained to encode language- and task-specific information that is complementary to the knowledge already stored in the pretrained transformer's parameters. -
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Annual Meeting of the Association for Computational Linguistics
Xiao Liu, Kaixuan Ji, Yicheng Fu, W. Tam, Zhengxiao Du, Zhilin Yang, Jie Tang (2022)
TLDR
The method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU and can serve as an alternative to finetuning and a strong baseline for future research. -
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
arXiv.org
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, Maosong Sun (2022)
TLDR
The theoretical principles underlying the effectiveness of delta tuning are discussed and frameworks to interpret delta tuning from the perspective of optimization and optimal control are proposed, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. -
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Annual Meeting of the Association for Computational Linguistics
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa (2021)
TLDR
A unified framework, UniPELT, is proposed, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism, indicating that a mixture of multiple P ELT methods may be inherently more effective than single methods.
Applications
-
Simple, Scalable Adaptation for Neural Machine Translation
Conference on Empirical Methods in Natural Language Processing
Ankur Bapna, N. Arivazhagan, Orhan Firat (2019)
TLDR
The proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model, which adapt the model to multiple individual tasks simultaneously, paving the way towards universal machine translation. -
Monolingual Adapters for Zero-Shot Neural Machine Translation
Conference on Empirical Methods in Natural Language Processing
Jerin Philip, Alexandre Berard, Matthias Gallé, L. Besacier (2020)
TLDR
A novel adapter layer formalism for adapting multilingual models is proposed, which is more parameter-efficient than existing adapter layers while obtaining as good or better performance. -
UDapter: Language Adaptation for Truly Universal Dependency Parsing
Conference on Empirical Methods in Natural Language Processing
A. Ustun, Arianna Bisazza, G. Bouma, Gertjan van Noord (2020)
TLDR
A novel multilingual task adaptation approach based on recent work in parameter-efficient transfer learning, which allows for an easy but effective integration of existing linguistic typology features into the parsing network, and consistently outperforms strong monolingual and multilingual baselines on both high-resource and low-resource languages. -
Single-dataset Experts for Multi-dataset Question Answering
Conference on Empirical Methods in Natural Language Processing
Dan Friedman, Ben Dodge, Danqi Chen (2021)
TLDR
This work trains a collection of lightweight, dataset-specific adapter modules that share an underlying Transformer model, and finds that these Multi-Adapter Dataset Experts (MADE) outperform all the authors' baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance. -
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)
TLDR
This work proposes a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts and demonstrates that they can yield improvements for low- resource languages written in scripts covered by the pretrained model. -
Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
Conference on Machine Translation
Asa Cooper Stickland, Alexandre Berard, Vassilina Nikoulina (2021)
TLDR
This work study the compositionality of language and domain adapters in the context of Machine Translation, and aims to study parameter-efficient adaptation to multiple domains and languages simultaneously and cross-lingual transfer in domains where parallel data is unavailable for certain language pairs. -
Multilingual Unsupervised Neural Machine Translation with Denoising Adapters
Conference on Empirical Methods in Natural Language Processing
A. Ustun, Alexandre Berard, L. Besacier, Matthias Gallé (2021)
TLDR
This paper proposes to use _denoising adapters_, adapter layers with a denoising objective, on top of pre-trained mBART-50, and shows that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally. -
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
Conference on Empirical Methods in Natural Language Processing
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig (2021)
TLDR
This paper finds that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters, and proposes EMA, a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. -
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
arXiv.org
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Y. Qiao (2023)
TLDR
A zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge on traditional vision and language tasks, demonstrating the superior generalization capacity of the approach.
Serving
-
Punica: Multi-Tenant LoRA Serving
arXiv.org
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy University of Washington, Duke University (2023)
TLDR
Punica is a system to serve multiple LoRA models in a shared GPU cluster that contains a new CUDA kernel design that allows batching of GPU operations for differentLoRA models, significantly enhancing GPU efficiency in terms of both memory and computation. -
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
arXiv.org
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica (2023)
TLDR
S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine- Tuning services.
Computer Vision
Methods
-
Learning multiple visual domains with residual adapters
Neural Information Processing Systems
Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2017)
TLDR
This paper develops a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains and introduces the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very differentVisual domains and measures their ability to recognize well uniformly. -
Efficient Parametrization of Multi-domain Deep Neural Networks
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2018)
TLDR
This paper proposes to consider universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters, and shows that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques. -
Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
arXiv.org
Hao Chen, R. Tao, Han Zhang, Yidong Wang, Weirong Ye, Jindong Wang, Guosheng Hu, M. Savvides (2022)
TLDR
Conv-Adapter, a PET module designed for ConvNets, is light-weight, domain-transferable, and architecture-agnostic with generalized performance on different tasks with comparable or surpasses the performance of full fine-tuning on 23 classification tasks of various domains. -
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Neural Information Processing Systems
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo (2022)
TLDR
AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks.
Audio Processing
Applications
-
Lightweight Adapter Tuning for Multilingual Speech Translation
Annual Meeting of the Association for Computational Linguistics
Hang Le, J. Pino, Changhan Wang, Jiatao Gu, D. Schwab, L. Besacier (2021)
TLDR
This paper proposes a comprehensive analysis of adapters for multilingual speech translation (ST) and shows that adapters can be used to efficiently specialize ST to specific language pairs with a low extra cost in terms of parameters. -
Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition
IEEE International Conference on Acoustics, Speech, and Signal Processing
Bethan Thomas, Samuel Kessler, S. Karout (2022)
TLDR
Adapters are applied to wav2vec 2.0 to reduce the number of parameters required for downstream ASR tasks, and increase scalability of the model to multiple tasks or languages, and support the theory that higher pre-trained layers encode more phonemic information.
Multi-Modal
Methods
-
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Computer Vision and Pattern Recognition
Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2021)
TLDR
The results demonstrate that training the adapter with the weight-sharing technique can match the performance of fine-tuning the entire model, and enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. -
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Neural Information Processing Systems
Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2022)
TLDR
LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections, and achieves higher accuracy than Adapter and LoRA in a low-memory regime. -
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
Neural Information Processing Systems
Tao Lei, Junwen Bai, Siddhartha Brahma, J. Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang (2023)
TLDR
This work proposes Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency and achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency. -
VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control
IEEE International Conference on Computer Vision
Zi-Yuan Hu, Yanyang Li, M. Lyu, Liwei Wang (2023)
TLDR
A Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose effective control over modular modifications via a novel granularity-controlled mechanism and a variety of model-agnostic VL-PET modules can be instantiated from this framework for better efficiency and effective-ness trade-offs is proposed.
Contributing
Contributions of new awesome adapter-related resources are very welcome! Before contributing, make sure to read this repository’s contributing guide.
Acknowledgments
Paper metadata is partially retrieved via Semantic Scholar’s API. Paper TLDRs are provided by Semantic Scholar’s TLDR feature.