Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? F i n d i n g s - E M N L P

Yi Tay | Mostafa Dehghani | Samira Abnar | Hyung Chung | William Fedus | Jinfeng Rao | Sharan Narang | Vinh Tran | Dani Yogatama | Donald Metzler |

DSI++: Updating Transformer Memory with New Documents EMNLP

Sanket Vaibhav Mehta | Jai Gupta | Yi Tay | Mostafa Dehghani | Vinh Q. Tran | Jinfeng Rao | Marc Najork | Emma Strubell | Donald Metzler |

Transcending Scaling Laws with 0.1% Extra Compute EMNLP

Yi Tay | Jason Wei | Hyung Chung | Vinh Tran | David So | Siamak Shakeri | Xavier Garcia | Steven Zheng | Jinfeng Rao | Aakanksha Chowdhery | Denny Zhou | Donald Metzler | Slav Petrov | Neil Houlsby | Quoc Le | Mostafa Dehghani |

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks ACL IJCNLP

Rabeeh Karimi Mahabadi | Sebastian Ruder | Mostafa Dehghani | James Henderson |

Are Pretrained Convolutions Better than Pretrained Transformers? ACL IJCNLP

Yi Tay | Mostafa Dehghani | Jai Prakash Gupta | Vamsi Aribandi | Dara Bahri | Zhen Qin | Donald Metzler |