GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints EMNLP

Joshua Ainslie | James Lee-Thorp | Michiel de Jong | Yury Zemlyanskiy | Federico Lebron | Sumit Sanghai |

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference F i n d i n g s - A C L

Michiel de Jong | Yury Zemlyanskiy | Joshua Ainslie | Nicholas FitzGerald | Sumit Sanghai | Fei Sha | William Cohen |

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding EMNLP

Andrea Burns | Krishna Srinivasan | Joshua Ainslie | Geoff Brown | Bryan Plummer | Kate Saenko | Jianmo Ni | Mandy Guo |

CoLT5: Faster Long-Range Transformers with Conditional Computation EMNLP

Joshua Ainslie | Tao Lei | Michiel de Jong | Santiago Ontanon | Siddhartha Brahma | Yury Zemlyanskiy | David Uthus | Mandy Guo | James Lee-Thorp | Yi Tay | Yun-Hsuan Sung | Sumit Sanghai |

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences F i n d i n g s - E M N L P

David Uthus | Santiago Ontanon | Joshua Ainslie | Mandy Guo |

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction ACL

Chen-Yu Lee | Chun-Liang Li | Hao Zhang | Timothy Dozat | Vincent Perot | Guolong Su | Xiang Zhang | Kihyuk Sohn | Nikolay Glushnev | Renshen Wang | Joshua Ainslie | Shangbang Long | Siyang Qin | Yasuhisa Fujii | Nan Hua | Tomas Pfister |

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction ACL

Chen-Yu Lee | Chun-Liang Li | Timothy Dozat | Vincent Perot | Guolong Su | Nan Hua | Joshua Ainslie | Renshen Wang | Yasuhisa Fujii | Tomas Pfister |

Making Transformers Solve Compositional Tasks ACL

Santiago Ontanon | Joshua Ainslie | Zachary Fisher | Vaclav Cvicek |

FNet: Mixing Tokens with Fourier Transforms NAACL

James Lee-Thorp | Joshua Ainslie | Ilya Eckstein | Santiago Ontanon |

LongT5: Efficient Text-To-Text Transformer for Long Sequences Findings NAACL

Mandy Guo | Joshua Ainslie | David Uthus | Santiago Ontanon | Jianmo Ni | Yun-Hsuan Sung | Yinfei Yang |

Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing COLING

Yury Zemlyanskiy | Michiel de Jong | Joshua Ainslie | Panupong Pasupat | Peter Shaw | Linlu Qiu | Sumit Sanghai | Fei Sha |

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT F i n d i n g s - E M N L P

James Lee-Thorp | Joshua Ainslie |

ReadTwice: Reading Very Large Documents with Memories NAACL

Yury Zemlyanskiy | Joshua Ainslie | Michiel de Jong | Philip Pham | Ilya Eckstein | Fei Sha |

RealFormer: Transformer Likes Residual Attention Findings

Ruining He | Anirudh Ravula | Bhargav Kanagal | Joshua Ainslie |

Improving Compositional Generalization in Classification Tasks via Structure Annotations ACL IJCNLP

Juyong Kim | Pradeep Ravikumar | Joshua Ainslie | Santiago Ontanon |

ETC: Encoding Long and Structured Inputs in Transformers EMNLP

Joshua Ainslie | Santiago Ontanon | Chris Alberti | Vaclav Cvicek | Zachary Fisher | Philip Pham | Anirudh Ravula | Sumit Sanghai | Qifan Wang | Li Yang |