GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints EMNLP

Joshua Ainslie | James Lee-Thorp | Michiel de Jong | Yury Zemlyanskiy | Federico Lebron | Sumit Sanghai |

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference F i n d i n g s - A C L

Michiel de Jong | Yury Zemlyanskiy | Joshua Ainslie | Nicholas FitzGerald | Sumit Sanghai | Fei Sha | William Cohen |

CoLT5: Faster Long-Range Transformers with Conditional Computation EMNLP

Joshua Ainslie | Tao Lei | Michiel de Jong | Santiago Ontanon | Siddhartha Brahma | Yury Zemlyanskiy | David Uthus | Mandy Guo | James Lee-Thorp | Yi Tay | Yun-Hsuan Sung | Sumit Sanghai |

Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing COLING

Yury Zemlyanskiy | Michiel de Jong | Joshua Ainslie | Panupong Pasupat | Peter Shaw | Linlu Qiu | Sumit Sanghai | Fei Sha |

ETC: Encoding Long and Structured Inputs in Transformers EMNLP

Joshua Ainslie | Santiago Ontanon | Chris Alberti | Vaclav Cvicek | Zachary Fisher | Philip Pham | Anirudh Ravula | Sumit Sanghai | Qifan Wang | Li Yang |