CoLT5: Faster Long-Range Transformers with Conditional Computation EMNLP

Joshua Ainslie | Tao Lei | Michiel de Jong | Santiago Ontanon | Siddhartha Brahma | Yury Zemlyanskiy | David Uthus | Mandy Guo | James Lee-Thorp | Yi Tay | Yun-Hsuan Sung | Sumit Sanghai |

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences F i n d i n g s - E M N L P

David Uthus | Santiago Ontanon | Joshua Ainslie | Mandy Guo |

Making Transformers Solve Compositional Tasks ACL

Santiago Ontanon | Joshua Ainslie | Zachary Fisher | Vaclav Cvicek |

FNet: Mixing Tokens with Fourier Transforms NAACL

James Lee-Thorp | Joshua Ainslie | Ilya Eckstein | Santiago Ontanon |

LongT5: Efficient Text-To-Text Transformer for Long Sequences Findings NAACL

Mandy Guo | Joshua Ainslie | David Uthus | Santiago Ontanon | Jianmo Ni | Yun-Hsuan Sung | Yinfei Yang |

Improving Compositional Generalization in Classification Tasks via Structure Annotations ACL IJCNLP

Juyong Kim | Pradeep Ravikumar | Joshua Ainslie | Santiago Ontanon |

ETC: Encoding Long and Structured Inputs in Transformers EMNLP

Joshua Ainslie | Santiago Ontanon | Chris Alberti | Vaclav Cvicek | Zachary Fisher | Philip Pham | Anirudh Ravula | Sumit Sanghai | Qifan Wang | Li Yang |