Measuring and Narrowing the Compositionality Gap in Language Models F i n d i n g s - E M N L P

Ofir Press | Muru Zhang | Sewon Min | Ludwig Schmidt | Noah Smith | Mike Lewis |

What Language Model to Train if You Have One Million GPU Hours? F i n d i n g s - E M N L P

Teven Le Scao | Thomas Wang | Daniel Hesslow | Stas Bekman | M Saiful Bari | Stella Biderman | Hady Elsahar | Niklas Muennighoff | Jason Phang | Ofir Press | Colin Raffel | Victor Sanh | Sheng Shen | Lintang Sutawika | Jaesung Tae | Zheng Xin Yong | Julien Launay | Iz Beltagy |

Transformer Language Models without Positional Encodings Still Learn Positional Information F i n d i n g s - E M N L P

Adi Haviv | Ori Ram | Ofir Press | Peter Izsak | Omer Levy |

Shortformer: Better Language Modeling using Shorter Inputs ACL IJCNLP

Ofir Press | Noah A. Smith | Mike Lewis |

Improving Transformer Models by Reordering their Sublayers ACL

Ofir Press | Noah A. Smith | Omer Levy |

Using the Output Embedding to Improve Language Models EACL

Ofir Press | Lior Wolf |