Radical Reads

At the Elevate Festival in Toronto, Aidan Gomez, Co-Founder and CEO of Cohere, was joined by journalist and author Stephen Marche to discuss large language models and what lies ahead.

Cohere enables businesses to use AI to understand natural language. Before co-founding the company, Aidan co-authored the 2017 paper “Attention Is All You Need” which introduced the architecture of “Transformers”, the foundational technology being applied to language and image generation.

Stephen Marche is a journalist and scholar of early modern English drama. Previously, he used Cohere’s NLP technology to authenticate versions of Shakespeare’s works in a collaboration in The New York Times.

Aidan and Stephen sat down for a discussion on language technology and what the future holds. Below is an excerpt of their conversation.

Stephen: My “holy shit” moment using language models was when I was writing a piece for The New Yorker. I asked GPT-3 to finish famous unfinished poems, and they worked. They truly sounded like Coleridge or Shakespeare. When was your “holy shit” moment?

Aidan: It happened shortly after the Transformer paper came out. I was in Toronto, and I got an email from my colleague back at Google. He sent me what was seemingly a Wikipedia page on the Transformer. I started reading and it went into a story about a Japanese punk band, how the members had broken up. At the end of the email he said, “I just put in ‘the Transformer,’ everything else was written by the model.” I was floored. Up until that point, our models could barely do anything. They couldn’t spell correctly. He trained a language model on Wikipedia and it crafted a super compelling story about Transformer, the Japanese punk band.

Stephen: Obviously, my PhD is in Shakespeare so I’m not supposed to understand it, but you are. Can you tell me what you do and don’t understand about the process?  

Aidan: I’m so close to the nuts and bolts of it that I often just see a bunch of matrix multiplications and floating point numbers. But when I step back and I look at the outputs — at a system we built where you can say “hey solve this problem for me” and it solves the problem — that is so extraordinary. It’s still magical for me. There’s still so much to be understood.

I understand how you source the data. I understand how the model is trained on that data. I understand how to scale up. When you put those three things together and actually get the output — you’re sitting in front of a trained model — I still don’t fully understand why the outputs are the outputs. Why does a model pick one option over another option? Getting into the way it makes those selections, that’s still an area of active research.

Stephen: Why aren’t we seeing more of this technology out in the world?

Aidan: I know what AI is capable of, but it’s not out there. One of the largest barriers that I’m sure a lot of people are aware of is that the people who know how to do this stuff — MLEs or machine learning engineers — we can’t train enough of them. There’s a supply-demand dynamic where there’s not enough talent on the face of the planet and there won’t be.

Stephen: Why?

Aidan: It’s going to take us so long to meet that demand — decades of education and new students. The way to bring AI into the products of today is not to train a bunch of people with highly specialized knowledge, instead it’s to present the technology in a different way.

At Cohere, we’re creating an interface to Transformers and supercomputers that’s accessible to anyone, to any developer. Using this technology should be intuitive and natural. That’s the mission; that’s our product vision.

Stephen: When you imagine where language AI is going to be in 5-10 years, what do you see?

Aidan: In gaming today, for instance, if you’re interacting with a character, there’s a dialogue tree that someone’s written, and there’s maybe ten paths through that dialogue tree. Every single player has the same experience. I imagine a world where games have a dialogue tree of 8 or 10 billion paths, and everyone experiences a different conversation with that character. Every single play-through is unique.

When you introduce this concept to the rest of the world, you have more mundane examples, like in customer support. As soon as I get a customer support chatbot, the first thing I write is “I want to talk to a human.” If we actually had compelling models of language — if we could actually create a chatbot that a human wants to talk to — it would change the interface of tech. Dialogue would be the interface. You could talk to your technology.

Right now, we must learn the computer’s language. I went to school for five years to learn how to talk to a computer, to tell it what to do, and to code it. I learned to speak its language. We’re not yet in a place where our products speak our language. That will change.

Cohere is a startup founded by ex-Googlers that enables leading-edge machine understanding of natural language for every business. Backed by Radical Ventures and AI luminaries such as Geoffrey Hinton and Fei-Fei Li, Cohere provides access to world-leading NLP without supercomputing infrastructure or a big tech budget. The use cases for Cohere’s language models continue to expand: read more about our investment in Cohere.