By Editorial Team
Image: Aidan Gomez and Stephen March at Elevate Festival 2022, Cohere
At the Elevate Festival in Toronto, Aidan Gomez, Co-Founder and CEO of Cohere, was joined by journalist and author Stephen Marche to discuss large language models and what lies ahead.
Cohere enables businesses to use AI to understand natural language. Before co-founding the company, Aidan co-authored the 2017 paper “Attention Is All You Need” which introduced the architecture of “Transformers”, the foundational technology being applied to language and image generation.
Stephen Marche is a journalist and scholar of early modern English drama. Previously, he used Cohere’s NLP technology to authenticate versions of Shakespeare’s works in a collaboration in The New York Times.
Aidan and Stephen sat down for a discussion on language technology and what the future holds. Below is an excerpt of their conversation.
Stephen: My “holy shit” moment using language models was when I was writing a piece for The New Yorker. I asked GPT-3 to finish famous unfinished poems, and they worked. They truly sounded like Coleridge or Shakespeare. When was your “holy shit” moment?
Aidan: It happened shortly after the Transformer paper came out. I was in Toronto, and I got an email from my colleague back at Google. He sent me what was seemingly a Wikipedia page on the Transformer. I started reading and it went into a story about a Japanese punk band, how the members had broken up. At the end of the email he said, “I just put in ‘the Transformer,’ everything else was written by the model.” I was floored. Up until that point, our models could barely do anything. They couldn’t spell correctly. He trained a language model on Wikipedia and it crafted a super compelling story about Transformer, the Japanese punk band.
Stephen: Obviously, my PhD is in Shakespeare so I’m not supposed to understand it, but you are. Can you tell me what you do and don’t understand about the process?
Aidan: I’m so close to the nuts and bolts of it that I often just see a bunch of matrix multiplications and floating point numbers. But when I step back and I look at the outputs — at a system we built where you can say “hey solve this problem for me” and it solves the problem — that is so extraordinary. It’s still magical for me. There’s still so much to be understood.
I understand how you source the data. I understand how the model is trained on that data. I understand how to scale up. When you put those three things together and actually get the output — you’re sitting in front of a trained model — I still don’t fully understand why the outputs are the outputs. Why does a model pick one option over another option? Getting into the way it makes those selections, that’s still an area of active research.
Stephen: Why aren’t we seeing more of this technology out in the world?
Aidan: I know what AI is capable of, but it’s not out there. One of the largest barriers that I’m sure a lot of people are aware of is that the people who know how to do this stuff — MLEs or machine learning engineers — we can’t train enough of them. There’s a supply-demand dynamic where there’s not enough talent on the face of the planet and there won’t be.
Aidan: It’s going to take us so long to meet that demand — decades of education and new students. The way to bring AI into the products of today is not to train a bunch of people with highly specialized knowledge, instead it’s to present the technology in a different way.
At Cohere, we’re creating an interface to Transformers and supercomputers that’s accessible to anyone, to any developer. Using this technology should be intuitive and natural. That’s the mission; that’s our product vision.
Stephen: When you imagine where language AI is going to be in 5-10 years, what do you see?
Aidan: In gaming today, for instance, if you’re interacting with a character, there’s a dialogue tree that someone’s written, and there’s maybe ten paths through that dialogue tree. Every single player has the same experience. I imagine a world where games have a dialogue tree of 8 or 10 billion paths, and everyone experiences a different conversation with that character. Every single play-through is unique.
When you introduce this concept to the rest of the world, you have more mundane examples, like in customer support. As soon as I get a customer support chatbot, the first thing I write is “I want to talk to a human.” If we actually had compelling models of language — if we could actually create a chatbot that a human wants to talk to — it would change the interface of tech. Dialogue would be the interface. You could talk to your technology.
Right now, we must learn the computer’s language. I went to school for five years to learn how to talk to a computer, to tell it what to do, and to code it. I learned to speak its language. We’re not yet in a place where our products speak our language. That will change.
Cohere is a startup founded by ex-Googlers that enables leading-edge machine understanding of natural language for every business. Backed by Radical Ventures and AI luminaries such as Geoffrey Hinton and Fei-Fei Li, Cohere provides access to world-leading NLP without supercomputing infrastructure or a big tech budget. The use cases for Cohere’s language models continue to expand: read more about our investment in Cohere.
5 Noteworthy AI and Deep Tech Articles: week of October 2, 2022
1) European Medicines Agency Qualifies Unlearn’s AI-powered Method for Running Smaller, Faster Clinical Trials (Bloomberg)
Unlearn, a Radical Ventures portfolio company and the world’s only AI company developing prognostic digital twins to run smaller, faster clinical trials for pharma, announced that the European Medicines Agency (EMA) has released its final qualified opinion on their technology. This is the first time a regulatory body has formally supported a learning-based method for reducing sample size in pivotal trials. The EMA’s validation of Unlearn’s digital twin technology signifies broader regulatory acceptance of AI in clinical development.
2) Meta unveils an AI that generates video based on text prompts (MIT Technology Review)
This week Meta revealed Make-a-Video, bringing the world of generative AI to videos. Make-a-Video uses word prompts, much like Dall-E and other image generators, while it can also make quick videos from photos or other videos that look similar. Researchers used caption-trained photos and unlabeled (unsupervised) video training data to build the AI model, which enabled the system to replicate motion in the real world. According to reports, the videos are distorted and frequently fuzzy. Make-A-Video was trained on web-scraped data, like many AI models, so it has learned and likely amplified some social biases. Meta is not yet releasing the application for public use.
3) Machine learning drives better clinical decision-making for tackling anti-microbial resistance (Digital Journal)
Hospital settings continue to face a serious danger from bacteria that are resistant to antibiotics. Some antimicrobials may be preferred above others at the local level by understanding patterns of resistance in the population. Researchers at Cambridge University are using machine learning to increase the regional-centric understanding of antibiotic resistance patterns. By examining metagenomics datasets, previously unknown antibiotic resistance determinants can be unearthed. The research undertakes predictive analysis at the very local, niche level of the International Space Station. The results have been published in the journal Microbiome part of Springer Nature.
4) Opinion | My Year With Woebot, an A.I. Chatbot Therapist (The New York Times)
Anthropologist, writer, and producer, Barclay Bram, shares his experience with an AI chatbot therapist during the height of Covid-19 related lockdowns. While the tool is not appropriate for people experiencing suicidal thoughts, it can be a tool to triage people toward better-equipped services when it detects suicidal ideation. AI chat therapists have been rolled out in settings as diverse as a maternity hospital in Kenya and refugee camps for people fleeing the war in Syria, and by the Singaporean government as part of its pandemic response. The AI chatbot trial in Kenya was led by Eric Green, an associate professor of global health at Duke University. On the topic of replacing in-person care, Green said, “You can’t replace something that doesn’t exist… globally, more people have access to phones than to mental health professionals.”
5) Canada’s National Day for Truth and Reconciliation: Indigenous AI Resources (Indigenous AI)
Each year, September 30 marks the National Day for Truth and Reconciliation. The day honours the children who never returned home and Survivors of residential schools and the lasting impacts of the residential system on Indigenous communities. We are sharing a list of Indigenous AI resources curated by Indigenous AI – the authors of the position paper on Indigenous Protocol and Artificial Intelligence (IP AI). We encourage our readers to explore these resources and continue learning about global Indigenous perspectives throughout the year.