Radical Reads

In the rapidly developing field of AI, accurately evaluating multimodal models remains a critical and unsolved issue. To address this challenge, Reka, a Radical Ventures portfolio company, has introduced innovative ideas through a paper on their new evaluation method, Vibe-Eval. Radical Partner and Reka board member Rob Toews sat down with three members of the Reka team – Chief Scientist Yi Tay and Vibe-Eval lead authors Piotr Padlewski and Max Bain – to discuss the significance of Vibe-Eval and its potential to advance the industry. Comprising 269 high-quality image-text prompts, Vibe-Eval is designed to challenge even the most advanced multimodal models, highlighting the need for robust evaluations to drive progress in AI development. The following interview has been edited for clarity and conciseness. 

Rob Toews (RT): Why did you create Vibe-Eval?

Piotr Padlewski, Max Bain and Yi Tay (Reka): We started by expanding our already large internal evaluation. While doing it, especially constructing challenging examples that Reka Core (and other frontier multimodal models) were unable to solve, we gained novel insights. We decided to shed some light on how we do evaluations internally and to share the dataset with the community, hopefully pushing the field forward.

RT: What sets Vibe-Eval apart from other industry benchmarks?

Reka: The diversity of examples in terms of difficulty, required capabilities, and location for starters. A perfect benchmark can capture incremental improvements (not so easy that it saturates, and not so hard that the model can’t capture any of it). To measure general progress towards AGI, the examples should test a variety of skills, often combining them (e.g. requiring image understanding with word knowledge or reasoning). Having a diverse set of problems coming from around the globe is also an improvement so that the resulting models can be useful to everyone. Constructing such benchmarks while maintaining high-quality references is extremely difficult and time-consuming. Ensuring no leakage to the training set is another thing, that is why almost all of the pictures in Vibe-Eval were captured by us.

RT: What are the challenges when it comes to evaluating today’s multimodal models?

Reka: Evaluating open-ended questions is challenging as it is hard to quantify the accuracy (e.g. as opposed to multiple choice questions). So far the best and most trusted way was to compare models with human evaluators. However, this is expensive and tedious work. We also found that generic human annotators were deficient at scoring difficult examples that required domain knowledge – even though the ground truth response was present. This is why we decided to use our best LLM for this task. We found that not only is it well correlated with human evaluation on a normal subset, but it performs better than the average evaluator on the difficult subset, as it was correlated strongly with evaluations done by experts.

RT: How did your team go about creating the Vibe-Eval benchmarks?

Reka: We used our own photos and created 169 challenging “normal” examples. After the first round of evaluations, it turned out that frontier models could solve, or get very close to solving, most of the examples. This is when we decided to change strategy and create 100 hard examples intended to break our own best model (Reka Core). Unsurprisingly, the examples were also hard for other frontier models, such as GPT-4V, Gemini 1.5, Claude Opus, with more than half of the examples being unsolvable. By carefully reviewing the examples and then failure cases of different models we were able to fix numerous bugs, making the reference annotation high-quality.

RT: What are the kinds of tasks or prompts that multimodal models struggle to answer correctly?

Reka: Multimodal models often struggle with counting, detailed scene understanding (e.g. what is left or right, details on a particular person’s clothing), understanding of charts, plots and tables, etc. Examples from the long tail requiring specialized knowledge or multistep reasoning will also likely fail. Current multimodal models also tend to hallucinate a lot.

RT: How will Vibe-Eval evolve with the growing sophistication of multimodal models?Reka: Our internal evals expand every day as we find new failure cases or when we learn about the needs of our customers. With the next wave of improvements on the frontier, we will most likely release a new updated version of Vibe-Eval.

RT: What is the most important thing for the average person to know about AI model evaluation?

Reka: There are a lot of benchmarks and, unfortunately, many of them are being gamed with tricks, making them less and less legitimate with time. So long as the prompts are diverse and not cherry-picked, the benchmarks done by human evaluation tend to be much more trustworthy. The advice to, “Just play with the model for 15 minutes” is the best recommendation if you want to know if a multimodal model is any good. Vibe-Eval might be the next best option.