OpenAI gets caught vibe graphing
- OpenAI’s GPT-5 livestream showcased impressive charts, but upon closer inspection, some graphs were found to be inaccurate.
- The most egregious error was in a chart comparing deception rates between GPT-5 and OpenAI’s o3 model, where the scale was inconsistent and led to a misleading comparison.
- CEO Sam Altman called out the mistake, labeling it a “mega chart screwup,” but noted that a corrected version of the chart is available on OpenAI’s blog post.
- An OpenAI marketing staffer apologized for the error, stating that they had fixed the chart in the blog post and acknowledged it as an “unintentional chart crime.”
- The incident has raised questions about whether OpenAI used GPT-5 to create the charts, which could have been a mistake given the company’s claims of significant advances in reducing hallucinations with its new model.
During its big GPT-5 livestream on Thursday, OpenAI showed off a few charts that made the model seem quite impressive — but if you look closely, some graphs were a little bit off.
In one, ironically showing how well GPT-5 does in “deception evals across models,” the scale is all over the place. For “coding deception,” for example, the chart shown onstage says GPT-5 with thinking apparently gets a 50.0 percent deception rate, but that’s compared to OpenAI’s smaller 47.4 percent o3 score which somehow has a larger bar. OpenAI appears to have accurate numbers for this chart in its GPT-5 blog post, however, where GPT-5’s deception rate is labeled as 16.5 percent.
With this chart, OpenAI showed onstage that one of GPT-5’s scores is lower than o3’s but is shown with a bigger bar. In this same chart, o3 and GPT-4o’s scores are different but shown with equally-sized bars. It was bad enough that CEO Sam Altman commented on it, calling it a “mega chart screwup,” though he noted that a correct version is in OpenAI’s blog post.
An OpenAI marketing staffer also apologized, saying, “We fixed the chart in the blog guys, apologies for the unintentional chart crime.”
OpenAI didn’t immediately respond to a request for comment. And while it’s unclear if OpenAI used GPT-5 to actually make the charts, it’s still not a great look for the company on its big launch day — especially when it is touting the “significant advances in reducing hallucinations” with its new model.