MathVista: Evaluating Math Reasoning in Visual Contexts

Code for the Paper "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts".

For more details, please refer to the project page with dataset exploration and visualization tools: https://mathvista.github.io/.

:bell: If you have any questions or suggestions, please don't hesitate to let us know. You can comment on the Twitter, or post an issue on this repository.

[Webpage] [Paper] [Huggingface Dataset] [Leaderboard] [Visualization] [Result Explorer] [Twitter]

<p align="center"> <img src="assets/logo_v1.png" width="40%"> <br> Tentative logo for <b>MathVista</b>. Generated by DALL·E 3 prompted by <br>"A photo-based logo with a gradient of soft blue and modern typography, accompanied by the title 'MathVista'". </p>

Outlines

• 🔦 Spotlight 🔦
• 💥 News 💥
• 👀 About MathVista
• 🏆 Leaderboard 🏆
  • Contributing the Leaderboard
  • Leaderboard on the testmini subset
  • Leaderboard on the test subset
• 📊 Dataset Examples
• 📖 Dataset Usage
  • Data Source
  • Data Downloading
  • Data Format
  • Data Visualization
  • Usage Demos
• 🔮 Evaluations on MathVista
  • Requirements (Optional)
  • Downloading Images (Optional)
  • Evaluation Pipelines
• 📝 Evaluation Scripts of Our Models
  • Evaluating Multimodal Bard
  • Evaluating Chain-of-Thought GPT-4
  • Evaluating Program-of-Thought GPT-4
  • Evaluating More Settings
  • Evaluating Large Multimodal Models
• 📈 Evaluation Results
• 📜 License
• ☕ Stay Connected!
• ✅ Cite
• 🧠 Related Work
• 🤝 Contributors

💥 Spotlight: Performance Update (Sept 12, 2024) 💥

• Eight models have now surpassed the average human performance level (based on AMT workers with at least a high school diploma).
• The top performers include:
  • 🥇 OpenAI o1
  • 🥈 Grok-2
  • 🥉 Grok-2 mini

💥 News 💥

• [2024.09.12] 💥 OpenAI o1 🥇 Sets New SOTA on MathVista with 73.9! OpenAI’s latest large multimodal model breaks the 70% barrier on MathVista, setting a new SOTA. Read more on the OpenAI blog.
• [2024.06.20] 💥 Claude 3.5 Sonnet achieves new SOTA on MathVista with 67.7! Learn more at the Anthropic blog.
• [2024.05.13] 💥 OpenAI's GPT-4o Outperforms Humans on MathVista! For the first time, OpenAI's new GPT-4o model has achieved a higher score than the human average on MathVista, scoring 63.8 compared to humans' 60.3. Learn more at the OpenAI blog.
• [2024.01.16] 🌟 Our MathVista paper has been accepted for an Oral presentation at ICLR 2024 (only top 85 out of over 7200 submissions)! 🎉 Cheers!
• [2023.12.21] 🚀 Qwen-VL-Plus achieves 43.3%, establishing itself as the best-performing one in open-sourced models. 🎉 Congratulations!
• [2023.12.08] 🔍 We've updated the leaderboard and radar graphs with the fine-grained scores of the Gemini family models. Thanks to the Gemini Team and Google for providing us with these results! 👏
• [2023.12.06] 🚀 Google's newly released multimodal model, Gemini, shows impressive abilities on MathVista, achieving a new SOTA performance with 50.3%! 🎉 Cheers!!
• [2023.11.17] 🌟 Congratulations to SPHINX (V2), which is now the SOTA open-source multimodal model on MathVista, reaching 36.7%. 👏
• [2023.10.25] 🚀 Dive into our comprehensive 112-page evaluation of GPT-4V, Bard, and other Large Multimodal Models, encompassing both quantitative and qualitative insights. Explore the full paper now! 📄✨
• [2023.10.16] 🔍 We are working on a comparative study on the GPT-4V model. Stay tuned for the detailed report! 📑.
• [2023.10.15] We finished the manual evaluation of GPT-4V with the playground chatbot on the testmini set on MathVista. 🚀 GPT-4V achieves a substantial gain of 15.1% ⬆️ over Bard, reaching a new record of 49.9%! 🎉
• [2023.10.15] Our dataset is now accessible at Huggingface Datasets.
• [2023.10.15] Our dataset is now accessible at Paper With Code.
• [2023.10.03] The top-performing model, 🎭 Multimodal Bard, achieved a score of 34.8% on the testmini set for MathVista 📊.
• [2023.10.03] Our work was featured by Aran Komatsuzaki on Twitter. Thanks!
• [2023.10.03] Our paper is now accessible at https://arxiv.org/abs/2310.02255.

👀 About MathVista

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging.

<p align="center"> <img src="assets/data-composition.png" width="40%"> <br> Source dataset distribution of <b>MathVista</b>. </p>

In October 2023, we conducted a comprehensive, quantitative evaluation of 12 prominent foundation models with MathVista. The best-performing GPT-4V model achieved an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis revealed that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often str