SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models

This project accompanies the research paper,

SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models <br> Mingze Xu*, Mingfei Gao*, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, Afshin Dehghan

<p align="center"> <img src="assets/teaser.png" width="600"> </p>

SlowFast-LLaVA is a training-free multimodal large language model (LLM) for video understanding and reasoning. Without requiring fine-tuning on any data, it achieves comparable or even better performance compared to state-of-the-art Video LLMs on a wide range of VideoQA tasks and benchmarks, as shown in the figure.

Table of contents

• Getting Started
  • Installation
  • Data Preparation
• Configuration
• Inference and Evaluation
  • Output Structures
• Demo
• License
• Citations

Getting Started

Installation

• The code is developed with CUDA 11.7, Python >= 3.10.12, PyTorch >= 2.1.0

  0. [Optional but recommended] Create a new conda environment.

     conda create -n sf_llava python=3.10.12
     

     And activate the environment.

     conda activate sf_llava
     

  1. Install the requirements.

     bash setup_env.sh
     

  2. Add OpenAI key and organization to the system environment to use GPT-3.5-turbo for model evaluation.

     export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY
     export OPENAI_ORG=$YOUR_OPENAI_ORG  # optional
     

  3. Download pre-trained LLaVA-NeXT weights from HuggingFace, and put them under the ml-slowfast-llava folder.

     git lfs clone https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b liuhaotian/llava-v1.6-vicuna-7b
     git lfs clone https://huggingface.co/liuhaotian/llava-v1.6-34b liuhaotian/llava-v1.6-34b
     

Data Preparation

1. We prepare the ground-truth question and answer files based on IG-VLM, and put them under playground/gt_qa_files.

   • MSVD-QA
     • Download the MSVD_QA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_msvd_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • MSRVTT-QA
     • Download the MSRVTT_QA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_msrvtt_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • TGIF-QA
     • Download the TGIF_FrameQA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_tgif_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • Activitynet-QA
     • Download the Activitynet_QA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_activitynet_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • NExT-QA
     • Download the NExT_QA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_nextqa_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • EgoSchema
     • Download the EgoSchema.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_egoschema_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • IntentQA
     • Download the IntentQA.csv from the here
     • Reformat the files by running

       python scripts/data/prepare_intentqa_qa_file.py --qa_file $PATH_TO_CSV_FILE
       

   • VCGBench
     • Download all files under text_generation_benchmark
     • Reformat the files by running

       python scripts/data/prepare_vcgbench_qa_file.py --qa_folder $TEXT_GENERATION_BENCHMARK
       

2. Download the raw videos from the official websites.

   • Openset VideoQA

     • [Recomanded] Option 1: Follow the instruction in Video-LLaVA to download raw videos.
     • Option 2: Download videos from the data owners.
       • MSVD-QA
       • MSRVTT-QA
       • TGIF-QA
       • ActivityNet-QA

   • Multiple Choice VideoQA

     • Download datasets from the data owners.
       • NExT-QA
       • EgoSchema
       • IntentQA

   • Text Generation

     • The videos are based on ActivityNet, and you can reuse the one from Openset VideoQA.

3. Organize the raw videos under playground/data.

   • To directly use our data loaders