UrbanGPT: Spatio-Temporal Large Language Models

A pytorch implementation for the paper: [UrbanGPT: Spatio-Temporal Large Language Models]

Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, Chao Huang* (*Correspondence)

Data Intelligence Lab@University of Hong Kong, South China University of Technology, Baidu Inc

This repository hosts the code, data, and model weights of UrbanGPT.

🎉 News

[x] 🚀🔥 [2024.05] 🎯🎯📢📢 Exciting News! We are thrilled to announce that our 🌟UrbanGPT🌟 has been accepted by KDD'2024! 🎉🎉🎉 Thanks to all the team members 🤗

🎯🎯📢📢 We upload the models and data used in our UrbanGPT on 🤗 Huggingface. We highly recommend referring to the table below for further details:

| 🤗 Huggingface Address | 🎯 Description | | ------------------------------------------------------------ | ------------------------------------------------------------ | | https://huggingface.co/bjdwh/UrbanGPT | It's the checkpoint of our UrbanGPT based on Vicuna-7B-v1.5-16k tuned on instruction data train-data | | https://huggingface.co/datasets/bjdwh/ST_data_urbangpt | We released a portion of the instruction dataset for evaluation. | | https://huggingface.co/datasets/bjdwh/UrbanGPT_ori_stdata | We released the original dataset used in UrbanGPT. |

[x] [2023.02.23] 🚀🚀 Release the code of UrbanGPT.
[x] [2023.02.29] Add video.
[x] [2023.03.05] Release the full paper.
[x] [2023.03.11] Upload the new checkpoint of our UrbanGPT.
[x] [2023.06.07] Release instruction generation codes and the original dataset used in UrbanGPT.

👉 TODO

[ ] Release baselines codes.
[ ] ...

Introduction

In this work, we present a spatio-temporal large language model that can exhibit exceptional generalization capabilities across a wide range of downstream urban tasks. To achieve this objective, we present the UrbanGPT, which seamlessly integrates a spatio-temporal dependency encoder with the instruction-tuning paradigm. This integration enables large language models (LLMs) to comprehend the complex inter-dependencies across time and space, facilitating more comprehensive and accurate predictions under data scarcity. Extensive experimental findings highlight the potential of building LLMs for spatio-temporal learning, particularly in zero-shot scenarios.

The detailed framework of the proposed UrbanGPT.

Demo Video

https://github.com/HKUDS/UrbanGPT/assets/90381931/9cd094b4-8fa3-486f-890d-631a08b19b4a

Getting Started

<a href='#Code Structure'>1. Code Structure</a>
<a href='#Environment'>2. Environment </a>
<a href='#Training UrbanGPT'>3. Training UrbanGPT </a>
- <a href='#Prepare Pre-trained Checkpoint'>3.1. Prepare Pre-trained Checkpoint</a>
- <a href='#Instruction Tuning'>3.2. Instruction Tuning</a>
<a href='#Evaluating UrbanGPT'>4. Evaluating UrbanGPT</a>
- <a href='#Preparing Checkpoints and Data'>4.1. Preparing Checkpoints and Data</a>
- <a href='#Running Evaluation'>4.2. Running Evaluation</a>
- <a href='#Evaluation Metric Calculation'>4.3. Evaluation Metric Calculation</a>
<a href='#Instructions Generation'>5. Instructions Generation </a>

1. Code Structure <a href='#all_catelogue'>[Back to Top]</a>

.
|   README.md
|   urbangpt_eval.sh
|   urbangpt_train.sh
|   
+---checkpoints
|   \---st_encoder
|           pretrain_stencoder.pth
|           
+---playground
|   |   inspect_conv.py
|   |   
|   +---test_embedding
|   |       README.md
|   |       test_classification.py
|   |       test_semantic_search.py
|   |       test_sentence_similarity.py
|   |       
|   \---test_openai_api
|           anthropic_api.py
|           openai_api.py
|           
+---tests
|       test_openai_curl.sh
|       test_openai_langchain.py
|       test_openai_sdk.py
|       
\---urbangpt
    |   constants.py
    |   conversation.py
    |   utils.py
    |   __init__.py
    |   
    +---eval
    |   |   run_urbangpt.py                     # evaluation
    |   |   run_vicuna.py
    |   |   
    |   \---script
    |           run_model_qa.yaml
    |           
    +---model
    |   |   apply_delta.py
    |   |   apply_lora.py
    |   |   builder.py
    |   |   compression.py
    |   |   convert_fp16.py
    |   |   make_delta.py
    |   |   model_adapter.py
    |   |   model_registry.py
    |   |   monkey_patch_non_inplace.py
    |   |   STLlama.py                          # model
    |   |   utils.py
    |   |   __init__.py
    |   |   
    |   \---st_layers
    |           args.py
    |           ST_Encoder.conf
    |           ST_Encoder.py                   # ST-Encoder
    |           __init__.py
    |           
    +---protocol
    |       openai_api_protocol.py
    |       
    +---serve
    |   |   api_provider.py
    |   |   bard_worker.py
    |   |   cacheflow_worker.py
    |   |   cli.py
    |   |   controller.py
    |   |   controller_graph.py
    |   |   gradio_block_arena_anony.py
    |   |   gradio_block_arena_named.py
    |   |   gradio_css.py
    |   |   gradio_patch.py
    |   |   gradio_web_server.py
    |   |   gradio_web_server_graph.py
    |   |   gradio_web_server_multi.py
    |   |   huggingface_api.py
    |   |   inference.py
    |   |   model_worker.py
    |   |   model_worker_graph.py
    |   |   openai_api_server.py
    |   |   register_worker.py
    |   |   test_message.py
    |   |   test_throughput.py
    |   |   __init__.py
    |   |   
    |   +---examples
    |   |       extreme_ironing.jpg
    |   |       waterview.jpg
    |   |       
    |   +---gateway
    |   |       nginx.conf
    |   |       README.md
    |   |       
    |   \---monitor
    |           basic_stats.py
    |           clean_battle_data.py
    |           elo_analysis.py
    |           hf_space_leaderboard_app.py
    |           monitor.py
    |           
    \---train
            llama2_flash_attn_monkey_patch.py
            llama_flash_attn_monkey_patch.py
            stchat_trainer.py
            train_lora.py
            train_mem.py
            train_st.py                         # train

2.Environment <a href='#all_catelogue'>[Back to Top]</a>

Please first clone the repo and install the required environment, which can be done by running the following commands:

conda create -n urbangpt python=3.9.13

conda activate urbangpt

# Torch with CUDA 11.7
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2

# To support vicuna base model
pip3 install "fschat[model_worker,webui]"

# To install pyg and pyg-relevant packages
pip install torch_geometric
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.0.1+cu117.html

# Clone our UrabnGPT or download it
git clone https://github.com/HKUDS/UrbanGPT.git
cd UrbanGPT

# Install required libraries
# (The recommendation is to install separately using the following method)
pip install deepspeed
pip install ray
pip install einops
pip install wandb
# （There is a version compatibility issue between "flash-attn" and "transformers". Please refer to the flash-attn [GitHub URL](https://github.com/Dao-AILab/flash-attention) for more information.）
pip install flash-attn==2.3.5  # or download from (https://github.com/Dao-AILab/flash-attention/releases, e.g. flash_attn-2.3.5+cu117torch2.0cxx11abiFALSE-cp39-cp39-linux_x86_64.whl)
pip install transformers==4.34.0

# （or you can install according to the requirements file.）
pip install -r requirements.txt

3. Training UrbanGPT <a href='#all_catelogue'>[Back to Top]</a>

3.1. Preparing Pre-trained Checkpoint <a href='#all_catelogue'>[Back to Top]</a>

UrabnGPT is trained based on following excellent existing models. Please follow the instructions to prepare the checkpoints.

Vicuna: Prepare our base model Vicuna, which is an instruction-tuned chatbot and base model in our implementation. Please download its weights here. We generally utilize v1.5 and v1.5-16k model with 7B parameters. You should update the 'config.json' of vicuna, for example, the 'config.json' in v1.5-16k can be found in config.json
`Spatio-temporal Enco

UrbanGPT

Install / Use

README