CodeLLMSurvey
[TOSEM'25] The official GitHub page for the survey paper "A Survey on Large Language Models for Code Generation".
Install / Use
/learn @juyongjiang/CodeLLMSurveyREADME
<div align="center">
<h1>:robot: A Survey on LLMs for Code Generation</h1>
<a href="https://awesome.re">
<img src="https://awesome.re/badge.svg" alt="Awesome">
</a>
<a href="https://img.shields.io/badge/PRs-Welcome-red">
<img src="https://img.shields.io/badge/PRs-Welcome-red" alt="PRs Welcome">
</a>
<a href="https://img.shields.io/github/last-commit/juyongjiang/CodeLLMSurvey?color=green">
<img src="https://img.shields.io/github/last-commit/juyongjiang/CodeLLMSurvey?color=green" alt="Last Commit">
</a>
</div>
<!--  -->
<p align="center" width="100%">
<img src="assets/codellms.png" alt="Code LLMs Workflow" style="width: 100%; min-width: 100px; display: block; margin: auto;">
<br>
A diagram illustrating the general training, inference, and evaluation workflow for Code LLMs and their associated databases.
The training workflow is mainly divided into four distinct stages:
Stage ① and ② are the pre-training phase, whereas Stages ③ and ④ represent the post-training phases. It is important to note that Stage ② and ④ are optional.
For instance, StarCoder incorporates only Stage ①. WizardCoder, fine-tuned upon StarCoder, includes only Stage ③, while Code Llama, continually pre-trained on Llama 2, encompasses Stages ② and ③. DeepSeek-Coder-V2, continually pre-trained on DeepSeek-V2, covers Stages ②, ③, and ④.
Note that pre-trained model can be directly used for inference by prompt engineering.
</p>
[!IMPORTANT] Good news! :tada: Our survey paper has been successfully accepted by TOSEM. :fire::fire::fire:
A curated collection of papers and resources on Large Language Models for Code Generation.
Please refer to our survey "A Survey on Large Language Models for Code Generation" for the detailed contents.
Please let us know if you discover any mistakes or have suggestions by emailing us: csjuyongjiang@gmail.com
If you find our survey beneficial for your research, please consider citing the following paper:
@article{jiang2024survey,
title={A Survey on Large Language Models for Code Generation},
author={Jiang, Juyong and Wang, Fan and Shen, Jiasi and Kim, Sungju and Kim, Sunghun},
journal={arXiv preprint arXiv:2406.00515},
year={2024}
}
Table of Contents
- Paper
- Pre-Training & Foundation Model
- Instruction Tuning & Parameter-Efficient Fine-tuning
- Reinforcement Learning with Feedback
- Prompting for Improving Code Generation
- High-Quality Data Synthesis
- Repository Level & Long Context
- Retrieval Augmented
- Autonomous Coding Agents
- Code LLMs Alignment: Green, Responsibility, Efficiency, Safety, and Trustworthiness
- Evaluation & Benchmark & Metrics
- Useful Resources
- Contributors
- Acknowledgements
- Star History
Paper
Pre-Training & Foundation Model
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation,
TSE, 2024 - CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation,
EMNLP, 2021 - Bias Assessment and Mitigation in LLM-based Code Generation,
arXiv, 2023 - CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation,
ACL, 2024 - Synchromesh: Reliable code generation from pre-trained language models,
ICLR, 2022 - Automatic Code Generation using Pre-Trained Language Models,
arXiv, 2021 - CoderEval: {A} Benchmark of Pragmatic Code Generation with Generative Pre-trained Models,
ICSE, 2024 - ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation,
ACL, 2021 - ChatCoder: Chat-based Refine Requirement Improves LLMs' Code Generation,
arXiv, 2023 - Test-Driven Development and LLM-based Code Generation,
ASE, 2024 - Toward a New Era of Rapid Development: Assessing GPT-4-Vision's Capabilities in UML-Based Code Generation,
LLM4CODE, 2024 - When LLM-based Code Generation Meets the Software Development Process,
arXiv, 2024 - Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation,
arXiv, 2024 - Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer,
arXiv, 2024 - CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X,
KDD, 2023 - Evaluating and Enhancing the Robustness of Code Pre-trained Models through Structure-Aware Adversarial Samples Generation,
EMNLP, 2023 - ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification,
arXiv, 2023 - Code Generation Tools (Almost) for Free? {A} Study of Few-Shot, Pre-Trained Language Models on Code,
arXiv, 2022 - Classification-Based Automatic {HDL} Code Generation Using LLMs,
arXiv, 2024 - Exploring the Effectiveness of {LLM} based Test-driven Interactive Code Generation: User Study and Empirical Evaluation,
ICSE, 2024 - Make Every Move Count: LLM-based High-Quality {RTL} Code Generation Using {MCTS},
arXiv, 2024 - Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond,
arXiv, 2024 - Game Agent Driven by Free-Form Text Command: Using LLM-based Code Generation and Behavior Branch,
arXiv, 2024 - Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation,
ICSE, 2024 - LLM-based and Retrieval-Augmented Control Code Generation,
LLM4Code, 2024 - LLM-Based Code Generation Method for Golang Compiler Testing,
FSE, 2023 - CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models,
ICSE, 2024 - LLM-based Control Code Generation using Image Recognition,
LLM4Code, 2024 - Benchmarking and Explaining Large Language Model-based Code Generation: {A} Causality-Centric Approach,
arXiv, 2023 - Code Llama: Open Foundation Models for Code,
arXiv, 2023 - Execution-based Code Generation using Deep Reinforcement Learning,
TMLR, 2023 - LongCoder: A Long-Range Pre-trained Language Model for Code Completion,
ICML, 2023 - Automating Code Review Activities by Large-Scale Pre-training,
arXiv, 2022 - Qwen2.5-Coder Technical Report,
arXiv, 2024 - DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence,
arXiv, 2024 - PanGu-Coder: Program Synthesis with Function-Level Language Modeling,
arXiv, 2022 - CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis,
ICLR, 2023 - Jigsaw: Large Language Models meet Program Synthesis,
ICSE, 2022 - ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages,
ACL, 2022 - SantaCoder: don't reach for the stars!,
arXiv, 2023 - CodeGen2: Lessons for Training LLMs on Programming and Natural Languages,
ICLR, 2023 - StarCoder: may the source be with you!,
arXiv, 2023 - Textbooks Are All You Need,
arXiv, 2023 - DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence,
arXiv, 2024 - StarCoder 2 and The Stack v2: The Next Generation,
arXiv, 2024 - CodeT5+: Open Code Large Language Models for Code Understanding and Generation,
EMNLP, 2023
Instruction Tuning & Parameter-Efficient Fine-tuning
- LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation,
arXiv, 2024 - Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models,
arXiv, 2023 - An Exploratory Study on Fine-Tuning Large Language Models for Secure Code Generation,
arXiv, 2024 - {ITERTL:} An Iterative Framework for Fine-tuning LLMs for {RTL} Code Generation,
arXiv, 2024 - Fine Tuning Large Language Model for Secure Code Generation,
Forge, 2024 - Eliciting Instruction-tuned Code Language Models' Capabilities to Utilize Auxiliary Function for Code Generation,
arXiv, 2024 - Code Less, Align More: Efficient {LLM} Fine-tuning for Code Generation with Data Pruning,
arXiv, 2024 - Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation,
arXiv, 2023 - DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions,
arXiv, 2023 - Code Alpaca: An Instruction-following LLaMA Model trained on code generation instructions, ``, 2023
- OctoPack: Instruction Tuning Code Large Language Models,
arXiv, 2023 - WizardCoder: Empowering Code Large Language Models with Evol-Instruct,
arXiv, 2023 - Magicoder: Source Code Is All You Need,
arXiv, 2023 - Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs,
COLING, 2024 - Multi-Programming Language Ensemble for Code Generation in Large Language Model,
arXiv, 2024 - Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation,
EMNLP, 2023 - Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models,
arXiv, 2024
Reinforcement Learning with Feedback
- Aligning Crowd-sourced Human Feedback for Code Generation wi
View on GitHub80/100
Security Score
Audited on Mar 27, 2026
No findings