XMainframe
Language Model for Mainframe Modernization
Install / Use
/learn @FSoft-AI4Code/XMainframeREADME
XMAiNframe: A Large Language Model for Mainframe Modernization
Table of Contents
- Introduction
- Demonstration
- Procedure of Data Construction
- Model Download
- Evaluation Results
- Usage
- License
- Acknowledgements
- Contact Us
- Citation Information
Introduction
We are introducing XMAiNframe, a state-of-the-art large language model (LLM) specifically designed with knowledge of mainframe legacy systems and COBOL codebases. XMAiNframe is built on top of DeepSeek-Coder 7B and is available with 7B and 10.5B parameters. Additionally, we present MainframeBench, a comprehensive benchmark for assessing mainframe knowledge, including multiple-choice questions, question answering, and COBOL code summarization. Our empirical evaluations demonstrate that XMAiNframe consistently outperforms existing state-of-the-art LLMs across these tasks. Specifically, XMAiNframe achieves 30% higher accuracy than DeepSeek-Coder on multiple-choice questions, doubles the BLEU score of Mixtral-Instruct 8x7B on question answering, and scores six times higher than GPT-3.5 on COBOL summarization. Our work highlights the potential of XMAiNframe to drive significant advancements in managing and modernizing legacy systems, thereby enhancing productivity and saving time for software developers.
Demonstration
In this section, we demonstrate the capabilities of XMAiNframe by comparing it with the leading language model, DeepSeek-Coder-7B. We evaluate the performance of each model by showcasing their responses to a series of realistic questions related to mainframe knowledge. The images below illustrate how each model handles identical prompts. As shown, the responses generated by XMAiNframe are not only accurate but also more detailed and comprehensive compared to those from the base model, DeepSeek-Coder-7B. This makes XMAiNframe particularly valuable for developers seeking a reliable and thorough AI assistant in the mainframe environment.
<div align="center"> <img src="./asset/sample_1.png" > <img src="./asset/sample_2.png" > <img src="./asset/sample_3.png" > </div>Procedure of Data Construction
Mainframe-Training
We utilized two different sources: using the GitHub API to collect COBOL projects hosted on GitHub and gathering online document data relevant to mainframes. In total, Mainframe-Training Dataset consists of 236 million tokens from documents about the mainframe technology and COBOL constructs. In the pre-training process, we combined our Mainframe-Training Dataset with SlimOrca-Dedup to enrich the model’s mainframe knowledge while retaining its general capabilities.
Mainframe-Instruct
<div align="center"> <img src="./asset/cobol_diagram-Page-8.drawio.png" > </div>Mainframe-Instruct is a high-quality synthetic dataset created through 5 steps:
-
Step 1: 300 seed data instances about Mainframe and COBOL are gathered and annotated by our domain experts.
-
Step 2: Using popular LLMs to enrich Mainframe-Instruct from seed data.
-
Step 3: Utilizing GPT-4 as an evaluator to judge model responses, scoring the outputs and ranking responses in a pairwise manner.
-
Step 4: Filtering and manually checking.
-
Step 5: Dividing Mainframe-Instruct into three tasks: Multiple Choice Questions, Question Answering, and COBOL summarization.
Below are the statistics of Mainframe-Instruct Dataset:
<table> <tr> <td> </td> <td>Training Samples</td> <td>Validating Samples</td> <td>Testing Samples</td> </tr> <tr> <td>Multiple Choice Questions</td> <td>13.894</td> <td>1.544</td> <td>1.931</td> </tr> <tr> <td>Question Answering</td> <td>18.692</td> <td>2.078</td> <td>2.598</td> </tr> <tr> <td>COBOL Summarization</td> <td>9.081</td> <td>1.010</td> <td>2.523</td> </tr> </table>MainframeBench, our benchmark for mainframe knowledge, is the testing set in Mainframe-Instruct Dataset. This benchmark is used to evaluate our LLMs with others which is now available at Huggingface datasets.
from datasets import load_dataset
# Load each sub-set in MainframeBench
QA_set = load_dataset("Fsoft-AIC/MainframeBench", 'question_answering')
MC_set = load_dataset("Fsoft-AIC/MainframeBench", 'multiple_choice_question')
Summarization_set = load_dataset("Fsoft-AIC/MainframeBench", 'COBOL_code_summarization')
Model Download
We release XMAiNframe with 7B and 10.5B parameters, including base and instruct models, to the public. XMAiNframe 10.5B is expanded from DeepSeek-Coder 7B by the depth up-scaling method without introducing additional modules or dynamic expert selection methods.
<div align="center">| Model | Download | | :-----------------------------: | :----------------------------------------------------------: | | XMAiNframe-base-7b | 🤗 HuggingFace | | XMAiNframe-instruct-7b | 🤗 HuggingFace | | XMAiNframe-base-10.5b | 🤗 HuggingFace | | XMAiNframe-instruct-10.5b | 🤗 HuggingFace |
</div>