FMEngine

<img src="https://raw.githubusercontent.com/eth-easl/fmengine/init/assets/logo.png" width="240"/>

FMEngine is a utility library for training large foundation models. It is inspired by and built on top of many existing libraries and frameworks, such as DeepSpeed, torchtitan, torchtune.

Acknowledgement

FMEngine is primarily implemented and maintained by the Efficient Architecture and Systems Labs @ ETH Zurich.

<a href="https://systems.ethz.ch/research/easl.html"><img src="https://systems.ethz.ch/research/easl/_jcr_content/par/textimage_842607556/image.imageformat.textsingle.745562631.png" width="120"></a>

We thank our friends for their generous support:

<a href="https://laion.ai/"><img src="https://avatars.githubusercontent.com/u/92627801?s=200&v=4" width="80"/></a> <a href="https://github.com/ontocord/"><img src="https://avatars.githubusercontent.com/u/8900094?v=4" width="80"/></a> <a href="https://huggingface.co/Multi-Domain-Expert-Learning"><img src="https://aeiljuispo.cloudimg.io/v7/https://cdn-uploads.huggingface.co/production/uploads/5fc6879e1c5ee87b1164876d/IoeynCnY_cMdjPAzrdU-2.jpeg?w=200&h=200&f=face" width="80"/></a> <a href=""><img src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width="200"/></a>

Contributors

• @xzyaoi
• @LorrinWWW
• @fishiu: Scaling up Low Rank Adapters + Longer Contexts
• @Taishi-N324: Benchmarking and Scaling Experiments
• @chiennv2000: Mistral Integration