hoshi-hiyouga 74a79cc059 fix test | 17 hours ago | |
---|---|---|
.github | 1 month ago | |
assets | 5 days ago | |
data | 1 week ago | |
docker | 1 week ago | |
evaluation | 4 months ago | |
examples | 1 month ago | |
scripts | 1 month ago | |
src | 17 hours ago | |
tests | 3 weeks ago | |
.dockerignore | 3 months ago | |
.env.local | 1 month ago | |
.gitattributes | 1 year ago | |
.gitignore | 2 months ago | |
CITATION.cff | 4 months ago | |
LICENSE | 1 year ago | |
MANIFEST.in | 4 months ago | |
Makefile | 1 month ago | |
README.md | 1 week ago | |
README_zh.md | 1 week ago | |
pyproject.toml | 5 months ago | |
requirements.txt | 1 week ago | |
setup.py | 1 week ago |
👋 Join our WeChat or NPU user group.
[ English | 中文 ]
Fine-tuning a large language model can be easy as...
https://github.com/user-attachments/assets/7c96b465-9df7-45f4-8053-bf03e58386d3
Choose your path:
[!NOTE] Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.
Compared to ChatGLM's P-Tuning, LLaMA Factory's LoRA tuning offers up to 3.7 times faster training speed with a better Rouge score on the advertising text generation task. By leveraging 4-bit quantization technique, LLaMA Factory's QLoRA further improves the efficiency regarding the GPU memory.
Definitions
pre_seq_len=128
for ChatGLM's P-Tuning and lora_rank=32
for LLaMA Factory's LoRA tuning.[24/10/09] We supported downloading pre-trained models and datasets from the Modelers Hub. See this tutorial for usage.
[24/09/19] We support fine-tuning the Qwen2.5 models.
[24/08/30] We support fine-tuning the Qwen2-VL models. Thank @simonJJJ's PR.
[24/08/27] We support Liger Kernel. Try enable_liger_kernel: true
for efficient training.
[24/08/09] We support Adam-mini optimizer. See examples for usage. Thank @relic-yuexi's PR.
Full Changelog
[24/07/04] We support contamination-free packed training. Use neat_packing: true
to activate it. Thank @chuan298's PR.
[24/06/16] We support PiSSA algorithm. See examples for usage.
[24/06/07] We supported fine-tuning the Qwen2 and GLM-4 models.
[24/05/26] We supported SimPO algorithm for preference learning. See examples for usage.
[24/05/20] We supported fine-tuning the PaliGemma series models. Note that the PaliGemma models are pre-trained models, you need to fine-tune them with paligemma
template for chat completion.
[24/05/18] We supported KTO algorithm for preference learning. See examples for usage.
[24/05/14] We supported training and inference on the Ascend NPU devices. Check installation section for details.
[24/04/26] We supported fine-tuning the LLaVA-1.5 multimodal LLMs. See examples for usage.
[24/04/22] We provided a Colab notebook for fine-tuning the Llama-3 model on a free T4 GPU. Two Llama-3-derived models fine-tuned using LLaMA Factory are available at Hugging Face, check Llama3-8B-Chinese-Chat and Llama3-Chinese for details.
[24/04/21] We supported Mixture-of-Depths according to AstraMindAI's implementation. See examples for usage.
[24/04/16] We supported BAdam optimizer. See examples for usage.
[24/04/16] We supported unsloth's long-sequence training (Llama-2-7B-56k within 24GB). It achieves 117% speed and 50% memory compared with FlashAttention-2, more benchmarks can be found in this page.
[24/03/31] We supported ORPO. See examples for usage.
[24/03/21] Our paper "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models" is available at arXiv!
[24/03/20] We supported FSDP+QLoRA that fine-tunes a 70B model on 2x24GB GPUs. See examples for usage.
[24/03/13] We supported LoRA+. See examples for usage.
[24/03/07] We supported GaLore optimizer. See examples for usage.
[24/03/07] We integrated vLLM for faster and concurrent inference. Try infer_backend: vllm
to enjoy 270% inference speed.
[24/02/28] We supported weight-decomposed LoRA (DoRA). Try use_dora: true
to activate DoRA training.
[24/02/15] We supported block expansion proposed by LLaMA Pro. See examples for usage.
[24/02/05] Qwen1.5 (Qwen2 beta version) series models are supported in LLaMA-Factory. Check this blog post for details.
[24/01/18] We supported agent tuning for most models, equipping model with tool using abilities by fine-tuning with dataset: glaive_toolcall_en
.
[23/12/23] We supported unsloth's implementation to boost LoRA tuning for the LLaMA, Mistral and Yi models. Try use_unsloth: true
argument to activate unsloth patch. It achieves 170% speed in our benchmark, check this page for details.
[23/12/12] We supported fine-tuning the latest MoE model Mixtral 8x7B in our framework. See hardware requirement here.
[23/12/01] We supported downloading pre-trained models and datasets from the ModelScope Hub. See this tutorial for usage.
[23/10/21] We supported NEFTune trick for fine-tuning. Try neftune_noise_alpha: 5
argument to activate NEFTune.
[23/09/27] We supported $S^2$-Attn proposed by LongLoRA for the LLaMA models. Try shift_attn: true
argument to enable shift short attention.
[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See examples for usage.
[23/09/10] We supported FlashAttention-2. Try flash_attn: fa2
argument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.
[23/08/12] We supported RoPE scaling to extend the context length of the LLaMA models. Try rope_scaling: linear
argument in training and rope_scaling: dynamic
argument at inference to extrapolate the position embeddings.
[23/08/11] We supported DPO training for instruction-tuned models. See examples for usage.
[23/07/31] We supported dataset streaming. Try streaming: true
and max_steps: 10000
arguments to load your dataset in streaming mode.
[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2 / Baichuan) for details.
[23/07/18] We developed an all-in-one Web UI for training, evaluation and inference. Try train_web.py
to fine-tune models in your Web browser. Thank @KanadeSiina and @codemayq for their efforts in the development.
[23/07/09] We released FastEdit ⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please follow FastEdit if you are interested.
[23/06/29] We provided a reproducible example of training a chat model using instruction-following datasets, see Baichuan-7B-sft for details.
[23/06/22] We aligned the demo API with the OpenAI's format where you can insert the fine-tuned model in arbitrary ChatGPT-based applications.
[23/06/03] We supported quantized training and inference (aka QLoRA). See examples for usage.
Model | Model size | Template |
---|---|---|
Baichuan 2 | 7B/13B | baichuan2 |
BLOOM/BLOOMZ | 560M/1.1B/1.7B/3B/7.1B/176B | - |
ChatGLM3 | 6B | chatglm3 |
Command R | 35B/104B | cohere |
DeepSeek (Code/MoE) | 7B/16B/67B/236B | deepseek |
Falcon | 7B/11B/40B/180B | falcon |
Gemma/Gemma 2/CodeGemma | 2B/7B/9B/27B | gemma |
GLM-4 | 9B | glm4 |
InternLM2/InternLM2.5 | 7B/20B | intern2 |
Llama | 7B/13B/33B/65B | - |
Llama 2 | 7B/13B/70B | llama2 |
Llama 3-3.2 | 1B/3B/8B/70B | llama3 |
LLaVA-1.5 | 7B/13B | llava |
LLaVA-NeXT | 7B/8B/13B/34B/72B/110B | llava_next |
LLaVA-NeXT-Video | 7B/34B | llava_next_video |
MiniCPM | 1B/2B/4B | cpm/cpm3 |
Mistral/Mixtral | 7B/8x7B/8x22B | mistral |
OLMo | 1B/7B | - |
PaliGemma | 3B | paligemma |
Phi-1.5/Phi-2 | 1.3B/2.7B | - |
Phi-3 | 4B/7B/14B | phi |
Qwen (1-2.5) (Code/Math/MoE) | 0.5B/1.5B/3B/7B/14B/32B/72B/110B | qwen |
Qwen2-VL | 2B/7B/72B | qwen2_vl |
StarCoder 2 | 3B/7B/15B | - |
XVERSE | 7B/13B/65B | xverse |
Yi/Yi-1.5 (Code) | 1.5B/6B/9B/34B | yi |
Yi-VL | 6B/34B | yi_vl |
Yuan 2 | 2B/51B/102B | yuan |
[!NOTE] For the "base" models, the
template
argument can be chosen fromdefault
,alpaca
,vicuna
etc. But make sure to use the corresponding template for the "instruct/chat" models.Remember to use the SAME template in training and inference.
Please refer to constants.py for a full list of models we supported.
You also can add a custom chat template to template.py.
Approach | Full-tuning | Freeze-tuning | LoRA | QLoRA |
---|---|---|---|---|
Pre-Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
Supervised Fine-Tuning | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
Reward Modeling | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
PPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
DPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
KTO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
ORPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
SimPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
[!TIP] The implementation details of PPO can be found in this blog.
Pre-training datasets
Supervised fine-tuning datasets
Preference datasets
Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.
pip install --upgrade huggingface_hub
huggingface-cli login
Mandatory | Minimum | Recommend |
---|---|---|
python | 3.8 | 3.11 |
torch | 1.13.1 | 2.4.0 |
transformers | 4.41.2 | 4.43.4 |
datasets | 2.16.0 | 2.20.0 |
accelerate | 0.30.1 | 0.32.0 |
peft | 0.11.1 | 0.12.0 |
trl | 0.8.6 | 0.9.6 |
Optional | Minimum | Recommend |
---|---|---|
CUDA | 11.6 | 12.2 |
deepspeed | 0.10.0 | 0.14.0 |
bitsandbytes | 0.39.0 | 0.43.1 |
vllm | 0.4.3 | 0.5.0 |
flash-attn | 2.3.0 | 2.6.3 |
* estimated
Method | Bits | 7B | 13B | 30B | 70B | 110B | 8x7B | 8x22B |
---|---|---|---|---|---|---|---|---|
Full | AMP | 120GB | 240GB | 600GB | 1200GB | 2000GB | 900GB | 2400GB |
Full | 16 | 60GB | 120GB | 300GB | 600GB | 900GB | 400GB | 1200GB |
Freeze | 16 | 20GB | 40GB | 80GB | 200GB | 360GB | 160GB | 400GB |
LoRA/GaLore/BAdam | 16 | 16GB | 32GB | 64GB | 160GB | 240GB | 120GB | 320GB |
QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | 140GB | 60GB | 160GB |
QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | 72GB | 30GB | 96GB |
QLoRA | 2 | 4GB | 8GB | 16GB | 24GB | 48GB | 18GB | 48GB |
[!IMPORTANT] Installation is mandatory.
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
Extra dependencies available: torch, torch-npu, metrics, deepspeed, liger-kernel, bitsandbytes, hqq, eetq, gptq, awq, aqlm, vllm, galore, badam, adam-mini, qwen, modelscope, openmind, quality
[!TIP] Use
pip install --no-deps -e .
to resolve package conflicts.
For Windows users
If you want to enable the quantized LoRA (QLoRA) on the Windows platform, you need to install a pre-built version of bitsandbytes
library, which supports CUDA 11.1 to 12.2, please select the appropriate release version based on your CUDA version.
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl
To enable FlashAttention-2 on the Windows platform, you need to install the precompiled flash-attn
library, which supports CUDA 12.1 to 12.2. Please download the corresponding version from flash-attention based on your requirements.
For Ascend NPU users
To install LLaMA Factory on Ascend NPU devices, please specify extra dependencies: pip install -e ".[torch-npu,metrics]"
. Additionally, you need to install the Ascend CANN Toolkit and Kernels. Please follow the installation tutorial or use the following commands:
# replace the url according to your CANN version and devices
# install CANN Toolkit
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run
bash Ascend-cann-toolkit_8.0.RC1.alpha001_linux-"$(uname -i)".run --install
# install CANN Kernels
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Milan-ASL/Milan-ASL%20V100R001C17SPC701/Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run
bash Ascend-cann-kernels-910b_8.0.RC1.alpha001_linux.run --install
# set env variables
source /usr/local/Ascend/ascend-toolkit/set_env.sh
Requirement | Minimum | Recommend |
---|---|---|
CANN | 8.0.RC1 | 8.0.RC1 |
torch | 2.1.0 | 2.1.0 |
torch-npu | 2.1.0 | 2.1.0.post3 |
deepspeed | 0.13.2 | 0.13.2 |
Remember to use ASCEND_RT_VISIBLE_DEVICES
instead of CUDA_VISIBLE_DEVICES
to specify the device to use.
If you cannot infer model on NPU devices, try setting do_sample: false
in the configurations.
Download the pre-built Docker images: 32GB | 64GB
Please refer to data/README.md for checking the details about the format of dataset files. You can either use datasets on HuggingFace / ModelScope / Modelers hub or load the dataset in local disk.
[!NOTE] Please update
data/dataset_info.json
to use your custom dataset.
Use the following 3 commands to run LoRA fine-tuning, inference and merging of the Llama3-8B-Instruct model, respectively.
llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml
llamafactory-cli chat examples/inference/llama3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/llama3_lora_sft.yaml
See examples/README.md for advanced usage (including distributed training).
[!TIP] Use
llamafactory-cli help
to show help information.
llamafactory-cli webui
For CUDA users:
cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash
For Ascend NPU users:
cd docker/docker-npu/
docker compose up -d
docker compose exec llamafactory bash
For AMD ROCm users:
cd docker/docker-rocm/
docker compose up -d
docker compose exec llamafactory bash
Build without Docker Compose
For CUDA users:
docker build -f ./docker/docker-cuda/Dockerfile \
--build-arg INSTALL_BNB=false \
--build-arg INSTALL_VLLM=false \
--build-arg INSTALL_DEEPSPEED=false \
--build-arg INSTALL_FLASHATTN=false \
--build-arg PIP_INDEX=https://pypi.org/simple \
-t llamafactory:latest .
docker run -dit --gpus=all \
-v ./hf_cache:/root/.cache/huggingface \
-v ./ms_cache:/root/.cache/modelscope \
-v ./om_cache:/root/.cache/openmind \
-v ./data:/app/data \
-v ./output:/app/output \
-p 7860:7860 \
-p 8000:8000 \
--shm-size 16G \
--name llamafactory \
llamafactory:latest
docker exec -it llamafactory bash
For Ascend NPU users:
# Choose docker image upon your environment
docker build -f ./docker/docker-npu/Dockerfile \
--build-arg INSTALL_DEEPSPEED=false \
--build-arg PIP_INDEX=https://pypi.org/simple \
-t llamafactory:latest .
# Change `device` upon your resources
docker run -dit \
-v ./hf_cache:/root/.cache/huggingface \
-v ./ms_cache:/root/.cache/modelscope \
-v ./om_cache:/root/.cache/openmind \
-v ./data:/app/data \
-v ./output:/app/output \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-p 7860:7860 \
-p 8000:8000 \
--device /dev/davinci0 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
--shm-size 16G \
--name llamafactory \
llamafactory:latest
docker exec -it llamafactory bash
For AMD ROCm users:
docker build -f ./docker/docker-rocm/Dockerfile \
--build-arg INSTALL_BNB=false \
--build-arg INSTALL_VLLM=false \
--build-arg INSTALL_DEEPSPEED=false \
--build-arg INSTALL_FLASHATTN=false \
--build-arg PIP_INDEX=https://pypi.org/simple \
-t llamafactory:latest .
docker run -dit \
-v ./hf_cache:/root/.cache/huggingface \
-v ./ms_cache:/root/.cache/modelscope \
-v ./om_cache:/root/.cache/openmind \
-v ./data:/app/data \
-v ./output:/app/output \
-v ./saves:/app/saves \
-p 7860:7860 \
-p 8000:8000 \
--device /dev/kfd \
--device /dev/dri \
--shm-size 16G \
--name llamafactory \
llamafactory:latest
docker exec -it llamafactory bash
Details about volume
hf_cache
: Utilize Hugging Face cache on the host machine. Reassignable if a cache already exists in a different directory.ms_cache
: Similar to Hugging Face cache but for ModelScope users.om_cache
: Similar to Hugging Face cache but for Modelers users.data
: Place datasets on this dir of the host machine so that they can be selected on LLaMA Board GUI.output
: Set export dir to this location so that the merged result can be accessed directly on the host machine.API_PORT=8000 llamafactory-cli api examples/inference/llama3_vllm.yaml
[!TIP] Visit this page for API document.
If you have trouble with downloading models and datasets from Hugging Face, you can use ModelScope.
export USE_MODELSCOPE_HUB=1 # `set USE_MODELSCOPE_HUB=1` for Windows
Train the model by specifying a model ID of the ModelScope Hub as the model_name_or_path
. You can find a full list of model IDs at ModelScope Hub, e.g., LLM-Research/Meta-Llama-3-8B-Instruct
.
You can also use Modelers Hub to download models and datasets.
export USE_OPENMIND_HUB=1 # `set USE_OPENMIND_HUB=1` for Windows
Train the model by specifying a model ID of the Modelers Hub as the model_name_or_path
. You can find a full list of model IDs at Modelers Hub, e.g., TeleAI/TeleChat-7B-pt
.
To use Weights & Biases for logging experimental results, you need to add the following arguments to yaml files.
report_to: wandb
run_name: test_run # optional
Set WANDB_API_KEY
to your key when launching training tasks to log in with your W&B account.
If you have a project that should be incorporated, please contact via email or create a pull request.
Click to show
This repository is licensed under the Apache-2.0 License.
Please follow the model licenses to use the corresponding model weights: Baichuan 2 / BLOOM / ChatGLM3 / Command R / DeepSeek / Falcon / Gemma / GLM-4 / InternLM2 / Llama / Llama 2 (LLaVA-1.5) / Llama 3 / MiniCPM / Mistral / OLMo / Phi-1.5/Phi-2 / Phi-3 / Qwen / StarCoder 2 / XVERSE / Yi / Yi-1.5 / Yuan 2
If this work is helpful, please kindly cite as:
@inproceedings{zheng2024llamafactory,
title={LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models},
author={Yaowei Zheng and Richong Zhang and Junhao Zhang and Yanhan Ye and Zheyan Luo and Zhangchi Feng and Yongqiang Ma},
booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
address={Bangkok, Thailand},
publisher={Association for Computational Linguistics},
year={2024},
url={http://arxiv.org/abs/2403.13372}
}
This repo benefits from PEFT, TRL, QLoRA and FastChat. Thanks for their wonderful works.