TL;DR
- Start with representative evaluations and test whether better prompts meet the task requirements before changing model weights.
- Set aside a diverse holdout dataset before training. Use it to compare the adapted model with the base model.
- For a self-hosted example, TRL’s supervised fine-tuning workflow trains Qwen3-0.6B on the Capybara dataset. Its LoRA option trains adapter weights while retaining the base model’s weights.
- Choose GPU memory, storage, and an execution environment for your workload.
- Save the trained model and test it on held-out examples. Then run a separate generation endpoint check to confirm that the serving container responds.
Fine-tuning adapts a pretrained LLM to a specific task using examples of the responses you want. It can improve output consistency and instruction following, though training takes time. Start with prompt tests and representative evaluations to decide whether to change the model’s weights.
Once you commit to training, your choice of method shapes the feedback and data you need. You must also choose GPU resources, compare the trained model with the base model on held-out examples, and test how you will serve it.
Read on to choose an adaptation method, follow a supervised fine-tuning example in TRL, and take the model through validation and inference hosting. Use case examples and common failure points will help you make those decisions for your own workload.
When is LLM fine-tuning worth the work
Consider fine-tuning when prompt changes still produce inconsistent results across representative inputs and you have examples of the responses you want. Training may improve output formatting and instruction following. It may also let you use shorter prompts. Establish a performance baseline first so you can measure the gain against the time and compute spent training.
Service availability can affect the choice of workflow. OpenAI’s managed fine-tuning platform is winding down, so check access and base-model deprecation dates before planning around it. The step-by-step example below uses self-hosted training with TRL.
Identify the fine-tuning method that suits your task
Choose a method based on the feedback you can collect. Supervised fine-tuning learns from examples of correct responses and suits tasks such as classification, translation, and fixed output formats. Direct preference optimization uses preferred and rejected responses for goals such as summary focus or chat tone. Reinforcement fine-tuning uses expert grades for complex tasks on supported reasoning models.
Next, decide how many model weights to train. LoRA adapters add trainable weights to selected layers. TRL also supports combining adapters with quantization for QLoRA. Your model architecture and available GPU memory will influence that setup.
The worked instruction-tuning example uses Qwen3-0.6B and the Capybara conversational dataset. Use it to follow the training sequence, then assess the result on examples that reflect your own task.
Prepare the data and training environment
Reserve a representative holdout set before training. Include the range of inputs and intended responses your model will encounter, and keep those examples out of the training split. You will use them to compare the trained model with the base model.
Format the training examples for your chosen method. TRL’s SFTTrainer accepts plain text or conversational records for language modeling and prompt-completion tasks. Conversational records contain roles and content; the trainer applies a chat template and tokenizes them. Inspect the formatted examples to confirm that prompts and target responses line up.
Set up the libraries required by your training path. A single GPU code fine-tuning example uses Transformers, Datasets, PEFT, bitsandbytes, and flash-attn. Before launching SFTTrainer, check the tokenizer’s padding behavior: it uses the end-of-sequence token when no padding token is set.
Choose GPU compute for training and inference
Fluence GPU Cloud provides GPU capacity for teams running their own fine-tuning and inference stack. Choose a deployment format that suits how you package the workload and access the machine:
- Containers run packaged training jobs with GPU access. You can use a predefined image or supply a custom one; a private registry requires credentials.
- VMs provide OS-level access for workloads that need control over the operating environment. Register an SSH key before launch.
- Bare metal provides a dedicated physical server without virtualization overhead. It also requires a registered SSH key for access.
Compare the available GPU model, VRAM, GPU count, vCPU, RAM, storage, location, and hourly rate against your training requirements. A listed configuration does not establish that your chosen model and training settings will fit in memory. Check the selected configuration’s price: the rate shown for a GPU model may be the lowest among several offerings.
Account for launch and replacement costs. The rental requirements call for a balance covering three hours, with the first hour charged immediately and the next two reserved. Changing a running VM or bare metal configuration requires terminating and recreating the instance, so plan how you will retain training checkpoints before changing machines.
How to finetune LLM and check the result
The worked example trains Qwen3-0.6B on conversational data with TRL’s SFTTrainer. Follow its model, dataset, and configuration choices for the first run, then inspect what the trainer will actually learn from each sequence.
1. Load the model and training data
Import load_dataset, SFTConfig, and SFTTrainer. The Qwen3 instruction-tuning example uses Qwen/Qwen3-0.6B-Base, the train split of trl-lib/Capybara, the HuggingFaceTB/SmolLM3-3B chat template, and Qwen3-0.6B-Instruct as its output directory. Pass the model, dataset, and configuration to SFTTrainer, then call train().
2. Check sequence length and loss settings
Review the SFTConfig defaults before starting a long run. max_length defaults to 1,024 tokens, so longer sequences are truncated according to truncation_mode. Gradient checkpointing is enabled by default; the learning rate defaults to 2e-5, and logging occurs every 10 steps. BF16 is enabled by default when FP16 has not been selected, so check precision against your GPU setup.
By default, the trainer computes loss across the full sequence. For conversational data, assistant_only_loss=True limits loss to assistant responses. Qwen3 templates receive the required generation markers automatically; other chat templates need those markers configured.
3. Add an adapter when appropriate
To train with LoRA, pass a LoraConfig to SFTTrainer. PEFT adds trainable weights to the targeted model layers while retaining the wrapped base layers. Check target_modules for your model architecture before running an adapter configuration.
4. Save and smoke-test the checkpoint
After training, use save_model to write a result that can be reloaded with from_pretrained(). For a functional check, the post-training example loads a saved Qwen3 checkpoint into a text-generation pipeline and submits a conversation as role-and-content messages. A generated response confirms the checkpoint runs. Use the holdout set to assess whether it performs better on your task.
Validate the model against the baseline
Use the holdout set to decide whether training improved the task. SFTTrainer accepts an eval_dataset in the same formats as the training data and a compute_metrics function for measuring results. Its evaluation and training signals include loss and mean token accuracy. Set your acceptance criteria around the responses your application needs.
Run the same representative inputs through the base and fine-tuned models. Use graders to compare their outputs, then inspect failures such as missed instructions or incorrect formatting. Revise the prompt or training examples and test again where results fall short.
Keep those test cases for later releases. LLM output can vary, and changes to prompts, model versions, or application behavior can affect results. Repeat the base-model comparison before deploying an updated version.
Host the fine-tuned model for inference
Point the inference server at the model artifact you validated. TGI’s model loader accepts a Hub model ID or a local directory containing files saved with Transformers’ save_pretrained() methods. Make that artifact available to the serving environment before starting the server.
The TGI Docker quick start uses teknium/OpenHermes-2.5-Mistral-7B as a separate serving example. Its text-generation-inference:3.3.5 container receives GPU access, a persistent data volume, 1 GB of shared memory, and a mapping from host port 8080 to container port 80. Check server compatibility before adapting the setup to your fine-tuned model.
Once it is running, send a local request to /generate and inspect the response. On Fluence, a GPU container can expose its configured ports through a domain or IP address. Set access controls before making the endpoint available to other users, and keep testing model responses against your held-out cases.
Troubleshoot training and inference failures
Start with the stage that failed. These checks address four common problems:
- Interrupted training: Use checkpoint recovery with a saved path, or set resume_from_checkpoint=True to load the latest checkpoint in args.output_dir. This restores the model, optimizer, and scheduler states.
- Overfitting: If the final model memorizes training examples, test earlier checkpoints against the holdout set where your training workflow has saved them.
- Inference memory errors: Measure batch performance as you increase sequence length. Check for out-of-memory errors and make sure the batching process can recover.
- Failed Fluence deployment: With a predefined OS image, SSH_PUBKEY is a reserved variable. Using it as a custom environment variable causes the deployment request to fail.
Review security and readiness before launch
Model repositories can contain custom Python code. TRL leaves trust_remote_code disabled by default. If your serving setup must run Hub-hosted model code, inspect it and pin a specific revision so later changes are not loaded automatically.
If you trained a LoRA adapter, keep the compatible base model available when you serve it. The adapter adds trained weights to selected base-model layers.
A successful checkpoint test and /generate response establish that the model and endpoint run. Before release, check outputs against your task criteria, control access to the endpoint, and complete a model safety review. Repeat your evaluations when the model or prompts change.
Conclusion
Fine-tuning earns its place when the trained model performs better on representative tasks and the improvement justifies the training and serving work.
Use a holdout set to make that decision, then keep testing as you update the model or prompts. Once the results meet your criteria, plan deployment around memory, access, and cost.
For a self-hosted workflow, compare the container, VM, and bare metal options on Fluence GPU Cloud and choose a configuration for your training and inference stack.
FAQ
Should I try prompting before fine-tuning?
Yes. Test revised prompts on representative inputs before committing to a training run.
How do I choose between supervised fine-tuning and DPO?
Use supervised fine-tuning when you have examples of correct responses. DPO uses preferred and rejected responses, such as examples that distinguish the tone or focus you want.
Can I train fewer than all the model’s weights?
Yes. LoRA adapters add trainable weights to selected layers while retaining the base model’s weights.
How should I choose a GPU configuration?
Compare your model and training settings with the available GPU model, VRAM, GPU count, memory, storage, and hourly rate. Confirm that the selected configuration meets your workload’s requirements.
How do I tell whether fine-tuning helped?
Run the base and trained models on the same held-out examples and score the outputs against your task criteria.
Can I host the fine-tuned model after training?
Yes. An inference server such as TGI can load a Hub model ID or a saved local model directory. Test the endpoint separately from model quality.
