Train a character LoRA: dataset, captions, configuration
A character LoRA is a small set of weights that teaches a base model one specific face. Whether it works is decided almost entirely by the twenty to forty images you feed it, which is why most of this page is about building that set and not about the training flags.
What you get from this page
- How to build a training set that teaches the character rather than the background
- A captioning strategy that keeps the trigger word doing the work
- A starting configuration, and which two values to change first
- How to evaluate the result objectively instead of by vibes
Before you start
- A working local or rented generation environment you can install nodes or scripts into
- A synthetic character concept with a settled appearance — see character consistency methods
- Roughly 20 to 40 candidate images of the character, or the means to generate them
- A GPU with enough memory for your chosen trainer, rented if you do not own one
On this page
A character LoRA is a small set of weights that teaches a base model one specific person. Trained well, it holds a face across poses, lighting and camera distances that prompting alone cannot. Trained badly, it produces a character who only exists in one pose against one background, which is a common and dispiriting outcome.
The difference is almost entirely in the dataset. So that is where most of this guide goes.
Step by step
Generate a candidate pool, larger than you need
Produce 80 to 150 images of the character using reference conditioning or a locked prompt. You are going to reject most of them, and having a large pool is what lets you be strict later. Vary pose, angle, expression, lighting and distance deliberately rather than generating a hundred variations of one shot.
Gotcha — Resist the urge to generate only flattering images. A dataset of nothing but three-quarter portraits produces an adapter that can only make three-quarter portraits.
Cull to 20 to 40, optimising for variety
Select for coverage first and quality second. You want several camera distances, a spread of head angles, a range of expressions, and multiple lighting conditions. Every selected image must be unambiguously the same person, in focus, and free of artefacts in the face.
Gotcha — Near-duplicates are actively harmful. Two nearly identical images teach the model that this exact framing is what the character is.
Neutralise the backgrounds
If every training image shares a background, the adapter learns the background as part of the character. Either vary backgrounds across the set, or use plain and varied ones. This is the single most common cause of a LoRA that cannot place its character anywhere new.
Standardise resolution and crop
Match your trainer's expected resolution, and crop so the subject occupies a consistent, generous share of the frame. Include some tighter and some wider crops for coverage, but do not include images where the character is a small element of a busy scene.
Caption with a rare trigger token and describe what varies
This is the step people get backwards. Use a distinctive trigger token, then describe everything in the image that is not an intrinsic property of the character: pose, clothing, background, lighting. Leave the character's permanent features undescribed, so the trigger token has to carry them.
captions/img_014.txttextohwx_lyra, sitting on a wooden chair, three-quarter view, soft window light from the left, plain grey wall, wearing a black knit sweater, neutral expression # Note what is absent: no eye colour, no hair colour, no face # shape, no build. Those belong to ohwx_lyra and must stay # attached to the trigger, not to a describable phrase.Gotcha — If you caption 'brown eyes' in every image, the model attaches brown eyes to the phrase rather than to the trigger. Describe the variable, omit the invariant.
Pick a starting configuration and run it
Use the baseline below. Do not tune anything on the first run: you need one result to compare against before changes mean anything.
character-lora.baseline.yamlyaml# Conventional starting point for a character adapter. # Treat every value as a hypothesis, not a setting. dataset: images: 24 # 20-40; variety over volume resolution: 1024 repeats: 10 # images x repeats x epochs = total steps training: epochs: 10 total_steps: ~2400 # 1000-3000 is the usual working range batch_size: 1 gradient_accumulation: 4 optimizer: type: adamw8bit learning_rate: 1.0e-4 # first thing to change if results are poor scheduler: cosine warmup_steps: 100 lora: rank: 16 # 8-32; higher is not better for one face alpha: 16 # commonly matched to rank target: attention_only # keep the adapter small and portable checkpoints: save_every_epochs: 2 # you want intermediates to compareGotcha — Save intermediate checkpoints. The best adapter is frequently not the final epoch, and without intermediates you cannot recover it without retraining.
Evaluate against a fixed prompt set, not new prompts
Build a list of 10 to 15 prompts covering close portrait, full body, profile, unusual lighting, an outfit not in the training data, and a setting not in the training data. Run every checkpoint against the identical list at the identical seeds. This is the only way to see which checkpoint is actually better.
eval/prompt-set.txttextohwx_lyra, close portrait, neutral background, soft light ohwx_lyra, full body, standing, plain studio backdrop ohwx_lyra, profile view, hard side light ohwx_lyra, laughing, outdoors, overcast ohwx_lyra, wearing a red raincoat, city street at night ohwx_lyra, seated at a cafe table, window light ohwx_lyra, looking over shoulder, low angle ohwx_lyra, harsh direct flash, close crop # Fixed seeds. Same list every checkpoint, every retrain, forever.Sweep the adapter weight before you retrain
Run your best checkpoint at several adapter weights across the same fixed prompt set. A LoRA that looks under-trained is often just being applied too weakly, and a sweep costs minutes where a retrain costs an hour.
Measure the keeper rate and record it
Generate an uncurated batch of at least 200 images with the finished adapter and score them against your real publication standard. That figure is what the training run actually bought you, and it is the number to beat next time. Method in the cost per usable image guide.
Diagnosing a bad result
| What you see | Most likely cause | What to change |
|---|---|---|
| Character only appears in training-set poses | Overfitting, or a dataset with too little pose variety | Reduce steps, or rebuild the dataset with more variety. Usually the dataset. |
| Same background keeps reappearing | Backgrounds were constant across the training set | Vary or neutralise backgrounds and retrain. |
| Face dissolves at higher guidance | Under-training, or adapter weight too low | Sweep weight first, then increase steps. |
| Identity is weak but images look good | Invariant features were described in captions | Recaption: remove permanent features, keep only what varies. |
| Everything looks like the base model | Learning rate too low, or the trigger token is not rare | Raise learning rate; change to a rarer token. |
| Artefacts on skin and edges | Over-training, or low-quality source images | Take an earlier checkpoint; cull the dataset harder. |
A diagnostic map, not a dataset. Each row pairs a failure people report with the cause that most often explains it in community training practice; treat it as a list of things to test in order, not as measured frequencies.
What it costs
A character LoRA is not an expensive training job. Runpod's published rate for an H100 PCIe 80GB is $2.89 per hour on Secure Cloud, and a run of the size described here is measured in fractions of an hour to a couple of hours depending on trainer and settings.
How many training steps do I need?
The commonly cited working range is 1,000 to 3,000 for a character adapter, and the correct answer inside that range depends on your dataset size and learning rate. This is exactly why you save intermediate checkpoints and evaluate them against a fixed prompt set instead of trusting a step count from someone else's setup.
Should I use a higher rank for better quality?
Usually not. A single human face is a small amount of information. Higher ranks give the adapter more capacity to memorise the training set, which tends to produce overfitting rather than fidelity, along with a larger file. Start in the 8 to 32 band and change the dataset before you change the rank.
Can I train on my own generated output?
Yes, and it is the normal path for a synthetic character: generate a candidate pool with reference conditioning, cull hard, train on the survivors. The risk is compounding the base model's biases, which is another reason to be strict at the culling stage.
Do I need to retrain when a new base model comes out?
Generally yes, if you want to move to it. Adapters are tied to the model family they were trained against. Budget for a retrain as part of adopting any new base model, and keep the dataset and caption files version-controlled so retraining is an hour rather than a project.
Where these numbers come from
The configuration values below are conventional starting points drawn from widely used community training recipes, offered as a baseline to iterate from rather than as tuned optima. Training behaviour varies by base model, trainer implementation and dataset; treat any specific number here as a starting hypothesis you will test.
Sources
- 01
Runpod GPU pricingRates for the card classes used in a training run
Who wrote this
AIOF Editorial DeskResearch & editorial
AIOF is an editorial desk that works from primary sources: platform terms, vendor pricing pages and published company data, read and dated rather than repeated from other write-ups. Where a figure comes from someone else we cite it and grade it. Where we have not measured something ourselves we say so, which today is most of the places you might expect a first-hand number.
- Works from primary platform documents, with the date each page was last checked published on the page itself
- Grades every figure as sourced, third-party estimate, modelled or measured, and labels which
- Publishes no first-party measured benchmarks yet, and says so on the methodology page rather than implying otherwise
- Corrections are made in place with a dated changelog entry, listed on the updates page
Rules and pricing here change often. This page was last touched on . Found something out of date? Tell us and we will fix it in place with a dated note.
Read next
Character consistency methods
The practical methods for keeping an AI character's face and body consistent across a content library, with the failure mode of each.
ProductionAnalysis6 minProductionAnalysis6 minCost per usable image
A model for the true cost of AI content production, based on keeper rate rather than raw generation cost, with published GPU and API rates.
EconomicsReference5 minEconomicsReference5 minOnlyNudes quick start
What a hosted AI studio costs per render, the sequence to follow, and the three signals that mean you have outgrown it. Worked from published credit prices.
ProductionTutorial6 minProductionTutorial6 min