The most important sentence in Laya's documentation is the one about zero-shot performance. Install the package, run it on your data, and you will see near-random accuracy. That is expected. The model is a base to fine-tune, not a service to call.
Step 1 — Install and confirm the environment
pip install layaThree checkpoints are involved: an English encoder (~421M), a multilingual encoder (~322M) and a language router. Loading all of them at once costs roughly 2 GB of memory; on edge devices load only the checkpoint you need.
Interface details are still moving quickly, so check the repository README for the current call signature rather than copying snippets from blog posts — including this one.
Step 2 — Collect labelled data before anything else
Budget for this honestly. You need enough examples per category for the model to learn a decision boundary, and they must be labelled consistently. Merge or rename categories whose distinction your own labelers cannot agree on.
Two rules that prevent most failures:
- Write the criteria table first. The options you define *are* the task.
- Hold out a slice you never train on. Without it you cannot detect overfitting.
Step 3 — Fine-tune
Training is the short part. The tooling is open, and the model is small by modern standards, so a single GPU is enough for most tasks.
Step 4 — Fit temperature on your own data
This is the step people skip, and it is the one that determines whether you can safely automate.
Raw checkpoints are overconfident. Reported calibration error drops from around 0.466 to around 0.081 after temperature fitting. Fit per question type and per option count — a single global temperature is a compromise that leaves one class of question badly calibrated.
If your confidence numbers are not calibrated, an automation threshold is a coin flip with a percentage sign attached.
Step 5 — Evaluate on the axis that matters
Accuracy is the least useful metric for a routing system. Track instead:
- Calibration error on held-out data
- Accuracy at your threshold, not at the argmax
- The cost asymmetry of each error type — routing abuse to sales is not the same size of mistake as the reverse
Realistic effort
Teams that have done this before report days, not weeks, for a narrow task with clean labels. The variable is data, not compute. If you cannot produce a few thousand consistent labelled examples, the fine-tuning path is not the right one for you yet.