Describe a good result
Quality means different things for different tasks. An answer based on internal manuals must be correct and link to the current source. A customer-service draft must also be clear, use the appropriate tone and avoid promises the organisation cannot keep. A system that sorts enquiries must use the categories employees actually work with.
Choose a small number of requirements that can be assessed consistently. Write one example that passes, one that needs correction and one that must be rejected. This makes it easier to discover when two specialists interpret a requirement differently.
Build a test set that resembles reality
Collect cases from the period, service and user group the system will encounter. Include common cases, rare but important exceptions, incomplete questions, outdated sources and phrases users actually write. Remove or anonymise any personal data you do not need.
Label each example with what you expect: a particular answer, essential facts, the correct category or a request for human help. Keep the test set with a version number and date so that you can run the same tests after changes.
Ask specialists to assess and explain the results
Automated checks can validate formatting, missing links or the presence of particular facts. They cannot determine on their own whether advice is appropriate for a specific case. Ask people who know the subject to assess a representative sample.
Record both the rating and the reason. If two people judge the same answer differently, clarify the criterion before adjusting the model. The disagreement may reveal a missing organisational rule rather than a poor AI system.
Test the whole system, not only the text
- Do users understand what the system can and cannot help with?
- Can they see which sources support the answer?
- Can they correct, reject and pass the case on?
- What do they see when a source is missing or the system is unavailable?
- Are necessary edits and approvals recorded?
Measure the time employees spend on the complete task as well. If a draft arrives quickly but checking and correcting it takes longer than before, the system has not produced the improvement you wanted.
Keep testing after launch
Sources, models and user behaviour will change. Run the test set when you switch models, change instructions or update important sources. Monitor which suggestions employees correct or reject and which cases have to be handled manually.
I can help you build the test set, incorporate assessment into the system and monitor quality over time. The work is tailored to what employees or customers will actually use AI for.