The Analogy: The Author and The Editor

Suppose you were authoring a book. A critical step in this process is the submission of your work to your editor — the proxy between you and the world eager to engage with your writing. Your editor is tasked with correcting errors, suggesting stylistic improvements, and grading comprehensibility, amongst other things.

There’s nothing novel or uniquely special about this process, and it has been the norm from draft, to draft, to published, for a long time. In fact, it is the obvious and expected process for anything that is to be presented to the world as “finished”; the peer-review, the draft revision, the QA process.

Introducing "Evals"

But when it comes to Generative AI and LLM application, it’s a very easy step to overlook. Important, but overlooked nonetheless. We call this process Evaluation, and refer the process and the metrics/dimensions by which we grade the AI output in their entirety as evals for short.

Just as you wouldn’t publish a book without any oversight or peer review, neither would you want your Generative AI system (especially one that takes real action) to function without oversight. The primary reason being that things go wrong, and it is valuable to have a basis by which to define what went wrong, and where.

Real-World Application: The AI Receptionist

Consider an example of an AI receptionist phone agent that is able to answer questions, qualify callers, and schedule meetings. You may wonder how evals are useful, and why, which are great questions.

One dimension that has the highest impact in this application is verbosity.

• Reading a paragraph of text doesn’t take long.

• Listening to a paragraph, however, does, and it’s one of the main areas that users complain about.

AI can be overly verbose, especially in voice applications, and this alone can be the difference between acceptable and “sounds too robotic.”

The Solution: Gold Standards

So how do we fix this? It’s important to define by what measure we are judging verbosity so that we can understand how to resolve it. In essence, what does a “normal” phone conversation sound like?

One strategy involves creating reference data, or Gold Standards; conversations that exemplify what an ideal, natural phone conversation sounds like. Then, by comparing the transcript of a real example to the gold standards and grading it against a rubric, you can evaluate the output of the system. Congratulations, you’ve implemented a basic eval!

You can imagine that in this conversational example you have your gold standards that represent the most ideal phone conversations, graded against a rubric that may contain dimensions such as:

• Verbosity

• Objective adherence

• Compliance and safety

• Tone

• Whatever else you may wish to measure and improve.

With enough gold standards, and enough real conversation examples to grade, you can assess the performance of the system to a high resolution, and use these findings to tweak the LLM.

As the old saying goes, you cannot improve what you don’t measure.

Consistency and Functionality

Furthermore, for any action this receptionist agent may take (transferring calls or scheduling meetings, for example), you can have both the input and output passed through evaluators to track the consistency of the payloads.

In real systems you may come to find that perhaps the data that the LLM passes to the tool to schedule a meeting may be of an incorrect format — and APIs are rigid so they expect the same format of data all the time — thus resulting in a failure to book the meeting. Having an eval system monitoring these outputs can flag failures, providing clarity on when it failed, why it failed, and overtime, and idea of how often it fails.

Of course, most, if not all, improvements will be made to the system prompt, but even a simple eval framework provides an insight into the quality and consistency of your AI product outputs and not too dissimilar to “fine tuning”, it allows you to track the impact of prompt changes over time to perfect the system for the desired outcome.

Changes without evals are like shooting in the dark.

Scaling the Process

Now that you understand the absolute basics, you can imagine how to do this full scale (there are definitely components that require arduous human hours e.g. the development of these gold standards — the more examples the better — and the grading rubric).

For example, you may:

1. Create a strong and varied set of human-defined gold standards.

2. Use an LLM to generate additional comparable examples.

3. Automate the handoff of conversation transcripts to another LLM as the evaluator, tasked to compare the chosen conversation to the gold standards, grade them, and even return feedback/suggestions.

You may also choose to utilize utilities such as Phoenix which is a utility built for evals.

Conclusion

Ultimately, regardless of how you choose to start, some evals are better than no evals, and the sooner you begin, the more data you have to refine your application.

Just like compound interest, the more time you can monitor and average the output, the more clarity you will have around which components of your application need help, and finally resolve the “sounds too robotic” support ticket.