How We Test the Quality of AI Responses at Greetio

At 10:47 p.m., a family messages a hotel: “We will arrive after midnight with our six-year-old. Can you prepare an extra bed? Our card did not go through, so please hold the room until morning.”

It looks like one simple request. It is actually a compact test of the entire guest-service system. The response has to preserve the correct date and party details, find the current late-arrival policy, understand the child and extra-bed rules for that particular room, distinguish a request to hold a booking from a confirmed hold, and handle a failed payment without inventing authority the system does not have.

An AI assistant can answer warmly, confidently, and in perfect prose—and fail every one of those checks. It may quote an outdated arrival time, promise a bed that does not fit the room, or say, “Your room is secured,” while no record has changed. The message feels like hospitality for a few minutes. The failure appears later, when the guest arrives or a staff member has to undo the promise.

That is why response quality cannot be reduced to how human the writing sounds. A high-quality answer is grounded in current hotel information, follows the property’s rules, understands the conversation, stays within its authority, and gives the guest a clear next step. If it claims that an action happened, the operational record must agree.

This article sets out the public standard against which Greetio responses should be assessed. It is not a claim of universal accuracy or an invented success rate. Available controls depend on the product version, connected systems, enabled actions, and each hotel’s policies. Honest evaluation starts with a defined scenario, a known version, and explicit limits.

The public Greetio website identifies the core safeguards: responses grounded in hotel data with visible sources, confidence thresholds, a choice between automatic replies, drafts, and human handoff, plus separate controls for risky topics. The sections below show how those principles can become a reproducible quality standard. They do not imply that every evaluation method described here appears as a separate control in the product interface.

The polished-answer trap

A single score hides distinctions that matter. Friendly language cannot compensate for a fabricated breakfast time. A correct cancellation policy does not help if it was applied to the wrong rate. “Done” is not a successful outcome if the requested task was never created.

We therefore separate response quality into dimensions. Some can be scored on a scale. Others must act as hard gates: a severe policy, privacy, or action failure should not disappear inside an attractive average.

The crucial distinction is between a writing flaw and a consequential failure. An unnecessary sentence can be edited. Awkward wording can be improved. A booking cancelled by mistake, a false payment assurance, or another guest’s information exposed in a reply requires a different safety mechanism. Those cases should fail the evaluation regardless of how strong the rest of the response appears.

Build a golden set that behaves like a small hotel

Systematic evaluation starts with a trusted set of test scenarios. The set should not be a collection of easy prompts designed to make the assistant look capable. It should be a small but faithful model of the hotel’s real operating world.

It should include common questions about check-in, breakfast, parking, children, pets, and services, alongside multi-intent messages, missing or conflicting details, spelling errors, transliteration, and mixed-language conversations. It also needs less frequent but costly failures: the wrong cancellation rule, a payment issue, a shifted date, false availability, or an action taken without guest confirmation.

Where privacy rules allow, cases can come from anonymized conversations and hotel specialists who know operational exceptions. Deliberately adversarial cases are useful too: requests to ignore a rule, disclose another reservation, “confirm now and sort out payment later,” or act without required information.

Each test needs more than an input message and a model answer. It should record the relevant policy and knowledge-base version, required facts, allowed and forbidden actions, expected end state, risk level, and handoff conditions. Without that frame, a test easily becomes a contest for the nicest sentence.

There should not be only one acceptable wording. “I’ll check that for you” and “Let me confirm what is available” can both be good. The evaluation should focus on meaning and outcome: Did the system find the correct rule? Did it avoid unsupported details? Did it complete only an authorized action? Did it explain honestly what it could not do?

OpenAI’s evaluation guidance recommends combining historical production examples, domain-expert cases, and deliberately created edge cases, then continuously expanding the test set with failures found in use. That advice is especially relevant to hotels, where rates, seasonal services, restrictions, and available integrations change over time. A trusted test set is a maintained asset, not a document completed once.

Pre-release testing: catch the error before a guest does

Whenever a model, instruction, knowledge source, retrieval rule, or available action changes, the same test set should be run against both the current and proposed versions. The purpose is not only to prove an improvement. It is to find what the improvement may have broken.

One run is not enough because generative systems vary. The same assistant can phrase an answer differently on repeated attempts, and one unstable decision inside a multi-step workflow can change the final outcome. Anthropic’s practical guide to evaluating agents recommends multiple trials, a distinction between tests that push new capability and tests that protect known behavior, and a combination of different grader types.

The first grader type is deterministic. Code can catch the wrong date, a missing required field, a forbidden action, an incorrect parameter, or a mismatch between a confirmation message and the booking state. If a reservation must never change without explicit confirmation, that condition can be tested objectively.

The second is a model-based grader working from narrow, written criteria. It is helpful for relevance, completeness, tone, and clarity, where several answers may be equally valid. But this grader is also an AI system. It needs an “insufficient information” option, careful calibration, and periodic comparison with human judgment.

The third is a hotel-domain reviewer. Human review matters most for exceptions, payments, cancellations, complaints, and decisions reserved for staff. A specialist may also spot an operational consequence the formal rubric misses, such as a technically cautious answer so vague that the guest must ask again.

No grader is sufficient alone. Deterministic checks are fast and reproducible but miss nuance. A model grader handles open language but can be confidently wrong. A hotel specialist brings context but cannot read every conversation. Reliability comes from layering them, not from choosing a single “smart” judge.

Test the journey, not only the final message

If the assistant only answers a policy question, evaluating the text and its sources may be enough. Once it checks availability, creates a task, changes a reservation, or hands a request to staff, the final sentence reveals only part of what happened.

The evaluation needs an end-to-end record: what information was retrieved, which policy was applied, which action was selected, which parameters were sent, whether the action returned success, what changed in the connected system, and what the guest was told afterward. OpenAI’s agent-evaluation documentation makes the same distinction by examining traces that include model calls, tool calls, safeguards, and handoffs—not merely the visible reply.

“Your room has been booked” is not evidence of a booking. The evidence is a corresponding record. “Housekeeping has been notified” is true only if the task exists for the correct room and time. If an external action fails, the assistant must distinguish intention from outcome: “I could not confirm the change, so I’m passing this to the team,” rather than “Everything is arranged.”

This principle is central to τ-bench, a research benchmark that evaluates conversational agents through policy compliance, tool use, and the final state of the environment. In hotel operations, the lesson is straightforward: the words and the operational truth have to match.

Grounding: every hotel fact needs a reliable home

For many guest questions, a general-purpose model should not rely on general knowledge. Check-in time, pet rules, breakfast availability, deposit amount, and the conditions of a particular rate must come from controlled, current hotel information.

Testing this requires at least two separate questions. First, did the system retrieve the relevant information, including exceptions and effective dates? Second, did the answer remain faithful to what was retrieved instead of adding a plausible detail? The RAGAs research framework approaches retrieval-augmented systems in this component-wise way: the relevance of retrieved context, the faithfulness of the answer to that context, and the quality of the generated response are distinct problems.

Conflicting sources need their own test cases. If the public website lists one arrival time and an internal instruction lists another, the assistant should not choose whichever version sounds more likely. A correct outcome may be to avoid a categorical claim, flag the conflict, and send it to the responsible staff member. Knowing when certainty is not justified is part of response quality.

Freshness matters as much as factuality. An answer can be perfectly grounded in a document that should no longer be used. Every knowledge source therefore needs an owner, a version or effective date, and a way to be retired. When a policy changes, the related cases should be rerun before the new information reaches live conversations.

Ukrainian and English are separate quality surfaces

An English test translated into Ukrainian is not yet a robust Ukrainian test. People express time, dates, politeness, refusal, urgency, and frustration differently across languages. Ukrainian messages may include transliteration, code-switching, local names for services, and familiar abbreviations. A literal translation can be grammatically correct while sounding unnatural or quietly changing the strength of a promise.

Each language needs scenarios written or reviewed by native speakers, a naturalness assessment, and a strict comparison of facts, amounts, dates, and policies. Tests should cover switching language mid-conversation, keyboard-layout mistakes, and ambiguous date formats. Results should be reported by language; a blended score can conceal weak Ukrainian behavior.

The Multi3WOZ study on multilingual task-oriented dialogue documents the limitations of translation-derived datasets, including translation artifacts, non-native language, and missing cultural adaptation. Its authors used language-specific, native-speaker localization rather than treating multilingual quality as a translation afterthought. The same principle applies here: each Greetio language experience should be evaluated as a service in its own right.

Cross-language consistency and native quality should also remain separate dimensions. Two answers can contain exactly the same facts while one is stiff or culturally inappropriate. Conversely, a beautifully localized answer can quietly alter a deadline or condition. Both checks are necessary.

Post-launch monitoring: real guests find the missing cases

Even a strong test set cannot anticipate every phrase, policy conflict, or operational situation. After release, teams need ongoing observation with appropriate personal-data protection, restricted access, and a defined retention policy.

Useful signals include a repeated question, a staff correction, a reopened conversation, a failed tool call, contradictory sources, a premature success message, or a missed human handoff. The same intent should not produce materially different outcomes simply because one guest wrote in Ukrainian and another in English.

Bookings, response speed, and positive reactions are valuable business signals, but none proves correctness alone. Conversion may rise because of seasonality, a discount, or a marketing campaign. A guest may like the friendly tone without knowing that the policy was quoted incorrectly. A fast false answer is worse than an accurate answer delivered a few seconds later.

The NIST Generative AI Profile warns against extrapolating broad capability claims from narrow, anecdotal assessments. It recommends source verification, tracking confidently stated falsehoods, documenting versions, recording errors and near misses, and maintaining post-deployment monitoring. ISO/IEC 42001 similarly treats AI management as a continuing cycle of governance, performance evaluation, accountability, and improvement rather than a one-time launch test.

The improvement loop should be closed. A detected failure receives a category and root-cause review. The team changes the relevant source, instruction, routing rule, or action. The real case becomes a permanent test. Related old tests run again to ensure the local repair did not cause another regression.

Give every failure a precise name

“The AI gave a bad answer” is not actionable. A source failure is different from a retrieval failure. A misunderstood date is different from an unsupported claim. A policy violation is different from a tool outage.

A useful failure taxonomy covers missing, stale, or conflicting sources; failed retrieval; misunderstood intent, date, party size, or negation; unsupported claims; policy breaches; incorrect actions or parameters; words that disagree with the final state; unclear next steps; localization errors; and missed human handoffs.

The label determines the repair. If the policy is stale, changing the response tone achieves nothing. If the correct rule was found but the wrong date reached the booking tool, the extraction and confirmation steps need attention. If Ukrainian wording is unnatural while every fact is correct, the solution is language-specific authoring and review, not a complete rebuild of the knowledge base.

Severity matters alongside category. A slightly repetitive answer and an unauthorized cancellation may both count as failures, but they do not deserve the same release decision. High-severity cases should block deployment or trigger immediate containment even when they are rare.

What an honest evaluation system cannot promise

No number of tests proves that an AI assistant will never be wrong. A trusted set covers known scenarios, not every future guest. Automated graders can misclassify an answer. Human reviewers disagree. Guest behavior changes. A model, policy, integration, or knowledge update can introduce a new failure.

That is why a claim such as “99% accurate” says very little unless it names the dataset, languages, system version, success criteria, and critical failures. A responsible quality report should state what was tested, when, under which configuration, in which languages, what counted as success, how severe failures were handled, and where human approval remained necessary.

This limitation is not an argument against AI. It is a sign of a mature operating model. Trust in hotel service does not come from a promise of perfection. It comes from predictable behavior: the system can show where a fact came from, stays inside policy, verifies the result of an action, acknowledges missing information, and involves a person when the risk exceeds its authority.

Back to the 10:47 p.m. message

A high-quality system retrieves the arrival and child-accommodation rules, checks whether an extra bed fits, separates the failed payment from a confirmed hold, and waits for verification before calling the room secured.

If an exception is required, it hands the conversation to a staff member with the context already collected instead of making the guest start again. Its reply explains, briefly, what is known, what has been checked, what is still missing, and what will happen next.

That is the quality target: not a flawless imitation of a person, but dependable assistance that survives contact with facts, policies, and operational reality.

A practical starting point for a hotel

Choose ten of your most frequent guest questions and ten situations where a mistake would be especially costly. For each one, identify the current source, permitted and forbidden actions, expected outcome, and handoff condition. Run the same scenarios several times in both Ukrainian and English.

When evaluating any hotel AI product, bring anonymized situations from your own operation rather than relying only on a prepared demonstration. Ask to see not just the final answer but also the source of the fact, the limits of authority, and proof of the claimed action. That is how “smart service” becomes testable service quality.

Frequently asked questions

Is manually reading a few good responses enough?

No. Manual review is valuable for tone and context, but a few convenient examples do not reveal consistency, rare high-impact failures, or regressions after a change. Repeatable scenarios, explicit success criteria, and version comparisons are needed.

Can an AI grader replace human review completely?

No. A model-based grader can scale assessments of completeness, relevance, and tone, but it can also be wrong. Hotel specialists remain important for payments, cancellations, complaints, safety, exceptions, and for calibrating automated judgments.

Why verify the system state if the response itself is correct?

Because correct wording is not proof that an action happened. If the assistant says a booking, task, or date change is confirmed, a corresponding operational record must exist. Otherwise the guest has received a false assurance.

Why not translate the English tests into Ukrainian?

Translation does not reproduce the full range of native phrasing, mixed-language messages, local service names, or cultural nuance. It can also alter how firm a refusal or promise sounds. Ukrainian cases should be authored or at least reviewed by native speakers.

Which quality metric matters most?

There is no universal single metric. For a low-risk information request, factuality and clarity may dominate. For a payment or cancellation, policy compliance, action correctness, and verified outcome come first. Critical failures should not be averaged away by good tone or fast response time.