LangChain Part 2: Structured Answers for a FAQ Desk

In part 1 Kora Desk returned a string. That is fine for a CLI demo. It is awkward the moment Nest or Expo needs to decide what to show: green badge, yellow "maybe", or "escalate to a human". Parsing prose with regex is how FAQ desks quietly invent confidence levels.

This is part 2 of the five-part Kora Desk build. We keep the same fictional FAQ assistant for Nigerian small businesses, still without retrieval. Today we shape the reply as typed data: an answer string and a confidence float the product code can trust.

Why free text stops working

A Nest handler that gets "Yes, you should register with CAC before opening the account." has no contract. Is that legal advice? How sure is the model? Should the UI show a disclaimer?

If the chain returns JSON that matches a schema, the TypeScript side can validate once and branch:

• confidence >= 0.8: show the answer
• 0.4 <= confidence < 0.8: show answer + "verify with a professional"
• confidence < 0.4: show escalate CTA, hide the guess

That decision belongs in your API, not in another prompt.

Messages, not one blob

Part 1 used a system string and a human question. For FAQ tone you usually want three roles:

1. System: who Kora Desk is, what it must refuse, output shape reminders.
2. Few-shot human/AI pairs: short examples of good shape (not a second knowledge base).
3. The real user question.

Few-shot is for format and refusal style. Do not paste your entire CAC handbook into examples. That belongs in RAG (part 3).

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are Kora Desk, a concise FAQ assistant for Nigerian small businesses. "
            "Answer only from general business-registration knowledge. "
            "If unsure, lower confidence and say what is missing. "
            "Never invent CAC fees, deadlines, or legal outcomes.",
        ),
        (
            "human",
            "Do I need a TIN before I register a business name with CAC?",
        ),
        (
            "ai",
            '{{"answer": "A TIN is usually obtained after registration for tax use; '
            'confirm current CAC and FIRS steps for your case.", '
            '"confidence": 0.55}}',
        ),
        (
            "human",
            "What is the exact CAC fee for a limited company today?",
        ),
        (
            "ai",
            '{{"answer": "I do not have a verified live fee schedule. '
            'Check the official CAC portal or a registered agent.", '
            '"confidence": 0.2}}',
        ),
        ("human", "{question}"),
    ]
)

Double braces in the AI examples escape curly braces so LangChain does not treat them as template variables. The live {question} stays single-braced.

Define the contract with Pydantic

Pick a small schema and keep it stable. Product code should depend on field names, not on whatever prose the model prefers this week.

from typing import Literal

from pydantic import BaseModel, Field


class DeskReply(BaseModel):
    answer: str = Field(description="Short reply the user can read.")
    confidence: float = Field(
        ge=0.0,
        le=1.0,
        description="0 = guess / refuse; 1 = high certainty for this FAQ.",
    )
    stance: Literal["answer", "uncertain", "refuse"] = Field(
        description="Coarse label for UI branching."
    )

stance is optional sugar. You can derive it from confidence in Python if you want a smaller model surface. I keep it when the UI needs an explicit refuse path that is not just "low number".

Wire structured output into the chain

Modern LangChain chat models expose with_structured_output. The model is instructed (and, when the provider supports it, constrained) to fill your schema.

from langchain_ollama import ChatOllama

model = ChatOllama(model="qwen2.5:7b", temperature=0)
structured = model.with_structured_output(DeskReply)

chain = prompt | structured

reply: DeskReply = chain.invoke(
    {
        "question": (
            "In one short paragraph: why do people register a business name "
            "with CAC before opening a business bank account?"
        )
    }
)

print(reply.model_dump())
# {'answer': '...', 'confidence': 0.72, 'stance': 'uncertain'}

Hosted variant (same chain shape):

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4.1-mini", temperature=0)
structured = model.with_structured_output(DeskReply)
chain = prompt | structured

Keep temperature=0 for FAQ desks until you have evals. You want stable JSON, not creative confidence scores.

What to do when the model ignores the schema

Even with structured output you will see:

1. Missing fields on weaker local models. Catch ValidationError, retry once with a tighter system line, then return a hard-coded refuse payload to the client.
2. Confidence always 0.9. Few-shot refuse examples help. So do evals in part 5 that fail the run if every fixture scores above 0.85.
3. Answer contradicts stance. Trust stance / low confidence for UI. Log the mismatch for later prompt fixes; do not show a bold answer next to refuse.

A thin guard in your Python service:

from pydantic import ValidationError


FALLBACK = DeskReply(
    answer="I cannot answer that from verified FAQ material. Please escalate.",
    confidence=0.0,
    stance="refuse",
)


def ask_desk(question: str) -> DeskReply:
    try:
        reply = chain.invoke({"question": question})
    except (ValidationError, Exception):
        return FALLBACK
    if reply.stance == "refuse" or reply.confidence < 0.4:
        return reply.model_copy(
            update={
                "answer": reply.answer
                or FALLBACK.answer,
            }
        )
    return reply

Nest still owns auth and rate limits. It receives DeskReply JSON over HTTP (part 5). Do not call Ollama from the Expo app.

Few-shot hygiene

• Three to five examples max. More examples burn context and start acting like a brittle knowledge base.
• Include at least one refuse and one uncertain example. If every shot is a confident answer, the model copies that energy.
• Keep examples short. Long essays teach verbosity.
• Refresh examples when you change the schema. Old AI messages that omit stance train the wrong shape.

How this fits the series

Structured output does not make the desk truthful. It only makes the lie machine-readable. Retrieval and evals still have to earn the confidence number.

Takeaways

• Product code needs a schema (answer, confidence, optional stance), not a paragraph to scrape.
• Use chat messages plus a few refuse/uncertain shots for format, not for encyclopedic knowledge.
• with_structured_output(DeskReply) keeps LCEL short: prompt | structured.
• On parse failure, return a refuse fallback; never ship half-parsed JSON to mobile.
• Link back to part 1; next is RAG over your own markdown folder.