Building a Multimodal Support Agent

Part of a series on running AI workflows across model providers — this one is the actual build.

The earlier post in this series on Gemini's multimodal handling made the case in the abstract. This one builds the thing: a support agent that takes a customer's screenshot and their text description together, classifies what's actually wrong, and decides whether to answer directly or hand off to a human — using DNotifier's real defineAgent and Workflow primitives with Gemini doing the multimodal reasoning.

The shape of it

[ Text + optional image ] ──► Triage agent (Gemini Flash)
         │
    ┌────┴────┐
    ▼         ▼
Resolution  Escalation + handoff note

One multimodal call replaces what would otherwise be a separate image pipeline stitched to a text pipeline.

A message comes in with an optional image attachment. A triage agent — the one actually looking at the image, if there is one — classifies the issue and decides whether it's a known, safely-automatable fix or something that needs a person. Known issues get a drafted resolution. Everything else gets flagged for a human, along with the triage agent's reasoning, so the handoff isn't a cold start for whoever picks it up.

Step 1: Define the triage agent

import { DNotifier } from "@dnotifier-realtime/dnotifier";

const triageAgent = DNotifier.defineAgent({
  name: "triage-agent",
  model: "gemini-3.8-flash",
  async run(ctx) {
    const messageContent = {
      text: `A customer sent this support request: "${ctx.input.text}"

      Classify it into one of: billing-question, technical-error, account-access, other.
      If an image is attached, use what's actually shown in it to inform the classification —
      for example, an error code visible in a screenshot is more reliable than the customer's
      own description of it.
      Then rate your confidence (high/medium/low) and state briefly why.`,
    };

    if (ctx.input.imageUrl) {
      messageContent.attachments = [{ type: "image", url: ctx.input.imageUrl }];
    }

    const result = await ctx.sendAI({ message: messageContent });
    ctx.state.triage = result.text;
    return result.text;
  },
});

Flash is doing the triage step here deliberately — it's a classification task, run on every incoming message, where speed and cost matter more than deep reasoning. The multimodal capability doesn't require a heavier model; Gemini's native image handling is available across the lineup, not gated to the most expensive tier.

Step 2: Define the resolution and escalation paths

const resolutionAgent = DNotifier.defineAgent({
  name: "resolution-agent",
  model: "gemini-3.8-flash",
  async run(ctx) {
    const draft = await ctx.sendAI({
      message: {
        text: `Draft a helpful, specific response to this customer issue.
        Triage notes: ${ctx.state.triage}
        Original message: "${ctx.input.text}"`,
      },
    });
    ctx.state.draftResponse = draft.text;
    return draft.text;
  },
});

const escalationAgent = DNotifier.defineAgent({
  name: "escalation-agent",
  model: "gemini-3.8-flash",
  async run(ctx) {
    const summary = await ctx.sendAI({
      message: {
        text: `Write a short handoff note for a human support agent, summarizing:
        the customer's issue, what the triage step found, and why this needs a person
        rather than an automated response.
        Triage notes: ${ctx.state.triage}`,
      },
    });
    ctx.state.handoffNote = summary.text;
    return summary.text;
  },
});

Step 3: Wire it into a workflow with a real decision point

const supportWorkflow = new DNotifier.Workflow({
  name: "multimodal-support-triage",
  description: "Triages support requests, using image context when available, and routes to automated resolution or human escalation",
  observability: true,
  async entry(ctx) {
    await ctx.agents.run(triageAgent, {
      text: ctx.input.text,
      imageUrl: ctx.input.imageUrl,
    });

    const isConfidentAndSimple =
      ctx.state.triage.toLowerCase().includes("high") &&
      !ctx.state.triage.toLowerCase().includes("account-access");

    if (isConfidentAndSimple) {
      await ctx.agents.run(resolutionAgent, { text: ctx.input.text });
      return { path: "automated", response: ctx.state.draftResponse };
    } else {
      await ctx.agents.run(escalationAgent, { text: ctx.input.text });
      return { path: "escalated", handoffNote: ctx.state.handoffNote };
    }
  },
});

supportWorkflow.registerAgents([triageAgent, resolutionAgent, escalationAgent]);

Account-access issues route to a human regardless of confidence — deliberately. Getting someone locked out of or into an account wrong has a different risk profile than a billing question, and that's a judgment call worth encoding explicitly rather than leaving to the model's general confidence score.

Step 4: Run it against a real ticket

const notifier = new DNotifier({
  appId: process.env.DNOTIFIER_APP_ID,
  secret: process.env.DNOTIFIER_SECRET,
  userId: "support-system",
  transport: "ws",
  WebSocketImpl: WebSocket,
});

await notifier.connect();

const outcome = await notifier.runWorkflow(supportWorkflow, {
  text: "I tried to check out and got this error, not sure what it means",
  imageUrl: "https://cdn.example.com/tickets/4471/screenshot.png",
});

if (outcome.path === "automated") {
  console.log("Sending response:", outcome.response);
} else {
  console.log("Routing to human, handoff note:", outcome.handoffNote);
}

The screenshot is doing real work here — a checkout error code visible in the image is exactly the kind of detail that turns "the customer says it's broken" into "here's specifically what's broken," and it's the triage agent seeing that directly rather than working from the customer's own paraphrase of what the error said.

Why this beats a separate vision pipeline

One call, one latency hit. A traditional pipeline — OCR or a captioning model first, then a separate text-classification call — means two round trips and two places for something to go wrong. This is one.

Nothing gets lost in translation. The triage agent reasons over the actual screenshot, not somebody's earlier decision about what was worth extracting from it into a caption.

It's still cheap. Using Flash for both triage and resolution keeps the whole pipeline inexpensive per ticket, reserving heavier models for the cases that are actually hard — which, per the workflow above, get routed to a human anyway rather than a more expensive model call.

What to add before this is production-ready

This example keeps things simple on purpose. A real deployment would want: a confidence threshold that's actually tuned against your own historical tickets rather than an assumed "high/medium/low" self-report, rate limiting against low-quality repeated attachments, and almost certainly a feedback loop where human corrections to escalated tickets get used to improve the triage prompt over time. None of that changes the core shape — it's refinement on top of the same three-agent pattern.

Frequently asked questions

What if the customer doesn't attach an image?

The triage agent handles that case explicitly — imageUrl is optional, and the workflow runs fine on text alone. Multimodal input strengthens the triage step when it's available; it isn't a requirement for the pipeline to function.

Could this use a different provider for one of the three agents?

Yes — each defineAgent call sets its own model, so nothing stops mixing providers here the same way the earlier research-agent build in this series mixed Claude models. This example keeps everything on Gemini because the multimodal triage step is the part that specifically benefits from it.

How do you decide the confidence threshold for automation versus escalation?

Start conservative — route more to humans than you think you need to — and loosen the threshold only after comparing automated responses against what a human would have actually said, on real historical tickets. Guessing a threshold and shipping it is how automated systems earn a bad reputation fast.

Does DNotifier's observability show the image that was sent, not just the text?

Yes — attachments are logged alongside the rest of the call in the observability dashboard, so reviewing why a triage decision came out a certain way includes seeing exactly what image the model was looking at.

Is this pattern specific to support tickets?

No — the shape (multimodal triage → confident automation or human handoff) generalizes to anything where visual context changes the right response: insurance claim intake, quality-control photo review, content moderation with an image component.

The Bottom Line

Next in this series: the practical question of which Gemini model to actually reach for — Flash, Pro, or Deep Think — and when the more expensive option is genuinely worth it.

Read part one: Gemini on DNotifier: Setup Guide. Explore dnotifier.com.