Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
AssemblyAI gets about 1,000 API sign-ups every single day. Until two days before this conversation, Matt Lawler was the only onboarding engineer at the company. That is a ratio no amount of hustle can fix, and Lawler’s response to it is the kind of statement that would get him side-eye at most companies: he told his team to automate him out of a job.
Speaking on the AI Engineer podcast, Lawler — a forward-deployed engineer, or FDE, at the speech-to-text infrastructure company — described how his team scrapped an off-the-shelf support bot that resolved 10% of tickets and replaced it with a custom agent named Joey. Joey now resolves 80% of inbound conversations end-to-end, with no human in the loop, at a cost of about $700 a month. The story matters beyond AssemblyAI because it lands in the middle of a broader shift: companies like Meta, Crypto.com, and Robinhood are racing to deploy autonomous agents for customer-facing tasks, and most of them are discovering that the gap between a chatbot that answers questions and an agent that actually finishes a job is wide. AssemblyAI’s response to that gap was to own every layer of the stack.
AssemblyAI trains its own foundation models for speech-to-text and sells voice AI infrastructure to companies building everything from meeting note-takers to real-time phone agents for drive-through ordering. Lawler’s job as an FDE is to get embedded with those customers — learn their use case, sometimes commit code directly to their repositories, and win the business. It is a high-touch model that works beautifully for relationship depth and fails catastrophically for scale.
At roughly 1,000 sign-ups per day, the math is unforgiving. Lawler was the sole onboarding engineer until two days before the talk. The company could not hire its way out of the problem, and it could not hand-build every customer win. His conclusion is deliberately provocative:
“I would encourage you to take that knowledge that you have as an FDE and try to automate yourself out of a job. I think that should be your role every single day.”
The logic is not self-sabotage. It is a recognition that the FDE’s real value is not being a human router for support tickets — it is understanding the product deeply enough to teach a machine how to do the routing. As Lawler put it:
“If you want to deliver a better customer experience at scale, you can’t be the bottleneck to having a good customer experience.”
The team’s first move was deflection: buy a support bot, point it at the documentation, and hope it absorbs the simple questions. It did not. The off-the-shelf bot resolved about 10% of conversations end-to-end. At 1,000 conversations a day, that meant roughly 100 handled by the bot and 900 still fielded by humans.
The deeper failure was control. The vendor owned the system prompt, the tools, and the RAG infrastructure. Every change request went to the vendor and came back as “it’s on the roadmap.” That inability to iterate — not the 10% number itself — is what pushed AssemblyAI to build its own agent. The distinction matters because it reframes the buy-vs-build debate. The problem was never that the bot was bad at answering questions. The problem was that AssemblyAI could not make it better.
Lawler broke Joey’s architecture into four parts, and the design choices reveal why the agent outperforms the vendor bot by 8x.
First, documentation lives as local Markdown files. All AssemblyAI docs are checked out locally and kept in sync whenever the docs system updates. Changelog, pricing, and major website pages are converted to Markdown. If the entire doc site went down, Joey could still answer questions about the newest shipped features from his own file system.
Second, retrieval runs through Voyage embeddings. AssemblyAI partners with Voyage to surface the most relevant documents up front, cutting search time and improving answer latency.
Third, Joey has agentic file-system search. Because the docs are local files, the agent can stitch together multiple resources when a question requires it — what Lawler describes as a “deep thinking mode.”
Fourth, deployment happens on Railway. The agent is containerized in a Docker file and deployed on Railway so the team avoids managing EC2 instances. A pull request can be written and deployed in about 30 seconds.
The behavioral glue is a Claude MD file Lawler describes as roughly 30,000 lines of guardrails and operating advice, updated every time Joey gives a wrong answer or has a bad customer interaction. Joey runs on the Claude Agent SDK and can manage his own infrastructure, write and debug code, call tools, and give himself new abilities — operating more like an FDE than a website chatbot. He is also a member of the support team in Pylon, which tracks his metrics and lets him remember past conversations.
The headline number is the jump.
“We went from 10% to 80% end-to-end resolution rate in just the first week of deploying this build, with a pretty naive implementation.”
And the cost:
“So we went from 10% to 80% end-to-end resolution rate in just the first week of deploying this build, with a pretty naive implementation. And we did that all for around $700 a month in both token and infrastructure costs.”
Lawler preempts the skepticism that these figures are gamed by filtering easy tickets. Customers cannot reach a human directly unless they go through Joey and ask him to escalate. Joey handles 100% of inbound tickets and escalates only 20%. That escalation list is substantive: rate changes, data opt-outs, and agreements that require a signature — not comprehension failures. Lawler frames the residual 20% as a to-do list. If Joey cannot send a BAA, give him a link to reference. If he cannot discuss pricing, teach him. Customers can already negotiate with Joey on the site today — tell him how many hours you have and he will quote a rate, and you can haggle from there.
The reason AssemblyAI added voice to Joey is dogfooding in its purest form. The company works in voice, and its customers build voice agents. So the team built one themselves. The voice agent API was integrated into Joey during the week of the talk, letting visitors switch to voice mode on the site and talk to him speech-in, speech-out over a single WebSocket connection that strings together AssemblyAI’s speech-to-text model, an LLM, and text-to-speech. It handles pauses, interruption handling, and barge-in. The same tools available in text mode are available in voice.
Lawler demonstrated the feature with a medical scribe scenario. Asked whether he can execute a BAA for an ambient medical scribe build, Joey answers:
“Yes, you can execute the BAA yourself right away. Just complete our standard HIPAA business associate agreement. But note that you’ll need a paid account with a card on file first. And signing the BAA automatically opts you out of model training. The relevant docs are linked in the chat.”
Two years earlier, that conversation required a human. If no support engineer was immediately available, the customer waited for a canned response. Now a customer asking Joey how to build a voice agent is already using one. That is the meta layer Lawler highlights — the support interaction is itself a sales demo of the product being evaluated. The team planned to hook Joey up to a phone number within days of the talk.
Lawler’s closing argument is that FDEs should build the exact product their customers are building, on their own stack, rather than waiting to be asked for help. He had to work through the same latency, turn-taking, and interruption-handling problems his customers face, and that gives him better empathy and better advice because he shipped something with the same API.
“I ultimately think for any FDE here, no matter how much you’ve been deployed with your customers, no matter how embedded you are with them, the best way to understand what your customers are building is to actually build their same product yourself.”
The operational proof is the iteration loop. The team has monitored a live conversation, caught a bug, deployed a fix, and had it approved for the rest of the session — without the customer knowing a fix shipped behind the scenes. That is the pace that a vendor-controlled bot cannot match, and it is the real return on owning the stack.
“It’s not enough to just be teaching your customers how to get started with it. Any new feature that you launch, any new product that you offer, you need to be teaching your customers with all the knowledge that you’ve built from actually using and building and trying to ship the same product.”
The unresolved thread is the 20% Joey still escalates. Lawler treats it as a roadmap rather than a ceiling, and says there is “plenty more to come.” But the categories — rate changes, data opt-outs, signed agreements, legal and FDE involvement — are exactly the ones where automation carries commercial and compliance risk, not just engineering difficulty. The $700-a-month figure covers token and infrastructure costs only; the engineering time to build and maintain the 30,000-line guardrail file is where the real ongoing investment sits.
For anyone evaluating autonomous agents in production — whether a support bot, a personal AI assistant, or a trading agent — the AssemblyAI result offers a concrete benchmark. A custom-built agent with full control over prompt, tools, retrieval, and deployment outperformed an off-the-shelf bot by 8x at a cost that is effectively a rounding error for most software budgets. The tradeoff is that you own the guardrails, the iterations, and the residual risk. That is a trade most companies say they want but few are willing to make. The ones that do — and that staff it with engineers who treat their own automation as the product — will be the ones that stop treating customer support as a cost center and start treating it as a live stress test for every feature they ship.
References:
Once added, BigGo Finance appears first in Google Search Top Stories, so you get the broadest, most up-to-the-minute, and most comprehensive global financial news first.
The news and data on this website are for reference only and do not constitute investment advice or an offer to buy or sell. Information is sourced from exchanges and public sources, and may be delayed, interrupted, or updated. While we strive for accuracy, we do not guarantee timeliness, correctness, or completeness. Content may include external links for which we are not responsible. By using this site, you agree that we and our partners are not liable for any losses. Investment carries full responsibility; please carefully assess risks and consult professionals. If there are errors in the content, please contact us for correction.
