AI agents can browse the web, write code, and orchestrate entire workflows, but the moment one needs to make a phone call or receive a text, the abstraction breaks. You’re suddenly stitching together a telecom provider, a speech-to-text pipeline, a text-to-speech pipeline, conversation state, and compliance tooling, none of which were designed with an LLM as the thing on the other end of the line.
AgentPhone.ai, a YC-backed startup founded by brothers Meet and Manav Modi, is betting the fix is a single API and a single webhook shape for both channels. The pitch is simple on the surface: one phone number, one event format, voice and SMS normalized into the same envelope. What’s more interesting is what’s actually happening underneath that pitch, because the architecture is publicly documented, and it’s more specific than most infra startups let you see before you sign up.
This is a walkthrough of that architecture: how the webhook contract is shaped, how voice and SMS get unified without losing channel-specific detail, how the hosted-vs-webhook voice split works, and where the real engineering tradeoffs live.
The core abstraction: one event, many channels
The foundational design decision is this: every inbound message, regardless of whether it arrived as SMS, iMessage, or a live voice transcript, is delivered to your webhook as the same top-level event type, agent.message. A channel field tells you the source. This is the thing that makes “unified webhook” more than a marketing phrase, because it’s a real constraint on how the backend must be structured, not just a docs page framing.
Here’s the shared envelope for an inbound SMS:
{
"event": "agent.message",
"channel": "sms",
"timestamp": "2025-01-15T12:00:00Z",
"agentId": "agt_abc123",
"data": {
"conversationId": "conv_def456",
"numberId": "num_xyz789",
"from": "+15559876543",
"to": "+15551234567",
"message": "Hi, I need help with my order",
"mediaUrl": null,
"direction": "inbound",
"receivedAt": "2025-01-15T12:00:00Z"
},
"conversationState": {
"customerName": "Jane Doe",
"orderId": "ORD-12345"
},
"recentHistory": [
{ "content": "Hello", "direction": "inbound", "channel": "sms", "at": "2025-01-15T11:59:00Z" }
]
}
And here’s the same event type carrying a live voice transcript instead:
{
"event": "agent.message",
"channel": "voice",
"timestamp": "2025-01-15T14:00:05Z",
"agentId": "agt_abc123",
"data": {
"callId": "call_abc123",
"numberId": "num_xyz789",
"from": "+15559876543",
"to": "+15551234567",
"status": "in-progress",
"transcript": "I need help with my order",
"confidence": 0.95,
"direction": "inbound"
},
"conversationState": null,
"recentHistory": [
{ "content": "Hello, how can I help?", "direction": "outbound", "channel": "voice", "at": "2025-01-15T14:00:00Z" }
]
}
The envelope (event, channel, timestamp, agentId, conversationState, recentHistory) is identical. Only the shape of data changes: SMS carries message and mediaUrl, voice carries transcript and confidence. This means a single handler function can route on channel and dispatch to channel-specific logic, without maintaining two separate webhook endpoints, two separate signature schemes, or two separate state models.
The design decision worth noticing here is that conversationState and recentHistory are present in every payload, not fetched separately. That’s a real architectural choice: instead of forcing you to call back into the API to reconstruct context before you can respond, AgentPhone pushes the last N messages (configurable 0–50, default 10) directly into the webhook body. Your backend doesn’t need its own conversation database for basic context; it can be entirely stateless if the use case allows it.
Voice has to carry more than transcripts, and it does
The place where “unified” could have failed is voice, because a phone call is a fundamentally different beast from a text message: it’s a stream, it has turns, it has an end state with a summary. AgentPhone handles this with a second event type, agent.call_ended, that fires when the call terminates and delivers the full transcript, not the windowed recentHistory, plus duration and optional analysis:
{
"event": "agent.call_ended",
"channel": "voice",
"timestamp": "2025-01-15T14:05:30Z",
"agentId": "agt_abc123",
"data": {
"callId": "call_ghi012",
"numberId": "num_xyz789",
"from": "+15559876543",
"to": "+15551234567",
"direction": "inbound",
"status": "completed",
"startedAt": "2025-01-15T14:00:00Z",
"endedAt": "2025-01-15T14:05:30Z",
"durationSeconds": 330,
"disconnectionReason": "agent_hangup",
"transcript": [
{ "role": "agent", "content": "Hello! How can I help you today?" },
{ "role": "user", "content": "I need help with my order." },
{ "role": "agent", "content": "Sure! Could you provide your order number?" }
],
"summary": "Customer called about an order inquiry. Agent helped locate the order.",
"userSentiment": "Positive",
"callSuccessful": true
}
}
This is fire-and-forget: your server doesn’t need to respond with anything meaningful, just a 200 OK. That’s a sensible design choice, since there’s nothing left to say to a caller who has already hung up.
The harder problem is what happens during the call, in the seconds between the caller finishing a sentence and your webhook returning a response. A phone call has no tolerance for the kind of latency a chat interface can absorb. If your webhook takes three seconds to think, the caller sits in dead air for three seconds, which reads as broken, not thoughtful.
AgentPhone’s answer is a streaming response contract using newline-delimited JSON:
{"text": "Let me check that for you.", "interim": true}
{"text": "I found 3 results for your order."}
The interim: true chunk is spoken immediately by the TTS layer while your server keeps working; the final chunk (without interim) closes the turn. This is the single most important detail in the whole voice architecture, because it directly addresses the thing that kills most naive phone-agent implementations: silence during tool calls.
Here’s what that looks like wired up to a real tool-calling loop, using Claude with function calling, adapted from AgentPhone’s own documentation:
from flask import Flask, request, Response
import json, anthropic, os
app = Flask(__name__)
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
TOOLS = [
{
"name": "search_orders",
"description": "Look up a customer's recent orders.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
]
TOOL_HANDLERS = {
"search_orders": lambda args: search_order_db(args["query"]),
}
def run_tool_call(user_message: str, history: list) -> str:
messages = [{"role": "user", "content": user_message}]
for _ in range(5):
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
system="You are a helpful phone assistant. Keep responses to 2-3 sentences.",
tools=TOOLS,
messages=messages,
)
if response.stop_reason == "tool_use":
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "tool_use":
handler = TOOL_HANDLERS.get(block.name)
result = handler(block.input) if handler else "Unknown tool"
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result,
})
messages.append({"role": "user", "content": tool_results})
else:
return " ".join(b.text for b in response.content if hasattr(b, "text"))
return "Sorry, I'm having trouble processing that."
@app.post("/webhook")
def webhook():
payload = request.json
if payload.get("channel") != "voice":
return "OK", 200
transcript = payload["data"].get("transcript", "")
history = payload.get("recentHistory", [])
def generate():
# Speak immediately so the caller isn't sitting in silence
yield json.dumps({"text": "Let me check on that.", "interim": True}) + "\n"
try:
answer = run_tool_call(transcript, history)
except Exception:
answer = "Sorry, I ran into a problem. Could you try again?"
yield json.dumps({"text": answer}) + "\n"
return Response(generate(), content_type="application/x-ndjson")
The pattern is worth internalizing even outside AgentPhone specifically: acknowledge, then compute, then answer. Any voice-agent architecture that skips the acknowledgment step will feel broken the moment a tool call takes longer than a second, which is most tool calls involving a real database or a real LLM round-trip.
Two voice modes, one underlying engine
AgentPhone splits voice handling into two modes, set on the agent object: webhook (the default) and hosted.
In webhook mode, every turn of the call is proxied to your server exactly like the tool-calling example above. You own the LLM, the logic, the tool calls, everything. In hosted mode, AgentPhone runs the LLM itself using a systemPrompt you provide, with no webhook or backend server required at all:
curl -X POST "https://api.agentphone.ai/v1/calls" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"agentId": "agt_abc123",
"toNumber": "+15559876543",
"initialGreeting": "Hi, this is Acme Corp calling about your recent order.",
"systemPrompt": "You are a friendly support agent from Acme Corp."
}'
This single request provisions and runs an entire outbound phone call, no server needed on your end at all. Both modes share the same underlying voice engine: streaming STT, streaming TTS, barge-in (the caller can interrupt mid-sentence), and turn detection. The mode switch is purely about who’s generating the response text, not about voice quality or latency characteristics, and it can be changed live via a PATCH on the agent with no downtime, according to the docs, because the backend re-provisions voice infrastructure behind the scenes.
This is a genuinely useful design choice for anyone building a spectrum of products: you can prototype fast in hosted mode, then graduate to webhook mode once you need real tool access, RAG, or a specific model you’ve already integrated elsewhere, without re-platforming.
Group chats break the one-to-one assumption, and the schema handles it
One detail that reveals real engineering thought rather than a v1 shortcut: iMessage group threads. A naive “unified webhook” design would treat every message as one sender talking to one number, but group chats have a roster, a group identity, and per-message attribution that a 1:1 model can’t express.
AgentPhone’s solution is to keep the same agent.message envelope and add two fields to data only when the message is part of a group: data.group (roster and metadata) and data.senderIdentifier (which member sent this specific message). For one-to-one conversations, these fields are omitted entirely, not set to null, which is a deliberate and useful contract: if (data.group) is a reliable, unambiguous check for “is this a group thread,” rather than something you have to infer from a mix of null-checks and message counts.
{
"event": "agent.message",
"channel": "imessage",
"data": {
"message": "Are we still on for Friday?",
"senderIdentifier": "+15554444444",
"group": {
"isGroup": true,
"groupId": "grp_abc123",
"groupName": "Trip Planning",
"participants": [
{ "identifier": "+15554444444", "name": "Alice" },
{ "identifier": "+15555555555", "name": null }
]
}
}
}
To reply, you send to groupId, not the sender’s number, which would incorrectly start a private 1:1 thread with just that one participant. It’s a small detail, but it’s the kind of detail that only shows up in a schema once someone has actually built and shipped group-chat support, not just imagined it.
Security: HMAC signatures, replay protection, idempotency
Every webhook delivery carries four headers: X-Webhook-Signature, X-Webhook-Timestamp, X-Webhook-ID, and X-Webhook-Event. The signature is HMAC-SHA256 over the string {timestamp}.{raw_body}, verified against a secret issued when you register the webhook:
import hmac
import hashlib
import time
def verify_webhook(raw_body: bytes, signature: str, timestamp: str, secret: str) -> bool:
if abs(time.time() - int(timestamp)) > 300:
return False # reject anything older than 5 minutes
signed_string = f"{timestamp}.".encode() + raw_body
expected = hmac.new(secret.encode(), signed_string, hashlib.sha256).hexdigest()
return hmac.compare_digest(f"sha256={expected}", signature)
This is standard, correct practice: signing over the timestamp plus body (not just the body) prevents replay attacks even if a signature were somehow intercepted, and the 5-minute window is a reasonable balance between clock drift tolerance and security. Nothing exotic here, but it’s implemented the way you’d want it implemented, which matters more than novelty for a security primitive.
Retries follow exponential backoff — immediate, then 5 minutes, 30 minutes, 2 hours, 6 hours, 12 hours, for a total of 6 attempts before a delivery is marked failed. The X-Webhook-ID header is there specifically so you can deduplicate: since retries mean your endpoint might receive the same event more than once, you’re expected to track processed IDs (in Redis or a database in production) and short-circuit repeats.
Where the real engineering cost lives
None of the individual pieces here are exotic. Signed webhooks, streaming NDJSON responses, and event-driven architectures are all well-understood patterns. What AgentPhone is actually selling is the integration of all of them into one coherent contract that handles SMS, iMessage, group chats, and live voice without forcing the developer to hold five different mental models simultaneously.
That’s also exactly the part that’s hardest to verify from the outside. The webhook contract, the SDKs (published on PyPI and npm), and the MCP server (open source on GitHub) are all real, inspectable artifacts, not just landing-page claims. What isn’t independently verifiable from the docs alone is how the system behaves under real production load: call quality at scale, transcription accuracy across accents, and how gracefully the platform degrades when a customer’s webhook times out mid-call in webhook mode. The docs are explicit that a 30-second default timeout (configurable 5–120 seconds) exists and that a caller hears silence if you blow through it, which is an honest admission of a real constraint rather than a glossed-over one, but it also means the architecture puts a hard ceiling on how slow your own tool calls are allowed to be.
If you’re evaluating this kind of infrastructure, or building your own version of it, the schema decisions covered here are the ones worth stealing regardless of which vendor you end up using: one event envelope with a channel discriminator, context pushed into the payload rather than requiring a callback, an explicit contract for streaming partial responses during slow turns, and a null-vs-omitted convention that makes edge cases like group chats checkable with a single boolean, not a chain of inferences.
Sources: AgentPhone documentation (docs.agentphone.ai), AgentPhone open-source MCP server (github.com/AgentPhone-AI), AgentPhone Python SDK (PyPI), Y Combinator launch page.
