On talking to agents
I've spent a lot of time thinking about how to call my agents, and get called by them. It's just something that happens a lot. I don't like prompting on my phone by text, but by voice it's totally fine. And I know I can dictate my prompt to Claude on their official apps, but that's fiddly while I'm driving, and impossible on really bad mobile connection.
And, well, calling an agent is just... fun.
So, how do we hook this up? After a lot of thinking, I landed on regular old mobile phone calls.
Regular old phone calls are great:
- They ring exceptionally reliably, much more so than any internet messaging system ever does (in my experience)
- They have zero auth (that's not always great though...)
- You don't need to install an app.
- You can usually place a plain old phone call on connection so terrible no internet calling could ever work out. And you can even have a proper conversation.
- Hands-free systems - such as that of my car's - handles regular phone calls a lot better than it does internet calls.
- A lot more small reliability tidbits that i'm not coming up with right now.
So that's great. How do we get agents to use the phone?
One can get a modem. I got this one. Long story short - it didn't work. The ESP32 on board wasn't wired to any audio ins out outs of the modem. I could have wired a small USB sound card to analog pins on the board, but I could not be bothered. These modem breakout/dev boards actually seem to almost always be shady, sketchy, and difficult to rig up in general, so I decided to give up on this idea entirely.
An arguably easier route that I did look into aswell are the plethora of call providers - such as Twilio. They aren't exactly expensive - cents per minute - and are supposed to be easy to set up. But I got lost in the SIP terminology and the great amount of different options offered by the providers. At some point I thought to just ask Claude to set my company's account up and forget about it, but even Claude had trouble coming up with a solid plan for getting a Lithuanian number.
Then, during one of my many brainstorming sessions with Claude, a few weeks later, and seemingly by accident, Claude mentioned the easiest way to actually make and receive calls programatically is by using any plain old phone with a good old SIM card, and Bluetooth! The modem will just work (that's kind of the main point of a phone), they all have bluetooth, I have one lying around so no need to buy any hardware, and I also had a NUC with bluetooth/wifi. The NUC connects to the phone via bluetooth, and by default it can make and receive calls, hang up, see the incoming number - literally all that's needed for this setup! And on top of that, you can literally use any phone that has bluetooth - something that probably most everyone has in a drawer. I get a Lithuanian phone number, and a cheap flat rate call plan. Fantastic!
So today I actually rolled this out. Claude was able to set up taking and placing of calls, audio handling, reconnecting, et cetera. It wired up the OpenAI Realtime model, and now I can dial my agent's number, speak a passphrase, and task it to do things while I drive (hand's free, of course!), or get a call when something during a session goes seriously wrong.
If a colleague texts me on Google Chat with a scary looking message while I'm driving, I don't have to stop by the side to launch an investigative Claude Code session - i can just call my agent and ask it to start that session.
The handover protocol is really simple:
- My agents get an escalation MCP server - they can essentially instruct ChatGPT Reatime on exactly what it needs to tell me, or what info it needs from me. ChatGPT Realtime then does this without any tools or datasources itself, operating purely on what it was given by the Claude Code session. Realtime then returns the MCP call with my answer.
- I can call the agent too. It picks up and says "A hedgehog has an apple stuck on one of it's spikes". I then need to reply "Sadly, that voids the warranty of the iPhone" (just an example, not my real passhprase). If the passphrase is correct, the agent asks me what "Scenario" i want to work on. A scenario here is just a Claude Code Cloud Routine pointer - I have a few for different customers, so i can invoke my question in the right... context. After picking the scenario, I can then ask for stuff: "Please count all configmaps in the Kubernetes cluster". Realtime summarizes: "I'll ask the downstream agent to count all configmaps on the default Kubernetes cluster. That right?". I say "yep". Realtime thanks me and hangs up, and invokes the Routine. The Routine is then explicitly prompted to always use the very same escalation MCP server to deliver it's final result or request for clarification, thus closing the loop.
If i didn't buy that annoying modem from before, this project would have costed 4€ in total (monthly prepaid SIM plan for phone calls). It was fun, and most importantly - it works.
Why not just get ChatGPT Realtime to actually do the work, instead of the weird handover? Well, it'd only work for incoming calls, as I use Claude Code for my work, so Claude needs to make the escalation. And I didn't feel like setting up a separate environment just to handle incoming calls.

0 comments