MusicTech Insights #14 | Curated by Joaquin Saavedra

This issue is guest curated by Joaquin Saavedra, Co-Founder & CEO of Edge Audio Labs, where he builds audio software across real-time audio, DSP, acoustics, and AI. Joaquin brings a mix of technical, academic, and hands-on experience to his work. Before founding Edge Audio Labs, he worked as a tech lead and CTO, taught Audio Technology at ORT University, and worked as a sound engineer across theatre, live shows, and studio environments.

In this edition, Joaquin explores what happens after you have a working AI audio model: where it runs, how it handles the demands of real-time audio, and how the choice between cloud and local inference can shape both the product and the business behind it.

Where Does the AI Actually Run?

Hi, I'm Joaquin.

I run Edge Audio Labs, a studio that builds audio software for other companies: plugins, DSP engines, and lately a lot of AI audio.

Almost every brief that reaches us now has AI in it. Some of it is real, some of it is a slide. And nearly all of it arrives with the same assumption, borrowed from ChatGPT: that there is a model somewhere, you call it, and it answers.

I want to spend this edition on what that assumption hides, because in my experience the model is not always the hardest part. A lot of times the hard part is the question nobody asks until month three: where is this thing actually going to run? How?

These are not the models you are used to

An LLM is a general model. Someone else trained it, someone else hosts it, and you rent it by the token. The models that process audio are the opposite: small, narrow, and trained for exactly one job. Separate a voice from a mix. Take the room out of a dialogue recording. Emulate one specific tape machine. Turn a written score into a singing voice.

They rarely work on words. They work on samples, or on features pulled out of the audio first (spectrograms, pitch curves, envelopes, MFCCs, etc). And they are yours. There is no public endpoint for the model that removes the room from this particular microphone.

There are services that sell pieces of this, and some of them are genuinely good. But the moment your product needs a specific behaviour, a specific sound, or a specific latency, you stop being a customer and become a builder. You train or fine-tune a model, and then you build the machinery that runs it.

That machinery starts with a chore that never makes it into the demo: getting the model out of Python. You cannot ship a Python runtime inside a plugin, so the model has to be exported, and then verified number by number against the original, because the export is rarely clean the first time.

Getting the model out of Python

The deadline that sorts the whole industry

Then comes the constraint that decides everything downstream.

A plugin lives inside the host's audio callback, and that is a hard deadline job. At a 128 sample block and 48 kHz you have roughly 2.7 milliseconds to hand the audio back. Miss it and the user does not get a slightly late result. They get a click.

The real-time budget

Inside that window you cannot allocate memory freely, you cannot spawn threads, and you certainly cannot wait on a network. Local inference does not always fit either, and when it does not you move it to a worker thread and pay for it in reported latency instead. That is a trade you can make. A network round trip is not.

So there are two places the model can live, and everything else is a detail.

Option one: your servers

The plugin captures audio, sends it to your infrastructure, a GPU does the work, the result comes back.

Cloud inference

The engineering here is familiar: an API, a queue, workers, autoscaling. The economics are not.

GPUs are expensive, and the trap is in how you pay for them. In most cases you do not rent a GPU by the inference, you rent it by the hour. A machine that is up and idle costs exactly the same as one working flat out. So you either keep utilisation high, which means queuing jobs and making people wait, or you keep latency low, which means paying for capacity that sits there doing nothing. Every serious cloud audio product lives somewhere on that line, and where it sits is a product decision, not an infrastructure one.

The consequence lands on the price tag, and this is the part I would push any founder to think about early. When every use carries a real marginal cost, a one time purchase quietly becomes a liability: you are promising to serve that customer forever on a single payment. Which is why so much AI audio has moved to subscriptions, or to credits bought up front. The pricing model is downstream of the architecture, and teams tend to work that out far too late to change either one.

What you get in exchange is control. The model never leaves your infrastructure. Nobody can extract it, nobody can run it without you, and you can improve it on a Tuesday without shipping anything to anyone.

Option two: user's machine

The other architecture puts the model inside the plugin. The audio never leaves the computer, and neither does the compute.

Local inference

The first consequence is the enjoyable one: processing becomes free. Not cheap, free. No bill grows with usage, so you can let people render all night without watching a meter, and selling the product once works again.

The second consequence is that you inherit hardware you will never see.

We had a desktop machine that kept quietly falling back to the CPU even though it had a perfectly good dedicated graphics card sitting inside it. The card was there. The driver reported its name and its memory correctly. It simply never set the one flag that says I am a dedicated GPU, so our code, quite reasonably, refused to trust it. Nothing was broken, in our software or in theirs. We infer it from the amount of video memory now.

And the reason that flag mattered so much is this:

The same render, three machines

Same model, same audio, three machines. The integrated GPU is slower than the CPU it was meant to help, because these models are bound by memory bandwidth and an integrated chip shares that bandwidth with the CPU. So use the GPU is not a strategy. Working out which GPU, and refusing the ones that would make things worse, is.

So, which one

Neither, cleanly. Most real products end up hybrid: the interactive path running locally, the heavy one shot jobs in the cloud. Some even let the user choose depending on their computer specs and pocket.

But this is not a technical preference, and it should not be settled by whoever happens to sit closest to the code. It sets your latency, your support burden, your exposure to having the model lifted, and whether you can sell your product once or have to charge for it every month.

Pick the architecture and you have picked the business.

This is most of what we do at Edge Audio Labs: taking models that work in a notebook and making them run inside somebody's plugin, on both of these architectures, on hardware we do not control. If you are somewhere in the middle of this decision, I am always happy to compare notes.

anira: real-time-safe neural inference for audio

The clearest explanation of why you cannot simply call a general inference engine on the audio thread, and the pattern that fixes it. If you read one thing on this list, read this one.

Learn more

ONNX Runtime execution providers

The map of the backends: CPU, CUDA, DirectML, CoreML. Skim it and the compiler versus kernel distinction above stops being abstract, because it is exactly what separates these entries.

Learn more

ONNX Runtime Execution Providers