Can LLMs Generate Audio With Text?
Sep 08, 2026 • 7 min read
While scrolling X, I found this post from @maxxrubin_: Astra identifying sounds from mel spectrograms zero-shot, with light reasoning.
So Astra is able to identify sounds from mel spectrograms
— Max (@maxxrubin_) September 7, 2026
zero-shot.
I don't think we've scratched the surface of what this model can do (and this is light reasoning btw) pic.twitter.com/eYQecO55sD
Basically, asking the vision side of an LLM to read a picture of sound. I think it's pretty cool when people discover capabilities like this. It feels similar to how a rudimentary version of thinking was "discovered" in the GPT-3 playground, by asking the model to think about how it would solve the problem and then output its attempt, before reasoning models were publicly available.
Then this reply from @AdamHoltererer pushed it one step further: GPT-6 generating a spectrogram in SVG of "Mary Had a Little Lamb". Not just hearing, but talking — through an image.
Not only can GPT-6 hear, it can talk!
— Adam Holter (@AdamHoltererer) September 7, 2026
It created a spectrogram in SVG of "Mary Had a Little Lamb" https://t.co/sIz9Ot9xEy pic.twitter.com/8t8LmOmA9h
And it gave me an idea: what if LLMs could generate audio using text?
Not a description of a sound. Text that we could actually turn into sound. I tried it, and the results went from three farts to a voice saying "Hello, dear human!"
Text as sound
If we could use text instead of an image, that would give the ability to generate sounds to models with no native audio output.
The first thing that came to mind was a domain-specific language (DSL) for sound. DSLs are everywhere in programming, and they are one of the most powerful tools you can use to represent a process or data.
So I asked GPT-6-Astra (low) to come up with a DSL capable of expressing sound by drawing structures in time and frequency. It made a good draft, but after reviewing it I found that it fixated too much on making it compatible with SVG. Which makes sense, given that during the brainstorm about the characteristics of the DSL I showed it that second tweet.
After some back and forth, it came up with SPL (SPectraL), a DSL for describing audio.
SPL in 30 seconds
SPL describes sound by drawing structures in time and frequency. An LLM writes curves, harmonic spectra, noise regions, and attacks. A deterministic renderer supplies phase and generates the waveform. No sample values, no FFT bins, no instrument names.
Every file starts with a header:
spl 2 RATE DURATION SEEDFor example, spl 2 24000 1.5 0 means 24 kHz, 1.5 seconds, seed 0. Then come four block types, each ending with end:
track: one frequency curve (TIME FREQUENCY GAINrows)harmonics: a pitch curve plus a spectral shape (spectrumthencurve)noise: a moving frequency region (TIME LOW HIGH GAIN SLOPErows)hit: short broadband attacks, one event per row
A minimal example:
spl 2 24000 0.5 0
track
0 440 0
0.02 440 0.3
0.4 440 0.3
0.5 440 0
endThe full authoring guide, exact syntax, and rendering formulas live in the SPL specification. The intended workflow is one-shot generation: the model receives the guide, writes SPL, and the app validates and renders it.
From spec to sound
I set up a basic loop to validate that the language was descriptive enough for our purpose, and with the help of Muse-spark-1.3 I implemented an interpreter for the DSL in Go.
The main goal of using Go was to easily have a WASM version available for the browser. Same engine, natively and in the page: parse, validate, render deterministic float64 audio, compute a spectrogram, export WAV and PNG.
That led to the next step: a small web interface to interpret SPL files in the browser. Editor, example library, spectrogram player with seeking, WAV and PNG downloads. Everything runs locally, no synthesis server or uploads. You can play with it right here: SPL playground.
I asked Muse-spark-1.3 to generate samples for preview, and it did a pretty good job! We could get sound out of the DSL. But could we get speech?
The experiment
I gave various LLMs the DSL specification and the same prompt, then listened to what came out:
Using the attached DSL specification, generate an SPL file representing what your voice would sound like saying, "Hello, dear human!"
Results
These are my very scientific categories: Meh, Interesting, and Wow. Each card has my reaction and the audio that goes with it. Expand the details to inspect the spectrogram and grab the SPL source. Listen for yourself; knowing the phrase might make it easier to hear.
Meh
This is weird. Is it just me, or does it sound like it could be another language?
I hear: "Call me Bob." Wrong phrase, but now I can't unhear it.
Quite similar to the high-effort attempt, but I can't make out the words.
This is just three farts.
Some resemblance to words, but nothing I can identify.
I honestly expected better. This one makes me wonder how much benchmark performance tells us about a task like this.
Interesting
Immediately, the spectrogram shows more detail. It sounds like an attempt at a human voice. I think the "Hello" is pretty good, even if the rest is unintelligible.
Similar to the max-effort attempt: I can clearly make out "Hello."
Wow
I can clearly hear "Hello, dear human." Wow!
A high-pitched voice, but I can still clearly hear the phrase.
Incredible. To my ears, this is saying the phrase without a doubt.
What's next
Everything from this post is public: the specification, the Go/WASM renderer, and the SPL sources linked above all live in the SPL repo. You can also experiment without installing anything — the SPL playground runs the full studio in your browser: paste SPL, render, listen, export.
Try one of the result cards' SPL files in the playground, change a few values, and listen again. I'm curious how far this can go: different voices, other phrases, maybe sounds that are harder to describe than to recognize.
Conclusion
I started with a tweet about reading sound from an image and ended up listening to text-only responses say "Hello, dear human!" Some of them, anyway.
I think there's an interesting experiment here, and maybe even the start of a benchmark. How much does a model understand about sound if it can describe the structure of a voice well enough for us to hear the words? These few samples don't answer that, but the difference between the attempts makes me want to keep testing.
For now, the part that surprised me most is that this works at all.
Give the playground a try and see what you can make it say.