[Prompt]
Herman and Corn, in last night's episode on voice engineering you said the future is many LoRAs on one base voice model: one per character, one to lift the prosody of the whole episode, maybe LoRAs that work on spans of speech. I pictured a DJ's mixing desk, a whole panel of LoRAs with a dial each. Where does that actually stand today? Image and video tools typically support about two LoRAs at once. Is the next generation going to support a big panel of them, each with its own dial, and how close is that for voice?

[Response]
Corn: A mixing desk. That's what Daniel saw when we said the future is many LoRAs on one voice model.

Herman: Which, I want to say, is the most flattering possible reaction to something I said at eleven at night.

Corn: He wrote in overnight. He's been training our voice adapters, the ones you hear right now, and he'd been treating them the way he puts it, "train an adapter, do a bit of EQ, done." Fairly crude, light-touch voice design. Then last night's episode put the idea of many LoRAs on one base model in his head, and the image arrived fully formed: a DJ desk. A panel of LoRAs, each with its own fader, someone turning them up and down before the output gets made.

Herman: And he wants to walk from that image to reality. Where do things actually stand, why do image and video tools seem to top out around two LoRAs at once, and how far is voice from that desk, including, this is the part that got me, LoRAs that work on individual spans of speech.

Corn: The short answer is that the two-LoRA figure is a quality ceiling, not a software cap. That distinction is the whole episode, so let's earn it.

Herman: A LoRA, for anyone who doesn't want the full derivation, is a small trained adapter. You freeze the base model and learn a weight delta, a set of low-rank updates, applied to specific layers. The bit that matters for Daniel's desk is that at inference you can scale that delta. There's a strength value, usually zero to one, sometimes higher, that multiplies how much of the adapter you actually apply.

Corn: The fader.

Herman: That's the one true dial every LoRA already has. Set it to 0.5 and you get half the adapter, set it to zero and you get the base model back. So the desk isn't fantasy hardware, it's arithmetic.

Corn: He assumed the two-LoRA thing was the software drawing a line.

Herman: It mostly isn't. Diffusers, the Hugging Face library, has an activate-adapters call where you pass a list of LoRAs and a weight each, and there's no documented count limit at all. Two adapters at 0.5 and 1.0, fine. If you give two adapters equal weights of 0.5 and 0.5, you've just averaged them. The library will happily take five.

Corn: But every forum post says one or two.

Herman: Because two or three is where the quality holds. When you stack LoRAs, you're summing their weight deltas into the same layers. Two adapters trained independently can push the same weights in conflicting directions, and the model feels both instructions at once. Vendor and community guidance across image and video lands in the same place: one or two well-chosen LoRAs, three only if you lower the strengths.

Corn: Two dials that fight each other isn't two dials. It's one dial with an argument attached.

Herman: And where the "two LoRAs" impression really comes from is that hosted APIs do sometimes cap it. WaveSpeed's Wan 2.2 endpoint takes at most three LoRAs, with strengths from zero to four. fal's endpoint for the same model takes a list and documents no cap at all. Same base model, two products, different decisions.

Corn: The cap is a product manager, not physics.

Herman: Right, the physics is the interference. Though video has one new wrinkle that voice doesn't have yet, and I want to spend a minute on it because it's a second kind of dial. Wan 2.2's fourteen-billion-parameter model runs two separate transformers: a high-noise one and a low-noise one. The high-noise transformer handles the early denoising steps, where the broad structure of the image or video is decided. The low-noise one handles the end, the fine detail.

Corn: And you can send a LoRA to just one of them?

Herman: On fal, each LoRA targets high-noise, low-noise, or both. And WaveSpeed's own guide says put your style LoRA on high-noise and your detail LoRA on low-noise. Which is a per-layer dial in disguise. You're not just choosing how strong an adapter is, you're choosing which half of the model it whispers to.

Corn: So the desk already has channel strips. Same fader, routed to different buses.

Herman: That's a better sentence than the one I was building toward, and I resent it. The third dial is the one that gets closest to what Daniel's asking for. In image tooling you can set a LoRA's strength per region of the picture. The Regional Prompter extension and masked sampling in ComfyUI let you bind one character LoRA to one area of the image, so if you're generating two characters, each one's adapter only operates where that character is.

Corn: Hold on, that's not just a dial. That's a dial with a where.

Herman: It is. And there's even a per-layer version, the LoRA Block Weight extension, which sets a strength separately for each block of the image model. For Stable Diffusion that's a string of about twenty-six numbers, one per block, and people share presets of those strings like EQ curves.

Corn: Twenty-six faders per LoRA.

Herman: And the catch, which I find quietly delightful, is that the presets don't carry across architectures. A twenty-six-number string tuned for Stable Diffusion is meaningless for Flux, because Flux has a different number of blocks. You can't even carry your EQ settings between mixing desks.

Corn: So to be clear, none of this is exotic. The per-LoRA fader, the per-layer routing, the per-region masking, all shipping today in image tools. And the reason people still say "about two LoRAs" isn't that the panel doesn't exist.

Herman: It's that the dials aren't independent. Turning one up moves the others, because they all sum into the same weights. A mixing desk where the bass fader also boosts the vocals.

Corn: Daniel asked whether the next generation heads toward a big panel. What's the research actually doing about the interference?

Herman: Two directions, roughly. One is smarter composition. There's a line of work called Multi-LoRA Composition, Zhong and colleagues, that proposes two alternatives to plain merging. LoRA Switch activates only one LoRA per denoising step, rotating through them, so no two adapters ever fight at the same instant. LoRA Composite averages each LoRA's guidance at every step instead of merging weights. Both beat plain merging, and the advantage grows as you add more LoRAs, which is the telling part.

Corn: What's the cost?

Herman: Composite needs an extra model pass per LoRA per step. Five LoRAs, five times the compute, roughly. Switch is cheaper but coarser.

Corn: Cheaper but coarser is usually what I end up buying.

Herman: The other direction is shrinking the problem instead of solving it. ZipLoRA and K-LoRA, which merge pairs, one subject plus one style, with learned coefficients. Elegant work. Built for pairs, not panels. Mix-of-Show from NeurIPS 2023 is the interesting middle case, it fuses separately trained character LoRAs with regional control so attributes don't leak between characters.

Corn: So the field keeps discovering that two is a comfortable number and building nicer furniture around it.

Herman: There's also the automatic hand. X-LoRA trains only a small gating head, and everything underneath stays frozen. The head sets each LoRA's strength per token and per layer. It's a mixer where the faders move themselves based on what's playing.

Corn: I hate that mixer.

Herman: You'd absolutely hate that mixer.

Corn: If I wanted the levels decided for me I'd let Daniel keep doing the EQ.

Herman: LoraHub is similar in spirit, it finds mixing coefficients automatically from a few examples, but its own authors note the search space explodes as you add modules, so they pre-select a subset first. Even the automatic mixer starts by throwing dials off the desk.

Corn: So the answer to Daniel's first question, the big panel. Is it coming?

Herman: The direction is yes, but not by raising a cap, because there was no cap. It comes from interference management, rotation, gating, orthogonal subspaces. The ceiling moves when the fighting stops, not when the API lets more entries into a list.

Larry: Friends, have you ever wanted a mixing desk for your voice? Of course you have. Introducing the LoRA Dial Panel Pro, the only desktop console with a real dial for every single character in your head, and when you turn two dials at once, it simply adds them together, which is exactly what the engineers do, so you get twice the personality and a faint buzzing sound that experts call interference, plus a bonus dial for prosody that lifts your episode by up to forty percent, guaranteed.

Larry: And now the small print, read quickly: results vary, dials are decorative on some models, stacking beyond two adapters may summon conflicting directions, the high noise dial must never be turned by children or donkeys, prosody lift measured on a good day, no refunds, no returns, and absolutely no phone numbers because we forgot to book any.

Corn: That was nearly word for word. The buzzing sound, the bit about interference. Herman, you said that a minute ago.

Herman: I noticed that too, and I've decided not to think about it.

Corn: Sensible. Where were we.

Herman: The regional dial. Which is the bridge to the part Daniel actually cares about, because a per-region dial in an image is the closest existing analogue to a per-span dial in speech. So the question is what voice has actually shipped.

Corn: His framing was that voice adapters are crude and light-touch. Is that fair?

Herman: Less than it was even a year ago. There's a paper from January, ReStyle-TTS, out of Zhejiang with the Shanghai Innovation Institute and Ant Group, that does almost exactly what Daniel pictured. They train style-specific LoRAs, one per style dimension, and fuse them orthogonally, which means the adapters are constrained to not overlap in weight space.

Corn: Orthogonal as in, the faders stop fighting?

Herman: That's the idea, each delta gets its own subspace. On a zero-shot TTS model, that gives continuous relative control of pitch, energy, and several emotions. You can turn pitch up to a degree and anger up to a degree, independently. And because moving style dials drags the speaker's timbre with it, they add a separate module that corrects timbre drift as the dials move.

Corn: The channel strip remembers whose voice it is.

Herman: There's also GLASS, from June, which I'd call the closest thing to a two-dial voice desk in a published paper. One LoRA per control axis, speaking rate and pitch, trained with reinforcement learning and acoustic rewards on a frozen zero-shot backbone. At inference you can swap the adapters, interpolate between them, combine them with plain LoRA arithmetic, all while the speaker prompt stays fixed. The voice doesn't change because the dials move.

Corn: And the character-plus-prosody split he imagined?

Herman: Fed-PISA, last year, splits a private speaker-identity LoRA from a shared style LoRA. The identity stays with the owner, the style gets shared. One catch worth knowing: a survey from March notes most LoRA work in TTS adapts the acoustic decoder, not the language backbone. Which matters for where the dials can live, because the other recent finding, from CoCoEmo, is that emotional prosody comes mainly from the language-model stage of a hybrid TTS system. The prosody dial may need to sit on a part of the model that people aren't currently adapting.

Corn: The fader's on the wrong end of the desk.

Herman: There's a rival kind of dial too, and this is where the panel gets weird. Steering vectors. EmoSteer-TTS steers emotion in F5-TTS, CosyVoice2 and E2-TTS without training any adapter at all, by adding directions to the model's internal activations. You can interpolate between emotions, or erase one. CoCoEmo reports mixed-emotion speech with controllable proportions, and that's the paper that found the emotion lives mostly in the language-model stage.

Corn: No adapter, no training data. Just directions in activation space.

Herman: Which is why I suspect the final desk is half LoRAs, half steering vectors. LoRAs for the things you train once, steering for the things you want to move live.

Corn: Okay. The question I actually care about, and I think it's the one he saved for last on purpose. Spans. LoRAs that work on individual spans of speech. The fader that moves mid-sentence.

Herman: I looked, and I'll say this carefully. I did not find a TTS system that varies a LoRA's strength over time, within a single utterance. Everything we just described is a whole-utterance dial. ReStyle, GLASS, Fed-PISA, the steering work, all of it sets a level and generates the whole output at that level.

Corn: Which isn't proof it can't be done.

Herman: No, and I want to be honest about that. It's absence of evidence. It's the kind of gap that means nobody's shipped it, not that it's been shown to fail.

Corn: So what's in the way? Say I wanted a fader on this very voice that lifts prosody between minute four and minute seven.

Herman: Four things, I think. First, a time-varying strength schedule during decoding. The nearest existing thing isn't in TTS at all, it's ComfyUI nodes that schedule a LoRA's strength to change during generation. Same idea, different modality. Second, cost. The composition methods that handle interference well in images, the Composite style, already pay an extra model pass per LoRA per step. If you then change the strength at every span boundary, you're recomputing or re-merging adapters constantly.

Corn: A fader that moves every four seconds costs more than one that sits still.

Herman: Third, you need a control signal that says which span gets which dial. Text spans, forced alignment, or a controller model, something that knows minute four is where the energy should lift. Fourth, and this is the sneaky one, timbre drift. ReStyle needed a correction module just for whole-utterance moves. Moving a style dial mid-utterance is exactly where speaker identity wobbles. You'd get to minute five sounding slightly like someone else.

Corn: A sloth with somebody else's chin.

Herman: I was going to say the audio equivalent of a face melting, but sure. There's one more confusion worth clearing up, because it's everywhere in the serving discussion. S-LoRA out of Berkeley serves thousands of LoRAs on one base model with paged adapter memory. vLLM and Predibase's LoRAX load adapters on demand per request, and vLLM extended multi-LoRA to mixture-of-experts models this February.

Corn: Thousands of dials.

Herman: No, and this is the point. That's one adapter per request. A library of voices, where each caller gets their own. Daniel's desk is a mix within one utterance. They sound like the same problem in a keynote and they are completely different problems.

Corn: The record library versus what the DJ does with two of them at once.

Herman: That. So if I'm answering him directly: the per-LoRA dial for voice exists, in research papers at least, GLASS and ReStyle show it. The per-layer dial exists implicitly in where you put the adapter, decoder versus backbone. The per-span dial isn't shipping anywhere. The desk exists, the faders just aren't time-coded.

Corn: Bring it home for us, though. Daniel's actual desk, for this show. What could he build now?

Herman: Now, realistically, three faders. He already has one channel per host, our character adapters, and each of those has a strength he can ride instead of leaving it at full. A second channel is a style adapter in the GLASS mould, a speaking-rate or pitch LoRA trained on our own clips, that he dials per episode, not per sentence. And emotion is probably better as a steering direction than another adapter, because that's the one you'd want to nudge without retraining anything.

Corn: And later?

Herman: Later is the time code. The same faders, but automated across the episode, lifted for the argument, eased for the quiet bit. That needs the span control signal and the timbre fix we just talked about, so I'd file it as plausible, not imminent.

Corn: You know what I'd want on ours? A dial that makes Herman's explanations come in at 0.7 strength after minute twenty.

Herman: I heard the operator testing that one during the last recording.

Corn: It had a faint buzzing sound.

Hilbert: The desk is the wrong picture, Herman. I ran overnight automation for a community radio station for a stretch, and it was never the faders that broke, it was the routing. We had a pre-recorded station ident, fifteen seconds, and it kept bleeding under the live voice for weeks. The fader looked correct. It was up where it was supposed to be, at the level I'd set it. The ident was assigned one channel off, so it was coming in under a different fader entirely, and everyone trusted the one they could see.

Corn: Nobody listening to the output?

Hilbert: Overnight, no. Corn, you'd have liked the hours. The fix was a routing chart, handwritten, and I still have it taped inside a cupboard door. Anything in my flat with more than two inputs, I check it against that chart before I plug it in, because I don't trust software to show me where a signal actually goes. The home setup goes through the same check. Corn, your cough at fourteen minutes is on the take, by the way, I'll cut it after.

Herman: The fader looked correct. That's it exactly, that's the whole interference story in four words. The strength value is where you set it, and the delta lands somewhere you didn't choose.

Corn: You've been at that desk twenty years and never once trusted a meter.

Hilbert: Meters show levels. They don't show where. The chart shows where.

Corn: Fair.

Herman: Here's the question I'd leave open, Corn. If the per-span dial does arrive, who sets it? A human with a fader, a controller model, or the text itself. The image field's answer is regional prompting plus learned gates, which suggests the human ends up setting policy rather than levels. You say which span matters, the machine sets the number.

Corn: So the DJ becomes the producer, and the desk plays itself. For voice specifically, nothing here is blocked by software support. It's blocked by three separate things. Interference, which has a maturing literature. Time-varying control, which barely exists outside ComfyUI. And a span-level control signal, which nobody has even agreed the shape of.

Herman: And Hilbert's routing point is the honest footnote to all three. Even when the dials arrive, the thing that will actually bite is where each delta lands.

Corn: Thanks as always to Hilbert Flumingtop, producing this one from behind a cupboard door with a chart on it, apparently. If you want more of this, try episode seventeen, Cloud Render Superpowers; and episode five, Fine-Tuning ASR For Maximal Usability. This has been My Weird Prompts. If you enjoyed it, the review button is right there on your app and it takes four seconds, I timed it.

Herman: See you tomorrow, Corn.

Corn: See you tomorrow, Herman.