ZipVoice-Dialog⚡️

Non-Autoregressive Spoken Dialogue Generation with Flow Matching


Paper | Codes and models | Datasets

Abstract Generating spoken dialogue is more challenging than monologue text-to-speech (TTS) due to the need for realistic turn-taking and distinct speaker timbres. Existing spoken dialogue generation models, being auto-regressive, suffer from slow and unstable inference. To overcome these limitations, we introduce ZipVoice-Dialog, a non-autoregressive zero-shot spoken dialogue generation model built upon flow matching. Key designs include: 1) speaker-turn embeddings for precise speaker turn-taking; 2) a curriculum learning strategy for stable speech-text alignment; 3) specialized strategies to enable stereo dialogue generation. Additionally, recognizing the lack of open-source large-scale spoken dialogue datasets, we curated OpenDialog, a 6.8k-hour spoken dialogue dataset from in-the-wild speech data. Furthermore, we established a benchmark to comprehensively evaluate various models. Experimental results demonstrate that ZipVoice-Dialog achieves superior performance in intelligibility, speaker turn-taking accuracy, speaker similarity, and inference speed. Our codes, model checkpoints, demo samples, and the OpenDialog dataset are all publicly available.


Contents

Model Overview

Figure 1: Illustration of ZipVoice-Dialog training (left) and inference (right).

Sinlge-Channel Dialogue Generation (English)

Prompts and texts are from the test-dialog-en test set.

Prompt [S1] Prompt [S2] Text ZipVoice-Dialog Mooncast Dia
[S1] I'm a big bookworm. [S2] Yeah, I have a few books- [S1] Yeah. [S2] ... that I've been trying to pick up, but, um, haven't had the time yet. But, um, I think I'll get into it at some point this summer. [S1] Uh, what are the titles? [S2] Oh, um, okay, well, one of them, it's, um, it's the, um, biography of, uh, Alexander Hamilton. I bought the book- [S1] Oh. [S2] ... when, when Hamilton first came out.
[S1] Yeah, well, what were, what were you gonna say? 'Cause I was just gonna change subject. [S2] Oh, well, I, I was just going to ask you, um, have you ever encountered any signs that you missed or any hints, like I did? [S1] Oh, not that I know of. Well, okay, for my life, oh my God, a sob story type, I'm just kidding. But, um, it's, it's always, um, uh, at least I, I have, I haven't known anyone to be interested, or like, if they did, they were, like, too subtle about it, you know? 'Cause, uh, I'm still on the, like, um, Yeah, like, uh, I mean, there's one person that I think maybe he could've been interested, but honestly, like, we're just, like, really close friends, so I don't know if, I, I don't even know if he was interested, you know? Um, it's just, um, it's just like occasionally, like, the, the vibes are like kinda off, but that's all.
[S1] I know, it could be worse, but- [S2] They didn't count the pages for you, right? [S1] Yeah, but, meh. What have you spent money on lately? [S2] I actually haven't been spending money on games that much. [S1] Oh, yeah, that's good.
[S1] Well, what was it, uh, like elementary to middle school, as long as I didn't fail, I was good. [S2] That was, yeah, that's, that was my mindset, uh, throughout high school. If you, if you pass it, uh, and you did your, and you did your best at the last, uh, at the final stretch, you have no shame. There's nothing to be shameful about. [S1] Yeah, but, uh, in high school, I actually started worrying about, like, GPAs and all that. [S2] Oh, yeah. [S1] Yeah, I'm not gonna lie, history, history class was, was a vibe, like.

Sinlge-Channel Dialogue Generation (Chinese)

Prompts and texts are from the test-dialog-zh test set.

Prompt [S1] Prompt [S2] Text ZipVoice-Dialog Mooncast
[S1] 是的 [S2] 像这种情，像这种情况下，我感觉以后服务业肯定，应，应该是会越来越发达的 [S1] 对这是必然的，但是因为我们人，人，人多的原因，所以说服，服务业可能就是发展得比较慢 [S2] 嗯，但是 但这种东西吧，我感觉肯定会发展，毕竟，毕竟这是一种趋势吧 [S1] 对
[S1] 好废呀，没什么用 [S2] 也不说没什么用吧 [S1] 就还是得看自己吧，师傅领进门修行靠自身 [S2] 嗯 [S1] 我能是什么样子都是这样子的
[S1] 我也觉得好像是，因为好像到最后齐铭，就是那个男二呀 还有那个顾森西啊好像都转学了 [S2] 嗯 [S1] 就证明好像那个女的那个女主好像已经不在了，就真的被那个河水给淹死了好像对吧？ 然后就像那个年少的你，我前面说的那个也是这种题材的嘛。 [S2] 嗯 [S1] 就是那个女的是
[S1] 那变成个那样的，你不觉得搞笑 [S2] 三个人 [S1] 铁三角 怎么成为黄金铁三角无法理解，我猜那女的暴力型看那个女的 [S2] 那个女的七十多级 [S1] 那他就负责指导嘛那 [S2] 那二龙和大师两个人是同一个宗门的 [S1] 嗯，二龙和大师他们是哪个宗门的？ [S2] 蓝电霸王龙的

Stereo Dialogue Generation

Stereo prompts and texts are from the test-dialog-en test sets, which is constructed from a stereo speech dataset.

Prompt [S1] Prompt [S2] Text ZipVoice-Dialog-Stereo
[S1] I'm a big bookworm. [S2] Yeah, I have a few books- [S1] Yeah. [S2] ... that I've been trying to pick up, but, um, haven't had the time yet. But, um, I think I'll get into it at some point this summer. [S1] Uh, what are the titles? [S2] Oh, um, okay, well, one of them, it's, um, it's the, um, biography of, uh, Alexander Hamilton. I bought the book- [S1] Oh. [S2] ... when, when Hamilton first came out.
[S1] Yeah, well, what were, what were you gonna say? 'Cause I was just gonna change subject. [S2] Oh, well, I, I was just going to ask you, um, have you ever encountered any signs that you missed or any hints, like I did? [S1] Oh, not that I know of. Well, okay, for my life, oh my God, a sob story type, I'm just kidding. But, um, it's, it's always, um, uh, at least I, I have, I haven't known anyone to be interested, or like, if they did, they were, like, too subtle about it, you know? 'Cause, uh, I'm still on the, like, um, Yeah, like, uh, I mean, there's one person that I think maybe he could've been interested, but honestly, like, we're just, like, really close friends, so I don't know if, I, I don't even know if he was interested, you know? Um, it's just, um, it's just like occasionally, like, the, the vibes are like kinda off, but that's all.
[S1] I know, it could be worse, but- [S2] They didn't count the pages for you, right? [S1] Yeah, but, meh. What have you spent money on lately? [S2] I actually haven't been spending money on games that much. [S1] Oh, yeah, that's good.
[S1] Well, what was it, uh, like elementary to middle school, as long as I didn't fail, I was good. [S2] That was, yeah, that's, that was my mindset, uh, throughout high school. If you, if you pass it, uh, and you did your, and you did your best at the last, uh, at the final stretch, you have no shame. There's nothing to be shameful about. [S1] Yeah, but, uh, in high school, I actually started worrying about, like, GPAs and all that. [S2] Oh, yeah. [S1] Yeah, I'm not gonna lie, history, history class was, was a vibe, like.