FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec
This project is still working on progress. To make FunCodec better, please let me know your concerns and feel free to comment them in the Issues part.
News
- 2023.12.22 🎉🎉: We release the training and inference recipes for LauraTTS as well as pre-trained models.
LauraTTS is a powerful codec-based zero-shot text-to-speech synthesizer,
which outperforms VALL-E in terms of semantic consistency and speaker similarity.
Please refer
egs/LibriTTS/text2speech_laura/README.mdfor more details.
Installation
git clone https://github.com/alibaba/FunCodec.git && cd FunCodec pip install --editable ./
Available models
🤗 links to the Huggingface model hub, while ⭐ refers the Modelscope.
| Model name | Model hub | Corpus | Bitrate | Parameters | Flops |
|---|---|---|---|---|---|
| audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch | 🤗 ⭐ | General | 250~8000 | 57.83 M | 7.73G |
| audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch | 🤗 ⭐ | General | 500~16000 | 14.85 M | 3.72 G |
| audio_codec-encodec-en-libritts-16k-nq32ds640-pytorch | 🤗 ⭐ | LibriTTS | 250~8000 | 57.83 M | 7.73G |
| audio_codec-encodec-en-libritts-16k-nq32ds320-pytorch | 🤗 ⭐ | LibriTTS | 500~16000 | 14.85 M | 3.72 G |
| audio_codec-freqcodec_magphase-en-libritts-16k-gr8nq32ds320-pytorch | 🤗 ⭐ | LibriTTS | 500~16000 | 4.50 M | 2.18 G |
| audio_codec-freqcodec_magphase-en-libritts-16k-gr1nq32ds320-pytorch | 🤗 ⭐ | LibriTTS | 500~16000 | 0.52 M | 0.34 G |
Model Download
Download models from ModelScope
Please refer egs/LibriTTS/codec/encoding_decoding.sh to download pretrained models:
cd egs/LibriTTS/codec model_name=audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch bash encoding_decoding.sh --stage 0 --model_name ${model_name} --model_hub modelscope # The pre-trained model will be downloaded to exp/audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch
Download models from Huggingface
Please refer egs/LibriTTS/codec/encoding_decoding.sh to download pretrained models:
cd egs/LibriTTS/codec model_name=audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch bash encoding_decoding.sh --stage 0 --model_name ${model_name} --model_hub huggingface # The pre-trained model will be downloaded to exp/audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch
Inference
Batch inference
Please refer egs/LibriTTS/codec/encoding_decoding.sh to perform encoding and decoding.
Extract codes with an input file input_wav.scp,
and the codes will be saved to output_dir/codecs.txt in a format of jsonl.
cd egs/LibriTTS/codec bash encoding_decoding.sh --stage 1 --batch_size 16 --num_workers 4 --gpu_devices "0,1" \ --model_dir exp/${model_name} --bit_width 16000 \ --wav_scp input_wav.scp --out_dir outputs/codecs/ # input_wav.scp has the following format： # uttid1 path/to/file1.wav # uttid2 path/to/file2.wav # ...
Decode codes with an input file codecs.txt,
and the reconstructed waveform will be saved to output_dir/logdir/output.*/*.wav.
bash encoding_decoding.sh --stage 2 --batch_size 16 --num_workers 4 --gpu_devices "0,1" \ --model_dir exp/${model_name} --bit_width 16000 --file_sampling_rate 16000 \ --wav_scp codecs.txt --out_dir outputs/recon_wavs # codecs.scp is the output of above encoding stage, which has the following format： # uttid1 [[[1, 2, 3, ...],[2, 3, 4, ...], ...]] # uttid2 [[[9, 7, 5, ...],[3, 1, 2, ...], ...]]
Training
Training on open-source corpora
For commonly-used open-source corpora, you can train a model using the recipe in egs directory.
For example, to train a model on the LibriTTS corpus, you can use egs/LibriTTS/codec/run.sh:
# entry the LibriTTS recipe directory cd egs/LibriTTS/codec # run data downloading, preparation and training stages with 2 GPUs (device 0 and 1) bash run.sh --stage 0 --stop_stage 3 --gpu_devices 0,1 --gpu_num 2
We recommend run the script stage by stage to have an overview of FunCodec.
Training on customized data
For uncovered corpora or customized dataset, you can prepare the data by yourself.
In general, FunCodec employs the kaldi-like wav.scp file to organize the data files.
wav.scp has the following format:
# for waveform files uttid1 /path/to/uttid1.wav uttid2 /path/to/uttid2.wav # for kaldi-ark files uttid3 /path/to/ark1.wav:10 uttid4 /path/to/ark1.wav:200 uttid5 /path/to/ark2.wav:10
As shown in the above example, FunCodec supports the combination of waveforms or kaldi-ark files
in one wav.scp file for both training and inference.
Here is a demo script to train a model on your customized dataset named foo:
cd egs/LibriTTS/codec # 0. make the directory for train, dev and test sets mkdir -p dump/foo/train dump/foo/dev dump/foo/test # 1a. if you already have the wav.scp file, just place them under the corresponding directories mv train.scp dump/foo/train/; mv dev.scp dump/foo/dev/; mv test.scp dump/foo/test/; # 1b. if you don't have the wav.scp file, you can prepare it as follows find path/to/train_set/ -iname "*.wav" | awk -F '/' '{print $(NF),$0}' | sort > dump/foo/train/wav.scp find path/to/dev_set/ -iname "*.wav" | awk -F '/' '{print $(NF),$0}' | sort > dump/foo/dev/wav.scp find path/to/test_set/ -iname "*.wav" | awk -F '/' '{print $(NF),$0}' | sort > dump/foo/test/wav.scp # 2. collate shape files mkdir exp/foo_states/train exp/foo_states/dev torchrun --nproc_per_node=4 --master_port=1234 scripts/gen_wav_length.py --wav_scp dump/foo/train/wav.scp --out_dir exp/foo_states/train/wav_length cat exp/foo_states/train/wav_length/wav_length.*.txt | shuf > exp/foo_states/train/speech_shape torchrun --nproc_per_node=4 --master_port=1234 scripts/gen_wav_length.py --wav_scp dump/foo/dev/wav.scp --out_dir exp/foo_states/dev/wav_length cat exp/foo_states/dev/wav_length/wav_length.*.txt | shuf > exp/foo_states/dev/speech_shape # 3. train the model with 2 GPUs (device 4 and 5) on the customized dataset (foo) bash run.sh --gpu_devices 4,5 --gpu_num 2 --dumpdir dump/foo --state_dir foo_states
Acknowledge
- We had a consistent design of FunASR, including dataloader, model definition and so on.
- We borrowed a lot of code from Kaldi for data preparation.
- We borrowed a lot of code from ESPnet. FunCodec follows up the training and finetuning pipelines of ESPnet.
- We borrowed the design of model architecture from Enocdec and Enocdec_Trainner.
License
This project is licensed under The MIT License. FunCodec also contains various third-party components and some code modified from other repos under other open source licenses.
Citations
@misc{du2023funcodec, title={FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec}, author={Zhihao Du, Shiliang Zhang, Kai Hu, Siqi Zheng}, year={2023}, eprint={2309.07405}, archivePrefix={arXiv}, primaryClass={cs.Sound} }