E5 Text Embeddings

Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

Improving Text Embeddings with Large Language Models. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

Text Embeddings by Weakly-Supervised Contrastive Pre-training. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022

LLM based Models

English Pre-trained Models

The models with -unsupervised suffix only pre-trains on unlabeled datasets.

Multilingual Pre-trained Models

Install Python Package Requirements

pip install -r requirements.txt

For e5-mistral-7b-instruct, it would require transformers>=4.34 to load Mistral model.

Evaluate on the BEIR Benchmark

After installing the required python packages, run the following command on GPU machines:

bash scripts/eval_mteb_beir.sh intfloat/e5-small-v2

By default, the evaluation script will use all the available GPUs.

Caution: it could take quite a long time (~10 hours) due to corpus encoding. For intfloat/e5-mistral-7b-instruct, it could take even longer (several days).

Evaluate on the MTEB Benchmark

Run the following command:

bash scripts/eval_mteb_except_retrieval.sh intfloat/e5-small-v2

For multilingual models, simply add a --multilingual suffix:

bash scripts/eval_mteb_except_retrieval.sh intfloat/multilingual-e5-base --multilingual

Other Resources

The data for our proposed synthetic task personalized passkey retrieval is available at https://huggingface.co/datasets/intfloat/personalized_passkey_retrieval.

Troubleshooting

If you encounter OOM error, please try to reduce the batch size.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

@article{wang2023improving,
  title={Improving Text Embeddings with Large Language Models},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2401.00368},
  year={2023}
}

@article{wang2022text,
  title={Text Embeddings by Weakly-Supervised Contrastive Pre-training},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2212.03533},
  year={2022}
}

License

This project is licensed under the license found in the LICENSE file in the root directory of this source tree. Microsoft Open Source Code of Conduct