IndicFT

fastText is a subword-aware word embedding model. It is particularly well-suited for Indian languages due to their highly agglutinative morphology. We train fastText models on our IndicNLP Corpora and evaluate them on a set of tasks to measure its performance.

Our fastText models are available for 11 Indian languages: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.

To use our fastText models, first download them. Next, install the fastText library:

pip3 install fasttext

and then load the models like this:

import fasttext
model = fasttext.load_model(path_to_binary_file)

For instructions on how to use these models, please refer to the official fastText documentation

For a full results of evaluation, check our paper. Here, we show some of the evaluations.

If you are using IndicFT, please cite the following paper:

@inproceedings{kakwani2020indicnlpsuite,
    title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
    author={Divyanshu Kakwani and Anoop Kunchukuttan and Satish Golla and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
    year={2020},
    booktitle={Findings of EMNLP},
}

The IndicFT embeddings are released under the MIT License.