At Gerulata Technologies, NLP technologies are a vital component of our toolkit. In 2020 we developed the first masked language model pretrained specifically for Slovak — and we’ve now decided to open-source it.
Slovak is classified as a “low-resource language” in NLP circles, characterised by a scarcity of published datasets suitable for training machine-learning models. While acquiring extensive, high-quality datasets for major languages like English or Spanish is straightforward through internet sources, comparable resources for smaller languages remain unavailable. Building custom datasets requires substantial web crawling, text cleaning and preprocessing work.
After assembling a training dataset, the next challenge is training the model — a computationally demanding process typically requiring access to specialised hardware. Once trained, however, the model operates efficiently on standard hardware and enables numerous practical applications.
Working alongside our KInIT research partners, we published a paper documenting the model’s training process and performance evaluation. SlovakBERT is compared against prominent cross-lingual alternatives. The paper is available on Arxiv, and the model itself, along with usage examples, is on Huggingface.
