Second only to humans: SberDevices language models best in the world at Russian text comprehension

SberDevices’ ruRoberta-large finetune text model received the best text comprehension score, coming second only to humans in terms of accuracy according to Russian SuperGLUE, the main Russian-language benchmark for evaluating large text models. Four more models designed by SberDevices made the top six: ruT5-large-finetune, ruBert-large finetune, ruT5-base-finetune, ruBert-base finetune.

Having successfully trained the ruBERT language model, Sber began to develop its more advanced version – ruRoBERTa. In terms of architecture, the advanced version is the same BERT trained on a large text corpus, but for the task of masked token recovery, featuring a large batch size and the BBPE tokenizer from the ruGPT-3 neural network. Training the model on the Christofari supercomputer took three weeks. The final dataset (250 GB of text) was similar to the one used for ruGPT-3, but with English and a section of the "dirty" Common Crawl removed.

The Russian SuperGLUE (General Language Understanding Evaluation) leaderboard is the first neural network ranking for the Russian language. The ranking is based on how well a neural network completes tasks on logic, common sense, goal-setting, and meaning. It is an open project that is used by all data researchers working with Russian-language neural networks.

The overall language comprehension evaluation begins with the diagnostic dataset, a set of tests reflecting different linguistic phenomena. It shows the language's linguistic phenomena and how well the ruRoberta-large finetune model understands certain particularities. The model’s high score (LiDiRus) means that it is not only memorizing tasks or guessing the result, but also learning the particularities and multitude of phenomena of the Russian language.

Each model is evaluated through a variety of tasks, including DaNetQA, a set of questions on common sense and knowledge with yes/no answers, RCB (Russian Commitment Bank), a classification of causal linkages between a text and a hypothesis, and PARus (Plausible Alternatives for Russian), which assesses goal-setting, choice of alternative options based on common sense, and more.

"Sber’s leading experts have spent several years perfecting Russian-language neural networks. For objective evaluation purposes, we developed the Russian SuperGLUE leaderboard, which is the first of its kind and clearly shows the progress being made. Our ultimate goal is the creation of reliable intelligent systems that will solve a variety of Russian-language tasks and will serve as the predecessors of robust, homegrown artificial intelligence," says David Rafalovsky, Executive Vice President, Sberbank; CTO, Head of Technology, Sber

PJSC Sberbank
Legal and Mailing Addresses:
19 Vavilova St., 117997 Moscow, Russia

Tel: 8 (800) 555-5550
Press office: media@sberbank.ru

PJSC Sberbank is Russia’s largest bank and a leading global financial institution. Holding almost one-third of aggregate Russian banking sector assets, Sberbank is the key lender to the national economy and one of the biggest deposit takers in Russia. The Government of the Russian Federation represented by the Ministry of Finance of the Russian Federation is the principal shareholder of PJSC Sberbank owning 50% plus one voting share of the bank’s authorized capital, with the remaining 50% minus one voting share held by domestic and international investors. Sberbank has customers in 18 countries. The bank has a major distribution network in Russia with about 14,000 branches, while its international operations – subsidiary banks, branches, and chapters – include the UK, US, CIS, Central and Eastern Europe, India, China, and other countries. It holds general banking license No. 1481 dd. August 11, 2015, from the Bank of Russia. Official websites of the bank: www.sberbank.com (Sberbank Group website), www.sberbank.ru.

In 2020 Sberbank underwent a rebranding, offering financial and non-financial services of the bank and Sberbank Group to individual and corporate customers. Today, the Sber ecosystem is a raft of services for life and daily assistance in handling pressing everyday issues for individual customers and businesses. The Sber ecosystem website: www.sber.ru.

This release was published on openPR.