Towards Linguistically Informed Multi-objective Transformer Pre-training for Natural Language Inference
We introduce a linguistically enhanced combination of pre-training methods for transformers. The pre-training objectives include POS-tagging, synset prediction based on semantic knowledge graphs, and parent prediction based on dependency parse trees. Our approach achieves competitive results on the Natural Language Inference task, compared to the state of the art. Specifically for smaller models, the method results in a significant performance boost, emphasizing the fact that intelligent pre-training can make up for fewer parameters and help building more efficient models. Combining POS-tagging and synset prediction yields the overall best results.
- Veröffentlicht in:
Advances in Information Retrieval. ECIR 2023 - Typ:
Inproceedings - Autoren:
- Jahr:
2023 - Source:
https://link.springer.com/chapter/10.1007/978-3-031-28238-6_46
Informationen zur Zitierung
: Towards Linguistically Informed Multi-objective Transformer Pre-training for Natural Language Inference, Advances in Information Retrieval. ECIR 2023, 2023, https://link.springer.com/chapter/10.1007/978-3-031-28238-6_46, Pielka.etal.2023a,
@Inproceedings{Pielka.etal.2023a,
author={Pielka, Maren; Schmidt, Svetlana; Pucknat, Lisa; Sifa, Rafet},
title={Towards Linguistically Informed Multi-objective Transformer Pre-training for Natural Language Inference},
booktitle={Advances in Information Retrieval. ECIR 2023},
url={https://link.springer.com/chapter/10.1007/978-3-031-28238-6_46},
year={2023},
abstract={We introduce a linguistically enhanced combination of pre-training methods for transformers. The pre-training objectives include POS-tagging, synset prediction based on semantic knowledge graphs, and parent prediction based on dependency parse trees. Our approach achieves competitive results on the Natural Language Inference task, compared to the state of the art. Specifically for smaller models,...}}
