Limitations of Language Models in The Oil & Gas Upstream Operations

Language models has powered a lot of applications and developments in the past years in different domains. One language model that became popular in the recent years is BERT which is used as a base for many different natural language processing tasks such as classification and question answering. We wanted to apply BERT on some upstream oil and gas related text and study its performance on this domain. We decided to choose classification as a task to experiment with it. We tried a mix of domain adaptation and transfer learning to understand the potential of language models in general and BERT in specific on our domain. Our results emphasized the need to create domain-specific datasets for the oil and gas world and subsequently build models using them.