BeTS: Abstractive Text Summarization with Transfer Learning

We introduce BeTS, which stands for Bert Transformer Summarizer, an abstractive summarization model that is built using transfer learning. Our model consists of a pre-trained BERT encoder and a generative transformer decoder to produce abstract summaries. The model is trained by a simple two step process, where the decoder is first trained and then the entire model is fine-tuned end-to-end. We train the model with CNN/Daily Mail summarization dataset and use the trained weights for transfer learning on the AMI dialog summarization dataset. We successfully demonstrate that our model outperforms the baseline models by at least 7.65 ROUGE points on the AMI data. We also demonstrate that our model shows good generalization performance by testing on different unseen input data and compare the generated results. We are able to generate significantly better summaries on inputs such as dialog compared to the current state-of-the-art model.