Abstract

In this study, the extractive summarization using sentence embeddings generated by the finetuned Bidirectional Encoder Representations from Transformers (BERT) models and the k-means clustering method has been investigated. To show how the BERT model can capture the knowledge in specific domains like engineering design and what it can produce after being finetuned based on domain-specific data sets, several BERT models are trained, and the sentence embeddings extracted from the finetuned models are used to generate summaries of a set of papers. Different evaluation methods are then applied to measure the quality of summarization results. Both the machine evaluation method Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and a human-based evaluation method are used for the comparison study. The results indicate that the BERT model finetuned with a larger dataset can generate summaries with more domain terminologies than the pretrained BERT model. Moreover, the summaries generated by BERT models have more contents overlapping with original documents than those obtained through other popular non-BERT-based models. The experimental results indicate that the BERT-based method can provide better and more informative summaries to engineers. It has also been demonstrated that the contextualized representations generated by BERT-based models can capture information in text and have better performance in applications like text summarizations after being trained by domain-specific data sets.

References

1.
Fleuren
,
W.
, and
Alkema
,
W.
,
2015
, “
Application of Text Mining in the Biomedical Domain
,”
Methods
,
74
(
3
), pp.
97
106
.
2.
Ferreira
,
R.
,
de Souza Cabral
,
L.
,
Lins
,
R.
,
Pereira e Silva
,
G.
,
Freitas
,
F.
,
Cavalcanti
,
G.
,
Lima
,
R.
,
Simske
,
S.
, and
Favaro
,
L.
, “
2013. Assessing Sentence Scoring Techniques for Extractive Text Summarization
,”
Expert Syst. Appl.
,
40
(
14
), pp.
5755
5764
.
3.
Lloret
,
E.
, and
Palomar
,
M.
,
2012
, “
Text Summarisation in Progress: A Literature Review
,”
Artif. Intell. Rev.
,
37
(
1
), pp.
1
41
.
4.
Mishra
,
R.
,
Bian
,
J.
,
Fiszman
,
M.
,
Weir
,
C. R.
,
Jonnalagadda
,
S.
,
Mostafa
,
J.
, and
Del Fiol
,
G.
,
2014
, “
Text Summarization in the Biomedical Domain: A Systematic Review of Recent Research
,”
J. Biomed. Inform.
,
52
(
12
), pp.
457
467
.
5.
Reeve
,
L. H.
,
Han
,
H.
, and
Brooks
,
A. D.
,
2007
, “
The Use of Domain-Specific Concepts in Biomedical Text Summarization
,”
Inf. Process. Manag.
,
43
(
6
), pp.
1765
1776
.
6.
Plaza
,
L.
,
Díaz
,
A.
, and
Gervás
,
P.
,
2011
, “
A Semantic Graph-Based Approach to Biomedical Summarization
,”
Artif. Intell. Med.
,
53
(
1
), pp.
1
14
.
7.
Ji
,
X.
,
Ritter
,
A.
, and
Yen
,
P. Y.
,
2017
, “
Using Ontology-Based Semantic Similarity to Facilitate the Article Screening Process for Systematic Reviews
,”
J. Biomed. Inform.
,
69
(
5
), pp.
33
42
.
8.
Kågebäck
,
M.
,
Mogren
,
O.
,
Tahmasebi
,
N.
, and
Dubhashi
,
D.
,
2014
, “
Extractive Summarization Using Continuous Vector Space Models
,”
Proceedings of the 2nd Workshop on Continuous Vector Space Models and Their Compositionality (CVSC)
,
Gothenburg, Sweden
,
Apr. 26–30
, pp.
31
39
.
9.
Camacho-Collados
,
J.
, and
Pilehvar
,
M. T.
,
2018
, “
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning
,”
J. Artif. Intell. Res.
,
63
(
1
), pp.
743
788
.
10.
Cheng
,
J.
, and
Lapata
,
M.
,
2016
, “
Neural summarization by extracting sentences and words
,”
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
,
Berlin, Germany
,
August
.
11.
Alami
,
N.
,
Meknassi
,
M.
, and
En-nahnahi
,
N.
,
2019
, “
Enhancing Unsupervised Neural Networks-Based Text Summarization With Word Embedding and Ensemble Learning
,”
Expert Syst. Appl.
,
123
(
6
), pp.
195
211
.
12.
Devlin
,
J.
,
Chang
,
M. W.
,
Lee
,
K.
, and
Toutanova
,
K.
,
2018
, “
BERT: Pre-training of deep bidirectional transformers for language understanding
,”
North American Association for Computational Linguistics (NAACL)
,
Minneapolis, MN
,
June
.
13.
Zhang
,
H.
,
Xu
,
J.
, and
Wang
,
J.
,
2019
, “
Pretraining-Based Natural Language Generation for Text Summarization
,”
In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
,
Hong Kong, China
,
November
, pp.
789
797
.
14.
Miller
,
D.
,
2019
, “
Leveraging BERT for Extractive Text Summarization on Lectures
,”
CoRR
.https://arxiv.org/abs/1906.04165
15.
Goldberg
,
Y.
, and
Hirst
,
G.
,
2017
,
Neural Network Methods for Natural Language Processing
, Vol.
10
,
Synth. Lect. Hum. Lang. Technol., Morgan & Claypool Publishers
, pp.
1
309
.
16.
Mikolov
,
T.
,
Chen
,
K.
,
Corrado
,
G.
, and
Dean
,
J.
,
2013
, “
Efficient estimation of word representations in vector space
,”
International Conference on Learning Representations: Workshops Track
,
Scottsdale, AZ
,
Jan
.
17.
Ling
,
W.
,
Dyer
,
C.
,
Black
,
A. W.
, and
Trancoso
,
I.
,
2015
, “
Two/too Simple Adaptations of Word2vec for Syntax Problems
,”
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
,
Denver, CO
,
May 31–June 5
, pp.
1299
1304
.
18.
Hendrycks
,
D.
, and
Gimpel
,
K.
,
2016
, “
Bridging Nonlinearities and Stochastic Regularizers With Gaussian Error Linear Units
,”
CoRR
.https://arxiv.org/abs/1606.08415
19.
Peters
,
M.
,
Neumann
,
M.
,
Iyyer
,
M.
,
Gardner
,
M.
,
Clark
,
C.
,
Lee
,
K.
, and
Zettlemoyer
,
L.
,
2018
, “
Deep Contextualized Word Representations
,”
The North American Chapter of the Association for Computational Linguistics (NAACL)
,
New Orleans, LA
,
June 1–6
.
20.
Radford
,
A.
,
Narasimhan
,
K.
,
Salimans
,
T.
, and
Sutskever
,
I.
,
2018
,
Improving Language Understanding by Generative Pre-Training
, https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
21.
Howard
,
J.
, and
Ruder
,
S.
, “
Universal language model finetuning for text classification
Proceeding of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)
,
Melbourne, Australia
,
July
.
22.
Vaswani
,
A.
,
Shazeer
,
N.
,
Parmar
,
N.
,
Uszkoreit
,
J.
,
Jones
,
L.
,
Gomez
,
A. N.
,
Kaiser
,
Ł
, and
Polosukhin
,
I.
,
2017
, “
Attention Is All You Need
,”
Proceeding of the 31st Conference on Neural Information Processing Systems (NIPS)
,
Long Beach, CA
,
December
.
23.
Radford
,
A.
,
Wu
,
J.
,
Child
,
R.
,
Luan
,
D.
,
Amodei
,
D.
, and
Sutskever
,
I.
,
2019
, “
Language Models are Unsupervised Multitask Learners
,”
OpenAI blog
,
1
(
8
), p.
9
.
24.
Brown
,
T. B.
,
Mann
,
B.
,
Ryder
,
N.
,
Subbiah
,
M.
,
Kaplan
,
J.
,
Dhariwal
,
P.
,
Neelakantan
,
A.
, et al
2020
, “
Language Models are Few-Shot Learners
,”
Proceeding of 2020 Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS)
,
Virtual
,
Dec. 6–12
.
25.
Zhu
,
Q.
, and
Luo
,
J.
,
2021
,
Generative Pre-Trained Transformer for Design Concept Generation: An Exploration; arXiv:2111.08489v1 [cs.CL]
.
26.
Zhu
,
Y.
,
Kiros
,
R.
,
Zemel
,
R.
,
Salakhutdinov
,
R.
,
Urtasun
,
R.
,
Torralba
,
A.
, and
Fidler
,
S.
,
2015
, “
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
,” Proceedings of the IEEE International Conference on Computer Vision, pp.
19
27
.
27.
Yao
,
J. G.
,
Wan
,
X.
, and
Xiao
,
J.
,
2017
, “
Recent Advances in Document Summarization
,”
Knowl. Inf. Syst.
,
53
(
2
), pp.
297
336
.
28.
Akbik
,
A.
,
Blythe
,
D.
, and
Vollgraf
,
R.
,
2018
, “
Contextual String Embeddings for Sequence Labeling
,”
Proceedings of the 27th International Conference on Computational Linguistics
,
Santa Fe, NM
,
Aug. 20–26
, pp.
1638
1649
.
29.
Lee
,
J.
,
Yoon
,
W.
,
Kim
,
S.
,
Kim
,
D.
,
Kim
,
S.
,
So
,
C. H.
, and
Kang
,
J.
,
2020
, “
BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining
,”
Bioinformatics
,
36
(
4
), pp.
1234
1240
.
30.
Alsentzer
,
E.
,
Murphy
,
J. R.
,
Boag
,
W.
,
Weng
,
W. H.
,
Jin
,
D.
,
Naumann
,
T.
, and
McDermott
,
M.
,
2019
, “
Publicly available clinical BERT embeddings
,”
Proceedings of the 2nd Clinical Natural Language Processing Workshop
,
Minneapolis, MN
,
June
.
31.
Si
,
Y.
,
Wang
,
J.
,
Xu
,
H.
, and
Roberts
,
K.
,
2019
, “
Enhancing Clinical Concept Extraction With Contextual Embeddings
,”
J. Am. Med. Inform. Assoc.
,
26
(
11
), pp.
1297
1304
.
32.
Peng
,
Y.
,
Yan
,
S.
, and
Lu
,
Z.
,
2019
, “
Transfer Learning in Biomedical Natural Language Processing: An Evaluation of Bert and Elmo on Ten Benchmarking Datasets
,”
Proceedings of the 18th BioNLP Workshop and Shared Task
,
Florence, Italy
,
August
.
33.
Beigbeder
,
M.
, and
Mercier
,
A.
,
2005
, “
An Information Retrieval Model Using the Fuzzy Proximity Degree of Term Occurences
,”
Proceedings of the 2005 ACM Symposium on Applied Computing
,
Santa Fe, NM
,
Mar. 13–17
, pp.
1018
1022
.
34.
Castells
,
P.
,
Fernandez
,
M.
, and
Vallet
,
D.
,
2006
, “
An Adaptation of the Vector-Space Model for Ontology-Based Information Retrieval
,”
IEEE Trans. Knowl. Data Eng.
,
19
(
2
), pp.
261
272
.
35.
Zhang
,
X.
,
Hou
,
X.
,
Chen
,
X.
, and
Zhuang
,
T.
,
2013
, “
Ontology-Based Semantic Retrieval for Engineering Domain Knowledge
,”
Neurocomputing
,
116
(
9
), pp.
382
391
.
36.
Sanya
,
I. O.
, and
Shehab
,
E. M.
,
2015
, “
A Framework for Developing Engineering Design Ontologies Within the Aerospace Industry
,”
Int. J. Prod. Res.
,
53
(
8
), pp.
2383
2409
.
37.
Zhang
,
C.
,
Zhou
,
G.
,
Lu
,
Q.
, and
Chang
,
F.
,
2017
, “
Graph-Based Knowledge Reuse for Supporting Knowledge-Driven Decision-Making in New Product Development
,”
Int. J. Prod. Res.
,
55
(
23
), pp.
7187
7203
.
38.
Shi
,
F.
,
Chen
,
L.
,
Han
,
J.
, and
Childs
,
P.
,
2017
, “
A Data-Driven Text Mining and Semantic Network Analysis for Design Information Retrieval
,”
ASME J. Mech. Des.
,
139
(
11
), p.
111402
.
39.
Martinez-Rodriguez
,
J. L.
,
Lopez-arevalo
,
I.
, and
Rios-alvarado
,
A. B.
,
2018
, “
OpenIE-Based Approach for Knowledge Graph Construction From Text
,”
Expert Syst. Appl.
,
113
(
12
), pp.
339
355
.
40.
Sarica
,
S.
,
Luo
,
J.
, and
Wood
,
K. L.
,
2020
, “
TechNet: Technology Semantic Network Based on Patent Data
,”
Expert Syst. Appl.
,
142
(
3
), p.
112995
.
41.
Sarica
,
S.
, and
Luo
,
J.
,
2021
, “
Design Knowledge Representation With Technology Semantic Network
,”
Proc. Des. Soc.
,
1
(
7
), pp.
1043
1052
.
42.
Siddharth
,
L.
,
Blessing
,
L. T. M.
,
Wood
,
K. L.
, and
Luo
,
J.
,
2021
, “
Engineering Knowledge Graph From Patent Database
,”
ASME J. Comput. Inf. Sci. Eng.
,
22
(
2
), p.
021008
.
43.
Hou
,
T.
,
Yannou
,
B.
,
Leroy
,
Y.
, and
Poirson
,
E.
,
2019
, “
Mining Changes of User Expectations Over Time From Online Reviews
,”
ASME J. Mech. Des.
,
141
(
9
), p.
091102
.
44.
Han
,
J.
,
Park
,
D.
,
Forbes
,
H.
, and
Schaefer
,
D.
,
2020
, “
A Computational Approach for Using Social Networking Platforms to Support Creative Idea Generation
,”
Procedia CIRP
,
91
, pp.
382
387
.
45.
Han
,
Y.
, and
Moghaddam
,
M.
,
2020
, “
Eliciting Attribute-Level User Needs From Online Reviews With Deep Language Models and Information Extraction
,”
ASME J. Mech. Des.
,
143
(
6
), p.
061403
.
46.
Akay
,
H.
, and
Kim
,
S. G.
,
2021
, “
Extracting Functional Requirements From Design Documentation Using Machine Learning
,”
Procedia CIRP
,
100
, pp.
31
36
.
47.
Ni
,
X.
,
Samet
,
A.
, and
Cavallucci
,
D.
,
2021
, “
Similarity-Based Approach for Inventive Design Solutions Assistance
,”
J. Intell. Manuf.
,
32
(
3
), pp.
1
18
.
48.
Gambhir
,
M.
, and
Gupta
,
V.
,
2017
, “
Recent Automatic Text Summarization Techniques: A Survey
,”
Artif. Intell. Rev.
,
47
(
1
), pp.
1
66
.
49.
Widyassari
,
A. P.
,
Rustad
,
S.
,
Shidik
,
G. F.
,
Noersasongko
,
E.
,
Syukur
,
A.
, and
Affandy
,
A.
,
2020
, “
Review of Automatic Text Summarization Techniques & Methods
,”
J. King Saud Univ. Comput. Inf. Sci.
,
34
(
4
), pp.
14
15
.
50.
Mani
,
I.
,
2001
,
Automatic Summarization
, Vol.
3
,
John Benjamins Publishing Co
, pp.
221
259
.
51.
Papineni
,
K.
,
Roukos
,
S.
,
Ward
,
T.
, and
Zhu
,
W. J.
,
2002
, “
Bleu: A Method for Automatic Evaluation of Machine Translation
,”
Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics
,
Philadelphia, PA
,
July 2002
.
52.
Lin
,
C. Y.
,
2004
,
Rouge: A Package for Automatic Evaluation of Summaries
. Text Summarization Branches Out.
53.
Denkowski
,
M.
, and
Lavie
,
A.
,
2014
, “
Meteor Universal: Language Specific Translation Evaluation for any Target Language
,”
Proceedings of the Ninth Workshop on Statistical Machine Translation
,
Baltimore, MD
,
June 26–27
, pp.
376
380
.
54.
Loper
,
E.
, and
Bird
,
S.
,
2002
, “
NLTK: The Natural Language Toolkit
,”
Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics
,
Philadelphia, PA
,
July
.
55.
Kingma
,
D. P.
, and
Ba
,
J.
,
2015
, “
Adam: A Method for Stochastic Optimization
,”
International Conference on Learning Representations (ICLR)
,
San Diego, CA
,
May
.
56.
Reimers
,
N.
, and
Gurevych
,
I.
,
2019
, “
Sentence-Bert: Sentence Embeddings Using Siamese Bert-Networks
,”
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
,
Hong Kong, China
,
November
.
57.
Bradley
,
P. S.
, and
Fayyad
,
U. M.
,
1998
, “Refining Initial Points for k-Means Clustering,”
ICML
, Vol.
98
, pp.
91
99
.
58.
Aria
,
H.
, and
Vanderwende
,
L.
,
2009
, “
Exploring Content Models for Multi-Document Summarization
,”
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics
,
Boulder, CO
,
June
.
59.
Mihalcea
,
R.
, and
Tarau
,
P.
,
2004
, “
Textrank: Bringing Order Into Text
,”
Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing
,
Barcelona, Spain
.
60.
Ozsoy
,
M.
,
Alpaslan
,
F.
, and
Cicekli
,
I.
,
2011
, “
Text Summarization Using Latent Semantic Analysis
,”
J. Inf. Sci.
,
37
(
4
), pp.
405
417
.
61.
Lin
,
C. Y.
,
2003
, “
Looking for a Few Good Metrics: Automatic Summarization Evaluation-how Many Samples are Enough?
Proceedings of the Fourth NII Testbeds and Community for Information Access Research Workshop (NTCIR)
,
Tokyo, Japan
,
April 2003–June 2004
.
62.
Allahyari
,
M.
,
Pouriyeh
,
S.
,
Assefi
,
M.
,
Safaei
,
S.
,
Trippe
,
E. D.
,
Gutierrez
,
J. B.
, and
Kochut
,
K.
,
2017
, “
Text Summarization Techniques: A Brief Survey
,”
CoRR
https://arxiv.org/abs/1707.02268.
63.
Gupta
,
V.
, and
Lehal
,
G. S.
,
2010
, “
A Survey of Text Summarization Extractive Techniques
,”
J. Emerg. Technol. Web Intell.
,
2
(
3
), pp.
258
268
.
64.
Schluter
,
N.
,
2017
, “
The Limits of Automatic Summarisation According to Rouge
,”
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics
,
Valencia, Spain
,
April
.
65.
Liu
,
F.
, and
Yang
,
L.
,
2008
, “
Correlation Between Rouge and Human Evaluation of Extractive Meeting Summaries
,”
Proceedings of the 2008 Annual Meeting of the Association for Computational Linguistics (ACL)
,
Columbus, OH
,
June
.
66.
Kawin Ethayarajh
,
K.
,
2019
, “
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
,”
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
,
Hong Kong, China
,
November
.
67.
Nguyen
,
Q. T.
,
Nguyen
,
T. L.
,
Luong
,
N. H.
, and
Ngo
,
Q. H.
,
2020
, “
Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews
,”
2020 7th NAFOSTED Conference on Information and Computer Science (NICS)
,
Ho Chi Minh, Vietnam
,
November
.
68.
Cer
,
D.
,
Yang
,
Y.
,
Kong
,
S. Y.
,
Hua
,
N.
,
Limtiaco
,
N.
,
John
,
R. S.
,
Constant
,
N.
,
Guajardo-Cespedes
,
M.
,
Yuan
,
S.
,
Tar
,
C.
,
Sung
,
Y. H.
,
Strope
,
B.
, and
Kurzweil
,
R.
,
2018
, “
Universal sentence encoder
,” CoRR, abs/1803.11175, Online.
69.
Conneau
,
A.
,
Kiela
,
D.
,
Schwenk
,
H.
,
Barrault
,
L.
, and
Bordes
,
A.
,
2017
, “
Supervised Learning of Universal Sentence Representations From Natural Language Inference Data
,”
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
,
Copenhagen, Denmark
,
September
.
70.
Van der Maaten
,
L.
, and
Hinton
,
G.
,
2008
, “
Visualizing Data Using t-SNE
,”
J. Mach. Learn. Res.
,
9
(
11
), pp.
1
27
.
You do not currently have access to this content.