ML►app►nlp►app►-text_match

Introduction

dataset

summary

MSRParaphraseCorpus

这个库，是二分类。tfidf的方法效果很差，因为不match的句子也有很多共有词汇。

msr_paraphrase_test.txt

转到excel，然后再转到markdown的表格格式。

1
2
3
4
5
Quality	#1 ID	#2 ID	#1 String	#2 String
1	1089874	1089925	PCCW's chief operating officer, Mike Butcher, and Alex Arena, the chief financial officer, will report directly to Mr So.	Current Chief Operating Officer Mike Butcher and Group Chief Financial Officer Alex Arena will report to So.
1	3019446	3019327	The world's two largest automakers said their U.S. sales declined more than predicted last month as a late summer sales frenzy caused more of an industry backlash than expected.	Domestic sales at both GM and No. 2 Ford Motor Co. declined more than predicted as a late summer sales frenzy prompted a larger-than-expected industry backlash.
1	1945605	1945824	According to the federal Centers for Disease Control and Prevention (news - web sites), there were 19 reported cases of measles in the United States in 2002.	The Centers for Disease Control and Prevention said there were 19 reported cases of measles in the United States in 2002.
0	1430402	1430329	A tropical storm rapidly developed in the Gulf of Mexico Sunday and was expected to hit somewhere along the Texas or Louisiana coasts by Monday night.	A tropical storm rapidly developed in the Gulf of Mexico on Sunday and could have hurricane-force winds when it hits land somewhere along the Louisiana coast Monday night.

STS_Data_2012_15

seq2seq的数据貌似也可以用。

text similarity的数据库也可用

semantic matching

  • Textual matching: TF-IDF, BM25
  • Topical matching: LDA
  • Embedding matching: word2vec
  • Translated text matching