文章类似度

一、环境   

单机版、windows系统、python3.六、gensim模块

参考文献：

 https://pypi.org/project/gensim/

 https://radimrehurek.com/gensim/

https://www.jianshu.com/p/6e07729c6c5b

二、gensim安装  https://pypi.org/project/gensim/

通常能够直接经过  pip install -U gensim安装

若是没有网络环境，能够在下载相应安装包 安装（会有其余包依赖问题，须要逐个安装）

三、经过gensim计算文章类似度 https://radimrehurek.com/gensim/similarities/docsim.html

3.1  cosine类似度 cosine similarity 

 a)gensim.similarities.docsim.MatrixSimilarity （矩阵向量，内存运算）

  b) gensim.similarities.docsim.Similarity （动态运算，若是MatrixSimilarity、SparseMatrixSimilarity数据量大，没法计算时，可以使用）

  c) gensim.similarities.docsim.SparseMatrixSimilarity (稀疏向量输入，内存运算）

3.2 wmd类似度

 gensim.similarities.docsim.WmdSimilarity  

 

四、简易代码

数据输入(text)： 分词完以后的词向量, 如[["love","China"], ["weather", "sunny"]]

from gensim.models import Word2Vec
from gensim.similarities import WmdSimilarity, Similarity, MatrixSimilarity, SparseMatrixSimilarity

from gensim import corpora, models
#文章输入
text = [["love","China"], ["weather", "sunny"]]

#将类似度向量转成list
def index2list(index):
doc_sim_list = []
for s in index:
try:
doc_sim_list.append(s)
except:
print ("there is something woring at index : {0}".format(s))
return doc_sim_list

##WmdSimilarity
#获取词向量模型
model = Word2Vec(text, min_count=1)
#计算WmdSimilarity
index = WmdSimilarity(text, model)
doc_sim_list = index2list(index)

##cosine similarity
#构建词语字典
dictionary = corpora.Dictionary(text)
#将文章转成此向量
corpus = [dictionary.doc2bow(t) for t in text]

#SparseMatrixSimilarity
index = SparseMatrixSimilarity(corpus, num_features=len(dictionary))
doc_sim_list = index2list(index)

#MatrixSimilarity
index = MatrixSimilarity(corpus, num_features=len(dictionary))
doc_sim_list = index2list(index)

#Similarity
#idf computation
tfidf_model = models.TfidfModel(corpus)

tfidf = tfidf_model[corpus]

index = Similarity("Similarity-index", tfidf, num_features=len(dictionary))doc_sim_list = index2list(index)