中文词向量训练

1. 英文预训练词向量很不错,  https://nlp.stanford.edu/projects/glove/

使用时首行加入一行行数和向量维度, gensim便可调用.

# sed -i '1i 400000 300' glove.6b.300d.txt

from gensim.models.keyedvectors import KeyedVectors

model = KeyedVectors.load_word2vec_format('glove.6b.300d.txt', binary=False)

# 获取最类似
for w, s in model.most_similar('apple', topn=5):
    print w, s

# 获取向量
print model['apple']

 

2. 网上找了不少中文,不尽人意,直接本身训练, 也不会很复杂.

2.1 构建中文语料库, 下载推荐: http://www.sogou.com/labs/resource/list_news.php

# 搜狐新闻 2.1G
tar -zxvf news_sohusite_xml.full.tar.gz 
cat news_sohusite_xml.full.tar.gz | iconv -f gb18030 -t utf-8 | grep "<content>" > news_sohusite.txt
sed -i 's/<content>//g' news_sohusite.txt
sed -i 's/<\/content>//g' news_sohusite.txt
python -m jieba -d ' ' news_sohusite.txt > news_sohusite_cutword.txt

# 全网新闻 1.8G
tar -zxvf news_tensites_xml.full.tar.gz 
cat news_tensites_xml.full.tar.gz | iconv -f gb18030 -t utf-8 | grep "<content>" > news_tensite.txt
sed -i 's/<content>//g' news_tensite.txt
sed -i 's/<\/content>//g' news_tensite.txt
python -m jieba -d ' ' news_tensite.txt > news_tensite_cutword.txt

# 其它自身的结合业务须要的预料, 如公司简介
python -m jieba -d ' ' other_entdesc.txt > other_entdesc_cutword.txt

# 合并切割好的语料
cat news_sohusite_cutword.txt news_tensite_cutword.txt other_entdesc_cutword.txt > w2v_chisim_corpus.txt

2.2 利用gensim库进行训练#!/usr/bin/env python

from gensim.models.word2vec import Word2Vec from gensim.models.word2vec import LineSentence sentences = LineSentence('w2v_chisim_corpus.txt') model = Word2Vec(sentences, size=300, window=8, min_count=10, sg=1, workers=4) # sg=0 使用cbow训练, sg=1对低频词较为敏感 model.save('w2v_chisim.300d.txt') 
 for w, s in model.most_similar(u'苹果'): print w, s for w, s in model.most_similar(u'中国'): print w, s for w, s in model.most_similar(u'中山大学'): print w, s

 

如何, 是否是也很简单, your show time now, good luck!