北大开源中文分词器被打脸现场...

有作过搜索的朋友知道，分词的好坏直接影响咱们最终的搜索结果。在分词的领域，英文分词要简单不少，由于英文语句中都是经过一个个空格来划分的，而咱们的中文博大精深，一样的词在不一样的语境中所表明的含义千差万别，有时候必须联系上下文才能知道它准确的表达意思，所以中文分词一直是分词领域的一大挑战。

以前介绍过一款北大新开源的分词器，根据做者的测试结果，这是一个准确率和速度都超过 jieba 等其余分词的分词器。

因此我就想来作个简单的测试！因而我想用《三国演义》来作一个测试，提取其中著名人名出现的频率。

首先搜索下三国中的人物名单，

获得人名以后，作成一我的名的列表，以前设置成一个以人物名为键，值为 0 的字典。我只取了曹魏和蜀汉的部分人名，代码以下:

wei = ["许褚","荀攸","贾诩","郭嘉","程昱","戏志","刘晔","蒋济","陈群","华歆","钟繇","满宠","董昭","王朗","崔琰","邓艾","杜畿","田畴","王修","杨修","辛毗",
       "杨阜",
       "田豫","王粲","蒯越","张继","于禁","枣祗","曹操","孟德","任峻","陈矫","郗虑","桓玠","丁仪","丁廙","司马朗","韩暨","韦康","邴原","赵俨","娄圭","贾逵",
       "陈琳",
       "司马懿","张辽","徐晃","夏侯惇","夏侯渊","庞德","张郃","李典","乐进","典韦","曹洪","曹仁","曹彰"]

wei_dict = dict.fromkeys(wei, 0)
shu_dict = dict.fromkeys(shu, 0)
复制代码

接着去下载一部三国的电子书，并读取

def read_txt():
    with open("三国.txt", encoding="utf-8") as f:
        content = f.read()

    return content
复制代码

pkuseg 测试结果

pkuseg 的用法很简单，首先实例化 pkuseg 的对象，获取人物名称数量的思路是这样的：循环咱们分词后的列表，若是检测到有人物名称字典中的人物名，就将该数据加 1，代码以下:

def extract_pkuseg(content):
    start = time.time()
    seg = pkuseg.pkuseg()
    text = seg.cut(content)

    for name in text:
        if name in wei:
            wei_dict[name] = wei_dict.get(name) + 1
        elif name in shu:
            shu_dict[name] = shu_dict.get(name) + 1

print(f"pkuseg 用时：{time.time() - start}")
print(f"pkuseg 读取人名总数：{sum(wei_dict.values()) + sum(shu_dict.values())}")
复制代码

执行结果以下：

jieba 测试结果

代码基本差很少，只是分词器的用法有些不一样。

def extract_jieba(content):
    start = time.time()
    seg_list = jieba.cut(content)
    for name in seg_list:
        if name in wei:
            wei_dict[name] = wei_dict.get(name) + 1
        elif name in shu:
            shu_dict[name] = shu_dict.get(name) + 1

    print(f"jieba 用时：{time.time() - start}")
    print(f"jieba 读取人名总数：{sum(wei_dict.values()) + sum(shu_dict.values())}")
复制代码

执行结果以下：

emmm 测试结果好像好像有点出乎意料，说好的 pkuseg 准确率更高呢？？？pkuseg 用时将近 jieba 的三倍，并且提取效果也没有 jieba 分词好！因而我就去逼乎搜了一下 pkuseg ，结果是这样的....

整体而言 pkuseg 吹的有点过了，并无做者说的那么神奇，有点博眼球的成分，也许它更是一款更注重细分领域的分词器！