Neo4j中实现自定义中文全文索引

<div class="markdown-text"><p>数据库检索效率时，通常首要优化途径是从索引入手，而后根据需求再考虑更复杂的负载均衡、读写分离和分布式水平/垂直分库/表等手段；<br>索引经过信息冗余来提升检索效率，其以空间换时间并会下降数据写入的效率；所以对索引字段的选择很是重要。</p> <ul> <li>Neo4j可对指定Label的Node Create Index，当新增/更新符合条件的Node属性时，Index会自动更新。Neo4j Index默认采用Lucene实现（可定制，如Spatial Index自定义实现的RTree索引），但默认新建的索引只支持精确匹配（get），模糊查询（query）的话须要以全文索引，控制Lucene后台的分词行为。</li> <li>Neo4j全文索引默认的分词器是针对西方语种的，如默认的exact查询采用的是lucene KeywordAnalyzer（关键词分词器）,fulltext查询采用的是 white-space tokenizer（空格分词器），大小写什么的对中文没啥意义；因此针对中文分词须要挂一个中文分词器，如IK Analyzer,Ansj，至于相似梁厂长家的基于深度学习的分词系统pullword，那就更厉害啦。</li> </ul> <p>本文以经常使用的IK Analyzer分词器为例，介绍如何在Neo4j中对字段新建全文索引实现模糊查询。</p> <hr> <h1>IKAnalyzer分词器</h1> <p><a href="https://github.com/wks/ik-analyzer" target="_blank">IKAnalyzer</a>是一个开源的，基于java语言开发的轻量级的中文分词工具包。<br>IKAnalyzer3.0特性:</p> <ul> <li>采用了特有的“正向迭代最细粒度切分算法“，支持细粒度和最大词长两种切分模式；具备83万字/秒（1600KB/S）的高速处理能力。</li> <li>采用了多子处理器分析模式，支持：英文字母、数字、中文词汇等分词处理，兼容韩文、日文字符优化的词典存储，更小的内存占用。支持用户词典扩展定义</li> <li>针对Lucene全文检索优化的查询分析器IKQueryParser(做者吐血推荐)；引入简单搜索表达式，采用歧义分析算法优化查询关键字的搜索排列组合，能极大的提升Lucene检索的命中率。<br>IK Analyser目前尚未maven库，还得本身手动下载install到本地库，下次空了本身在github作一个maven私有库，上传这些maven central库里面没有的工具包。</li> </ul> <h1>IKAnalyzer自定义用户词典</h1> <ul> <li>词典文件<br>自定义词典后缀名为.dic的词典文件，必须使用无BOM的UTF-8编码保存的文件。</li> <li>词典配置<br>词典和IKAnalyzer.cfg.xml配置文件的路径问题，IKAnalyzer.cfg.xml必须在src根目录下。词典能够任意放，可是在IKAnalyzer.cfg.xml里要配置对。以下这种配置，ext.dic和stopword.dic应当在同一目录下。 <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> <div>5</div> <div>6</div> <div>7</div> <div>8</div> <div>9</div> <div>10</div> <div>11</div> </td> <td> <div><span><span>&lt;?xml version=<span>"1.0" encoding=<span>"UTF-8"<span>?&gt;</span></span></span></span></span></div> <div><span>&lt;!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"&gt;</span></div> <div><span>&lt;<span>properties&gt; </span></span></div> <div><span>&lt;<span>comment&gt;IK Analyzer 扩展配置<span>&lt;/<span>comment&gt;</span></span></span></span></div> <div>&nbsp;</div> <div><span>&lt;!--用户能够在这里配置本身的扩展字典 --&gt;</span></div> <div><span>&lt;<span>entry <span>key=<span>"ext_dict"&gt;/ext.dic;<span>&lt;/<span>entry&gt;</span></span></span></span></span></span></div> <div>&nbsp;</div> <div><span>&lt;!--用户能够在这里配置本身的扩展中止词字典--&gt;</span></div> <div><span>&lt;<span>entry <span>key=<span>"ext_stopwords"&gt;/stopword.dic<span>&lt;/<span>entry&gt;</span></span></span></span></span></span></div> <div><span>&lt;/<span>properties&gt;</span></span></div> </td> </tr> </tbody> </table> </li> </ul> <h1>Neo4j全文索引构建</h1> <p>指定IKAnalyzer做为luncene分词的analyzer，并对全部Node的指定属性新建全文索引</p> <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> <div>5</div> <div>6</div> <div>7</div> <div>8</div> <div>9</div> <div>10</div> <div>11</div> <div>12</div> <div>13</div> <div>14</div> <div>15</div> <div>16</div> <div>17</div> </td> <td> <div><span>[@Override](/user/Override)</span></div> <div><span><span>public <span>void <span>createAddressNodeFullTextIndex <span>() {</span></span></span></span></span></div> <div> <span>try (Transaction tx = graphDBService.beginTx()) {</span></div> <div> IndexManager index = graphDBService.index();</div> <div> Index&lt;Node&gt; addressNodeFullTextIndex =</div> <div> index.forNodes( <span>"addressNodeFullTextIndex", MapUtil.stringMap(IndexManager.PROVIDER, <span>"lucene", <span>"analyzer", IKAnalyzer.class.getName()));</span></span></span></div> <div>&nbsp;</div> <div> ResourceIterator&lt;Node&gt; nodes = graphDBService.findNodes(DynamicLabel.label( <span>"AddressNode"));</span></div> <div> <span>while (nodes.hasNext()) {</span></div> <div> Node node = nodes.next();</div> <div> <span>//对text字段新建全文索引</span></div> <div> Object text = node.getProperty( <span>"text", <span>null);</span></span></div> <div> addressNodeFullTextIndex.add(node, <span>"text", text);</span></div> <div> }</div> <div> tx.success();</div> <div> }</div> <div>}</div> </td> </tr> </tbody> </table> <p>&nbsp;</p> <h1>Neo4j全文索引测试</h1> <p>对关键词（如’有限公司’），多关键词模糊查询（如’苏州 教育 公司’）默认都能检索，且检索结果按关联度已排好序。</p> <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> <div>5</div> <div>6</div> <div>7</div> <div>8</div> <div>9</div> <div>10</div> <div>11</div> <div>12</div> <div>13</div> <div>14</div> <div>15</div> <div>16</div> <div>17</div> <div>18</div> <div>19</div> <div>20</div> <div>21</div> <div>22</div> <div>23</div> <div>24</div> <div>25</div> <div>26</div> <div>27</div> <div>28</div> <div>29</div> <div>30</div> <div>31</div> <div>32</div> <div>33</div> <div>34</div> <div>35</div> <div>36</div> <div>37</div> <div>38</div> <div>39</div> <div>40</div> <div>41</div> <div>42</div> <div>43</div> <div>44</div> <div>45</div> </td> <td> <div><span>package uadb.tr.neodao.test;</span></div> <div>&nbsp;</div> <div><span>import org.junit.Test;</span></div> <div><span>import org.junit.runner.RunWith;</span></div> <div><span>import org.neo4j.graphdb.GraphDatabaseService;</span></div> <div><span>import org.neo4j.graphdb.Node;</span></div> <div><span>import org.neo4j.graphdb.Transaction;</span></div> <div><span>import org.neo4j.graphdb.index.Index;</span></div> <div><span>import org.neo4j.graphdb.index.IndexHits;</span></div> <div><span>import org.neo4j.graphdb.index.IndexManager;</span></div> <div><span>import org.neo4j.helpers.collection.MapUtil;</span></div> <div><span>import org.springframework.beans.factory.annotation.Autowired;</span></div> <div><span>import org.springframework.test.context.ContextConfiguration;</span></div> <div><span>import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;</span></div> <div><span>import org.wltea.analyzer.lucene.IKAnalyzer;</span></div> <div>&nbsp;</div> <div><span>import com.lt.uadb.tr.entity.adtree.AddressNode;</span></div> <div><span>import com.lt.util.serialize.JsonUtil;</span></div> <div>&nbsp;</div> <div><span>/**</span></div> <div> * AddressNodeNeoDaoTest</div> <div> *</div> <div> * <span>[@author](/user/author) geosmart</span></div> <div> */</div> <div><span>@RunWith(SpringJUnit4ClassRunner. <span><span>class)</span></span></span></div> <div>@<span>ContextConfiguration(<span>locations = { <span>"classpath:app.neo4j.cfg.xml" })</span></span></span></div> <div><span>public <span><span>class <span>AddressNodeNeoDaoTest {</span></span></span></span></div> <div> <span>[@Autowired](/user/Autowired)</span></div> <div> GraphDatabaseService graphDBService;</div> <div>&nbsp;</div> <div> <span>[@Test](/user/Test)</span></div> <div> <span><span>public <span>void <span>test_selectAddressNodeByFullTextIndex<span>() {</span></span></span></span></span></div> <div> <span>try (Transaction tx = graphDBService.beginTx()) {</span></div> <div> IndexManager index = graphDBService.index();</div> <div> Index&lt;Node&gt; addressNodeFullTextIndex = index.forNodes(<span>"addressNodeFullTextIndex" ,</span></div> <div> MapUtil. stringMap(IndexManager.PROVIDER, <span>"lucene", <span>"analyzer" , IKAnalyzer.class.getName()));</span></span></div> <div> IndexHits&lt;Node&gt; foundNodes = addressNodeFullTextIndex.query(<span>"text" , <span>"苏州 教育 公司" );</span></span></div> <div> <span>for (Node node : foundNodes) {</span></div> <div> AddressNode entity = JsonUtil.ConvertMap2POJO(node.getAllProperties(), AddressNode. <span><span>class, <span>false, <span>true);</span></span></span></span></div> <div> System. out.println(entity.getAll地址实全称());</div> <div> }</div> <div> tx.success();</div> <div> }</div> <div> }</div> <div>}</div> </td> </tr> </tbody> </table> <p>&nbsp;</p> <h1>CyperQL中使用自定义全文索引查询</h1> <h2>正则查询</h2> <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> </td> <td> <div>profile </div> <div>match (a:AddressNode{ruleabbr:'TOW',text:'惟亭镇'})&lt;-[r:BELONGTO]-(b:AddressNode{ruleabbr:'STR'})</div> <div>where b.text=~ '金陵.*'</div> <div>return a,b</div> </td> </tr> </tbody> </table> <h2>全文索引查询</h2> <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> <div>5</div> </td> <td> <div>profile</div> <div><span>START b=node:addressNodeFullTextIndex(<span>"text:金陵*")</span></span></div> <div><span>match (a:AddressNode{ruleabbr:<span>'TOW',<span>text:<span>'惟亭镇'})&lt;-[r:BELONGTO]-(b:AddressNode)</span></span></span></span></div> <div><span>where b.ruleabbr=<span>'STR'</span></span></div> <div><span>return a,b</span></div> </td> </tr> </tbody> </table> <h1>LegacyIndex中创建联合exact和fulltext索引</h1> <p>对label为AddressNode的节点，根据节点属性ruleabbr的分类addressnode_fulltext_index（省-&gt;市-&gt;区县-&gt;乡镇街道-&gt;街路巷/物业小区）/addressnode_exact_index(门牌号-&gt;楼幢号-&gt;单元号-&gt;层号-&gt;户室号)，对属性text分别建不一样类型的索引</p> <p> </p> <table> <tbody> <tr> <td> <div>1</div> <div>2</div> <div>3</div> <div>4</div> </td> <td> <div>profile</div> <div><span>START a=node:addressnode_fulltext_index(<span>"text:商业街"),b=node:addressnode_exact_index(<span>"text:二期19")</span></span></span></div> <div><span>match (a:AddressNode{ruleabbr:<span>'STR'})-[r:BELONGTO]-(b:AddressNode{ruleabbr:<span>'TAB'})</span></span></span></div> <div><span>return a,b <span>limit <span>10</span></span></span></div> </td> </tr> </tbody> </table>原文地址：http://neo4j.com.cn/topic/58184ea2cdf6c5bf145675c3</div>