ElastricSearch打分简介
一、Elasticsearch的打分公式node
Elasticsearch的默认打分公式是lucene的打分公式，主要分为两部分的计算，一部分是计算query部分的得分，另外一部分是计算field部分的得分，下面给出ES官网给出的打分公式：json
score(q,d) =
queryNorm(q)
· coord(q,d)
· ∑ (
tf(t in d)
· idf(t)²
· t.getBoost()
· norm(t,d)
) (t in q)
queryNorm(q)：
对查询进行一个归一化，不影响排序，由于对于同一个查询这个值是相同的，可是对term于ES来讲，必须在分片是1的时候才不影响排序，不然的话，仍是会有一些细小的区别，有几个分片就会有几个不一样的queryNorm值ui
queryNorm(q)=1 / √sumOfSquaredWeights spa
上述公式是ES官网的公式，这是在默认query boost为1，而且在默认term boost为1 的状况下的打分，其中code
sumOfSquaredWeights =idf(t1)*idf(t1)+idf(t2)*idf(t2)+...+idf(tn)*idf(tn)orm
其中n为在query里面切成term的个数，可是上面所有是在默认为1的状况下的计算，实际上的计算公式以下所示：排序
coord(q,d):
coord(q,d)是一个协调因子它的值以下：ip
coord(q,d)=overlap/maxoverlap
其中overlap是检索命中query中term的个数，maxoverlap是query中总共的term个数，例如查询词为“无线通讯”，使用默认分词器，若是文档为“通知他们开会”，只会有一个“通”命中，这个时候它的值就是1/4=0.25ci
tf(t in d):文档
即term t在文档中出现的个数，它的计算公式官网给出的是：
tf(t in d) = √frequency
即出现的个数进行开方，这个没什么能够讲述的，实际打分也是如此
idf(t):
这个的意思是出现的逆词频数，即召回的文档在总文档中出现过多少次，这个的计算在ES中与lucene中有些区别，只有在分片数为1的状况下，与lucene的计算是一致的，若是不惟一，那么每个分片都有一个不一样的idf的值，它的计算方式以下所示：
idf(t) = 1 + log ( numDocs / (docFreq + 1))
其中，log是以e为底的，不是以10或者以2为底，这点须要注意，numDocs是指全部的文档个数，若是有分片的话，就是指的是在当前分片下总的文档个数，docFreq是指召回文档的个数，若是有分片对应的也是在当前分片下召回的个数，这点是计算的时候与lucene不一样之处，若是想验证是否正确，只需将分片shard的个数设置为1便可。
t.getboost():
对于每个term的权值，没仔细研究这个项，我的理解的是，若是对一个field设置boost，那么若是在这个boost召回的话，每个term的boost都是该field的boost
norm(t,d):
对于field的标准化因子，在官方给的解释是field越短，若是召回的话权重越大，例如搜索无线通讯，一个是很长的内容，但都是包含这几个字，可是并非咱们想要的，另一个内容很短，可是完整包含了无线通讯，咱们不能由于后面的只出现了一次就认为权重是低的，相反，权重应当是更高的，其计算公式以下所示：
其中d.getboost代表若是该文档权重越大那么久越重要
f.getboost代表该field的权值越大，越重要
lengthnorm表示该field越长，越不重要，越短，越重要，在官方文档给出的公式中，默认boost所有为1，在此给出官方文档的打分公式：
norm(d) = 1 / √numTerms
以上的是理论上的，看看实际例子
GET act_shop-2018.01.12/shop/_search
{
"size": 1,
"query": {
"term": {
"name.keyword": "星巴克"
}
}
, "explain": true
}
结果是
{
"took": 25,
"timed_out": false,
"_shards": {
"total": 150,
"successful": 150,
"failed": 0
},
"hits": {
"total": 127667,
"max_score": 15.511484,
"hits": [
{
"_shard": "[act_shop-2018.01.12][80]",
"_node": "6vfIeV95QOK1vAcLdx6CEA",
"_index": "act_shop-2018.01.12",
"_type": "shop",
"_id": "187672",
"_score": 15.511484,
"_routing": "36341",
"_parent": "36341",
"_source": {
"status": 1,
"city": {
"id": 2084,
"name": "虹口区"
},
"update_time": "2017-10-23 15:23:00.329000",
"tel": [
"021-65200108"
],
"name": "星巴克(凉城店)",
"tags": [
"餐饮服务",
"咖啡厅",
"咖啡厅"
],
"tags_enrich": {
"name": "美食",
"id": 10
},
"id": 187672,
"label": "have_act",
"create_time": "2017-01-11 14:59:43.950000",
"city_enrich": {
"region": "华东地区",
"name": "上海",
"level": 1
},
"address": "车站南路330弄2号、6号第1、二层的4839F01059",
"coordinate": {
"lat": 31.29496,
"lon": 121.475442
},
"brand": {
"id": 490,
"name": "星巴克"
}
},
"_explanation": {
"value": 15.511484,
"description": "sum of:",
"details": [
{
"value": 15.511484,
"description": "sum of:",
"details": [
{
"value": 4.7601295,
"description": "weight(name:星 in 6914) [PerFieldSimilarity], result of:",
"details": [
{
"value": 4.7601295,
"description": "score(doc=6914,freq=1.0 = termFreq=1.0\n), product of:",
"details": [
{
"value": 4.314013,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 159,
"description": "docFreq",
"details": []
},
{
"value": 11920,
"description": "docCount",
"details": []
}
]
},
{
"value": 1.103411,
"description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
"details": [
{
"value": 1,
"description": "termFreq=1.0",
"details": []
},
{
"value": 1.2,
"description": "parameter k1",
"details": []
},
{
"value": 0.75,
"description": "parameter b",
"details": []
},
{
"value": 9.224329,
"description": "avgFieldLength",
"details": []
},
{
"value": 7.111111,
"description": "fieldLength",
"details": []
}
]
}
]
}
]
},
{
"value": 5.0423846,
"description": "weight(name:巴 in 6914) [PerFieldSimilarity], result of:",
"details": [
{
"value": 5.0423846,
"description": "score(doc=6914,freq=1.0 = termFreq=1.0\n), product of:",
"details": [
{
"value": 4.5698156,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 123,
"description": "docFreq",
"details": []
},
{
"value": 11920,
"description": "docCount",
"details": []
}
]
},
{
"value": 1.103411,
"description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
"details": [
{
"value": 1,
"description": "termFreq=1.0",
"details": []
},
{
"value": 1.2,
"description": "parameter k1",
"details": []
},
{
"value": 0.75,
"description": "parameter b",
"details": []
},
{
"value": 9.224329,
"description": "avgFieldLength",
"details": []
},
{
"value": 7.111111,
"description": "fieldLength",
"details": []
}
]
}
]
}
]
},
{
"value": 5.70897,
"description": "weight(name:克 in 6914) [PerFieldSimilarity], result of:",
"details": [
{
"value": 5.70897,
"description": "score(doc=6914,freq=1.0 = termFreq=1.0\n), product of:",
"details": [
{
"value": 5.173929,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 67,
"description": "docFreq",
"details": []
},
{
"value": 11920,
"description": "docCount",
"details": []
}
]
},
{
"value": 1.103411,
"description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
"details": [
{
"value": 1,
"description": "termFreq=1.0",
"details": []
},
{
"value": 1.2,
"description": "parameter k1",
"details": []
},
{
"value": 0.75,
"description": "parameter b",
"details": []
},
{
"value": 9.224329,
"description": "avgFieldLength",
"details": []
},
{
"value": 7.111111,
"description": "fieldLength",
"details": []
}
]
}
]
}
]
}
]
},
{
"value": 0,
"description": "match on required clause, product of:",
"details": [
{
"value": 0,
"description": "# clause",
"details": []
},
{
"value": 1,
"description": "_type:shop, product of:",
"details": [
{
"value": 1,
"description": "boost",
"details": []
},
{
"value": 1,
"description": "queryNorm",
"details": []
}
]
}
]
}
]
}
}
]
}
}
详细说明一下
一、在 "_shard": "[act_shop-2018.01.12][80]"这个分片里，按照es的标准分词，当match'星巴克'的时候，而后会分词为'星'，'巴'，'克'这三个词。每一个词的得分为:
'星'：4.7601295
'巴'：5.0423846
'克'：5.70897
总的得分：4.7601295+5.0423846+5.70897=15.511484
二、而后每一个词是怎么得分的，这里详细说一下，以'星'为例：
sorce'星'=idf.tfNorm（也就是词频*逆向词频）
idf计算以下：
{
"value": 4.7601295,
"description": "score(doc=6914,freq=1.0 = termFreq=1.0\n), product of:",
"details": [
{
"value": 4.314013,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 159,
"description": "docFreq",
"details": []
},
{
"value": 11920,
"description": "docCount",
"details": []
}
]
}
docFreq:在这个分片里，击中'星'的文档数量：159
docCount：在这个分片里，包括总的文档数量:11920
公式：log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))=4.314013
tfNorm计算以下
tf能够理解为，这个'星'，在某个文档里出现的次数的一些占比
{
"value": 1.103411,
"description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
"details": [
{
"value": 1,
"description": "termFreq=1.0",
"details": []
},
{
"value": 1.2,
"description": "parameter k1",
"details": []
},
{
"value": 0.75,
"description": "parameter b",
"details": []
},
{
"value": 9.224329,
"description": "avgFieldLength",
"details": []
},
{
"value": 7.111111,
"description": "fieldLength",
"details": []
}
]
}
tfNorm=(freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength))=1.103411
因此sorce'星'=idf.tfNorm=4.314013*1.103411=4.7601295
- 1. ElastricSearch的nested的应用
- 2. ElastricSearch的parent_child的应用
- 3. MacPorts打包过程简介
- 4. Android 签名打包简介
- 5. ＭＢＲ分区简介
- 6. Brinson分析简介
- 7. 分布式简介
- 8. git分支简介
- 9. Git分支-分支简介
- 10. 分形绘图简介（5）分形火焰算法简介（一）
- 更多相关文章...
- • Scala 简介 - Scala教程
- • AJAX 简介 - PHP教程
- • Github 简明教程
- • Java Agent入门实战（一）-Instrumentation介绍与使用
-
每一个你不满意的现在，都有一个你没有努力的曾经。
- 1. Duang!超快Wi-Fi来袭
- 2. 机器学习-补充03 神经网络之**函数(Activation Function)
- 3. git上开源maven项目部署 多module maven项目（多module maven+redis+tomcat+mysql）后台部署流程学习记录
- 4. ecliple-tomcat部署maven项目方式之一
- 5. eclipse新导入的项目经常可以看到“XX cannot be resolved to a type”的报错信息
- 6. Spark RDD的依赖于DAG的工作原理
- 7. VMware安装CentOS-8教程详解
- 8. YDOOK：Java 项目 Spring 项目导入基本四大 jar 包 导入依赖，怎样在 IDEA 的项目结构中导入 jar 包 导入依赖
- 9. 简单方法使得putty（windows10上）可以免密登录树莓派
- 10. idea怎么用本地maven
- 1. ElastricSearch的nested的应用
- 2. ElastricSearch的parent_child的应用
- 3. MacPorts打包过程简介
- 4. Android 签名打包简介
- 5. ＭＢＲ分区简介
- 6. Brinson分析简介
- 7. 分布式简介
- 8. git分支简介
- 9. Git分支-分支简介
- 10. 分形绘图简介（5）分形火焰算法简介（一）