【Python之路】特别篇--服务商API认证、Restful、一致性哈希

时间 2019-11-21

标签 Python之路特别务商 api 认证 restful 一致性哈希栏目 Zookeeper 繁體版

原文原文链接

API加密方式

1/ 加密方式:html

Md5 (随机字符串 + 时间戳)node

2/ 发送方式:python

http://127.0.0.1:8888/index?pid= MD5加密值 | 时间戳 | 序号

服务端接收:web

(1) 判断时间戳是否在有效期内json

(2) 判断url地址是否已经访问过api

(3) 判断md5加密值与服务端生成的md5 是否匹配缓存

import hashlib
access_record = [

]

PID_LIST = [
    'abcd',
    'ddd',
    'ks',
]
class MainHandler(tornado.web.RequestHandler):
    def get(self):
        import time
        # 获取url中所有数据
        pid = self.get_argument('pid', None)
        # 获取变量
        m5, client_time, i = pid.split('|')

        server_time = time.time()
        # 时间超过10s禁止
        if server_time > float(client_time) + 10:
            self.write(' Error !! ')
            return
        # 处理10s内容重复的请求
        if pid in access_record:
            self.write(' Error !! ')
            return
        access_record.append(pid)

        pid = PID_LIST[int(i)]
        ramdom_str = "%s|%s" %(pid, client_time)
        h = hashlib.md5()
        h.update(bytes(ramdom_str, encoding='utf-8'))
        server_m5 = h.hexdigest()
        # print(m5,server_m5)
        if m5 == server_m5:
            self.write("Hello, world")
        else:
            self.write(' Error !! ')

客户端: 生成带规则的url链接 -> SDK服务器

import requests
import time
import hashlib

url = 'http://127.0.0.1:8888/index?pid=%s'

pid = 'aabc'
client_time = time.time()

h = hashlib.md5()
ramdom_str = "%s|%s" %(pid, client_time)
h.update(bytes(ramdom_str,encoding='utf-8'))

m5 = h.hexdigest()
url_payloads = '%s|%s|%d'%(m5,client_time,0)
url = url%(url_payloads)

print(url)
ret = requests.get(url)
print(ret.text)

客户端

Access_LIST 能够用Redis 设置过时时间,到期限自动清空内容。restful

Restful

1、名称网络

REST，即Representational State Transfer的缩写。我对这个词组的翻译是 "表现层状态转化" 。

若是一个架构符合REST原则，就称它为RESTful架构。

2、资源（Resources）

REST的名称"表现层状态转化"中，省略了主语。"表现层"其实指的是"资源"（Resources）的"表现层"。

所谓"资源"，就是网络上的一个实体，或者说是网络上的一个具体信息。它能够是一段文本、一张图片、一首歌曲、一种服务，总之就是一个具体的实在。你能够用一个URL（统一资源定位符）指向它，每种资源对应一个特定的URL。要获取这个资源，访问它的URL就能够，所以URL就成了每个资源的地址或独一无二的识别符。

3、表现层（Representation）

"资源"是一种信息实体，它能够有多种外在表现形式。咱们把"资源"具体呈现出来的形式，叫作它的"表现层"（Representation）。

好比，文本能够用txt格式表现，也能够用HTML格式、XML格式、JSON格式表现，甚至能够采用二进制格式；图片能够用JPG格式表现，也能够用PNG格式表现。

URL只表明资源的实体，不表明它的形式。严格地说，有些网址最后的".html"后缀名是没必要要的，由于这个后缀名表示格式，属于"表现层"范畴，而URL应该只表明"资源"的位置。它的具体表现形式，应该在HTTP请求的头信息中用Accept和Content-Type字段指定，这两个字段才是对"表现层"的描述。

4、状态转化（State Transfer）

访问一个网站，就表明了客户端和服务器的一个互动过程。在这个过程当中，势必涉及到数据和状态的变化。

互联网通讯协议HTTP协议，是一个无状态协议。这意味着，全部的状态都保存在服务器端。所以，若是客户端想要操做服务器，必须经过某种手段，让服务器端发生"状态转化"（State Transfer）。而这种转化是创建在表现层之上的，因此就是"表现层状态转化"。

客户端用到的手段，只能是HTTP协议。具体来讲，就是HTTP协议里面，四个表示操做方式的动词：GET、POST、PUT、DELETE。它们分别对应四种基本操做：GET用来获取资源，POST用来新建资源（也能够用于更新资源），PUT用来更新资源，DELETE用来删除资源。

5、综述

综合上面的解释，咱们总结一下什么是RESTful架构：

　　（1）每个URL表明一种资源；

　　（2）客户端和服务器之间，传递这种资源的某种表现层；

　　（3）客户端经过四个HTTP动词，对服务器端资源进行操做，实现"表现层状态转化"。

6、误区

RESTful架构有一些典型的设计误区。

最多见的一种设计错误，就是URI包含动词。由于"资源"表示一种实体，因此应该是名词，URI不该该有动词，动词应该放在HTTP协议中。

举例来讲，某个URI是/posts/show/1，其中show是动词，这个URI就设计错了，正确的写法应该是/posts/1，而后用GET方法表示show。

好比网上汇款，从帐户1向帐户2汇款500元，错误的URI是：

POST /accounts/1/transfer/500/to/2

正确的写法是把动词transfer改为名词transaction，资源不能是动词，可是能够是一种服务：

POST /transaction HTTP/1.1
Host: 127.0.0.1
　　
from=1&to=2&amount=500.00

另外一个设计误区，就是在URI中加入版本号：

http://www.example.com/app/1.0/foo
http://www.example.com/app/1.1/foo
http://www.example.com/app/2.0/foo

由于不一样的版本，能够理解成同一种资源的不一样表现形式，因此应该采用同一个URI。版本号能够在HTTP请求头信息的Accept字段中进行区分

Accept: vnd.example-com.foo+json; version=1.0
Accept: vnd.example-com.foo+json; version=1.1
Accept: vnd.example-com.foo+json; version=2.0

👉 详情点击

一致性哈希

py2.7版本哈希

# -*- coding: utf-8 -*-
"""
    hash_ring
    ~~~~~~~~~~~~~~
    Implements consistent hashing that can be used when
    the number of server nodes can increase or decrease (like in memcached).

    Consistent hashing is a scheme that provides a hash table functionality
    in a way that the adding or removing of one slot
    does not significantly change the mapping of keys to slots.

    More information about consistent hashing can be read in these articles:

        "Web Caching with Consistent Hashing":
            http://www8.org/w8-papers/2a-webserver/caching/paper2.html

        "Consistent hashing and random trees:
        Distributed caching protocols for relieving hot spots on the World Wide Web (1997)":
            http://citeseerx.ist.psu.edu/legacymapper?did=38148


    Example of usage::

        memcache_servers = ['192.168.0.246:11212',
                            '192.168.0.247:11212',
                            '192.168.0.249:11212']

        ring = HashRing(memcache_servers)
        server = ring.get_node('my_key')

    :copyright: 2008 by Amir Salihefendic.
    :license: BSD
"""

import math
import sys
from bisect import bisect

if sys.version_info >= (2, 5):
    import hashlib
    md5_constructor = hashlib.md5
else:
    import md5
    md5_constructor = md5.new

class HashRing(object):

    def __init__(self, nodes=None, weights=None):
        """`nodes` is a list of objects that have a proper __str__ representation.
        `weights` is dictionary that sets weights to the nodes.  The default
        weight is that all nodes are equal.
        """
        self.ring = dict()
        self._sorted_keys = []

        self.nodes = nodes

        if not weights:
            weights = {}
        self.weights = weights

        self._generate_circle()

    def _generate_circle(self):
        """Generates the circle.
        """
        total_weight = 0
        for node in self.nodes:
            total_weight += self.weights.get(node, 1)

        for node in self.nodes:
            weight = 1

            if node in self.weights:
                weight = self.weights.get(node)

            factor = math.floor((40*len(self.nodes)*weight) / total_weight);

            for j in xrange(0, int(factor)):
                b_key = self._hash_digest( '%s-%s' % (node, j) )

                for i in xrange(0, 3):
                    key = self._hash_val(b_key, lambda x: x+i*4)
                    self.ring[key] = node
                    self._sorted_keys.append(key)

        self._sorted_keys.sort()

    def get_node(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned.

        If the hash ring is empty, `None` is returned.
        """
        pos = self.get_node_pos(string_key)
        if pos is None:
            return None
        return self.ring[ self._sorted_keys[pos] ]

    def get_node_pos(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned
        along with it's position in the ring.

        If the hash ring is empty, (`None`, `None`) is returned.
        """
        if not self.ring:
            return None

        key = self.gen_key(string_key)

        nodes = self._sorted_keys
        pos = bisect(nodes, key)

        if pos == len(nodes):
            return 0
        else:
            return pos

    def iterate_nodes(self, string_key, distinct=True):
        """Given a string key it returns the nodes as a generator that can hold the key.

        The generator iterates one time through the ring
        starting at the correct position.

        if `distinct` is set, then the nodes returned will be unique,
        i.e. no virtual copies will be returned.
        """
        if not self.ring:
            yield None, None

        returned_values = set()
        def distinct_filter(value):
            if str(value) not in returned_values:
                returned_values.add(str(value))
                return value

        pos = self.get_node_pos(string_key)
        for key in self._sorted_keys[pos:]:
            val = distinct_filter(self.ring[key])
            if val:
                yield val

        for i, key in enumerate(self._sorted_keys):
            if i < pos:
                val = distinct_filter(self.ring[key])
                if val:
                    yield val

    def gen_key(self, key):
        """Given a string key it returns a long value,
        this long value represents a place on the hash ring.

        md5 is currently used because it mixes well.
        """
        b_key = self._hash_digest(key)
        return self._hash_val(b_key, lambda x: x)

    def _hash_val(self, b_key, entry_fn):
        return (( b_key[entry_fn(3)] << 24)
                |(b_key[entry_fn(2)] << 16)
                |(b_key[entry_fn(1)] << 8)
                | b_key[entry_fn(0)] )

    def _hash_digest(self, key):
        m = md5_constructor()
        m.update(key)
        return map(ord, m.digest())

View Code

py3.5版本哈希

# -*- coding: utf-8 -*-
"""
    hash_ring
    ~~~~~~~~~~~~~~
    Implements consistent hashing that can be used when
    the number of server nodes can increase or decrease (like in memcached).

    Consistent hashing is a scheme that provides a hash table functionality
    in a way that the adding or removing of one slot
    does not significantly change the mapping of keys to slots.

    More information about consistent hashing can be read in these articles:

        "Web Caching with Consistent Hashing":
            http://www8.org/w8-papers/2a-webserver/caching/paper2.html

        "Consistent hashing and random trees:
        Distributed caching protocols for relieving hot spots on the World Wide Web (1997)":
            http://citeseerx.ist.psu.edu/legacymapper?did=38148


    Example of usage::

        memcache_servers = ['192.168.0.246:11212',
                            '192.168.0.247:11212',
                            '192.168.0.249:11212']

        ring = HashRing(memcache_servers)
        server = ring.get_node('my_key')

    :copyright: 2008 by Amir Salihefendic.
    :license: BSD
"""

import math
import sys
from bisect import bisect

if sys.version_info >= (2, 5):
    import hashlib
    md5_constructor = hashlib.md5
else:
    import md5
    md5_constructor = md5.new

class HashRing(object):

    def __init__(self, nodes=None, weights=None):
        """`nodes` is a list of objects that have a proper __str__ representation.
        `weights` is dictionary that sets weights to the nodes.  The default
        weight is that all nodes are equal.
        """
        self.ring = dict()
        self._sorted_keys = []

        self.nodes = nodes

        if not weights:
            weights = {}
        self.weights = weights

        self._generate_circle()

    def _generate_circle(self):
        """Generates the circle.
        """
        total_weight = 0
        for node in self.nodes:
            total_weight += self.weights.get(node, 1)

        for node in self.nodes:
            weight = 1

            if node in self.weights:
                weight = self.weights.get(node)

            factor = math.floor((40*len(self.nodes)*weight) / total_weight);

            for j in range(0, int(factor)):
                b_key = self._hash_digest( '%s-%s' % (node, j) )

                for i in range(0, 3):
                    key = self._hash_val(b_key, lambda x: x+i*4)
                    self.ring[key] = node
                    self._sorted_keys.append(key)

        self._sorted_keys.sort()

    def get_node(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned.

        If the hash ring is empty, `None` is returned.
        """
        pos = self.get_node_pos(string_key)
        if pos is None:
            return None
        return self.ring[ self._sorted_keys[pos] ]

    def get_node_pos(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned
        along with it's position in the ring.

        If the hash ring is empty, (`None`, `None`) is returned.
        """
        if not self.ring:
            return None

        key = self.gen_key(string_key)

        nodes = self._sorted_keys
        pos = bisect(nodes, key)

        if pos == len(nodes):
            return 0
        else:
            return pos

    def iterate_nodes(self, string_key, distinct=True):
        """Given a string key it returns the nodes as a generator that can hold the key.

        The generator iterates one time through the ring
        starting at the correct position.

        if `distinct` is set, then the nodes returned will be unique,
        i.e. no virtual copies will be returned.
        """
        if not self.ring:
            yield None, None

        returned_values = set()
        def distinct_filter(value):
            if str(value) not in returned_values:
                returned_values.add(str(value))
                return value

        pos = self.get_node_pos(string_key)
        for key in self._sorted_keys[pos:]:
            val = distinct_filter(self.ring[key])
            if val:
                yield val

        for i, key in enumerate(self._sorted_keys):
            if i < pos:
                val = distinct_filter(self.ring[key])
                if val:
                    yield val

    def gen_key(self, key):
        """Given a string key it returns a long value,
        this long value represents a place on the hash ring.

        md5 is currently used because it mixes well.
        """
        b_key = self._hash_digest(key)
        return self._hash_val(b_key, lambda x: x)

    def _hash_val(self, b_key, entry_fn):
        return (( b_key[entry_fn(3)] << 24)
                |(b_key[entry_fn(2)] << 16)
                |(b_key[entry_fn(1)] << 8)
                | b_key[entry_fn(0)] )

    def _hash_digest(self, key):
        m = md5_constructor()
        m.update(bytes(key, encoding='utf-8'))
        return list(m.digest())

View Code

区别在于py2.7下

m = md5_constructor()
m.update(key)
return map(ord, m.digest())

digest() 返回的是字符串, 最后map函数调用ord,返回数字列表

py3.5中: digest() 返回的是字节类型，for循环后就能获得数字，省去了ord这一步，即便用 list进行for循环便可生成数字列表！

def _hash_digest(self, key):
    m = md5_constructor()
    m.update(bytes(key, encoding='utf-8'))
    return list(m.digest())

例子:

假设有5台memcache服务器 ( 利用ip进行哈希 )

memcache_servers = ['192.168.0.245:11212',
                    '192.168.0.246:11212',
                    '192.168.0.247:11212',
                    '192.168.0.248:11212',
                    '192.168.0.249:11212']

主机ip 哈希完成后造成的哈希值以下

    cacheA       key1
    cacheB       key2
    cacheC       key3
    cacheD       key4
    cacheE       key5

而后5台节点围绕称一个环形，如图

主机造成哈希值后，若是须要对某个值进行hash,(这里和普通hash不同的就是这里只hash不求余)，当hash求出一个数值后，

查看这个值介于哪两个值之间。好比介于key4和key5之间，而后从key4这个落点开始顺时针查找，最早遇到的第一个cache服务器后，就把这个服务器看成这次缓存的落点，

从而把这个值放到这个落点上。如图:

在使用普通的哈希时，当其中任意一台机器down掉后，咱们须要重计算落点，由于除数已经发生了改变，因此都要从新设置缓存。

而使用一致性哈希，当其中一台机器down掉后，只有它上面到它这一段的环路缓存失效了，如：当cacheC down掉后，掉落在cacheB到cacheC之间的值都将失效，那么失效的落点将继续顺时针查找cache服务器，因此新的落点就会降落到cacheD上。

memcache服务器中，为某些ip添加权重，就是对环上设置多个相同的key增长几率

memcache_servers = ['192.168.0.246:11212',
                    '192.168.0.247:11212',
                    '192.168.0.249:11212']
weights = {
    '192.168.0.246:11212': 1,
    '192.168.0.247:11212': 2,
    '192.168.0.249:11212': 10000
}

ring = HashRing(memcache_servers, weights)
server = ring.get_node('my_key1')
print(server)

# 链接mem(server)
# mem.set()