python 爬取网易 buff 饰品数据及 steam 饰品市场数据 达到折上折

时间  2020-10-03 标签 python 网易 buff 饰品 数据 steam 市场 达到

这里写自定义目录标题

前言

最近因为steam政策改变，steam礼品卡折上折难搞了，我一直买的那家tb店50$要270￥，在接近8折的条件下还须要提供帐号密码代充，安全性有待考量，因此想着用py爬虫爬buff数据和steam数据进行处理，最后获得买卖饰品的折值，以达到等同于礼品卡的效果。
在学习Charles-D的文章后发现他的目的是炼金，其中并不涉及steam的信息爬取，而puppylpg的文章中对于steam信息的处理是buff的近七天交易记录，而折上折的要点在于销量，因此我又找了一个steam的.jsonhttps://steamcommunity.com/market/priceoverview/?country=CN&currency=23&appid=570&market_hash_name=Exalted%20Manifold%20Paradox来进行爬取。
在这里插入图片描述html

运行效果图

PS.本文例子为dota2，buff上的其他饰品同理python

环境

import requests
import re
import pandas as pd
import time

根据我所用到的引用模块，须要的库为
requests库，用于获取buff及steam的html，安装教程：
re库，用于正则匹配获取所需数据，为内置库。
pandas库，用于保存最终结果，安装教程：
time库，用于延时（防止被检测请求过多，获得html为null）、记录运行时间，为内置库。web

开始前

环境配置完毕后让咱们理一下逻辑，最终获得的结果应该包含[饰品名字]、[BUFF价格]、[steam价格]、[steam24小时售出数量]、[折率]。
那么：
第一步——爬取BUFF的[饰品名字]和[BUFF价格]。
第二步——爬取steam的[steam价格]和[steam24小时售出数量]。
第三步——对得到的数据进行处理。json

第一步：爬取BUFF的[饰品名字]和[BUFF价格]

爬取BUFF数据遇到的第一个问题是登录
可以使用登陆后的cookie进行访问。
详细参考api

1.获取cookie和header

访问https://buff.163.com/登录BUFF后按F12打开开发者工具，选中网络+标头，刷新页面，找到Cookie和User-Agentsass

在这里插入图片描述

# 表头
    headers = { 
 
  
        'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Mobile Safari/537.36 Edg/85.0.564.63'
    }

    # BUFF cookie
    cookie_str = r'Device-Id=yFZJ64QHkCtznv0xgxqY; _ga=GA1.2.1833906180.1599195822; P_INFO=18581573728|1601021166|1|netease_buff|00&99|null&null&null#jil&220100#10#0|&0||18581573728; remember_me=U1093767863|vtjnXD4iEtuLVHis1vNpStAd0qoV56Oo; Locale-Supported=zh-Hans; _gid=GA1.2.1530976571.1601513433; game=csgo; session=1-k2SvP24G4lp7mVi7on-6KWL_AgR3y4wyEphsI_QXDFEf2046758383; _gat_gtag_UA_109989484_1=1; csrf_token=ImU1OWQwN2M3YmM4NTBhY2RhNTljZDA3OTY3NDZkN2Y2NjI5ZTIzMTki.ElcQxQ.wgB--s7F06wV64qbnKXHQjX9I_k'
    cookies = { 
 
  }
    for line in cookie_str.split(';'):
        key, value = line.split('=', 1)
        cookies[key] = value

2.访问buff返回html

在BUFF中输入筛选价格能够帮咱们过滤一部分数据，我这里选的35~200。安全

在这里插入图片描述

访问https://buff.163.com/api/market/goods?game=dota2&page_num=1&min_price=35&max_price=200cookie

"items": [
      {
        "appid": 570, 
        "bookmarked": false, 
        "buy_max_price": "131", 
        "buy_num": 45, 
        "can_search_by_tournament": false, 
        "description": null, 
        "game": "dota2", 
        "goods_info": {
          "icon_url": "https://g.fp.ps.netease.com/market/file/5a0e956d6f049424e570876aRCofBmRW", 
          "info": {
            "tags": {
              "hero": {
                "category": "hero", 
                "internal_name": "npc_dota_hero_phantom_assassin", 
                "localized_name": "\u5e7b\u5f71\u523a\u5ba2"
              }, 
              "rarity": {
                "category": "rarity", 
                "internal_name": "arcana", 
                "localized_name": "\u81f3\u5b9d"
              }, 
              "slot": {
                "category": "slot", 
                "internal_name": "weapon", 
                "localized_name": "\u6b66\u5668"
              }, 
              "type": {
                "category": "type", 
                "internal_name": "wearable", 
                "localized_name": "\u53ef\u4f69\u5e26"
              }
            }
          }, 
          "item_id": 7247, 
          "original_icon_url": "https://g.fp.ps.netease.com/market/file/59926f895e60273b4cf3f424sv02msLE", 
          "steam_price": "29.48", 
          "steam_price_cny": "200.19"
        }, 
        "has_buff_price_history": true, 
        "id": 14575, 
        "market_hash_name": "Exalted Manifold Paradox", 
        "market_min_price": "0", 
        "name": "\u5c0a\u4eab \u65e0\u53cc\u8be1\u9b45", 
        "quick_price": "131.28", 
        "sell_min_price": "131.78", 
        "sell_num": 284, 
        "sell_reference_price": "131.78", 
        "steam_market_url": "https://steamcommunity.com/market/listings/570/Exalted%20Manifold%20Paradox", 
        "transacted_num": 0
      },

访问"steam_market_url"：https://steamcommunity.com/market/listings/570/Exalted%20Manifold%20Paradox，正是页面第一个饰品。
因此咱们要访问的url为https://buff.163.com/api/market/goods?game=dota2&page_num=+i+&min_price=35&max_price=200网络

for i in range(5):
        # 标准url:https://buff.163.com/api/market/goods?game=dota2&page_num=1&min_price=35&max_price=200
        buff_dota2_url = 'https://buff.163.com/api/market/goods?game=dota2&page_num=' + str(
            i + 1) + '&min_price=35&max_price=200'
        buff_dota2_text = requests.get(url=buff_dota2_url, headers=headers, cookies=cookies).text
        print(buff_dota2_text)

3.re正则匹配获得[饰品名字]和[BUFF价格]

再利用re正则匹配找到咱们须要[饰品名字]和[BUFF价格]。
发现[饰品名字跟在"steam_market_url"后面，在https://buff.163.com/api/market/goods?game=dota2&page_num=1&min_price=35&max_price=200中查找"steam_market_url": "https://steamcommunity.com/market/listings/570/(.*)",发现仅有20个，意思就是每一个item对应一个，那么这就是[饰品名字]的匹配规则，BUFF价格同理。
关于re.findall的使用参考悲恋花丶无意之人。session

for i in range(5):
        # 标准url:https://buff.163.com/api/market/goods?game=dota2&page_num=1&min_price=35&max_price=200
        buff_dota2_url = 'https://buff.163.com/api/market/goods?game=dota2&page_num=' + str(
            i + 1) + '&min_price=35&max_price=200'
        buff_dota2_text = requests.get(url=buff_dota2_url, headers=headers, cookies=cookies).text
        
        # 饰品名
        names_list_temp = re.findall(r'"steam_market_url": "https://steamcommunity.com/market/listings/570/(.*)",',
                                     buff_dota2_text, re.M)
        # BUFF售价
        price_list_temp = re.findall(r'"sell_min_price": "(.*)",', buff_dota2_text, re.M)

第二步：爬取steam的[steam价格]和[steam24小时售出数量]

1.访问steam返回html

[steam24小时售出数量]我只在库存中查看物品的时候看见过，因此进入库存，按F12打开开发者工具，选中网络，刷新页面后随便点一个物品。
在这里插入图片描述

红框的.json文件内容正是咱们要的内容。
访问https://steamcommunity.com/market/priceoverview/?country=CN&currency=23&appid=570&market_hash_name=Exalted%20Manifold%20Paradox

{"success":true,"lowest_price":"¥ 201.02","volume":"64","median_price":"¥ 167.51"}
steam_time = len(names_list_temp)
        # 取steam价格和在售数量
        for k in range(steam_time):
            item = names_list_temp[k]
            steam_item_text = requests.get(url=url + item, headers=headers).text
            print(steam_item_text)

2.re正则匹配获得[steam价格]和[steam24小时售出数量]

这里注意，re.findall获得的是列表，须要选择第一个才能进行比较与转换。

steam_24h_qty = int(re.findall(r'"volume":"([0-9]*)",', steam_item_text, re.M)[0])
	price_steam_temp = re.findall(r'"lowest_price":"¥ ([0-9]*.[0-9]*)",', steam_item_text, re.M)[0]

第三步：对得到的数据进行处理

首先理一下逻辑，已知参数[饰品名字]和[BUFF价格]，可经过[饰品名字]得到[steam价格]和[steam24小时售出数量]，当[steam24小时售出数量]<必定值，这组数据就应该被删去，[steam价格]也不须要爬取，也就是：
1.经过[饰品名字]得到[steam24小时售出数量]
2.比较[steam24小时售出数量]判断删除该组仍是爬取[steam价格]
3.进行删除
4.保存

1.经过[饰品名字]得到[steam24小时售出数量]

steam_time = len(names_list_temp)
        # 取steam价格和在售数量
        for k in range(steam_time):
            item = names_list_temp[k]
            steam_item_text = requests.get(url=url + item, headers=headers, cookies=steam_cookies).text
            steam_24h_qty_temp = int(re.findall(r'"volume":"([0-9]*)",', steam_item_text, re.M)[0])

2.比较[steam24小时售出数量]判断删除该组仍是爬取[steam价格]

cleanlist = []
        steam_time = len(names_list_temp)
        # 取steam价格和在售数量
        for k in range(steam_time):
            item = names_list_temp[k]
            steam_item_text = requests.get(url=url + item, headers=headers, cookies=steam_cookies).text
            print(k + 1, "/", steam_time, ":", steam_item_text, item)
            try:
                steam_24h_qty_temp = int(re.findall(r'"volume":"([0-9]*)",', steam_item_text, re.M)[0])
            except IndexError:
                steam_24h_qty_temp = 0
            if steam_24h_qty_temp < 10:
                cleanlist.append(k)
            else:
                try:
                    price_steam_temp0 = re.findall(r'"lowest_price":"¥ ([0-9]*.[0-9]*)",', steam_item_text, re.M)[0]
                    price_steam_temp.append(price_steam_temp0)
                    sell_num_list_temp.append(steam_24h_qty_temp)
                except IndexError:
                    cleanlist.append(k)

3.进行删除

for k in range(len(cleanlist) - 1, -1, -1):
            names_list_temp.pop(cleanlist[k])
            price_list_temp.pop(cleanlist[k])

4.保存

for k in range(len(names_list_temp)):
            soldprice_temp0 = float(price_steam_temp[k]) / 1.15
            percentage_temp0 = float(price_list_temp[k]) / soldprice_temp0
            soldprice_temp.append(soldprice_temp0)
            percentage_temp.append(percentage_temp0)
        # 饰品名
        name_list.extend(names_list_temp)
        # BUFF价格
        price_list.extend(price_list_temp)
        # steam价格
        price_steam_list.extend(price_steam_temp)
        # steam 24小时销售数量
        sell_num_list.extend(sell_num_list_temp)
        # 按steam市场最低价售出税后价格
        soldprice.extend(soldprice_temp)
        # 折值
        percentage.extend(percentage_temp)
        # 汇合信息写成表格并保存
        csv_name = ["name", "BUFF price", "steam price", "steam 24hour sold qty", "steam sellprice", "percentage"]
        csv_data = zip(name_list, price_list, price_steam_list, sell_num_list, soldprice, percentage)
        items_information = pd.DataFrame(columns=csv_name, data=csv_data)
        items_information.to_csv("items_information.csv")

总结

对于steam的访问须要梯子，不要忘记time.sleep()，若是访问steam .json返回为null，能够换个节点。
我本身使用时time.sleep(3)，结果爬了几页BUFF后steam .json返回null，一直没变回来，估计是被ban了，后面time.sleep(5)运行没问题。

附完整代码

import requests
import re
import pandas as pd
import time


def main():
    time_start = time.time()
    # steam appid=750 为 DOTA2
    url = r'https://steamcommunity.com/market/priceoverview/?country=CN&currency=23&appid=570&market_hash_name='

    # 表头
    headers = { 
 
  
        'User-Agent': ''
    }

    # BUFF cookie
    cookie_str = r''
    cookies = { 
 
  }
    for line in cookie_str.split(';'):
        key, value = line.split('=', 1)
        cookies[key] = value

    # 初始化
    name_list = []
    price_list = []
    price_steam_list = []
    sell_num_list = []
    soldprice = []
    percentage = []

    for i in range(5):
        time_page_start = time.time()
        dec = time_page_start - time_start
        minute = int(dec / 60)
        second = dec % 60
        print("%02d:%02d page" % (minute, second), i)
        # 标准url:https://buff.163.com/api/market/goods?game=dota2&page_num=1&min_price=35&max_price=200
        buff_dota2_url = 'https://buff.163.com/api/market/goods?game=dota2&page_num=' + str(
            i + 1) + '&min_price=35&max_price=200'
        buff_dota2_text = requests.get(url=buff_dota2_url, headers=headers, cookies=cookies).text

        # 饰品名
        names_list_temp = re.findall(r'"steam_market_url": "https://steamcommunity.com/market/listings/570/(.*)",',
                                     buff_dota2_text, re.M)
        # BUFF售价
        price_list_temp = re.findall(r'"sell_min_price": "(.*)",', buff_dota2_text, re.M)

        cleanlist = []
        price_steam_temp = []
        soldprice_temp = []
        percentage_temp = []
        sell_num_list_temp = []
        print("BUFF当前页爬取完成，开始访问steam")
        steam_time = len(names_list_temp)
        # 取steam价格和在售数量
        for k in range(steam_time):
            item = names_list_temp[k]
            steam_item_text = requests.get(url=url + item, headers=headers, cookies=steam_cookies).text
            print(k + 1, "/", steam_time, ":", steam_item_text, item)
            time.sleep(5)
            try:
                steam_24h_qty_temp = int(re.findall(r'"volume":"([0-9]*)",', steam_item_text, re.M)[0])
            except IndexError:
                steam_24h_qty_temp = 0
            if steam_24h_qty_temp < 10:
                cleanlist.append(k)
            else:
                try:
                    price_steam_temp0 = re.findall(r'"lowest_price":"¥ ([0-9]*.[0-9]*)",', steam_item_text, re.M)[0]
                    price_steam_temp.append(price_steam_temp0)
                    sell_num_list_temp.append(steam_24h_qty_temp)
                except IndexError:
                    cleanlist.append(k)
        for k in range(len(cleanlist) - 1, -1, -1):
            names_list_temp.pop(cleanlist[k])
            price_list_temp.pop(cleanlist[k])
        for k in range(len(names_list_temp)):
            soldprice_temp0 = float(price_steam_temp[k]) / 1.15
            percentage_temp0 = float(price_list_temp[k]) / soldprice_temp0
            soldprice_temp.append(soldprice_temp0)
            percentage_temp.append(percentage_temp0)
        # 饰品名
        name_list.extend(names_list_temp)
        # BUFF价格
        price_list.extend(price_list_temp)
        # steam价格
        price_steam_list.extend(price_steam_temp)
        # steam 24小时销售数量
        sell_num_list.extend(sell_num_list_temp)
        # 按steam市场最低价售出税后价格
        soldprice.extend(soldprice_temp)
        # 折值
        percentage.extend(percentage_temp)
        time_page_end = time.time()
        dec = time_page_end - time_page_start
        minute = int(dec / 60)
        second = dec % 60
        print("page_cost: %02dmin%02dsec" % (minute, second))
    # 汇合信息写成表格并保存
    csv_name = ["name", "BUFF price", "steam price", "steam 24hour sold qty", "steam sellprice", "percentage"]
    csv_data = zip(name_list, price_list, price_steam_list, sell_num_list, soldprice, percentage)
    items_information = pd.DataFrame(columns=csv_name, data=csv_data)
    items_information.to_csv("items_information.csv")


if __name__ == "__main__":`在这里插入代码片`
    # 当程序被调用执行时，调用函数
    main()
联系我们 沪ICP备13005482号-10