爬虫综合大做业

1、爬取的对象

 豆瓣图书的书籍，总共获取1万8千条数据。

2、保存数据为excel格式。

 

3、数据分析

1.经过在excel进行数据处理，筛选出20部评分高，评论多的做品，以下图所示。

因而，推荐阅读的书籍为：

闲暇时分，可经过了解做品详情，若感兴趣，能够阅读该做品，省去筛选做品的时间。

2.有部分做者，做品多，并且评分也高，好比：

若读者们感兴趣，也能够找这些做者的做品来阅读。

运行代码：

import re
import time
import pandas as pd
import random
import requests
from bs4 import BeautifulSoup

user = ["Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0",\
        "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50",\
        "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",\
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",\
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",\
        "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",\
        "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",\
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",\
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20"]
   
def get_user():
    use = random.choice(user)
    return use

def get_soup(url):
    headers = {'user-agent':get_user()}
    res = requests.get(url,headers=headers)
    res.encoding = 'utf-8'  
    soup=BeautifulSoup(res.text,"html.parser")
    time.sleep(random.random()*3)
    book_list = []
    for i in range(0,20):        
        book_dict = {}
        try:
            book_dict['做品']= soup.find_all("h2", {"class": ""})[i].find_all('a')[0].text.strip().replace("\n","") 
            book_dict['做者']= soup.select('.pub')[i].text.strip().split('/')[0]  
            book_dict['评分']=soup.findAll("span", {"class": "rating_nums"})[i].text.strip()
            book_dict['评论次数']=soup.select('.pl')[i].text.strip().lstrip("(").rstrip(")人评价")
            book_dict['价格']=soup.select('.pub')[i].text.strip().split('/')[-1]
            book_dict['详情']= soup.find_all("div", {"class": "info"})[i].find_all('p')[0].text.strip().replace("\n","") 
        except Exception as e:
            book_dict['做品']=''
            book_dict['做者']=''
            book_dict['评分']=''
            book_dict['评论次数']=''
            book_dict['价格']=''
            book_dict['详情']=''
        else:
            book_list.append(book_dict)
    return book_list
    
    
allbook_list =[]
for i in range(0,980,20):
    url='https://book.douban.com/tag/%E6%AD%A6%E4%BE%A0?start={}&type=T'.format(i)    
    allbook_list.extend(get_soup(url))

bookdf= pd.DataFrame(allbook_list)
bookdf

bookdf.to_csv(r'C:\Users\Administrator\Desktop\book.csv', encoding="utf_8_sig")