猫哥教你写爬虫 039--存储数据

时间 2019-12-04

原文原文链接

存储到mysql数据库

咱们须要新的模块 pymysql

pip install pymysql
复制代码

建立数据库book_store

create database book_store character set utf8;
复制代码

建立category分类表, 包含id和分类名

use book_store;
create table category(
    id int primary key auto_increment, 
    name varchar(255) not null
);
复制代码

建立数据表book, 包含id,分类id,图书名,价格, 还有外键

create table book(
	id int primary key auto_increment,
    cid int not null,
    title varchar(200) not null,
    price decimal(10,2) not null,
    foreign key(cid) references category(id)
);
复制代码

mysql> desc category;
+-------+--------------+------+-----+---------+----------------+
| Field | Type         | Null | Key | Default | Extra          |
+-------+--------------+------+-----+---------+----------------+
| id    | int(11)      | NO   | PRI | NULL    | auto_increment |
| name  | varchar(255) | NO   |     | NULL    |                |
+-------+--------------+------+-----+---------+----------------+
mysql> desc book;
+-------+---------------+------+-----+---------+----------------+
| Field | Type          | Null | Key | Default | Extra          |
+-------+---------------+------+-----+---------+----------------+
| id    | int(11)       | NO   | PRI | NULL    | auto_increment |
| cid   | int(11)       | NO   | MUL | NULL    |                |
| title | varchar(200)  | NO   |     | NULL    |                |
| price | decimal(10,2) | NO   |     | NULL    |                |
+-------+---------------+------+-----+---------+----------------+
4 rows in set (0.02 sec)
复制代码

使用python把爬取的数据保存到数据库

原来的代码...

import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get('http://books.toscrape.com/').text,'html.parser')
with open('books.txt','w',encoding='utf8') as file:
    for i in soup.find('ul',class_='nav nav-list').find('ul').find_all('li'):
        file.write(i.text.strip()+'\n')
        res = requests.get("http://books.toscrape.com/"+i.find('a')['href'])
        res.encoding='utf8'
        soup = BeautifulSoup(res.text,'html.parser')
        for j in soup.find_all('li',class_="col-xs-6 col-sm-4 col-md-3 col-lg-3"):
            print(j.find('h3').find('a')['title'])
            file.write('\t"{}" {}\n'.format(j.find('h3').find('a')['title'],j.find('p',class_='price_color').text))
复制代码

进行一下注释和改进

import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get('http://books.toscrape.com/').text,'html.parser')
for i in soup.find('ul',class_='nav nav-list').find('ul').find_all('li'):
    category = i.text.strip() # 分类名
    print(category)
    res = requests.get("http://books.toscrape.com/"+i.find('a')['href'])
    res.encoding='utf8'
    soup = BeautifulSoup(res.text,'html.parser')
    for j in soup.find_all('li',class_="col-xs-6 col-sm-4 col-md-3 col-lg-3"):
        title = j.find('h3').find('a')['title'] # 图书名
        price = j.find('p',class_='price_color').text # 图书价格
复制代码

研究一下pymysql 的使用方法

shockerli.net/post/python…

建立mysql链接

import pymysql
connection = pymysql.connect(host='localhost',
                             port=3306,
                             user='root',
                             password='root',
                             db='demo',
                             charset='utf8')
复制代码

生成游标, 执行sql语句...

# 获取游标
cursor = connection.cursor()
    
# 建立数据表
effect_row = cursor.execute(''' CREATE TABLE `users` ( `name` varchar(32) NOT NULL, `age` int(10) unsigned NOT NULL DEFAULT '0', PRIMARY KEY (`name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ''')
    
# 插入数据(元组或列表)
effect_row = cursor.execute('INSERT INTO `users` (`name`, `age`) VALUES (%s, %s)', ('mary', 18))
    
# 插入数据(字典)
info = {'name': 'fake', 'age': 15}
effect_row = cursor.execute('INSERT INTO `users` (`name`, `age`) VALUES (%(name)s, %(age)s)', info)
    
connection.commit()
复制代码

一次性执行多条sql语句

# 获取游标
cursor = connection.cursor()
    
# 批量插入
effect_row = cursor.executemany(
    'INSERT INTO `users` (`name`, `age`) VALUES (%s, %s) ON DUPLICATE KEY UPDATE age=VALUES(age)', [
        ('hello', 13),
        ('fake', 28),
    ])
    
connection.commit()
复制代码

获取自增id

cursor.lastrowid
复制代码

查询数据, 一条或者多条

# 执行查询 SQL
cursor.execute('SELECT * FROM `users`')
# 获取单条数据
cursor.fetchone()
# 获取前N条数据
cursor.fetchmany(3)
# 获取全部数据
cursor.fetchall()
复制代码

思路分析

代码拆解

面向对象的写法...

import requests
from bs4 import BeautifulSoup
import pymysql
class Spider():
    def __init__(self, base_url):
        self.base_url = base_url
        self.db = Database('localhost', 'root', 'root', 'book_store')
        self.soup = BeautifulSoup(requests.get(
            self.base_url).text, 'html.parser')
    def get_category(self):
        for i in self.soup.find('ul', class_='nav nav-list').find('ul').find_all('li'):
            category = i.text.strip()  # 分类名
            self.db.add_category(category)
            self.db.book_link_dict[category] = "http://books.toscrape.com/"+i.find('a')[
                'href']
    def get_book(self):
        for cat in self.db.book_link_dict:
            res = requests.get(self.db.book_link_dict[cat])
            res.encoding = 'utf8'
            soup = BeautifulSoup(res.text, 'html.parser')
            for j in soup.find_all('li', class_="col-xs-6 col-sm-4 col-md-3 col-lg-3"):
                title = j.find('h3').find('a')['title']  # 图书名
                price = j.find('p', class_='price_color').text[1:]  # 图书价格
                self.db.add_book(cat, title, price)
    def start(self):
        self.get_category()
        self.get_book()
class Database():
    category_dict = {}
    book_link_dict = {}
    def __init__(self, host, username, password, db):
        self.connect = pymysql.connect(
            host=host, port=3306, user=username, password=password, db=db, charset='utf8'
        )
        self.cursor = self.connect.cursor()
    def add_category(self, name):
        sql = "insert into `category`(`name`) values('{}')".format(name)
        print(sql)
        self.cursor.execute(sql)
        self.connect.commit()
        last_id = self.cursor.lastrowid
        self.category_dict[name] = last_id
        return last_id
    def add_book(self, category, title, price):
        if category in self.category_dict:
            cid = self.category_dict[category]
        else:
            cid = self.add_category(category)
        sql = "insert into `book`(`cid`,`title`,`price`) values({},{},{})".format(
            cid, repr(title), price)
        print(sql)
        self.cursor.execute(sql)
        self.connect.commit()
        return self.cursor.lastrowid
if __name__ == "__main__":
    spider = Spider('http://books.toscrape.com/')
    spider.start()
复制代码

小做业:爬取猫眼电影top100, 保存 电影名, 排名, 评分, 主演, 封面图片, 上映时间到数据库

maoyan.com/board/4

快速跳转:

猫哥教你写爬虫 039--存储数据

存储到mysql数据库

咱们须要新的模块 pymysql

建立数据库book_store

建立category分类表, 包含id和分类名

建立数据表book, 包含id,分类id,图书名,价格, 还有外键

使用python把爬取的数据保存到数据库

原来的代码...

进行一下注释和改进

研究一下pymysql 的使用方法

shockerli.net/post/python…

建立mysql链接

生成游标, 执行sql语句...

一次性执行多条sql语句

获取自增id

查询数据, 一条或者多条

思路分析

代码拆解

面向对象的写法...

小做业:爬取猫眼电影top100, 保存 电影名, 排名, 评分, 主演, 封面图片, 上映时间到数据库

maoyan.com/board/4

快速跳转:

小做业:爬取猫眼电影top100, 保存 `电影名`, `排名`, `评分`, `主演`, `封面图片`, `上映时间`到数据库