你都怎样看知乎日报？

时间 2019-11-12

原文原文链接

1.分析知乎日报网页（http://daily.zhihu.com/）html

<span class="title">这样一分析，你就明白该在哪里投广告了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269183" class="link-button"><img src="http://pic2.zhimg.com/05c67496e38f662958a141847a734ffd.jpg" class="preview-image"><span class="title">有些热闹的「共享经济」，恐怕只是一个美好的童话</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9268452" class="link-button"><img src="http://pic3.zhimg.com/da8d8d3bf282170379c51ea0cf1ae4a6.jpg" class="preview-image"><span class="title">让孩子拥有属于本身的无聊时光，到底有多重要？</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269792" class="link-button"><img src="http://pic2.zhimg.com/245f3cf8dd4bfdd1bf0911ba4b486295.jpg" class="preview-image"><span class="title">看不懂，说人话，否则公司就亏大发了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269818" class="link-button"><img src="http://pic3.zhimg.com/8b1588e23187bb05d160d599bbfe1752.jpg" class="preview-image"><span class="title">《金刚狼 3》中有哪些隐藏的彩蛋和有趣的细节？</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9266807" class="link-button"><img src="http://pic1.zhimg.com/4cb4d5dec4a68553e41dcbd483010e84.jpg" class="preview-image"><span class="title">瞎扯 · 如何正确地吐槽</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9259222" class="link-button"><img src="http://pic4.zhimg.com/16378a8129349aface9694cd27c71e2f.jpg" class="preview-image"><span class="title">小事 · 爱无能</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9267167" class="link-button"><img src="http://pic1.zhimg.com/fdf5e0ff47de69d615f706559a260168.jpg" class="preview-image">python

每个话题和图片都在一个span标签里git

<span class="title">这样一分析，你就明白该在哪里投广告了</span></a></div></div><div class="wrap"><div class="box"><a href="/story/9269183" class="link-button"><img src="http://pic2.zhimg.com/05c67496e38f662958a141847a734ffd.jpg" class="preview-image">web

很简单构造一个正则表达式去匹配上面（获取标题，图片，连接）正则表达式
pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)

2.下载页面匹配正则表达式多线程

先建立一个请求头app

self.header={
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'Accept-Encoding':'gzip, deflate, sdch',
            'Accept-Language':'zh-CN,zh;q=0.8',
            'Cache-Control':'max-age=0',
            'Connection':'keep-alive',
            'Host':'daily.zhihu.com',
            'Referer':'https://www.baidu.com/link?url=Eh6CKs72Buyf0LEjPd1795QSL8ZK74kwItBvzaybausT6proAZIr3UkkmMPSDfk7&wd=&eqid=d1bd8fd9004118c90000000258bd8149',
            'Upgrade-Insecure-Requests':'1',
            'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/56.0.2924.76 Chrome/56.0.2924.76 Safari/537.36'
        }

使用下载页面使用正则匹配url

html=self.req.get(url=self.url,headers=self.header)
        pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)
        T=list()
        self.l=re.findall(pattern,html.text)

3.使用多线程下载图片spa

T=list()
        self.l=re.findall(pattern,html.text)
        for i in self.l:
            self.w.write(str(i)+'\n')
            #if(self.n<29):
            t=Thread(target=self.getimg,args=(i[2],self.n))
                #self.getimg(i[2],self.n)
            T.append(t)
            t.start()
            #print(i)
            self.n+=1
            #time.sleep(1)
        for tt in T:
            tt.join()

def getimg(self,src,n):
        try:
            h=self.req.get(url=src)
            s=open(str(n)+'.jpg','wb')
            s.write(h.content)
            s.close()
        except requests.exceptions.MissingSchema:
            print('这个url无效',n)

4.结果.net

所有代码：

#!/usr/bin/python3
#coding:utf8
import requests
import re
import time
from threading import Thread
class main(object):
    def __init__(self):
        self.url='http://daily.zhihu.com/'
        self.l=list()
        self.n=0
        self.req=requests.Session()
        self.header={
            'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'Accept-Encoding':'gzip, deflate, sdch',
            'Accept-Language':'zh-CN,zh;q=0.8',
            'Cache-Control':'max-age=0',
            'Connection':'keep-alive',
            'Host':'daily.zhihu.com',
            'Referer':'https://www.baidu.com/link?url=Eh6CKs72Buyf0LEjPd1795QSL8ZK74kwItBvzaybausT6proAZIr3UkkmMPSDfk7&wd=&eqid=d1bd8fd9004118c90000000258bd8149',
            'Upgrade-Insecure-Requests':'1',
            'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/56.0.2924.76 Chrome/56.0.2924.76 Safari/537.36'
        }
    def getpage(self):
        html=self.req.get(url=self.url,headers=self.header)
        pattern=re.compile(u'<span class="title">(.*?)</span>.*?'+
        u'<a href="(.*?)".*?'+
        u'<img src="(.*?)".*?'
        ,re.S)
        T=list()
        self.l=re.findall(pattern,html.text)
        for i in self.l:
            self.w.write(str(i)+'\n')
            #if(self.n<29):
            t=Thread(target=self.getimg,args=(i[2],self.n))
                #self.getimg(i[2],self.n)
            T.append(t)
            t.start()
            #print(i)
            self.n+=1
            #time.sleep(1)
        for tt in T:
            tt.join()
    def getimg(self,src,n):
        try:
            h=self.req.get(url=src)
            s=open(str(n)+'.jpg','wb')
            s.write(h.content)
            s.close()
        except requests.exceptions.MissingSchema:
            print('这个url无效',n)


if __name__=='__main__':
    p=main()
    p.w=open('zh.txt','w')
    p.getpage()
    p.w.close()

项目地址：https://git.oschina.net/nanxun/zhihuribao