urllib模块

时间 2019-12-12

标签 urllib 模块繁體版

原文原文链接

1. 打开URL，读取返回的数据html

#!/usr/bin/env python

from urllib import request

url = 'https://www.jd.com'
req = request.urlopen(url)
res = req.read()
print(res.decode('utf-8'))

2. 对post数据进行编码python

#!/usr/bin/env python

from urllib import request
from urllib import parse

url = 'http://httpbin.org/post'
payload = {'key1': 'value1', 'key2': 'value2'}
newpayload = parse.urlencode(payload).encode('utf-8')  #post的数据必须是bytes或者iterable of bytes，不能是str，所以须要进行encode()编码
print(type(newpayload))                                #encode就是把字符串转换成字节
req = request.urlopen(url, data=newpayload)
res = req.read()
print(res.decode('utf-8'))                            #decode就是把字节转换成字符串

--------------------------------------------------------------------------------------->
<class 'bytes'>
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "key1": "value1",
    "key2": "value2"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Connection": "close",
    "Content-Length": "23",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.6"
  },
  "json": null,
  "origin": "183.48.35.148",
  "url": "http://httpbin.org/post"
}

构造请求头

#!/usr/bin/env python

from urllib import request

url = 'https://www.qiushibaike.com/'
ua = {'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"}

req = request.Request(url, headers=ua)
res = request.urlopen(req)
print(res.read().decode('utf-8'))

4. 下载git

#!/usr/bin/env python

from urllib import request
url = 'https://www.baidu.com/img/bd_logo1.png'
request.urlretrieve(url, filename='baidu.png')

也能够本身写下载方法

import codecs
from urllib import request

url = 'https://www.baidu.com/img/bd_logo1.png'

req = request.urlopen(url)
res = req.read()
with codecs.open('1.png', 'wb') as fd:
    fd.write(res)

5. 代理github

因为默认的opener（也就是urlopen()，它是opner的一个实例）只能携带get或者post的参数访问url，和设置过时时间，因此不能用于代理，也没法携带cookie跨域请求

因此要自行构造一个支持代理的opener

#!/usr/bin/env python

from urllib import request

url = 'http://2017.ip138.com/ic.asp'
print(request.urlopen(url).read().decode('gb2312'))

dic = {'http': '125.88.177.128:3128'}
proxy = request.ProxyHandler(dic)             //建立一个有代理功能的handler
opener = request.build_opener(proxy)          //用handler建立opener
req = opener.open(url)                        //用opener访问url
res = req.read().decode('gb2312')
print(res)

----------------------------------------------------------->
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=gb2312">
<title> 您的IP地址 </title>
</head>
<body style="margin:0px"><center>您的IP是：[113.68.17.83] 来自：广东省广州市 电信</center></body></html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=gb2312">
<title> 您的IP地址 </title>
</head>
<body style="margin:0px"><center>您的IP是：[119.129.229.185] 来自：广东省广州市 电信</center></body></html>

6. 构造能携带cookie的openerjson

在python2中，建立cookie对象是使用cookielib模块

在python3中，建立cookie对象是使用http.cookiejar模块

#!/usr/bin/env python

import http.cookiejar
from urllib import request

url = 'https://www.github.com'
cookie = http.cookiejar.CookieJar()                  //建立cookie对象
print(cookie)
handler = request.HTTPCookieProcessor(cookie)        //建立能保存cookie的handler
opener = request.build_opener(handler)               //用handler建立opener
req = opener.open(url)                               //用opener访问url，访问完cookie对象就有值了
print(cookie)

---------------------------------------------------------->
<CookieJar[]>
<CookieJar[<Cookie logged_in=no for .github.com/>, <Cookie _gh_sess=eyJzZXNzaW9uX2lkIjoiYzJkNzE0NTA1OWQ4ZDc5MDA1NjM4NWI1ZDIwYjkxNTgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjU0MDEzNzE0MCwiX2NzcmZfdG9rZW4iOiJOdTVMYXZjT0xMNDRYS1JmOTh3MFZNQnI0c3ZPOTNCWkFWRUZnN21yUFZ3PSJ9--c47838621e21bba5d72c7050345b40a9c513335a for github.com/>]>

7. 保存cookie信息到文件中跨域

MozillaCookieJar() 这个类继承了FileCookieJar() 这个类， FileCookieJar()的构造函数定义了一个filename参数，也就是存储cookie的文件，而能获取cookie 是由于FileCookieJar() 这个类继承了CookieJar() 这个类cookie

#!/usr/bin/env python

import http.cookiejar
from urllib import request

url = 'https://www.github.com'
filename = 'cookie.txt'
cookie = http.cookiejar.MozillaCookieJar(filename)      //实例化的时候传入filename参数，获得cookie对象
handler = request.HTTPCookieProcessor(cookie)
opener = request.build_opener(handler)
res = opener.open(url)
cookie.save()

8. 从文件中读取cookie并访问urlapp

#!/usr/bin/env python

import http.cookiejar
from urllib import request

filename = 'cookie.txt'
cookie = http.cookiejar.MozillaCookieJar(filename)
cookie.load(filename)
url = 'https://www.github.com'
handler = request.HTTPCookieProcessor(cookie)
opener = request.build_opener(handler)
res = opener.open(url).read()
print(res.decode('utf-8'))