MAC系统下基于Python3版本安装Scrapy
时间 2021-08-15
标签
爬虫
python
自然语言处理
输入 pip install scrapy 执行 scrapy 安装
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
|
➜ ~ pip install Scrapy
Collecting Scrapy
Using cached Scrapy-1.4.0-py2.py3-none-any.whl
Collecting lxml (from Scrapy)
Using cached lxml-4.1.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy)
Using cached PyDispatcher-2.0.5.tar.gz
Collecting Twisted>=13.1.0 (from Scrapy)
Using cached Twisted-17.9.0.tar.bz2
Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Scrapy)
Collecting queuelib (from Scrapy)
Using cached queuelib-1.4.2-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from Scrapy)
Using cached cssselect-1.0.1-py2.py3-none-any.whl
Collecting parsel>=1.1 (from Scrapy)
Using cached parsel-1.2.0-py2.py3-none-any.whl
Collecting service-identity (from Scrapy)
Using cached service_identity-17.0.0-py2.py3-none-any.whl
Collecting six>=1.5.2 (from Scrapy)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from Scrapy)
Using cached w3lib-1.18.0-py2.py3-none-any.whl
Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->Scrapy)
Collecting constantly>=15.1 (from Twisted>=13.1.0->Scrapy)
Using cached constantly-15.1.0-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from Twisted>=13.1.0->Scrapy)
Using cached incremental-17.5.0-py2.py3-none-any.whl
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->Scrapy)
Using cached Automat-0.6.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->Scrapy)
Using cached hyperlink-17.3.1-py2.py3-none-any.whl
Collecting pyasn1 (from service-identity->Scrapy)
Using cached pyasn1-0.3.7-py2.py3-none-any.whl
Collecting pyasn1-modules (from service-identity->Scrapy)
Using cached pyasn1_modules-0.1.5-py2.py3-none-any.whl
Collecting attrs (from service-identity->Scrapy)
Using cached attrs-17.2.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->Scrapy)
Installing collected packages: lxml, PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, Scrapy
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber
ensure_dir(destdir)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir
os.makedirs(path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml'
|
出现 OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml' 错误
4. 尝试重新安装lxml,执行 sudo pip install lxml
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
|
➜ ~ sudo pip install lxml
The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
Downloading lxml-4.1.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (8.7MB)
100% |████████████████████████████████| 8.7MB 97kB/s
Installing collected packages: lxml
Successfully installed lxml-4.1.0
➜ ~ sudo pip install scrapy
The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting scrapy
Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248kB)
100% |████████████████████████████████| 256kB 1.5MB/s
Requirement already satisfied: lxml in /Library/Python/2.7/site-packages (from scrapy)
Collecting PyDispatcher>=2.0.5 (from scrapy)
Downloading PyDispatcher-2.0.5.tar.gz
Collecting Twisted>=13.1.0 (from scrapy)
Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
100% |████████████████████████████████| 3.0MB 371kB/s
Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)
Collecting queuelib (from scrapy)
Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from scrapy)
Downloading cssselect-1.0.1-py2.py3-none-any.whl
Collecting parsel>=1.1 (from scrapy)
Downloading parsel-1.2.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
Downloading service_identity-17.0.0-py2.py3-none-any.whl
Collecting six>=1.5.2 (from scrapy)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from scrapy)
Downloading w3lib-1.18.0-py2.py3-none-any.whl
Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->scrapy)
Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)
Downloading constantly-15.1.0-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)
Downloading incremental-17.5.0-py2.py3-none-any.whl
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)
Downloading Automat-0.6.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)
Downloading hyperlink-17.3.1-py2.py3-none-any.whl (73kB)
100% |████████████████████████████████| 81kB 1.4MB/s
Collecting pyasn1 (from service-identity->scrapy)
Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)
100% |████████████████████████████████| 71kB 2.8MB/s
Collecting pyasn1-modules (from service-identity->scrapy)
Downloading pyasn1_modules-0.1.5-py2.py3-none-any.whl (60kB)
100% |████████████████████████████████| 61kB 2.5MB/s
Collecting attrs (from service-identity->scrapy)
Downloading attrs-17.2.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->scrapy)
Installing collected packages: PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, scrapy
Running setup.py install for PyDispatcher ... done
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Successfully uninstalled six-1.4.1
Running setup.py install for Twisted ... done
Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 attrs-17.2.0 constantly-15.1.0 cssselect-1.0.1 hyperlink-17.3.1 incremental-17.5.0 parsel-1.2.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0 six-1.11.0 w3lib-1.18.0
|
成功安装lxml-4.1.0
5. 再次尝试安装scrapy，执行 sudo pip install scrapy
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
➜ ~ sudo pip install scrapy
The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting scrapy
Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248kB)
100% |████████████████████████████████| 256kB 1.5MB/s
Requirement already satisfied: lxml in /Library/Python/2.7/site-packages (from scrapy)
Collecting PyDispatcher>=2.0.5 (from scrapy)
Downloading PyDispatcher-2.0.5.tar.gz
Collecting Twisted>=13.1.0 (from scrapy)
Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
100% |████████████████████████████████| 3.0MB 371kB/s
Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)
Collecting queuelib (from scrapy)
Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from scrapy)
Downloading cssselect-1.0.1-py2.py3-none-any.whl
Collecting parsel>=1.1 (from scrapy)
Downloading parsel-1.2.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
Downloading service_identity-17.0.0-py2.py3-none-any.whl
Collecting six>=1.5.2 (from scrapy)
Downloading six-1.11.0-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from scrapy)
Downloading w3lib-1.18.0-py2.py3-none-any.whl
Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->scrapy)
Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)
Downloading constantly-15.1.0-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)
Downloading incremental-17.5.0-py2.py3-none-any.whl
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)
Downloading Automat-0.6.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)
Downloading hyperlink-17.3.1-py2.py3-none-any.whl (73kB)
100% |████████████████████████████████| 81kB 1.4MB/s
Collecting pyasn1 (from service-identity->scrapy)
Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)
100% |████████████████████████████████| 71kB 2.8MB/s
Collecting pyasn1-modules (from service-identity->scrapy)
Downloading pyasn1_modules-0.1.5-py2.py3-none-any.whl (60kB)
100% |████████████████████████████████| 61kB 2.5MB/s
Collecting attrs (from service-identity->scrapy)
Downloading attrs-17.2.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->scrapy)
Installing collected packages: PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, scrapy
Running setup.py install for PyDispatcher ... done
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Successfully uninstalled six-1.4.1
Running setup.py install for Twisted ... done
Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 attrs-17.2.0 constantly-15.1.0 cssselect-1.0.1 hyperlink-17.3.1 incremental-17.5.0 parsel-1.2.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0 six-1.11.0 w3lib-1.18.0
|
6. 执行 scrapy 出现下面错误
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
➜ ~ scrapy
Traceback (most recent call last):
File
"/usr/local/bin/scrapy"
, line
7
,
in
<module>
from
scrapy.cmdline
import
execute
File
"/Library/Python/2.7/site-packages/scrapy/cmdline.py"
, line
9
,
in
<module>
from
scrapy.crawler
import
CrawlerProcess
File
"/Library/Python/2.7/site-packages/scrapy/crawler.py"
, line
7
,
in
<module>
from
twisted.internet
import
reactor, defer
File
"/Library/Python/2.7/site-packages/twisted/internet/reactor.py"
, line
38
,
in
<module>
from
twisted.internet
import
default
File
"/Library/Python/2.7/site-packages/twisted/internet/default.py"
, line
56
,
in
<module>
install
=
_getInstallFunction(platform)
File
"/Library/Python/2.7/site-packages/twisted/internet/default.py"
, line
50
,
in
_getInstallFunction
from
twisted.internet.selectreactor
import
install
File
"/Library/Python/2.7/site-packages/twisted/internet/selectreactor.py"
, line
18
,
in
<module>
from
twisted.internet
import
posixbase
File
"/Library/Python/2.7/site-packages/twisted/internet/posixbase.py"
, line
18
,
in
<module>
from
twisted.internet
import
error, udp, tcp
File
"/Library/Python/2.7/site-packages/twisted/internet/tcp.py"
, line
28
,
in
<module>
from
twisted.internet._newtls
import
(
File
"/Library/Python/2.7/site-packages/twisted/internet/_newtls.py"
, line
21
,
in
<module>
from
twisted.protocols.tls
import
TLSMemoryBIOFactory, TLSMemoryBIOProtocol
File
"/Library/Python/2.7/site-packages/twisted/protocols/tls.py"
, line
63
,
in
<module>
from
twisted.internet._sslverify
import
_setAcceptableProtocols
File
"/Library/Python/2.7/site-packages/twisted/internet/_sslverify.py"
, line
38
,
in
<module>
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError:
'module'
object
has no attribute
'OP_NO_TLSv1_1'
|
需要更新 OpenSSL 库，执行 sudo pip install --upgrade pyopenssl
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
➜ ~ sudo pip install
-
-
upgrade pyopenssl
Password:
The directory
'/Users/wangruofeng/Library/Caches/pip/http'
or
its parent directory
is
not
owned by the current user
and
the cache has been disabled. Please check the permissions
and
owner of that directory. If executing pip with sudo, you may want sudo's
-
H flag.
The directory
'/Users/wangruofeng/Library/Caches/pip'
or
its parent directory
is
not
owned by the current user
and
caching wheels has been disabled. check the permissions
and
owner of that directory. If executing pip with sudo, you may want sudo's
-
H flag.
Collecting pyopenssl
Downloading pyOpenSSL
-
17.3
.
0
-
py2.py3
-
none
-
any
.whl (
51kB
)
100
%
|████████████████████████████████|
51kB
132kB
/
s
Requirement already up
-
to
-
date: six>
=
1.5
.
2
in
/
Library
/
Python
/
2.7
/
site
-
packages (
from
pyopenssl)
Collecting cryptography>
=
1.9
(
from
pyopenssl)
Downloading cryptography
-
2.1
.
1
-
cp27
-
cp27m
-
macosx_10_6_intel.whl (
1.5MB
)
100
%
|████████████████████████████████|
1.5MB
938kB
/
s
Collecting cffi>
=
1.7
; platform_python_implementation !
=
"PyPy"
(
from
cryptography>
=
1.9
-
>pyopenssl)
Downloading cffi
-
1.11
.
2
-
cp27
-
cp27m
-
macosx_10_6_intel.whl (
238kB
)
100
%
|████████████████████████████████|
245kB
2.2MB
/
s
Collecting enum34; python_version <
"3"
(
from
cryptography>
=
1.9
-
>pyopenssl)
Downloading enum34
-
1.1
.
6
-
py2
-
none
-
any
.whl
Collecting idna>
=
2.1
(
from
cryptography>
=
1.9
-
>pyopenssl)
Downloading idna
-
2.6
-
py2.py3
-
none
-
any
.whl (
56kB
)
100
%
|████████████████████████████████|
61kB
3.1MB
/
s
Collecting asn1crypto>
=
0.21
.
0
(
from
cryptography>
=
1.9
-
>pyopenssl)
Downloading asn1crypto
-
0.23
.
0
-
py2.py3
-
none
-
any
.whl (
99kB
)
100
%
|████████████████████████████████|
102kB
2.7MB
/
s
Collecting ipaddress; python_version <
"3"
(
from
cryptography>
=
1.9
-
>pyopenssl)
Downloading ipaddress
-
1.0
.
18
-
py2
-
none
-
any
.whl
Collecting pycparser (
from
cffi>
=
1.7
; platform_python_implementation !
=
"PyPy"
-
>cryptography>
=
1.9
-
>pyopenssl)
Downloading pycparser
-
2.18
.tar.gz (
245kB
)
100
%
|████████████████████████████████|
256kB
3.6MB
/
s
Installing collected packages: pycparser, cffi, enum34, idna, asn1crypto, ipaddress, cryptography, pyopenssl
Running setup.py install
for
pycparser ... done
Found existing installation: pyOpenSSL
0.13
.
1
DEPRECATION: Uninstalling a distutils installed project (pyopenssl) has been deprecated
and
will be removed
in
a future version. This
is
due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling pyOpenSSL
-
0.13
.
1
:
Successfully uninstalled pyOpenSSL
-
0.13
.
1
Successfully installed asn1crypto
-
0.23
.
0
cffi
-
1.11
.
2
cryptography
-
2.1
.
1
enum34
-
1.1
.
6
idna
-
2.6
ipaddress
-
1.0
.
18
pycparser
-
2.18
pyopenssl
-
17.3
.
0
|
更新 OpenSSL 成功，再次尝试执行 scrapy
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
➜ ~ scrapy
Scrapy
1.4
.
0
-
no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre
-
defined templates
runspider Run a
self
-
contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version
Print
Scrapy version
view
Open
URL
in
browser, as seen by Scrapy
[ more ] More commands available when run
from
project directory
Use
"scrapy <command> -h"
to see more info about a command
|
出现上面内容，表明安装成功。现在可以通过 scrapy 创建一个爬虫项目了
7. 进入到你项目的目录,执行 scrapy startproject firstscrapy创建 firstscrapy 爬虫项目
|
1
2
3
4
5
6
7
8
|
➜ PycharmProjects scrapy startproject firstscrapy
New Scrapy project
'firstscrapy'
, using template directory
'/Library/Python/2.7/site-packages/scrapy/templates/project'
, created
in
:
/
Users
/
wangruofeng
/
PycharmProjects
/
firstscrapy
You can start your first spider with:
cd firstscrapy
scrapy genspider example example.com
➜ PycharmProjects
|
出现上面内容表明项目创建成功，但是使用的是2.7版本的Python怎么切换到3.6版本呢？
8. 使用 PyCharm IDE 打开刚才的项目，执行 command + , 打开偏好设置菜单，在Project里面选择 Projiect interpreter 来切换你需要依赖的Python库的版本，配置结束。
