Linux升级wget/curl用于下载https文件的过程

时间 2019-11-11

标签 linux 升级 wget curl 用于下载 https 文件过程栏目 Linux 繁體版

原文原文链接

由于须要提高服务器的性能以及支持mysql更新版本的某些特性，所以决定升级mysql版本从5.1.30到8.0，目标肯定下来就开始干。html

`Mysql`安装方式选择

在Linux上安装应用，通常有三种方式，优劣对比分别为：
java

所以咱们选择二进制安装，安装简单、方便，支持多个Mysql版本同时存在。mysql

在Linux上安装二进制版本的应用，统一为三步：linux

// 经过配置自动生成文件
./configure
// 编译文件
make
// 检查自测单元，看编译是否经过，能够省略该步，不影响安装
make check
// 安装
make install

卸载经过二进制安装的程序：git

// 方式一: 在编译目录里执行卸载
make uninstall
// 方式二：找到安装目录，而后删除，如nettle程序
$find / -name nettle
/usr/include/nettle
rm -rf /usr/include/nettle

`wget&curl`不支持`https`下载

咱们能够在mysql官网下载最新版本的mysql8.0.13二进制文件，注意官网提供的下载连接是https协议的，当咱们在服务器执行下载命令：github

// 使用wget或者curl来下载文件
wget https://dev.mysql.com/downloads/file/?id=480751
curl -O https://dev.mysql.com/downloads/file/?id=480751

会报错：redis

// wget 加上--no-check-certificate 依然不能够
$wget https://dev.mysql.com/downloads/file/?id=480751
--2018-12-12 16:57:54--  https://dev.mysql.com/downloads/file/?id=480751
Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11
Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [40]: Handshake failed
Unable to establish SSL connection.

// curl 加上--insecure依然不能够
$curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根据网上查询到的答案，缘由均为版本太低，当前的版本不支持https协议的下载:算法

So the error actually happens with www.coursera.org and the reason is missing support for SNI. You need to upgrade your version of wget.

当前的版本：sql

$wget --version
GNU Wget 1.16.3 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Locale: 
    /usr/local/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib 
    -DHAVE_LIBGNUTLS -DNDEBUG 
Link: 
    gcc -DHAVE_LIBGNUTLS -DNDEBUG -lpcre -lnettle -lgnutls -lz -lidn 
    -lrt ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a 

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

------------------

$curl.7.19.7 --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

`wget`升级

既然是版本太低，升级便可，直接安装新版本的wget，而后卸载掉原有的wget文件便可。vim

下载完成1.20版本以后

// 得到文件wget-1.20.tar.gz
wget http://mirror.sergal.org/gnu/wget/wget-1.20.tar.gz
// 解压缩
tar -xzvf wget-1.20.tar.gz
// 进入解压后的文件夹
cd wget-1.20
// 开始配置
./configure
// 然而报错了：
...
checking for GNUTLS... no
configure: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

报错信息显示咱们没有安装gnutls依赖，须要继续进行安装，更为详细的报错信息，能够查看config.log:

// 查看详细报错信息
vim config.log

...
PKG_CONFIG='/usr/local/bin/pkg-config'
...
configure:44443: checking for GNUTLS
configure:44450: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44453: $? = 1
configure:44467: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44470: $? = 1
configure:44484: result: no
No package 'gnutls' found
configure:44500: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you                                                                            
installed software in a non-standard prefix.                                                                                                  
                                                                                                                                              
Alternatively, you may set the environment variables GNUTLS_CFLAGS                                                                            
and GNUTLS_LIBS to avoid the need to call pkg-config.                                                                                         
See the pkg-config man page for more details.

这里能够看出配置脚本实际是执行了：

$ /usr/local/bin/pkg-config --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found

安装`gnutls`库

gnutls全称 GNU Transport Layer Security Library，即基于GNU版权协议的传输层安全协议，是wget支持https中的ssl协议的基础库。

咱们能够在官方提供的镜像库里快速下载并安装：

// 下载gnutls二进制文件
wget http://www.ring.gr.jp/pub/net/gnupg/gnutls/v3.6/gnutls-3.6.4.tar.xz
// 解压xz文件
xz -d gnutls-3.6.4.tar.xz
tar -xvf gnutls-3.6.4.tar
cd gnutls-3.6.4
./configure

// 报错：
...
checking for NETTLE... no
configure: error: 
  ***
  *** Libnettle 3.4 was not found.

// 若是以为可能不安全，能够下载md5签名文件验证文件，可是这个文件验证后发现签名过时了，因此没办法验证了
wget http://www.ring.gr.jp/pub/net...
gpg --verify gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar
gpg --recv-key F1679A65
gpg --verify --verbose gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar

查看详细报错信息：

$ vim config.log

...
configure:10032: checking for NETTLE
configure:10039: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10042: $? = 1
configure:10056: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10059: $? = 1
configure:10073: result: no
No package 'nettle' found
configure:10090: error:
  ***
  *** Libnettle 3.4 was not found.

结果显示咱们须要3.4版本以上的Libnettle库，继续安装。

安装`Libnettle`库

Nettle库是用于跨平台的底层密码库，包含加密和解密的不一样算法。咱们下载并安装nettle库：

wget ftp://ftp.gnu.org/gnu/nettle/nettle-3.4.1.tar.gz
tar -xzvf nettle-3.4.1.tar.gz
cd nettle-3.4.1
./configure
// 安装成功
...
configure: summary of build options:

  Version:           nettle 3.4.1
  Host type:         x86_64-unknown-linux-gnu
  ABI:               64
  Assembly files:    x86_64
  Install prefix:    /usr/local
  Library directory: ${exec_prefix}/lib64
  Compiler:          gcc
  Static libraries:  yes
  Shared libraries:  yes
  Public key crypto: no
  Using mini-gmp:    no
  Documentation:     yes

make
make install

根据官方文档，咱们安装完成后应该会有两个文件lib{hogweed,nettle}.so，然而咱们只能发现其中一个：

make install &&
chmod -v 755 /usr/lib/lib{hogweed,nettle}.so &&
install -v -m755 -d /usr/share/doc/nettle-3.4.1 &&
install -v -m644 nettle.html /usr/share/doc/nettle-3.4.1

$ ll | grep '\.so'
-rwxr-xr-x 1 root root 3675341 Dec 12 19:15 libnettle.so
$ ll | grep weed
-rw-rw-r-- 1 work work     529 Dec 10 15:30 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-rw-r-- 1 work work     298 Dec 10 15:30 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in

少了一个libhogweed.so文件，稍后咱们编译gnutls时会发现这个致使的问题。

继续编译`gnutls`

既然nettle安装完成了，咱们能够继续安装gnutls：

./configure

...
configure: error: 
  ***
  *** Libnettle 3.4 was not found.

依然报错缺失库，但咱们明明已经安装了，为何找不到呢？咱们用包管理工具查找一下：

$ pkg-config --modversion nettle
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found

咱们找下这个nettle.pc刚才安装到哪里去了：

$ locate nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc.in
/usr/lib64/pkgconfig/nettle.pc
/usr/local/lib64/pkgconfig/nettle.pc

而咱们pkg-config默认的管理包检索路径为/usr/lib/pkgconfig，因此没法正常找到，参考pkgconfig文档，有两种方案：

// 方案一：连接该文件到默认目录中
ln -s /usr/local/lib64/pkgconfig/nettle.pc /usr/lib/pkgconfig/nettle.pc  
// 方案二：全局变量中更改包的检索路径（只在本次终端窗口生效，退出后恢复，因此只能临时使用一下）
$ echo $PKG_CONFIG_PATH

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64/pkgconfig/
$ echo $PKG_CONFIG_PATH
:/usr/local/lib64/pkgconfig/

// 任一方案执行后结果
$ pkg-config --modversion nettle
3.4.1

此时，咱们继续安装：

$ ./configure
...
checking for NETTLE... yes
checking for HOGWEED... no
configure: error: 
  ***
  *** Libhogweed (nettle's companion library) was not found. Note that you must compile nettle with gmp support.

能够看到，咱们的nettle库已经安装成功，可是hogweed却检查没有找到，提示中也写明了libhogweed须要字gmp库已经安装完成的状况下从新编译nettle才能够被安装。

有人提出过相关的问题，咱们也能够从官网文档上更详细的知道这个Nettle对于libhogweed的依赖：

5 Linking
Nettle actually consists of two libraries, libnettle and libhogweed. The libhogweed library contains those functions of Nettle that uses bignum operations, and depends on the GMP library. With this division, linking works the same for both static and dynamic libraries.
If an application uses only the symmetric crypto algorithms of Nettle (i.e., block ciphers, hash functions, and the like), it’s sufficient to link with -lnettle. If an application also uses public-key algorithms, the recommended linker flags are -lhogweed -lnettle -lgmp. If the involved libraries are installed as dynamic libraries, it may be sufficient to link with just -lhogweed, and the loader will resolve the dependencies automatically.

总而言之，就是没有libhogweed.so这个文件不行，而它只能由nettle进行安装。根据nettle库官方资料显示，libhogweed.so应该在nettle安装时被自动生成，然而咱们在上面的安装过程当中并无生成。那是否是由于我没有安装gmp致使的呢？

安装`gmp`库

咱们下载gmp库并安装，能够在编译Nettle的config.log中查看有一条warning，指明了版本需求：

$ vim config.log

...
configure:6583: result: no
configure:6594: WARNING: GNU MP not found, or too old. GMP-6.0 or later is needed, see https://gmplib.org/.
    Support for public key algorithms will be unavailable.

因此咱们须要下载6.0版本后的：

// 这里我只找到了官网的https版本，没办法，只好本地下载，而后rz到服务器，由于是二进制文件，要带上-be参数
rz -be
// 而后正常编译
$ ./configure & make & make install
...
Libraries have been installed in:
   /usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

这里提醒咱们须要将动态库连接到缓存中，咱们采用第四种方案，能够参考ldconfig命令：

$ vim /etc/ld.so.conf

// 添上安装的.so文件路径
/usr/local/lib
:wq

$ ldconfig
$ ldconfig -v | grep gmp
        libgmp.so.10 -> libgmp.so.10.3.2
        libgmpxx.so.4 -> libgmpxx.so.4.1.0
        libgmp.so.3 -> libgmp.so.3.5.0

看到libgmp.so.10就是咱们安装的最新版本，如今OK了。

而后从新编译安装nettle，会生成libhogweed.so文件：

$ ll | grep weed
-rw-r--r-- 1 root root     541 Dec 12 22:12 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-r--r-- 1 root root 6154192 Dec 12 22:13 libhogweed.a
-rw-r--r-- 1 root root     298 Dec 12 22:12 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in
-rwxr-xr-x 1 root root 5519996 Dec 12 22:13 libhogweed.so
-rw-r--r-- 1 root root       8 Dec 12 22:13 libhogweed.stamp

请注意若是安装完成后，若是出现多个版本的gmp库，请删除老版本的。具体删除哪一项请自行斟酌，我删除了全部的，而后在编译的过程当中，会报错：can't find libgmp.so.3，说明libgmp.so.3这个是基础库，请不要动！
等我删除了老版本的，从新编译nettle就OK。若是你安装成功了新版本，依然编译不成功，请参考这个。

依赖地狱

用二进制来安装的时候，老是会出现各类各样的问题，缺乏各类依赖的包，解决方法就是缺什么就去安什么，可是会很是恐怖。为了解决nettle安装的问题，除了上面的gmp，我还安装了最新版本的各类库：

libunistring： https://www.gnu.org/software/...
p11-kit：https://github.com/p11-glue/p...
libffi：https://sourceware.org/libffi/
pkg-config: https://pkg-config.freedeskto...
libtasn1: https://ftp.gnu.org/gnu/libta...

同时，因为gnutls编译不经过的问题，又升级了pkg-config，它依赖于Libtasn1。

继续安装`gnutls`库（失败、暂时放弃更新）

./configure
// 此时没有错误信息了，可是还有不少WARNING信息
*** autogen not found. Will not link against libopts.
*** You will not be able to create source packages with 'make dist'
  because gtk-doc >= 1.14 is not found.
*** LIBIDN2 was not found. You will not be able to use IDN2008 support
*** libunbound was not found. Libdane will not be built.
*** trousers was not found. TPM support will be disabled.
*** `guile-snarf' from Guile not found.  Guile bindings not built.

*** The DNSSEC root key file in /etc/unbound/root.key was not found.
*** This file is needed for the verification of DNSSEC responses.
*** Use the command: unbound-anchor -a "/etc/unbound/root.key"
*** to generate or update it.

// 继续编译，又报错了
make
...
tlsproxy/buffer.c:40: error: redefinition of typedef 'buffer_t'
tlsproxy/buffer.h:31: note: previous declaration of 'buffer_t' was here

暂时放弃更新wget，过几天继续尝试，解决各类问题太费时间了

若是想要减小warning信息，能够更新autogen等库：

autogen: https://ftp.gnu.org/gnu/autog...
guile: http://alpha.gnu.org/gnu/guile/
gcc: http://ftp.tsukuba.wide.ad.jp...
mpc: https://ftp.gnu.org/gnu/mpc/

安装autogen过程当中须要依赖guile，然而安装guile时又报错：guile configure: error: Cannot find a type to use in place of socklen_t
放弃更新autogen。

尝试`curl`更新，层层依赖，放弃

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根据报错缘由和网上资料是因为版本过老，须要更新curl版本。

从官方地址下载curl后安装，再次用新版本的curl请求：

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

仍是报错，根据上面的资料，若是依然不能解决问题，须要更新NSS，NSS和OpenSSl相似，都属于底层密码学，由Mozilla维护，MDN文档提供安装说明，它跟前面的二进制文件略有不一样，不提供configure自动配置，详细的查看它的文档。

安装NSS库比较麻烦，还须要再安装GYP库，想起来又是层层嵌套的依赖关系，放弃更新。

解决方案：本地下载，`rz`上传

在耗费两天的时间后，我及时的终止了无畏的尝试，转而使用本地下载mysql8.0文件，而后rz -be上传到服务器，搞定。

总结

Linux上层层依赖的二进制文件安装简直是地狱版的体验，在给咱们带来高自由度的同时也有无尽的烦恼，然而yum安装版本又太低，不能知足需求。虽然最终仍是没有成功更新wget或者curl，可是在整个过程当中，也学习到了不少的新东西，在这篇文章总结一下过程，但愿也能帮助一些人在某一步骤遇到的问题。

解决问题总体思路和过程

参考资料

mysql8.0官网下载地址：https://dev.mysql.com/downloa...
Mysql三种安装方式详解：https://www.jianshu.com/p/a04...
[StackOverFlow] wget ssl alert handshake failure：https://stackoverflow.com/que...
卸载二进制程序：http://www.blogjava.net/zhyiw...
wget下载地址：http://mirror.sergal.org/gnu/...
gnutls下载地址：http://www.ring.gr.jp/pub/net...
利用.sig文件验证数据的完整性：https://blog.csdn.net/xiazhiy...
下载安装nettle http://www.linuxfromscratch.o...
nettle官方文档：http://www.lysator.liu.se/~ni...
gmp下载地址：https://gmplib.org/
ldconfig命令：http://man.linuxde.net/ldconfig
SecureCRT rz 上传文件失败问题：https://blog.csdn.net/heavend...
初识NSS，一文了解全貌：https://cloud.tencent.com/dev...
MDN文档 NSS：https://developer.mozilla.org...
curl: (35) SSL connect error：https://stackoverflow.com/que...
简述configure、pkg-config、pkg_config_path三者的关系：http://www.mike.org.cn/articl...
How to compile GnuTLS: https://stackoverflow.com/que...

Linux升级wget/curl用于下载https文件的过程

Mysql安装方式选择

wget&curl不支持https下载

wget升级

安装gnutls库

安装Libnettle库

继续编译gnutls

安装gmp库