最近有个监控需求,须要远程执行集群每一个节点上的脚本,并获取脚本执行结果,为了安全起见不须要帐号密码登录节点主机,要求只须要调用远程脚本模块的方法就能实现。html
总结下python进行远程调用脚本方法:python
登录主机执行脚本,python模块支持如 pssh、pexpect、paramiko、ansiblenginx
以远程方法调用(不须要登录主机),python模块 rpyc,支持分布式shell
socket 方式,稍显复杂,须要熟悉网络协议,起点比较高编程
rpyc支持远程调用、分布式计算,以较少代码量实现复杂socket编程,本文主要介绍 rpyc 并用它来实现一个 demo。安全
以代码方式介绍:网络
需求:分别执行集群每一个节点上 server 端的脚本,并返回执行结果给 client 端多线程
Monitor_RPC_Client.py #!/usr/bin/env python # coding=utf-8 # 测试utf-8编码 # python exec_cmd.py "ls -lrt /opt/data1/logs/nginx/pc/track/`date +'%Y%m%d'`|awk '{s+=\$5}END{print s}'" # python exec_cmd.py "wc -l /opt/data1/logs/nginx/pc/track/`date +'%Y%m%d'`/*|awk '{s+=\$1}END{print s}'" import sys reload(sys) sys.setdefaultencoding('utf-8') import rpyc from pyUtil import * from multiprocessing.dummy import Pool as ThreadPool hostDict = { '192.168.1.216': 12345, '192.168.1.217': 12345, '192.168.1.218': 12345 } localResultDict = {} def rpc_client(host_port_cmd): host = host_port_cmd[0] port = host_port_cmd[1] cmd = host_port_cmd[2] c = rpyc.connect(host, port) result = c.root.exposed_execCmd(cmd) localResultDict[host] = result c.close() def exec_cmd(cmd_str): host_port_list = [] for (host, port) in hostDict.items(): host_port_list.append((host, port, cmd_str)) pool = ThreadPool(len(hostDict)) results = pool.map(rpc_client, host_port_list) pool.close() pool.join() for ip, result in sorted(localResultDict.iteritems(), key=lambda d: int(d[0].replace(".", ""))): print ip + ":\t" + result if __name__ == "__main__": if len(sys.argv) == 2 and sys.argv[1] != "-h": print "======================" print " Your command is:\t" + sys.argv[1] print "======================" cmd_str = sys.argv[1] else: print """ 该脚本能够在集群中批量执行任意命令并返回结果,但需注意如下几点: 一、命令请先单机测试经过,而后提交给脚本批量执行; 二、不要执行 rm 等危险 || 极其耗时 || 影响机器性能的命令; 三、命令请用双引号引发来,另外命令中有 $ 符号须要转义成 \$ 不然会被 Shell 当作变量解析掉,具体请参见下面的例子。 Usage && for example: python exec_cmd.py "ls -lrt /opt/data1/logs/nginx/pc/track/{}|awk '{{s+=\$5}}END{{print s}}'" python exec_cmd.py "wc -l /opt/data1/logs/nginx/pc/track/{}/*|awk '{{s+=\$1}}END{{print s}}'" """.format(yesterday, yesterday) sys.exit(1) exec_cmd(cmd_str) Monitor_RPC_Server.py #!/usr/bin/env python # coding=utf-8 # 测试utf-8编码 # cd /opt/script/rpcMonitorFlume # pkill -f flumeFileMonitor_RPC_Server.py # nohup python -u flumeFileMonitor_RPC_Server.py >> logs/flumeFileMonitor_RPC_Server.log 2>&1 & import sys reload(sys) sys.setdefaultencoding('utf-8') import os, commands, glob, re import datetime from rpyc import Service from rpyc.utils.server import ThreadedServer from pyUtil import getNowTime, get_ip_address class remote_call_func(Service): def on_connect(self): print "[{0}]\t--------------<<< on_connect".format(getNowTime()) def on_disconnect(self): print "[{0}]\t-------------->>> on_disconnect".format(getNowTime()) def exposed_execCmd(self, cmd): exitCode, execResult = commands.getstatusoutput(cmd) nowTime = (datetime.datetime.now()).strftime("%Y-%m-%d %H:%M:%S") print "[{0}] → {1} → {2}".format(nowTime, cmd, execResult) return execResult rpycServer = ThreadedServer(remote_call_func, hostname=get_ip_address('eth0'), port=11111, auto_register=False) rpycServer.start()
官方文档中相似例子不少,就不详细介绍了,需注意3点:并发
server端定义方法须要被client调用,必须定义以exposed 开头的方法,否则会报错AttributeError: ‘remote_call_script’ object has no attribute ‘exposed_iamshell’app
server端默认不设认证机制,若是须要认证有推荐两种方法: ThreadedServer的authenticator参数与SSL模块
pip install rpyc ,若是 import rpyc 报错则 yum install openssl-devel,而后从新编译、安装 python
固然还须要考虑不少异常处理,如超时、验证失败等。
Refer:
[1] python远程调用脚本(一)
http://rpyc.readthedocs.org/en/latest/tutorial.html
[2] python学习——python中执行shell命令
http://zhou123.blog.51cto.com/4355617/1312791
[3] celery实现任务统一收集、分发执行
http://blog.csdn.net/vintage_1/article/details/47664187
[4] Timeout function if it takes too long to finish [duplicate]
http://stackoverflow.com/questions/2281850/timeout-function-if-it-takes-too-long-to-finish
[5] 源码之Queue
http://www.cnblogs.com/liqxd/p/5104051.html
[6] python多线程编程(9) Queue模块
http://beginman.cn/python/2015/12/01/python-threading-queue/
[7] Python 并行任务技巧
http://my.oschina.net/leejun2005/blog/194270?fromerr=mNcoWQlp
[8] 利用 Python yield 建立协程将异步编程同步化
http://my.oschina.net/leejun2005/blog/501448?fromerr=ynpLsTXB
[9] Python 多线程教程:并发与并行
http://my.oschina.net/leejun2005/blog/398826
[10] 理解 Python 中的多线程
http://my.oschina.net/leejun2005/blog/179265
[11] paramiko小记