工做中用到的linux、SQL、Python语句

时间 2019-12-08

标签用到 linux sql python 语句栏目 Linux 繁體版

原文原文链接

SQL:html

分组取最新值后求和python

select sum(fff.asset_balance) as "总资产",sum(fff.enable_balance) as "可用资金",sum(fff.fund_balance) as "现金资产" from (
select tt.fund_account,tt.fund_balance,tt.asset_balance,tt.enable_balance,date_time from fund_real_log as tt group by fund_account,fund_balance,asset_balance,date_time having
date_time=(select max(date_time) from fund_real_log where fund_account=tt.fund_account and fund_account in ('30001691','30001708'))
) fff;
复制代码

Python解决unicode编码
linux

s = '\xe9\x9d\x92\xe8\x9b\x99\xe7\x8e\x8b\xe5\xad\x90'
ss = s.encode('raw_unicode_escape')
print(ss)  # 结果：b'\xe9\x9d\x92\xe8\x9b\x99\xe7\x8e\x8b\xe5\xad\x90'
sss = ss.decode()
print(sss)复制代码

转码：shell

>>> t1 = '\xe6\x9f\xa5\xe8\xaf\xa2\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xe5\x87\xba\xe7\x8e\xb0\xe5\xbc\x82\xe5\xb8\xb8' >>> print t1.decode('utf-8') 查询数据库出现异常复制代码

python比对Excel（新版本openpyxl）数据库

#-*- coding:utf-8 -*-
from openpyxl import *


def compareFile(ename1, ename2):
    # A bool to verify if 2 xlsx is the same.
    fileSame = True
    # Load workbook, and get sheetname
    src_book = load_workbook(filename=ename1)
    dlo_book = load_workbook(filename=ename2)
    src_sheets = src_book.sheetnames
    dlo_sheets = dlo_book.sheetnames

    if (src_sheets != dlo_sheets):
        print "Two file has different sheets."
        print ename1, "has sheet names:", src_sheets
        print ename2, "has sheet names:", dlo_sheets
    else:
        print "Two file has the same sheets."
        sn = src_sheets
        for wsn in sn:
            # Get worksheet
            src_sheet_object = src_book[wsn]
            dlo_sheet_object = dlo_book[wsn]
            c = src_sheet_object.max_column
            r = src_sheet_object.max_row
            # This can be replace by src_sheet_object.get_dimension
            if ((dlo_sheet_object.max_column != c) or (dlo_sheet_object.max_row != r)):
                print "DIFFERDENT at SHEET-", wsn, ": Rows or columns not the same!"
                fileSame = False
            else:
                print "SHEET-", wsn, ":Rows or columns are same!"
                # Compare every cell.
                flag = True
                for i in range(1, r + 1):
                    for j in range(1, c + 1):
                        c1 = src_sheet_object.cell(i, j)
                        c2 = dlo_sheet_object.cell( i, j)
                        if (c1):
                            if (c2):
                                if (c1.value != c2.value):
                                    if ((wsn == "Internal Info") and ((i == 4) and (j == 2)) or (
                                        (i == 5) and (j == 3))):
                                        continue
                                    print "DIFFERDENT_VALUE at SHEET-", wsn, ": At (", i, ",", j, ")",
                                    print "diff FROM", c1.value, "TO", c2.value
                                    flag = False
                            else:
                                print "DIFFERDENT_TO_NONE at SHEET-", wsn, ": At (", i, ",", j, ")"
                                print "diff FROM", c1.value
                                flag = False
                        else:
                            if (c2):
                                print "DIFFERDENT_TO_NONE at SHEET-", wsn, ": At (", i, ",", j, ")"
                                print "diff FROM", c2.value
                                flag = False
                fileSame = fileSame and flag
        if fileSame:
            print "SAME_FILE:", ename1, ename2


print "------------------文件比对开始------------------"
srcfile = u"文件绝对路径"
dlofile = u"文件绝对路径"compareFile(srcfile,dlofile)
print "------------------文件比对结束------------------"


复制代码

出现echo展现命令执行结果未换行时加双引号便可:bash

echo "`cmd`"复制代码

开启FTP服务传文件:服务器

service vsftpd status
sudo service vsftpd start
复制代码

抓包:tcp

sudo tcpdump -i any host IP -s0 -w tmp.cap
strings tmp.cap > tmp.txt
复制代码

awk提取数据:性能

ps -fux | grep python | awk '{for(i=1;i<=10;i++)printf $i ";;";for(a=11;a<=NF;a++)printf $a " ";printf "\n"}'
复制代码

memUse=`awk -v x="$memUseRate" -v y=100 'BEGIN{printf "%.2f\n",x*y}'`
awk -F '|' '{print $3}' 1120capstats.csv| awk '{sum+=$1} END {print "Average = ", sum/(NR-1)}'
复制代码

一行命令（循环）:测试

for i in `cat 1214.txt` ; do if [[ i-tmp=1 ]]; then   tmp=i;   echo "right"; else   echo "wrong";   exit 2; fi; done
复制代码

查看进程的具体状态, 包括进程ID，进程启动的路径

jps -lv复制代码

杀进程

for pid in `ps fux | grep excu.py | grep -v grep | awk '{print $2}'`;do `kill -9 ${pid}`; done复制代码

检查返回结果是否连续:

#!/bin/bash
#getResult.sh
resultFile=$1
readonly resultFile
jobid=$2
grepCondition="\"40014\":\"${jobid}\""
result=`grep -ai ${grepCondition} ${resultFile} | awk -F ',' '{print $10}'`
#echo "$result"
if [ ! "$result" ]
then
  echo "No result!!!!!!!!!!Please check your jobid!!!!!!!!!!!"
  exit 2
fi
initNum=$3
tmp=($(echo "$result"))
for item in ${tmp[@]};do
  #echo $item
  len=${#item}
  part2=${item:9:-1}
  echo $part2
  differ=`expr ${part2}-${initNum}`
  if [[ $differ -eq 1 ]] 
  then
    echo "The result is right!!!"
    ((initNum++));
  else
    echo "Wrong result!!!!!!!!!!"
  exit 2
  fi
done
复制代码

执行语句:

./getResult.sh msg.log 745 0
复制代码

linux浮点数据计算问题

性能测试时统计服务器指标一（比较粗）

#!/bin/bash
#script to capture system statistics
OUTFILE=/home/test/tmp.csv
`:>$OUTFILE`
echo " 时间 用户链接数 系统平均负载 空闲内存 空闲CPU百分比" >> $OUTFILE

int=1

while(($int<=10000));do
    TIME=`date "+%Y-%m-%d %H:%M:%S"`
    TIMEOUT=`uptime`
    VMOUT=`vmstat 1 1`
    USERS=`echo $TIMEOUT | gawk -F ',' '{print $3}' | sed 's/^[ \t]*//g'`
    #LOAD=`echo $TIMEOUT | gawk '{for(i=9;i<=NF;i++)printf $i}'`
    LOAD=`echo $TIMEOUT | gawk '{print $9,$10}' | sed "s/,//"`
    FREE=`echo $VMOUT | gawk '{for(i=24;i<=NF;i++)printf $i " "}' | gawk '{print $4}'`
    #FREE=`echo $VMOUT | sed -n '/[0-9]/p' | sed -n '1p' | gawk '{print $4} '`
    IDLE=`echo $VMOUT | gawk '{for(i=24;i<=NF;i++)printf $i " "}' | gawk '{print $15}'`
    #IDLE=`echo $VMOUT | sed -n '/[0-9]/p' | sed -n '2p' |gawk '{print $15}'`
    echo " $TIME | $USERS | $LOAD | $FREE | $IDLE" >> $OUTFILE
    ((int++))
    sleep 1
done
复制代码

性能测试时统计服务器指标二（比较粗）

#!/bin/bash 
OUTFILE=/home/test/newTmp.csv

`:>newTmp.csv`

echo " 时间 系统平均负载 CPU使用率 内存使用率 " > $OUTFILE

int=1

while(($int<=100000));do
    TIME=`date "+%Y-%m-%d %H:%M:%S"`
    cmd="`top -b -n 1 | head -n 5|sed s/[[:space:]]//g`"
    loadaverage=`echo "$cmd" | sed -n "1p" | gawk -F ',' '{print $4,$5,$6}'`

    us=`echo "$cmd" | sed -n "3p" | gawk -F ',' '{print $1,$2,$4}' | gawk '{print $1}' | gawk -F ':' '{print $2}' | sed "s/us//"`
    #echo $us
    sy=`echo "$cmd" | sed -n "3p" | gawk -F ',' '{print $1,$2,$4}' | gawk '{print $2}' | sed "s/sy//"`
    #echo $sy
    cpuUs=`echo "$us+$sy"|bc`
    #echo $cpuUs

    memused=`echo "$cmd" | sed -n "4p"| gawk -F ':' '{print $2}' | gawk -F ',' '{print $3}'| sed "s/used//"`
    memtotal=`echo "$cmd" | sed -n "4p"| gawk -F ':' '{print $2}' | gawk -F ',' '{print $1}' | sed "s/total//"`
    memUseRate=`echo | awk -v a=$memused -v b=$memtotal '{printf("%.4f",a/b)}'`
    memUse=`echo "$memUseRate*100"|bc`   #linux shell浮点计算

    echo "$TIME | $loadaverage | $cpuUs | $memUse " >> $OUTFILE
    ((int++))
    sleep 1
done
复制代码

性能测试时统计服务器指标计算：

#!/bin/bash 
#index=$1

file=$1

#if [ ! $3 ];then
# $3=' '
#
#rmcode=$3

echo "12123213123213"

initlist=`cat $file | gawk -F '|' '{print $3}' | gawk -F ':' '{print $2}' | sed "s/ //"`

echo "34534534534543"

tmp=($(echo "$initlist"))
sum=0.0
nums=0
for item in ${tmp[@]};do
  sum=`echo "$sum+$item"|bc`
  ((nums+=1))
done

echo $sum

averageLoad=`echo | awk -v a=$sum -v b=$nums '{printf("%.2f",a/b)}'`

echo $averageLoad
复制代码