django server 500 error定位

环境信息

python使用的 2.7.6版本

[test@localhost supervisor]$ pip list
ansible (1.9.4)
APScheduler (3.0.3)
click (5.1)
Django (1.6)
ecdsa (0.13)
elasticsearch (1.3.0)
elasticsearch-curator (3.2.3)
futures (3.0.3)
gevent (1.0.2)
greenlet (0.4.9)
gunicorn (19.1.1)
ipython (2.3.1)
Jinja2 (2.8)
MarkupSafe (0.23)
meld3 (1.0.0)
MySQL-python (1.2.5)
paramiko (1.15.3)
pip (1.4.1)
pycrypto (2.6.1)
pyes (0.99.5)
pytz (2015.4)
PyYAML (3.11)
requests (2.6.0)
setuptools (5.8dev)
six (1.9.0)
supervisor (3.1.3)
tzlocal (1.2)
urllib3 (1.10)
wsgiref (0.1.2)

结构以下： Nginx-----Gunicorn----Django | | supervisor

gunicorn使用gevent模式

问题现象

用户一段时间没有登录，在登录系统后，页面显示server 500 error。用户刷新页面也没法解决。

解决过程

汇总信息进行思考

• 测试环境和研发环境没有此问题，只有生产环境有。
• 若是用户短期内登录成功过，则后继操做正常。

初步怀疑认证系统问题

经过汇总的信息，能够推论出 生产使用了登录验证的SDK，而且长时间无登录，就会毕现该问题

进行验证

对gunicorn进行抓包分析

抓包内容以下：

2016-03-01 15:45:25.248883 IP 127.0.0.1.36944 > 127.0.0.1.18500: P 1:474(473) ack 1 win 257
E...;G@.@............PHD..M0....P.......GET /590 HTTP/1.0
Host: pre.diag.ucweb.local
X-Real-IP: 100.84.46.79
X-Forwarded-For: 100.84.46.79
Connection: close
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 UBrowser/5.5.10106.5 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8


2016-03-01 15:45:25.248890 IP 127.0.0.1.18500 > 127.0.0.1.36944: . ack 474 win 265
E..(u.@.@..-........HD.P......O P..     ....
2016-03-01 15:45:25.257441 IP 127.0.0.1.18500 > 127.0.0.1.36944: P 1:174(173) ack 474 win 265
E...u.@.@...........HD.P......O P..     ....HTTP/1.0 500 INTERNAL SERVER ERROR
Server: gunicorn/19.1.1
Date: Tue, 01 Mar 2016 07:45:25 GMT
Connection: close
X-Frame-Options: SAMEORIGIN
Content-Type: text/html


2016-03-01 15:45:25.257455 IP 127.0.0.1.36944 > 127.0.0.1.18500: . ack 174 win 265
E..(;H@.@............PHD..O     ...wP.. ....
2016-03-01 15:45:25.257481 IP 127.0.0.1.18500 > 127.0.0.1.36944: P 174:201(27) ack 474 win 265
E..Cu.@.@...........HD.P...w..O P..     .7..<h1>Server Error (500)</h1>

gunicorn返回了 500的error，可是查看gunicorn没有发现任何异常。

对django进行抓包分析

这里再也不使用gunicorn，而是直接使用：python manage.py runserver 0.0.0.0:18500,启动服务

抓包内容以下：

2016-03-01 15:51:53.590777 IP 127.0.0.1.34323 > 127.0.0.1.18500: P 1:474(473) ack 1 win 257
E....-@.@.............HD.t.L..(.P.......GET /590 HTTP/1.0
Host: pre.diag.ucweb.local
X-Real-IP: 100.84.46.79
X-Forwarded-For: 100.84.46.79
Connection: close
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 UBrowser/5.5.10106.5 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8


2016-03-01 15:51:53.590783 IP 127.0.0.1.18500 > 127.0.0.1.34323: . ack 474 win 265
E..(K.@.@...........HD....(..t.%P..     2...
2016-03-01 15:51:53.715345 IP 127.0.0.1.18500 > 127.0.0.1.34323: P 1:37(36) ack 474 win 265
E..LK.@.@...........HD....(..t.%P..     .@..HTTP/1.0 500 INTERNAL SERVER ERROR

2016-03-01 15:51:53.715357 IP 127.0.0.1.34323 > 127.0.0.1.18500: . ack 37 win 257

这里能够看到django返回了500。锁定django，继续追查。 结合以前分析到登录验证系统的嫌疑。应该是登录验证SSD报错，但是这部分代码，没作任何修改，为什么这个版本就报错呢？

登录验证SDK添加异常捕获

分析了好久代码以为不多是登录验证SDK的问题，但是如何排除呢。 只能修改SDK代码：添加异常堆栈的输出 使用 traceback.format_exc() 打印堆栈

分析堆栈

在修改完SDK代码后，启动服务：python manage.py runserver 0.0.0.0:18500。 运行后获得堆栈以下：

Traceback (most recent call last):
  File "/home/project/app/project_pre/sso/sso_v2.py", line 47, in _deco
    return redirect('%s?%s=%s' % (SSO_LOGIN_URL, SSO_REDIRECT_FIELD_NAME, http_referer))
  File "/home/project/local/python/lib/python2.7/site-packages/django/shortcuts/__init__.py", line 78, in redirect
    return redirect_class(resolve_url(to, *args, **kwargs))
  File "/home/project/local/python/lib/python2.7/site-packages/django/shortcuts/__init__.py", line 151, in resolve_url
    return urlresolvers.reverse(to, args=args, kwargs=kwargs)
  File "/home/project/local/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 480, in reverse
    app_list = resolver.app_dict[ns]
  File "/home/project/local/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 310, in app_dict
    self._populate()
  File "/home/project/local/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 285, in _populate
    lookups.appendlist(pattern.callback, (bits, p_pattern, pattern.default_args))
  File "/home/project/local/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 229, in callback
    self._callback = get_callable(self._callback_str)
  File "/home/project/local/python/lib/python2.7/site-packages/django/utils/functional.py", line 32, in wrapper
    result = func(*args)
  File "/home/project/local/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 117, in get_callable
    (lookup_view, mod_name))
ViewDoesNotExist: Could not import porject.views.api.get_httpsf_api_status. View does not exist in module project.views.api.

看了工程的urls，的确是有配置，可是对应的model没有改方法。 原来django在执行redirect时，去检查urls。若是url对应的方法不存在，就报500异常。可是又不会打印相关信息，而且SDK也没有捕获