解决mysqldb查询大量数据致使内存使用太高的问题

1.源码

connection=MySQLdb.connect( host="thehost",user="theuser", passwd="thepassword",db="thedb") cursor=connection.cursor() cursor.execute(query) for row in cursor.fetchall(): print(row)

2.问题

普通的操做不论是fetchall()仍是fetchone()都是先将数据加载到本地再进行计算，大量的数据会致使内存资源消耗光。解决的方法是使用SSCurosr光标来处理。

3.优化后的代码  

import MySQLdb.cursors connection=MySQLdb.connect( host="thehost",user="theuser", passwd="thepassword",db="thedb", cursorclass = MySQLdb.cursors.SSCursor) cursor=connection.cursor() cursor.execute(query) for row in cursor: print(row)

參考文档：http://mysql-python.sourceforge.net/MySQLdb.html#

关键段落截取：

BaseCursor

The base class for Cursor objects. This does not raise Warnings.

CursorStoreResultMixIn

Causes the Cursor to use the  mysql_store_result() function to get the query result. The entire result set is stored on the client side.

CursorUseResultMixIn

Causes the cursor to use the  mysql_use_result() function to get the query result. The result set is stored on the server side and is transferred row by row using fetch operations.

CursorTupleRowsMixIn

Causes the cursor to return rows as a tuple of the column values.

CursorDictRowsMixIn

Causes the cursor to return rows as a dictionary, where the keys are column names and the values are column values. Note that if the column names are not unique, i.e., you are selecting from two tables that share column names, some of them will be rewritten as  table.column. This can be avoided by using the SQL  ASkeyword. (This is yet-another reason not to use  * in SQL queries, particularly where  JOIN is involved.)

Cursor

The default cursor class. This class is composed of  CursorWarningMixIn,  CursorStoreResultMixIn,  CursorTupleRowsMixIn, and  BaseCursor, i.e. it raises  Warning, uses mysql_store_result(), and returns rows as tuples.

DictCursor

Like  Cursor except it returns rows as dictionaries.

SSCursor

A "server-side" cursor. Like  Cursor but uses  CursorUseResultMixIn. Use only if you are dealing with potentially large result sets.

SSDictCursor

Like  SSCursor except it returns rows as dictionaries.