前言
用python 读取一个大于10g 的文件，自己电脑只有8g内存，一运行就报内存溢出：memoryerror
python 如何用open函数读取大文件呢？
读取大文件
首先可以自己先制作一个大于10g的txt文件
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
a = '''2021-02-02 21:33:31,678 [django.request:93] [base:get_response] [warning]- not found: /http:/123.125.114.144/2021-02-02 21:33:31,679 [django.server:124] [basehttp:log_message] [warning]- "head http://123.125.114.144/ http/1.1" 404 16782021-02-02 22:14:04,121 [django.server:124] [basehttp:log_message] [info]- code 400, message bad request version ('http')2021-02-02 22:14:04,122 [django.server:124] [basehttp:log_message] [warning]- "get ../../mnt/custom/productdefinition http" 400 -2021-02-02 22:16:21,052 [django.server:124] [basehttp:log_message] [info]- "get /api/login http/1.1" 301 02021-02-02 22:16:21,123 [django.server:124] [basehttp:log_message] [info]- "get /api/login/ http/1.1" 200 38762021-02-02 22:16:21,192 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/img/main_bg.png http/1.1" 200 28012021-02-02 22:16:21,196 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/style.css http/1.1" 200 16382021-02-02 22:16:21,229 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/img/bg.jpg http/1.1" 200 1359902021-02-02 22:16:21,307 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy http/1.1" 200 69002021-02-02 22:16:23,525 [django.server:124] [basehttp:log_message] [info]- "post /api/login/ http/1.1" 302 02021-02-02 22:16:23,618 [django.server:124] [basehttp:log_message] [info]- "get /api/index/ http/1.1" 200 184472021-02-02 22:16:23,709 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/commons.js http/1.1" 200 132092021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/admin.css http/1.1" 200 196602021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/common.css http/1.1" 200 10042021-02-02 22:16:23,714 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/app.js http/1.1" 200 208442021-02-02 22:16:26,509 [django.server:124] [basehttp:log_message] [info]- "get /api/report_list/1/ http/1.1" 200 146492021-02-02 22:16:51,496 [django.server:124] [basehttp:log_message] [info]- "get /api/test_list/1/ http/1.1" 200 248742021-02-02 22:16:51,721 [django.server:124] [basehttp:log_message] [info]- "post /api/add_case/ http/1.1" 200 02021-02-02 22:16:59,707 [django.server:124] [basehttp:log_message] [info]- "get /api/test_list/1/ http/1.1" 200 248742021-02-03 22:16:59,909 [django.server:124] [basehttp:log_message] [info]- "post /api/add_case/ http/1.1" 200 02021-02-03 22:17:01,306 [django.server:124] [basehttp:log_message] [info]- "get /api/edit_case/1/ http/1.1" 200 365042021-02-03 22:17:06,265 [django.server:124] [basehttp:log_message] [info]- "get /api/add_project/ http/1.1" 200 177372021-02-03 22:17:07,825 [django.server:124] [basehttp:log_message] [info]- "get /api/project_list/1/ http/1.1" 200 297892021-02-03 22:17:13,116 [django.server:124] [basehttp:log_message] [info]- "get /api/add_config/ http/1.1" 200 248162021-02-03 22:17:19,671 [django.server:124] [basehttp:log_message] [info]- "get /api/config_list/1/ http/1.1" 200 19532'''while true: with open("xxx.log", "a", encoding="utf-8") as fp: fp.write(a) |
循环写入到 xxx.log 文件，运行 3-5 分钟，pycharm 打开查看文件大小大于 10g
于是我用open函数 直接读取
|
1
2
3
|
f = open("xxx.log", 'r')print(f.read())f.close() |
抛出内存溢出异常:memoryerror
traceback (most recent call last):
file "d:/2021kecheng06/demo/txt.py", line 35, in <module>
print(f.read())
memoryerror
运行的时候可以看下自己电脑的内存已经占了100%， cpu高达91% ，不挂掉才怪了！
这种错误的原因在于，read()方法执行操作是一次性的都读入内存中，显然文件大于内存就会报错。
read() 的几种方法
1.read() 方法可以带参数 n， n 是每次读取的大小长度，也就是可以每次读一部分，这样就不会导致内存溢出
|
1
2
3
|
f = open("xxx.log", 'r')print(f.read(2048))f.close() |
运行结果
2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [warning]- not found: /http:/123.125.114.144/
2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [warning]- "head http://123.125.114.144/ http/1.1" 404 1678
2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [info]- code 400, message bad request version ('http')
2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [warning]- "get ../../mnt/custom/productdefinition http" 400 -
2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [info]- "get /api/login http/1.1" 301 0
2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [info]- "get /api/login/ http/1.1" 200 3876
2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/img/main_bg.png http/1.1" 200 2801
2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/style.css http/1.1" 200 1638
2019-10-24 22:16:21,229 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/img/bg.jpg http/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy http/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [info]- "post /api/login/ http/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [info]- "get /api/index/ http/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/commons.js http/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/admin.css http/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/common.css http/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/app.js http/1.1" 200 20844
2019-10-24 22:16:26,509 [django.server:124] [basehttp:log_message] [i
这样就只读取了2048个字符，全部读取的话，循环读就行
|
1
2
3
4
5
6
7
|
f = open("xxx.log", 'r')while true: block = f.read(2048) print(block) if not block: breakf.close() |
2.readline()：每次读取一行，这个方法也不会报错
|
1
2
3
4
5
6
7
8
|
f = open("xxx.log", 'r')while true: line = f.readline() print(line, end="") if not line: breakf.close() |
3.readlines()：读取全部的行，生成一个list，通过list来对文件进行处理，显然这种方式依然会造成：memoyerror
真正 pythonic 的方法
真正 pythonci 的方法，使用 with 结构打开文件，fp 是一个可迭代对象，可以用 for 遍历读取每行的文件内容
|
1
2
3
|
with open("xxx.log", 'r') as fp: for line in fp: print(line, end="") |
yield 生成器读取大文件
前面一篇讲yield 生成器的时候提到读取大文件，函数返回一个可迭代对象，用next()方法读取文件内容
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
def read_file(fpath): block_size = 1024 with open(fpath, 'rb') as f: while true: block = f.read(block_size) if block: yield block else: returnif __name__ == '__main__': a = read_file("xxx.log") print(a) # generator objec print(next(a)) # bytes类型 print(next(a).decode("utf-8")) # str |
运行结果
<generator object read_file at 0x00000226b3005258>
b'\r\n2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [warning]- not found: /http:/123.125.114.144/\r\n2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [warning]- "head http://123.125.114.144/ http/1.1" 404 1678\r\n2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [info]- code 400, message bad request version (\'http\')\r\n2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [warning]- "get ../../mnt/custom/productdefinition http" 400 -\r\n2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [info]- "get /api/login http/1.1" 301 0\r\n2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [info]- "get /api/login/ http/1.1" 200 3876\r\n2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/img/main_bg.png http/1.1" 200 2801\r\n2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/style.css http/1.1" 200 1638\r\n2019-10-24 22:16:21,229 [django.server:124] '
[basehttp:log_message] [info]- "get /static/assets/img/bg.jpg http/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy http/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [info]- "post /api/login/ http/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [info]- "get /api/index/ http/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/commons.js http/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/admin.css http/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/css/common.css http/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [info]- "get /static/assets/js/app.js http/1.1" 200 20844
2019-10-24 22:16:26,509 [django.server:124] [basehtt
到此这篇关于python读取一个大于10g的txt文件的方法的文章就介绍到这了,更多相关python读取大于10g文件内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://www.cnblogs.com/yoyoketang/p/14477562.html
