python讀取一個(gè)大于10G的txt文件的方法

第python讀取一個(gè)大于10G的txt文件的方法用python讀取一個(gè)大于10G的文件，自己電腦只有8G內(nèi)存，一運(yùn)行就報(bào)內(nèi)存溢出：MemoryError

python如何用open函數(shù)讀取大文件呢？

讀取大文件

首先可以自己先制作一個(gè)大于10G的txt文件

a='''

2025-02-0221:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/

2025-02-0221:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678

2025-02-0222:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion('HTTP')

2025-02-0222:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-

2025-02-0222:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010

2025-02-0222:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876

2025-02-0222:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801

2025-02-0222:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638

2025-02-0222:16:21,229[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-02-0222:16:21,307[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/fonts/icomoon.ttfu4m6fyHTTP/1.1"2006900

2025-02-0222:16:23,525[django.server:124][basehttp:log_message][INFO]-"POST/api/login/HTTP/1.1"3020

2025-02-0222:16:23,618[django.server:124][basehttp:log_message][INFO]-"GET/api/index/HTTP/1.1"20018447

2025-02-0222:16:23,709[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/commons.jsHTTP/1.1"20013209

2025-02-0222:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/admin.cssHTTP/1.1"20019660

2025-02-0222:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/common.cssHTTP/1.1"2001004

2025-02-0222:16:23,714[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/app.jsHTTP/1.1"20020844

2025-02-0222:16:26,509[django.server:124][basehttp:log_message][INFO]-"GET/api/report_list/1/HTTP/1.1"20014649

2025-02-0222:16:51,496[django.server:124][basehttp:log_message][INFO]-"GET/api/test_list/1/HTTP/1.1"20024874

2025-02-0222:16:51,721[django.server:124][basehttp:log_message][INFO]-"POST/api/add_case/HTTP/1.1"2000

2025-02-0222:16:59,707[django.server:124][basehttp:log_message][INFO]-"GET/api/test_list/1/HTTP/1.1"20024874

2025-02-0322:16:59,909[django.server:124][basehttp:log_message][INFO]-"POST/api/add_case/HTTP/1.1"2000

2025-02-0322:17:01,306[django.server:124][basehttp:log_message][INFO]-"GET/api/edit_case/1/HTTP/1.1"20036504

2025-02-0322:17:06,265[django.server:124][basehttp:log_message][INFO]-"GET/api/add_project/HTTP/1.1"20017737

2025-02-0322:17:07,825[django.server:124][basehttp:log_message][INFO]-"GET/api/project_list/1/HTTP/1.1"20029789

2025-02-0322:17:13,116[django.server:124][basehttp:log_message][INFO]-"GET/api/add_config/HTTP/1.1"20024816

2025-02-0322:17:19,671[django.server:124][basehttp:log_message][INFO]-"GET/api/config_list/1/HTTP/1.1"20019532

whileTrue:

withopen("xxx.log","a",encoding="utf-8")asfp:

fp.write(a)

循環(huán)寫(xiě)入到xxx.log文件，運(yùn)行3-5分鐘，pycharm打開(kāi)查看文件大小大于10G

于是我用open函數(shù)直接讀取

f=open("xxx.log",'r')

print(f.read())

f.close()

拋出內(nèi)存溢出異常:MemoryError

Traceback(mostrecentcalllast):

File"D:/2025kecheng06/demo/txt.py",line35,inmodule

print(f.read())

MemoryError

運(yùn)行的時(shí)候可以看下自己電腦的內(nèi)存已經(jīng)占了100%，cpu高達(dá)91%，不掛掉才怪了！

這種錯(cuò)誤的原因在于，read()方法執(zhí)行操作是一次性的都讀入內(nèi)存中，顯然文件大于內(nèi)存就會(huì)報(bào)錯(cuò)。

read()的幾種方法

1.read()方法可以帶參數(shù)n，n是每次讀取的大小長(zhǎng)度，也就是可以每次讀一部分，這樣就不會(huì)導(dǎo)致內(nèi)存溢出

f=open("xxx.log",'r')

print(f.read(2048))

f.close()

運(yùn)行結(jié)果

2025-10-2421:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/

2025-10-2421:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678

2025-10-2422:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion('HTTP')

2025-10-2422:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-

2025-10-2422:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010

2025-10-2422:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876

2025-10-2422:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801

2025-10-2422:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638

2025-10-2422:16:21,229[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-10-2422:16:21,307[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/fonts/icomoon.ttfu4m6fyHTTP/1.1"2006900

2025-10-2422:16:23,525[django.server:124][basehttp:log_message][INFO]-"POST/api/login/HTTP/1.1"3020

2025-10-2422:16:23,618[django.server:124][basehttp:log_message][INFO]-"GET/api/index/HTTP/1.1"20018447

2025-10-2422:16:23,709[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/commons.jsHTTP/1.1"20013209

2025-10-2422:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/admin.cssHTTP/1.1"20019660

2025-10-2422:16:23,712[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/css/common.cssHTTP/1.1"2001004

2025-10-2422:16:23,714[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/js/app.jsHTTP/1.1"20020844

2025-10-2422:16:26,509[django.server:124][basehttp:log_message][I

這樣就只讀取了2048個(gè)字符，全部讀取的話，循環(huán)讀就行

f=open("xxx.log",'r')

whileTrue:

block=f.read(2048)

print(block)

ifnotblock:

break

f.close()

2.readline()：每次讀取一行，這個(gè)方法也不會(huì)報(bào)錯(cuò)

f=open("xxx.log",'r')

whileTrue:

line=f.readline()

print(line,end="")

ifnotline:

break

f.close()

3.readlines()：讀取全部的行，生成一個(gè)list，通過(guò)list來(lái)對(duì)文件進(jìn)行處理，顯然這種方式依然會(huì)造成：MemoyError

真正Pythonic的方法

真正Pythonci的方法，使用with結(jié)構(gòu)打開(kāi)文件，fp是一個(gè)可迭代對(duì)象，可以用for遍歷讀取每行的文件內(nèi)容

withopen("xxx.log",'r')asfp:

forlineinfp:

print(line,end="")

yield生成器讀取大文件

前面一篇講yield生成器的時(shí)候提到讀取大文件，函數(shù)返回一個(gè)可迭代對(duì)象，用next()方法讀取文件內(nèi)容

defread_file(fpath):

BLOCK_SIZE=1024

withopen(fpath,'rb')asf:

whileTrue:

block=f.read(BLOCK_SIZE)

ifblock:

yieldblock

else:

return

if__name__=='__main__':

a=read_file("xxx.log")

print(a)#generatorobjec

print(next(a))#bytes類型

print(next(a).decode("utf-8"))#str

運(yùn)行結(jié)果

generatorobjectread_fileat0x00000226B3005258

b'\r\n2025-10-2421:33:31,678[django.request:93][base:get_response][WARNING]-NotFound:/44/\r\n2025-10-2421:33:31,679[django.server:124][basehttp:log_message][WARNING]-"HEAD44/HTTP/1.1"4041678\r\n2025-10-2422:14:04,121[django.server:124][basehttp:log_message][INFO]-code400,messageBadrequestversion(\'HTTP')\r\n2025-10-2422:14:04,122[django.server:124][basehttp:log_message][WARNING]-"GET../../mnt/custom/ProductDefinitionHTTP"400-\r\n2025-10-2422:16:21,052[django.server:124][basehttp:log_message][INFO]-"GET/api/loginHTTP/1.1"3010\r\n2025-10-2422:16:21,123[django.server:124][basehttp:log_message][INFO]-"GET/api/login/HTTP/1.1"2003876\r\n2025-10-2422:16:21,192[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/img/main_bg.pngHTTP/1.1"2002801\r\n2025-10-2422:16:21,196[django.server:124][basehttp:log_message][INFO]-"GET/static/assets/iconfont/style.cssHTTP/1.1"2001638\r\n2025-10-2422:16:21,229[django.server:124]'

[basehttp:log_message][INFO]-"GET/static/assets/img/bg.jpgHTTP/1.1"200135990

2025-10-2422:16:21,307[django.server:124][basehttp:log_messag