How do you read a very large file in Python without loading it all into memory?
Iterate over the file object directly with for line in f. It reads one line at a time through a buffer, so memory use stays small. read() and readlines() load everything at once and can exhaust memory on big files.
Explain it simply
Reading a 5 GB log file with read() is like trying to carry an entire library home in one trip. Looping over the file is like borrowing one book at a time and returning it before taking the next.
A file object is itself an iterator that yields lines lazily.
Example
with open("sample.txt", "w", encoding="utf-8") as f:
f.write("alpha\nbeta\ngamma\n")
count = 0
with open("sample.txt", encoding="utf-8") as f:
for line in f:
if "a" in line:
count += 1
print(count) # Output: 3
with open("sample.txt", encoding="utf-8") as f:
print(f.readline().strip()) # Output: alpha
with open("sample.txt", encoding="utf-8") as f:
print(len(f.readlines())) # Output: 3
with open("sample.txt", "rb") as f:
print(len(f.read(5))) # Output: 5For binary or very long lines, read in fixed-size chunks with f.read(65536) inside a loop until it returns an empty value.
What to say in the interview
"I iterate over the file object, which streams one line at a time with constant memory. read() and readlines() load the whole file, so I avoid them for large inputs. For binary files or huge single lines I read fixed-size chunks in a loop."
Common mistakes
- Calling
readlines()out of habit on a huge file and running out of memory.