What are streams in Node.js and why would you use them for large files?

Think about watching a cricket match on a streaming app. You do not download the whole 8-hour match before pressing play. The video arrives in small pieces, and each piece is shown as soon as it lands. That is a stream.

In Node.js, a stream lets you read or write data piece by piece. Without streams, reading a 2 GB log file with fs.readFile puts all 2 GB in memory before you can do anything. With a stream, Node reads it in chunks of 64 KB (the default for file streams), and you handle each chunk and then let it go. Memory stays small no matter how big the file is.

Sending a big file to the browser, the wrong way and the right way:

const http = require("node:http");
const fs = require("node:fs");
const { pipeline } = require("node:stream/promises");

http.createServer(async (req, res) => {
  // Bad for big files: loads the whole file into memory first.
  // const data = await fs.promises.readFile("./big-report.csv");
  // res.end(data);

  // Good: sends the file chunk by chunk.
  res.setHeader("Content-Type", "text/csv");
  try {
    await pipeline(fs.createReadStream("./big-report.csv"), res);
  } catch (err) {
    console.error("Download failed:", err.message);
  }
}).listen(3000);

Compressing a file while reading it:

const fs = require("node:fs");
const zlib = require("node:zlib");
const { pipeline } = require("node:stream/promises");

async function compressLog() {
  await pipeline(
    fs.createReadStream("./orders.log"),
    zlib.createGzip(),
    fs.createWriteStream("./orders.log.gz")
  );
}

compressLog().catch((err) => console.error(err));

There are four kinds of streams:

• Readable: you read from it (fs.createReadStream, an incoming HTTP request).
• Writable: you write to it (fs.createWriteStream, an HTTP response).
• Duplex: both readable and writable (a TCP socket).
• Transform: a duplex stream that changes data as it passes (zlib.createGzip).

An important idea is backpressure. If the reader is fast (a local disk) and the writer is slow (a user on a weak mobile network), data would pile up in memory. When a writable stream's buffer is full, write() returns false, and a well-behaved source pauses until the drain event. pipe() and pipeline() do this automatically.

• Serving video, PDFs and CSV exports.
• Uploading files to cloud storage without saving them to disk first.
• Processing large log or CSV files line by line (readline over a read stream).
• Gzip compression of HTTP responses.

• Using readFile for files that can be hundreds of MB. It works on a laptop and crashes in production.
• Using .pipe() chains without error handling. If one stream fails, the others are not cleaned up. Prefer pipeline().
• Ignoring the return value of write() in a loop, which removes backpressure and fills memory.

• What is backpressure and how does Node handle it?
• Why is pipeline() preferred over .pipe()?
• How would you read a large CSV file line by line?