If you still chain streams with .pipe(), a failure in one stage can leave the others open and leak file descriptors. The Node.js stream pipeline, pipeline() from node:stream/promises, connects streams, forwards errors from any stage and cleans everything up. This guide shows how to use it with async/await, transforms, cancellation and an Express endpoint.

Why .pipe() is not enough

With a.pipe(b).pipe(c), an error on b does not automatically destroy a or c. You have to attach an error listener to every stream and tear things down yourself. The official Node.js guide says pipeline() exists for this reason: it forwards errors from any stage, and if one stream fails or closes it destroys the others so resources are not left open.

Node.js stream pipeline basics with stream/promises

The promise version is the one to use in modern code. It resolves when the last stream finishes and rejects with the first error.

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';

async function compress(src, dest) {
  try {
    await pipeline(
      createReadStream(src),
      createGzip(),
      createWriteStream(dest)
    );
    console.log('Pipeline succeeded');
  } catch (err) {
    console.error('Pipeline failed', err);
  }
}

await compress('access.log', 'access.log.gz');

The first argument is the source, the middle ones are transforms, and the last is the destination. A callback version exists in node:stream, but await with try/catch reads better.

Backpressure: what pipeline handles for you

When a writable's .write() returns false, its buffer has passed the highWaterMark (16 KB by default, or 16 objects in object mode). The source should pause until the writable emits drain. Writing by hand means you must check that return value yourself:

import { once } from 'node:events';

for (const chunk of chunks) {
  if (!writable.write(chunk)) {
    await once(writable, 'drain');
  }
}

Inside a pipeline you do not write this code. The stages wire pause and resume for you, so memory stays bounded even when the destination is slower than the source.

Transforming data with async generators

You do not need to subclass Transform. pipeline() accepts async generator functions as middle stages. Each receives the previous stage as an async iterable and yields output chunks. This example reads a large CSV and uppercases the first column of each line:

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import readline from 'node:readline';

async function* transformLines(lines) {
  for await (const line of lines) {
    const [first, ...rest] = line.split(',');
    yield [first.toUpperCase(), ...rest].join(',') + '\n';
  }
}

const lines = readline.createInterface({
  input: createReadStream('users.csv'),
  crlfDelay: Infinity,
});

await pipeline(lines, transformLines, createWriteStream('users-out.csv'));

Because the generator is consumed with for await, it only pulls the next line when downstream is ready. That is backpressure by construction.

Cancelling a pipeline with AbortSignal

Pass an options object with a signal as the last argument. Aborting it destroys the pipeline and rejects with an AbortError.

const ac = new AbortController();
const timer = setTimeout(() => ac.abort(), 5000);

try {
  await pipeline(
    createReadStream('big.bin'),
    createGzip(),
    createWriteStream('big.bin.gz'),
    { signal: ac.signal }
  );
} catch (err) {
  if (err.name === 'AbortError') console.warn('Timed out');
  else throw err;
} finally {
  clearTimeout(timer);
}

If your own generator is the source, the Node.js docs tell you to honour the signal that is passed to it, otherwise it may keep running after the abort.

Streaming a file download from Express

An Express response is a writable stream, so you can pipeline a file straight into it. Handle the case where the client disconnects: the pipeline rejects, and you should not try to send a second response.

import express from 'express';
import { createReadStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';

const app = express();

app.get('/export', async (req, res, next) => {
  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Encoding', 'gzip');
  try {
    await pipeline(createReadStream('export.csv'), createGzip(), res);
  } catch (err) {
    if (res.headersSent) {
      res.destroy(err);
    } else {
      next(err);
    }
  }
});

For request-scoped logging of these failures, see AsyncLocalStorage in Node.js: Request Context for Express. Long-running streams also matter at deploy time, so pair this with Node.js graceful shutdown.

Common mistakes to avoid

  • Not awaiting the pipeline. A floating promise turns stream errors into unhandled rejections.
  • Mixing .pipe() inside a pipeline. You lose the error forwarding you wanted.
  • Buffering inside a transform. Collecting all chunks into an array defeats the point of streaming.
  • Ignoring the source. The Node.js docs warn a pipeline can hang if its first argument is not consumed correctly.
  • Raising highWaterMark blindly. The Node.js guide advises caution; larger buffers mean more memory per stream.

Conclusion

Use await pipeline(source, ...transforms, destination) by default: one promise, one place to catch errors, automatic cleanup and built-in backpressure. Reach for async generators for custom transforms and an AbortSignal for timeouts. For deeper reference, read the official backpressure guide and the stream API docs.