Nexline

Read file, stream, string, buffer line by line without putting them all in memory.
It supports cool features below.

• Lightweight
• Iterable protocol
• Handle large file with small memory footprint
• Support various input types (file, stream, string, buffer)
• Support various encodings
• Support reverse mode
• Support custom, multiple line separators
• Support multiple inputs

Why I made this?

Node.js's default readline module is great but it's pause() method does not work immediately.
I made wrapper to fix that issue before, but it was not enough.
Even if pause() works immediately, it is still inconvenient.
If I want to execute async function over line by line in large file, I have to call pause() and resume() at every line.
I needed better way to do that without putting them all in memory.

Breaking changes in 1.0.0

• lineSeparator default value is changed from ['\n', '\r\n'] to '\n' for better performance
  • If you want to handle both CRLF and LF, set lineSeparator: ['\n', '\r\n']
• Support reverse mode.
• Support file descriptor as input
• Improve performance
• Optimize memory usage

Install with npm

Required Node.js version >= 8.0.0

npm install nexline

How to use

Use next() method

const nexline = require('nexline');

async function main () {
  const nl = nexline(...);
  
  while(true) {
    const line = await nl.next();
    console.log(line);
    if (line === null) break; // If all data is read, returns null
  }
}

Use as iterator ( Requires node >= 10 )

const nexline = require('nexline');

async function main () {
  const nl = nexline(...);

  // nexline is iterable
  for await (const line of nl) console.log(line);
}

Use file as input

• Don't forget to close file descriptor after finish
• You can use autoCloseFile: true to close file descriptor automatically

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'),
  // autoCloseFile: true,
});

Use stream as input

const nl = nexline({
  input: fs.createReadStream(path_to_file),
});

Use string as input

const nl = nexline({
  input: 'foo\nbar\nbaz',
});

Use buffer as input

const nl = nexline({
  input: Buffer.from('foo\nbar\nbaz'),
});

Use multiple input

const nl = nexline({
  input: [
    fs.openSync(path_to_file1, 'r'),
    fs.createReadStream(path_to_file2),
    'foo\nbar\nbaz',
    Buffer.from('foo\nbar\nbaz'),
  ],
});

Use other encodings

See encodings supported by iconv-lite

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'), 
  encoding: 'cp949',
});

Use other lineSeparator

const nl = nexline({
  input: 'foo;bar;baz', 
  lineSeparator: ';',
});

Use multiple lineSeparator

const nl = nexline({
  input: 'foo\r\nbar\nbaz', 
  lineSeparator: ['\n', '\r\n'], // You can handle both LF and CRLF like this.
});

Reverse mode

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'), // NOTE: You cannot use stream in reverse mode. 
  reverse: true, 
});

Methods

| Name | Description | | ------------- | --------------- | | next() | async, It returns line until all data is read, after then returns null | | close() | Close nexline |

Options

| Name | Default | Description | | ------------- | --------------------------- | --------------- | | input | undefined | Required. File descriptor, stream, string, buffer, You can provide multiple inputs using array | | lineSeparator | '\n' | Any string more than one character. You can provide multiple line separator using array | | encoding | 'utf8' | See encodings supported by iconv-lite | | reverse | false | Reverse mode, Cannot use this option with stream input because stream cannot be read from end. Use file descriptor instead | | autoCloseFile | false | Automatically close file descriptor after all data is read |

Contribution

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue

License

Copyright © 2019 Youngho Jin. Released under the MIT License