awaiting between creating a readline interface, and using async iterator will cause iterator to sometimes skip.

Version:v12.13.0
Platform:Linux myuser 4.15.0-74-generic Proposal: Introduce CI builds to GitHub PR process #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Subsystem:
readline.createInterface
asyncIterator

What steps will reproduce the bug?

if you perform something like
https://github.com/nodejs/node/blob/2a7432dadec08bbe7063d84f1aa4a6396807305c/test/parallel/test-readline-async-iterators.js

async function testSimple() {
  for (const fileContent of testContents) {
    fs.writeFileSync(filename, fileContent);

    const readable = fs.createReadStream(filename);
    const rli = readline.createInterface({
      input: readable,
      crlfDelay: Infinity
    });

    const iteratedLines = [];
    for await (const k of rli) {
      iteratedLines.push(k);
    }

    const expectedLines = fileContent.split('\n');
    if (expectedLines[expectedLines.length - 1] === '') {
      expectedLines.pop();
    }
    assert.deepStrictEqual(iteratedLines, expectedLines);
    assert.strictEqual(iteratedLines.join(''), fileContent.replace(/\n/gm, ''));
  }
}

If you add some kind of await xxx() between creating the interface and iterating, the iterator will miss lines. In my case I input 100k lines from a file, then output those same lines to a new file. several thousand lines will go missing.

How often does it reproduce? Is there a required condition?

100%
Need to add some await async between creating the interface and using for await

What is the expected behavior?

not to miss iterations

What do you see instead?

Additional information

readline isn't a stream, if you don't attach a handler you will miss events. that being said, exposing an async iterable doesn't make that very clear.

@devsnek

As @devsnek outlined there's nothing actually wrong with the behavior. We might want to improve our docs when it comes to that?

@devsnek

@devsnek @BridgeAR I would like to take up this issue, could you guide me through the process for achieving the same as I am a first time contributor.

Similar problem shown here: https://stackoverflow.com/questions/62885667/why-does-this-readline-async-iterator-not-work-properly.

I see only this output in the console:

before for() loop
finished
finally
done

The for await (const line1 of rl1) loop never goes into the for loop - it just skips right over it:

const fs = require('fs');
const readline = require('readline');
const { once } = require('events');

async function test(file1, file2) {
    try {
        const stream1 = fs.createReadStream(file1);
        await once(stream1, 'open');
        const rl1 = readline.createInterface({input: stream1, crlfDelay: Infinity});

        const stream2 = fs.createReadStream(file2);
        await once(stream2, 'open');
        const rl2 = readline.createInterface({input: stream2, crlfDelay: Infinity});

        console.log('before for() loop');
        for await (const line1 of rl1) {
            console.log(line1);
        }
        console.log('finished');
    } finally {
        console.log('finally');
    }
}

test("data/numbers.txt", "data/letters.txt").then(() => {
    console.log(`done`);
}).catch(err => {
    console.log('Got rejected promise:', err);
})

doc: add a note about possible missing lines to readline.asyncIterator

So, there is no intent to ever actually "fix" this. You're just going to document it as is and leave it that way forever? So, no asynchronous operations are permitted between creating the interface and consuming the async iterator to read the lines? What if you're trying to use this along with some other asynchronous stuff together? That's just a busted implementation IMO. Documenting the existing behavior is OK for a stop-gap, but not really OK for just sweeping the issue under the rug forever.

@nodejs/readline Should this be re-opened?

doc: add a note about possible missing lines to readline.asyncIterator

@jfriend00

I agree with @jfriend00. This behaves like an improper implementation. Even after following the guidelines to prevent loss of line data, we are facing issues where have a function that is doing some heavy async operations on the result of each line (i.e. each line may take 5 or minutes to complete) and what we get is readline running some lines and eventually hanging and not issuing any more lines -> the program hangs forever.

async function heavyAsyncOperationsPerLine() {
    const rl = readline.createInterface({ input: getReadStream() }); // this file has 800K lines
 // called directly after 
    for await (const line of rl) {
        await operation1()
        await operation2()
    }
}

readline: buffer line events in async iterator

doc: add a note about possible missing lines to readline.asyncIterator

@govindrai

@govindrai I think you might have encountered another bug

The original problem is due to how readline implements AsyncIterable: it uses EventEmitter.on() to create it. EventEmitter.on() has implementation that starts buffering events from the moment on() was called

If there's a delay between creating rl and calling rl[Symbol.asyncIterator]() (which is called internally by for await), EventEmitter.on() has not been called yet, so the lines read within the delay are not saved anywhere and won't be in the iterable

But I don't see how that can cause your issue

I though that maybe your problem was caused by memory leak/too many callbacks pending in the event loop, but the implementation seems to handle backpressure

doc: add a note about possible missing lines to readline.asyncIterator

Navigation Menu

Repository navigation

awaiting between creating a readline interface, and using async iterator will cause iterator to sometimes skip. #33463

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

awaiting between creating a readline interface, and using async iterator will cause iterator to sometimes skip. #33463

Description

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Activity

devsnek commented on May 19, 2020

BridgeAR commented on May 23, 2020

preyunk commented on May 27, 2020

jfriend00 commented on Jul 13, 2020

jfriend00 commented on Aug 30, 2020

Trott commented on Aug 30, 2020

govindrai commented on Apr 9, 2021

azerum commented on Mar 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions