Repository navigation
streams - request for docs: can _transform or _write(v) be called in parallel? #3208
Description
The reason I ask is because I have a transform-stream that receives objects and turns those into a bunch of buffers for each single transform. Some of these buffers I have to push out always have the same length. If possible (safe) I would like to create those buffers in my transform stream constructor and simply stick new data in during each transform.
So my question: as long as I don't call the callback in _transform, is it safe to assume _transform will not be called again? If my transform is piped down to disk for example, is there a chance that me changing the buffer in the next _transform call could have an effect on the outcome of the first write to disk?
In either case, documentation on this would be most welcome. I think stream implementers (myself in this case) often really need to squeeze out all performance, and understanding how this behaves would help. Also, as long as it's not documented one way or the other, I would be scared to depend on the behavior (even if it behaves the way I hope it does), because it may change tomorrow.
Activity
brendanashworth commentedon Oct 12, 2015
I'm fairly sure you're correct - this is a guarantee for streams. I'm surprised that it isn't documented, actually.
bnoordhuis commentedon May 2, 2016
/cc @nodejs/documentation? Provisionally adding a doc label.
stevemao commentedon May 4, 2016
also CC @nodejs/streams
mcollina commentedon May 4, 2016
@ronkorving correct. For each stream there is only one
_transformand_writeand_writevis executed at any given time.My module throughv does parallel transform (and it's compatible with streams3).
Trott commentedon Jul 15, 2017
This has been open for quite some time with the
good first contributionlabel. Long enough that I think it might be best if someone experienced just goes ahead and does it. Any volunteers?1 remaining item
mcollina commentedon Jul 19, 2017
Fixed in 0e5283b.
doc,stream: _transform happens one at a time
ronkorving commentedon Jul 19, 2017
@mcollina Appreciate the documentation update. But it doesn't quite answer the question I had. Maybe my phrasing was simply unclear. So let me be more specific:
During every transform, I write a 10 byte buffer. The contents of that buffer will be different each time. Is it safe to reuse the same 10 byte Buffer instance each transform, or will the 2nd transform change the contents of the 1st transform?
My guess is that this is unsafe, but .. that's really where this question originated.
Apologies for the lack of clarity.
mcollina commentedon Jul 19, 2017
You cannot reuse buffers in any part of the streams API.
doc,stream: _transform happens one at a time
doc,stream: _transform happens one at a time
doc,stream: _transform happens one at a time
doc,stream: _transform happens one at a time
doc,stream: _transform happens one at a time
doc,stream: _transform happens one at a time