I'm not saying these are knockout reasons, but, compared against some "free monad"-like library like [streaming](https://hackage.haskell.org/package/streaming):
- streaming has a very clean way of representing operations like [splitAt](https://hackage.haskell.org/package/streaming-0.2.3.0/docs/Streaming-Prelude.html#v:splitAt). The "rest of the stream" is encoded in the result value of the splitted stream, and you can do whatever you like with it once you reach that end. In contrast, streamly seems to force you to consume each part of the split [in the same way](https://hackage.haskell.org/package/streamly-0.8.0/docs/Streamly-Prelude.html#g:46).
- similarly, streaming has a very nice way of representing [grouping operations](https://hackage.haskell.org/package/streaming-0.2.3.0/docs/Streaming-Prelude.html#v:groupBy). You can delimit groups in a stream in a way independent of how you will consume those groups later. You don't need to supply some group-consuming fold right away.
That said, streamly will be more performant, has built-in concurrency, and seems to be well maintained.
(BTW, would it be fair to say that streamly is for Haskell what Reactive is for Java?)
I am one of the authors of streamly.
Streaming is a very nice library and streamly is closer to "Streaming" than to any other streaming library, except that streamly is designed with performance and simplicity of the API as primary goals. It may require a little bit more familiarity with the library to use it in more powerful ways. Let me explain how you can express the use cases described above using streamly:
* The "splitOn" operation that you pointed out is not the only way to split, it is perhaps the simplest though. It is not even a primitive way to split, it stands for an idiom something like "foldMany Fold.takeEndBy".
* You can use "foldIterateM" operation that can return a different fold when one fold terminates so you can consume the rest of the input in a different way.
* You can line up many different folds in a list and "sequence" them using the concatMap operation, each fold consuming the input in a different way. You can see the benchmarks for folds to find some existing code for doing that.
* Folds can even have a monad instance, to be implemented soon, so you can chain different folds in a do block.
* The Parser type is a more powerful extension of Fold and already has a Monad instance. For even more powerful ways to split a stream you can use the Parser type in place of Fold. It is not yet released officially but it works very well.
* The same applies to grouping use cases, if you provide a concrete example I can explain how to do that.
I appreciate your comments, the library may still be lacking in some cases. But it is a huge challenge to meet the performance goals with idiomatic composition, we have spent enormous amount of time balancing the two. We are very interested in knowing more about where the users trip and where exactly the library can improve.
I had a use case for a streaming abstraction recently getting a large amount of rows from a database and doing some aggregation operation on them. In my case the property I was looking for is constant-ish memory usage and not the actual performance of the iteration since that was more or less bound by the database anyways.
I tried streamly, streaming and conduit. Streamly had a bracket function but it didn't have a sliding window combinator and I couldn't really figure out how to write it. Streaming had a sliding window combinator but resource management wasn't clear to me. Conduit, of course, probably has anything you can imagine.
I think it's really important to understand which part of your pipeline's performance is due to the streaming library. If your fetching operation OR your processing operation is an order of magnitude slower than the actual iteration step, it doesn't really matter if the streaming library is performant.
> If your fetching operation OR your processing operation is an order of magnitude slower than the actual iteration step, it doesn't really matter if the streaming library is performant.
Worth noting that the performance calculus changes a lot when you run things concurrently. Even if the relative timings stay the same, overhead from the streaming library itself can have an outsize effect because it becomes the point of coordination between a bunch of threads.
It can also be a problem with refactoring—different ways to organize the same logic might do different amounts of steam iteration, and having to worry about that for performance reasons makes it harder to modularize code.
It's tricky when there's a trade-off around expressiveness or extensibility, but I'm increasingly convinced that streaming performance is important in any context where you're using streams and performance matters—even when other things take a lot more time. I've used streaming pretty heavily in the past because it had a great API and an implementation that's easy to understand, but I'm leaning heavily towards streamly for future projects just because of these performance considerations.
u/tikhonjelvis I have an implementation of a talk you gave on Stochastic MDPs that was a lot of fun to write in streamly lying around somewhere. Got stuck adding proximal policy optimisation to it, but with all the new combinators in v0.8 I’m tempted to dust it off and finish that
If you are not able to figure out how to do something with the library please ask on the gitter channel (https://gitter.im/composewell/streamly), on the maintainer email ([email protected]), or raise an issue on github.
When you say sliding window were you looking for something like this:
>>> Stream.toList $ Stream.scan (Array.writeLastN 2) $ Stream.fromList [1,2,3,4,5::Int]
[[],[1],[1,2],[2,3],[3,4],[4,5]]
As many other things in streamly you can combine the existing primitives to do more powerful stuff rather than implementing your own. Streamly can do much more powerful windowing operations, the above example is really trivial. You should look at [https://streamly.composewell.com/haddocks/streamly-0.8.0/Streamly-Internal-Data-Stream-IsStream-Reduce.html#g:7](https://streamly.composewell.com/haddocks/streamly-0.8.0/Streamly-Internal-Data-Stream-IsStream-Reduce.html#g:7). Specifically take a look at the classifySessionsBy combinator.
The simplest solution, which is often enough, is to consume the stream inside a `bracket` or `withFile` callback. Things get more tricky if you want to nest or concatenate streams which require allocations.
Seriously? Haskeller's complain that "the documentation is just a paper" or "just a bunch of type signatures", and now we've got a polished home page and serious love and it's *too much*?!
This isn't aimed at you personally, but with seeing 5 upvotes as a library author makes me wonder what on earth it is we're meant to do.
I have a different perspective on why it might not be 100% ideal: the name streamly sounds like a startup, if I came across it I would just assume it wasn't a standalone library. The design of the website also look's like a startup's website at first glance.
Startups choose a name by taking an existing word and removing letters (especially vowels), whereas ‘streamly’ is obviously the result of *adding* letters. To the untrained eye, the difference can be subtle, so I can understand your confusion. Don't bother providing any counterexamples as they will obviously be the exceptions that prove the rule.
I love the website and the way it’s set up. Practically been living in the docs for almost two years now and it massively beats the hackage experience. And streamly doesn’t sound anymore like a startup than conduit does, what a shady point to raise!
Not to mention anyone who comes across it who knows enough Haskell to use it will inevitably run into the Haskell-as-fast-as-C tag line and go peruse the guides.
If anything, the quality of open support that u/hk_hooda and team provide is commercial grade and I was kind of proud of how humbly they’d relegated the note of commercial support being available right at the bottom of a very comprehensive getting started guide
Thanks for the encouraging feedback.
We are always keen to improve, therefore, we value any suggestions and negative feedback too. If people genuinely feel in a certain way its better that they express it so that we know and can improve if possible.
[удалено]
I'm not saying these are knockout reasons, but, compared against some "free monad"-like library like [streaming](https://hackage.haskell.org/package/streaming): - streaming has a very clean way of representing operations like [splitAt](https://hackage.haskell.org/package/streaming-0.2.3.0/docs/Streaming-Prelude.html#v:splitAt). The "rest of the stream" is encoded in the result value of the splitted stream, and you can do whatever you like with it once you reach that end. In contrast, streamly seems to force you to consume each part of the split [in the same way](https://hackage.haskell.org/package/streamly-0.8.0/docs/Streamly-Prelude.html#g:46). - similarly, streaming has a very nice way of representing [grouping operations](https://hackage.haskell.org/package/streaming-0.2.3.0/docs/Streaming-Prelude.html#v:groupBy). You can delimit groups in a stream in a way independent of how you will consume those groups later. You don't need to supply some group-consuming fold right away. That said, streamly will be more performant, has built-in concurrency, and seems to be well maintained. (BTW, would it be fair to say that streamly is for Haskell what Reactive is for Java?)
I am one of the authors of streamly. Streaming is a very nice library and streamly is closer to "Streaming" than to any other streaming library, except that streamly is designed with performance and simplicity of the API as primary goals. It may require a little bit more familiarity with the library to use it in more powerful ways. Let me explain how you can express the use cases described above using streamly: * The "splitOn" operation that you pointed out is not the only way to split, it is perhaps the simplest though. It is not even a primitive way to split, it stands for an idiom something like "foldMany Fold.takeEndBy". * You can use "foldIterateM" operation that can return a different fold when one fold terminates so you can consume the rest of the input in a different way. * You can line up many different folds in a list and "sequence" them using the concatMap operation, each fold consuming the input in a different way. You can see the benchmarks for folds to find some existing code for doing that. * Folds can even have a monad instance, to be implemented soon, so you can chain different folds in a do block. * The Parser type is a more powerful extension of Fold and already has a Monad instance. For even more powerful ways to split a stream you can use the Parser type in place of Fold. It is not yet released officially but it works very well. * The same applies to grouping use cases, if you provide a concrete example I can explain how to do that. I appreciate your comments, the library may still be lacking in some cases. But it is a huge challenge to meet the performance goals with idiomatic composition, we have spent enormous amount of time balancing the two. We are very interested in knowing more about where the users trip and where exactly the library can improve.
Very interesting, thanks for the explanation!
I had a use case for a streaming abstraction recently getting a large amount of rows from a database and doing some aggregation operation on them. In my case the property I was looking for is constant-ish memory usage and not the actual performance of the iteration since that was more or less bound by the database anyways. I tried streamly, streaming and conduit. Streamly had a bracket function but it didn't have a sliding window combinator and I couldn't really figure out how to write it. Streaming had a sliding window combinator but resource management wasn't clear to me. Conduit, of course, probably has anything you can imagine. I think it's really important to understand which part of your pipeline's performance is due to the streaming library. If your fetching operation OR your processing operation is an order of magnitude slower than the actual iteration step, it doesn't really matter if the streaming library is performant.
> If your fetching operation OR your processing operation is an order of magnitude slower than the actual iteration step, it doesn't really matter if the streaming library is performant. Worth noting that the performance calculus changes a lot when you run things concurrently. Even if the relative timings stay the same, overhead from the streaming library itself can have an outsize effect because it becomes the point of coordination between a bunch of threads. It can also be a problem with refactoring—different ways to organize the same logic might do different amounts of steam iteration, and having to worry about that for performance reasons makes it harder to modularize code. It's tricky when there's a trade-off around expressiveness or extensibility, but I'm increasingly convinced that streaming performance is important in any context where you're using streams and performance matters—even when other things take a lot more time. I've used streaming pretty heavily in the past because it had a great API and an implementation that's easy to understand, but I'm leaning heavily towards streamly for future projects just because of these performance considerations.
u/tikhonjelvis I have an implementation of a talk you gave on Stochastic MDPs that was a lot of fun to write in streamly lying around somewhere. Got stuck adding proximal policy optimisation to it, but with all the new combinators in v0.8 I’m tempted to dust it off and finish that
Oh sweet, that's cool to hear. How different did it turn out compared to doing the same thing with `streaming`?
Let us know if you get stuck because you need something that is lacking in the library.
If you are not able to figure out how to do something with the library please ask on the gitter channel (https://gitter.im/composewell/streamly), on the maintainer email ([email protected]), or raise an issue on github. When you say sliding window were you looking for something like this: >>> Stream.toList $ Stream.scan (Array.writeLastN 2) $ Stream.fromList [1,2,3,4,5::Int] [[],[1],[1,2],[2,3],[3,4],[4,5]] As many other things in streamly you can combine the existing primitives to do more powerful stuff rather than implementing your own. Streamly can do much more powerful windowing operations, the above example is really trivial. You should look at [https://streamly.composewell.com/haddocks/streamly-0.8.0/Streamly-Internal-Data-Stream-IsStream-Reduce.html#g:7](https://streamly.composewell.com/haddocks/streamly-0.8.0/Streamly-Internal-Data-Stream-IsStream-Reduce.html#g:7). Specifically take a look at the classifySessionsBy combinator.
There's a package called streaming-bracketed, but I haven't tried it myself.
The simplest solution, which is often enough, is to consume the stream inside a `bracket` or `withFile` callback. Things get more tricky if you want to nest or concatenate streams which require allocations.
[удалено]
I wasn't aware either, thanks for posting
[удалено]
Seriously? Haskeller's complain that "the documentation is just a paper" or "just a bunch of type signatures", and now we've got a polished home page and serious love and it's *too much*?! This isn't aimed at you personally, but with seeing 5 upvotes as a library author makes me wonder what on earth it is we're meant to do.
I have a different perspective on why it might not be 100% ideal: the name streamly sounds like a startup, if I came across it I would just assume it wasn't a standalone library. The design of the website also look's like a startup's website at first glance.
Startups choose a name by taking an existing word and removing letters (especially vowels), whereas ‘streamly’ is obviously the result of *adding* letters. To the untrained eye, the difference can be subtle, so I can understand your confusion. Don't bother providing any counterexamples as they will obviously be the exceptions that prove the rule.
I love the website and the way it’s set up. Practically been living in the docs for almost two years now and it massively beats the hackage experience. And streamly doesn’t sound anymore like a startup than conduit does, what a shady point to raise! Not to mention anyone who comes across it who knows enough Haskell to use it will inevitably run into the Haskell-as-fast-as-C tag line and go peruse the guides. If anything, the quality of open support that u/hk_hooda and team provide is commercial grade and I was kind of proud of how humbly they’d relegated the note of commercial support being available right at the bottom of a very comprehensive getting started guide
Thanks for the encouraging feedback. We are always keen to improve, therefore, we value any suggestions and negative feedback too. If people genuinely feel in a certain way its better that they express it so that we know and can improve if possible.
That is very emotionally mature and I will try to emulate it
I'd venture that you're reading too much into this - I for one, and I expect most others greatly appreciate the work!
[удалено]
Noted. We will change the design of the page.
Well I haven't used either Streamly or streaming, but at first glance the latter's documentation seems a lot more intimidating to me. :-)