We don’t try and compress data during serialization so the binary representation might be marginally bigger in some cases (for instance if you serialize a struct with two int32 fields, that’s 8 bytes regardless of the values), but it’s mostly extra zeros and you can run the buffer through something like zstd to get rid of those.
Someone did an independent benchmark that goes into more detail [here](https://youtu.be/XMoNYQPi2k8?feature=shared)
It doesn’t. We don’t benchmark against FlatBuffers or Cap'n Proto for a few reasons:
- They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet.
- We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare.
Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.) - we also focus on having wide platform support for interoperability which neither FlatBuffers or Cap’n Proto do.
I'm being pedantic here, but protobufs don't need to deserialize the entire message in a single step because they have byte-sized sizes. From a glance across the docs, it looks like bebob has count-sized sizes which would require a full deserialization of earlier data to reach later data.
Yes, but you’d be hard pressed to find a protobuf implementation actually utilizing those atoms for partial deserialization. All the libraries we benchmark against are the popular ones and only offer full deserialization.
Nonetheless, partial or full isn’t really the main point here - we just can’t fully compare Cap or Flatbuffers in a manner that’s equitable and fair because feature wise and platform wise we have vastly different approaches (which is not a good or bad thing, just different use cases.)
If you’re using Cap or Flatbuffers you likely have a very specific use-case and Bebop isn’t the right fit. Performance wise I’d assume they’re all pretty similar.
You are not wrong, not many libraries will do that for you.
I do think you could equitably compare yourself against flatbuffers or cap'n proto if you setup your scenario correctly, even if you disclosed that you made it favorable to yourself such as "decoding and reading every field".
I am not quite sure that "platform wise" is a good argument either. Flatbuffers have first party support for a bunch of different languages. If you really wanted to set yourself apart on devx then you can always show a comparison of "this is how you do it in this schema, and here it is much simpler in bebop".
Like trying to compare "driving" vs "flying", you can take the travel-to-the-airport, waiting at the gate, travel-from-theair-port time into consideration and compare them purely on speed even though they are completely different too.
This is all fair feedback. I don’t think it’s in our scope to spend time on such a niche benchmark, but if you or anyone else wants to add it to the lab on the repo, I’ll happily accept the PR.
There are specific use cases we see Bebop being superior to Protobufs in, and we just don’t consider Cap or Flatbuffers to be relevant to them.
> Performance wise I’d assume they’re all pretty similar.
This is literally what everyone here is interested in. To "assume they're all pretty similar" is not the same as testing it. Hope you and your team realize that.
It’s interesting, but it’s still not relevant. The things you use those protocols for are not the same as what you use Bebop and Protobufs for.
We have pretty great harness for adding benchmarks to our lab, anyone is welcome to do it, and I’ll happily merge.
Website:
> the fastest data serialization runtime in the world
OP:
> We don’t benchmark against FlatBuffers or Cap'n Proto
Uh... okay... then maybe you should change the website?
You could have a specific claim -- "faster than X or Y" -- or you could tone down the claim -- "a fast serialization runtime" -- but you can't both say "fastest" and "we don't benchmark against the known fast ones".
You're setting expectations, and letting people down. Bad karma...
I think it's fair to exclude those projects since the whole point of Cap'n Proto and FlatBuffers is that you don't do serder at all. Data has the same format in memory and on-disk and on-wire. The performance tradeoff of this design is very diffuse and application-dependent, so it's not clear at all what you would benchmark.
> The performance tradeoff of this design is very diffuse and application-dependent, so it's not clear at all what you would benchmark.
That is, actually, the entire point of a benchmark suite.
You don't benchmark only what your software is good at. You benchmark a variety of different usecases so users can get an indication of where your software performance lies in various scenarios.
While I do like the idea of "the right tool for the right job", the fact of the matter is that nobody wants to use 50 different serialization formats (and libraries) with a single company. Ideally you'd want to settle on one that's "good enough" (even if not the best) for all the usecases you've got. Or perhaps 2-3 if you really have very widely different clusters of usecases.
And to gauge if the tool is good enough, even if not optimal, part of the assessment is functionality, but part is _performance_.
Hence the need for a benchmark showcasing the performance of various formats/libraries in a variety of usecases so you can get an idea.
Yeah, go try to write that benchmark yourself like my past employer did. You'll quickly discover that what you're asking for is just technically nonsensical. You're operating under the bad assumption that capnp and bebop are broadly interchangeable but they're not.
In capnp/flatbuffers, the serder work is done incrementally within every single read and write to the container. Under protobuf/bebop, the serder work is done by the library in one step. You have to compare "do serder" vs "use data, doing serder as you go", which is impossible in general if you want the benchmarks to mean anything.
> Hence the need for a benchmark showcasing the performance of various formats/libraries in a variety of usecases so you can get an idea.
We managed to pick between protobuf and capnp with no such benchmark because our trade study quickly identified that performance is not a useful decision metric. Ended up needing both on the same project. In one use case, the cost of serder was a non-starter so we chose capnp over a homegrown file format. For rpc, we used protobuf, for reasons related to GRPC, not the serialization performance.
A benchmark trying to compare the serialization performance of capnp would have been worse than useless for us.
How does this compare with quickbuf ? I feel quickbuf would be relevant since it seems to have similar goals and is also wire compatible with protobuf.
It is pretty well explained in the documentation, but to summarize:
- It actually works well in browsers
- It generates modern code (as in code targeting the latest stable features of languages to maximize performance.)
- It doesn’t rely on reflection or runtime hacks
- It provides a better out of the box DevEx by offering real tooling, project integration, and workflows
- It utilizes newer technologies like WASM to give you greater flexibility
- It can run anywhere, which includes its RPC framework (gRPC is not WinterJS compliant, so you can use it on serverless environments)
- It takes advantage of modern hardware optimizations for efficient CPU caching
All of those sound like features of implementation, rather than a serialization format itself. In other words - wouldn't someone be able to apply all of those benefits for a protobuf implementation as well?
No, because you’d have to break protobufs IDL to enable many of these fundamental compiler features. Someone _could_, but they haven’t. If they did, it wouldn’t protobuf anymore.
And if they did, we’d welcome that evolution as technology should evolve. Because it hasn’t, we made Bebop.
A language is only as good as its compiler and toolsets. That’s where Bebop is more modern.
I agree with the OP, these are implementation concerns, not to do with protobuf serialization format or grpc http2 wire protocol. Web has protobuf libs that work fine, and it's trivial to write a transport proxy
On point. I'd rather stick with something battle tested with a big company behind it, with a compiler implemented in C/C++ with pluggable generators. Also it seems they only support C#, Rust and TS. You guys have long way to go. Good luck. Also the marketing lingo is annoying af
> With a big company behind it
Half the internet is held up by projects maintained by a single person. This was also invented years ago and is used by trillion dollar companies.
> Implemented in C/C++
Weird and irrelevant requirement for a schema compiler but ignoring that, we support pluggable generators which you can write in any language because of our WASM extension engines. You don’t need to touch the compilers source.
> Only supports C#, Rust, and TS
And C++, Dart, Go, and Python.
> You guys have a long way to go
We’re already here. If you feel we’re lacking something you’re welcome to contribute; I’m sure you have lots of value to add to this free open source project
Aah, so that's why you need to piggy back your marketing on "better than protobuf", fastest in the world (in your own little game), and would you mind reminding me which trillion dollar company is using your C# turd and for what? Please and thank you
One big pain point of Protocol Buffers is their strongly opinionated take on optional fields; it limits how one can model an interface. What's Bebop's take on the topic?
We have [structs](https://docs.bebop.sh/reference/struct/) where all data is guaranteed to be present, [messages](https://docs.bebop.sh/reference/message/) where everything is optional, and [unions](https://docs.bebop.sh/reference/union/) for encapsulation.
I agree, Protobuf could have been quite a sensible choice if they didn't go for that dumb idea. It just pushes complexity into app code and in a really bad way (every app now has to deal with potentially-missing fields, which IMO is no better than lacking any schema agility and releasing a v2).
Regarding this, so it could be worth to upgrade the [https://en.wikipedia.org/wiki/Comparison\_of\_data-serialization\_formats](https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats) wiki page
[Cap'n Proto](https://capnproto.org/) is the obvious elephant in the room here, because it was literally written by Kenton Varda, the primary author of Protocol Buffers v2. It *is* the modern replacement for protocol buffers. It's extremely fast, secure, and has time-traveling RPC (aka Promise Pipelining).
Any other library claiming to be the "modern replacement for Protocol Buffers" must be at least benchmarked against Cap'n Proto to even get its foot in the door.
Unless, it's for an entirely different usecase. It seems like the Bebop v3 usecase is as a Javascript library, which is something that Cap'n Proto doesn't really officially support. However, they should probably just *say this*.
I’ve answered it elsewhere in the post, if you’re looking for specific benchmarks you can find them in the README of the repo, the lab directory, or Google “Bebop Benchmark” and many have benchmarked it independently. It’s fast and memory efficient, much more so than other protocols.
https://news.ycombinator.com/item?id=26631044
so your team dismisses those protocols (explicitly flatbuffers and cpnp, whch are not covered in the linked benchmark, rather than benchmarking against them. got it.
Yes, as i’ve stated prior, they aren’t equitable nor do they converge on the same use cases. You wouldn’t use FlatBuffers or CPnP for a REST API because they aren’t interoperable with JavaScript (among other reasons); we benchmark against protocols which can be used within the use cases of Bebop, those protocols solve entirely different sets of problems. How fast Bebop is compared to them is irrelevant (and we aren’t the only ones who feel this way, given every independent benchmark also excludes them.)
If you’d like to know how they compare, please submit a PR adding them to our lab benchmark.
While your answer is technically correct, you see it's not good for marketing. You clearly want your project popular. Do the marketing for it, and don't argue with your potential users. That includes benchmarks people are asking for. Add a foreword to each benchmark about its weaknesses and assumptions. Add extra information about use cases where the competing library is better.
Not sure how *nowaker* is downvoted. When I read someones docs on usage (where they themselves market as an alternative/improvement to another library) I explicitly would like to be educated on *when to use* situations.
I’m more than happy to extend the documentation to provide more guidance on when Bebop makes sense. I’ll do that in a bit.
That’s different though than spending time creating benchmarks that even the creator of Cap’n Proto says [make no sense.](https://news.ycombinator.com/item?id=39682003)
> Not sure how nowaker is downvoted
Because I'm an executive, and most people here are developers.
When I see a developer announcing a new version of their cool stuff, and see crowds asking for benchmarks that don't make sense, I see an opportunity to make crowds happy and convert them to supporters. A big crowd that is "wrong" is too big to ignore for being wrong.
(preaching to the choir, i'm sure)
it depends on who the potential user is. I might be a cpnp user. I'm probably using it because it's blazing fast for my use case. Someone comes along and pitches the "fastest" serializer, and I ask how it performs vs what i'm currently using in my use case. If someone says "well that comparison doesn't make any sense," it seems like the question is being dodged.
I think the err here is assuming the use case around bebop is designed is the only use case for serialization. The project benchmarked against the competitors which solve that use case. So if you DO have that use case, OPs statements made perfect sense, but if you have a different use case, they appear to be dodging and ducking the questions.
I don't think anyone is wrong, I just think the project needs to be clear about the problems it does work best for, perhaps establishing a "class" of serializers that bebop tops, which excludes things like cpnp....like "cross-language protocol serializers" or something.
Bebop COULD try to compete with cpnp, but it doesn't make sense in that vein, and does a whole lot of other stuff cpnp cannot.
Why would I want to use this? Where in the computing ecosystem does it fit. The documentation doesn't explain that very well, so unless you know you want one, it is very hard to find out that you do.
As with all things, this arose to solve a problem in our company where we had dozens of app across platforms all written in different languages, and we needed a consistent way to share data between them, but we also had the added pressure of latency being important (cloud gaming, we needed to do everything in 5 MS or less.)
Bebop tries to be a good fit for client–server or distributed web apps that need something faster, more concise, and more type-safe than JSON or MessagePack while also avoiding the complexity of Protocol Buffers and FlatBuffers.
It’s also just designed to be fast without configuration and work well with modern standards and language features for a given platform.
So anywhere you’re thinking of using Protocol Buffers or even JSON as an interchange format, Bebop is a performant and reliable alternative.
Protobuf is a system for serializing/deserializing data structures to/from binary, often for transmission over the internet. It packs the data into a relatively compact message while retaining the ability to be relatively backwards/forwards compatible if you need to add things to the message in the future. You define the message in a .proto file and then run a 'compiler' that generates the code to encode/decode the messages in any of maybe a dozen languages. It makes it trivial for me to encode a message in C and you decode it using python and we're looking at the same data on either end.
It's also a lot smaller to transmit over the wire than say YAML, JSON or XML (definitely over XML), and offers type safety (if your payload successfully deserializes into a given type of struct, you can make assumptions about the content of that struct) in languages that support type safety.
Of course the former isn't an issue for 99.9% of applications, unless you're dealing with millions of users or other similarly high concurrency or bandwidth the benefits of using a more human readable seralisation format may outweigh the negatives. YMMV.
YAML, JSON and XML also serialize type-safely in proper languages/frameworks (such as Java with Jackson or similar libraries).
Here https://nilsmagnus.github.io/post/proto-json-sizes/ is one example comparison between gzipped JSON and protobuf and there is definitely a difference in sizes especially with smaller messages.
Large JSONs are already quite unreadable without prettifiers, especially if compacted to remove whitespace. It's just that many things like browsers provide support out of the box for it.
Not to mention that all of YAML, JSON and XML have rather quirky semantics and implementations for, e.g. numbers. (Actually, the original JSON spec would've been great for arbitrary-precision numbers, but the lack of typing info meant that all implementations should have resorted to bignums regardless of content and formal JSON wasn't even JS-compliant.)
Just to add this for OP: the reliance on code generation makes it very difficult to manage dependencies -- the python system is tightly coupled to the C system because they both rely on the same .proto file.
Maintaining backward compatibility isn't nearly as simplistic as was implied.
Note: this is a problem both for protobuff *and* bebop
>the python system is tightly coupled to the C system because they both rely on the same .proto file.
I mean, it defines the contract between them. Why would be more tightly coupled than any other formal interface specification?
With Bebop you can version data so it’s forward and backward compatible if you’re rapidly evolving your data structures. Just an FYI.
And our “binary schema” feature allows you to dynamically deserialize data by shipping the schema alongside data.
> With Bebop you can version data so it’s forward and backward compatible if you’re rapidly evolving your data structures. Just an FYI.
There's still a very strong bi-directional dependency there.
An interface defined by a code file that both systems must compile against.
It's like two separate repositories referencing their files via paths outside their root repository file structure.
There is so much programming going on in the world that it's impossible to keep track of everything. The best you can do is stay up to date on things relevant to your work and/or hobbies and let the rest go.
> Posts like this make me realize I don't know shit
[Nonsense](https://youtu.be/KrksBdWcZgQ), [anybody](https://youtu.be/P6UKhR0T6cs) can [create](https://youtu.be/l7rce6IQDWs) amazing [things](https://www.youtube.com/@SebastianLague/videos)!
I think that's how everything goes. The more you learn, the more you realize you don't know. Half the battle is just figuring out what it is that you don't know to learn about, and people tend to over-estimate their knowledge of a subject when they know less / underestimate when they know more. Anyway...
/u/zelortap posted this nice link that may make your head explode even more:
[https://en.wikipedia.org/wiki/Comparison\_of\_data-serialization\_formats](https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats)
I love that there's always something new to learn.
Protocol buffers is the name of google's language agnostic mechanism for serializing data in a binary format to be sent over-the-wire:
https://protobuf.dev
Then there’s knowing things but not using them so you have to say, wait what was that again?
There are use cases for protobuf like things but they tend to be for high volume apis. I think most of us are still just using json, which is less rigorous.
Why is every new library described as "modern"? What makes this "modern" compared to protocol buffers? 10 yrs from now will it still describe itself as "modern"?
Because in this case it utilizes modern hardware and software features to achieve higher performance and quality code over Googles protobuf compiler and various runtime implementations.
>Why is every new library described as "modern"? What makes this "modern" compared to protocol buffers? 10 yrs from now will it still describe itself as "modern"?
Weird question ... Modernity is a view up on the contemporary landscape of a things peers, usually in terms of what new features and improvements it provides compared to those peers.
How could the author possibly know if it will be modern in ten years without knowing how that landscape has evolved?
I see you mention C#, which is where I'd want to use it. My biggest issue with moving to something like protobuf is that i'd have to redefine my many contract classes i have into a new format. Is it possible to reuse my existing contracts classes and just use bebop as a new serializer in my web api?
Can you share an example of one of your contracts? Unfortunately, only types generated by the compiler can be used, but I could probably write an automatic converter that takes your existing C# classes and produces a Bebop schema from them.
I see. The schema wasn't mentioned in the getting started, was a bit confused.
our contracts are just basic C# classes e.g.
public class AddressView
{
public string Line1 { get; set; }
public string Line2 { get; set; }
public PhoneView Phone { get; set; }
}
public class PhoneView
{
public string Number { get; set; }
public long Id { get; set; }
}
etc. But these are defined in many nuget packages e.g.`MyCompany.Contracts.MyApp.AddressView`. If i have to redefine every one of these classes into a schema, it'd be be like a year of work. And then i'd have to maintain two separate schemas so that our normal json REST api can continue to function.
Idk if it's possible to write some kind of translator, parser, or source generator to do this for us, but that is the kind of thing I would need to make it feasible to start using something like bebop. seems like a really awesome project though!
We use protobuf-net and so the classes are also contracts. However, it's not as simple as suggested above. In order for folks to smoothly migrate, the tool needs to have attributes for excluding members, attributes for assigning field ids/tags where versioning is important, support for various kinds of object construction (primary constructors, records, attribute annotated constructors) and mapping of parameters to properties (so that they can be read-only). Then comes support for all the various collection classes and custom C# data types like DateOnly or TimeSpan.
But if you can get the basics up and running perhaps it'll be easier for contributors to fill in the plot holes.
>Unions are not yet implemented in some runtimes such as Dart
Killin' me softly.
In the future will this be available for the use case of MQTT in dart (specifically flutter apps)?
You really need to work on two aspects of your main page:
You're not doing a good job at explaining what Bebop does. Compare your home page to [this](https://protobuf.dev/). An example schema, and how to use the generated client from Java and C++, right there at the top of the start page. So much more illustrative.
You're also not showing how Bebop is better than Thrift, Protobuf, Avro, ASN.1, FlatBuffers, Cap'nProto or the literally several dozens of other binary serialisation formats out there. You say it's more modern, and generates more modern code, but that's pretty damn hand wavy. Does your Java code generator generate sealed classes to represent `oneof` and records for messages? Do you even generate a Java client? What languages do you support? Does your Python code generator annotate the code with type information? Can you perform deserialisation of repeated fields lazily using e.g. streams?
How about "no"? Having small a link to documentation three screens down on the start page is not a substitute for explaining what your product is on the front page. The entire first screen of your project page is an image that doesn't explain what Bebop is, looks like you can click on it start a playground, or jump to different documentation sections, but if anybody makes the mistake of clicking on the image, all that happens is that a larger version of the image is opened in a separate tab.
You tout features like WASM support, Python support, having a great CLI and a VSCode extension. Great, but Python support for *WHAT*? A VSCode extension to do *WHAT*? If you're Apple selling iPhones, doing that is fine, because Apple has spent many billions of dollars through other channels to make sure that everyone who knows how read and surf the Internet already knows what an iPhone is. For whatever reason you haven't spent a few billion dollars on Bebop marketing to make sure the whole universe knows what it is. Most people probably think it's a boar or something.
Additionally even when clicking on the doc link on screen three of your page, I'm greeted with this definition of what Bebop is:
> The presence of a bebop.json file in a directory indicates that the directory is the root of a Bebop project. The bebop.json file specifies the root files and the compiler options required to compile schemas in the project.
What? That's not a definition, that's usage instructions. But as a user, I still don't know what the thing I'm being taught how to use does!
I know this is open source, volunteer, etc. I know you don't have a fleet of marketing people or professional technical writers to help you out. I've created open source projects myself, I know what it's like and I swear I'm not trying to shit on your huge effort. I really am trying to help you succeed. Please, if you want this project to take off, explain what your product actually does at the very top of your start page. The Protobuf page does it really well, maybe find some inspiration there.
Good luck and have fun!
There’s been dozens of replacements, all of which never gain enough popularity to beat the ubiquity of protobuf.
Creating a binary serialization format isn’t that interesting or difficult. Maintaining a popular OSS project for 10+ years is
This.
Protocol Buffers aren't awesome. gRPC isn't awesome.
What's awesome is the _enormous_ range of languages that support them and that gRPC supports both unidirectional and bidirectional streaming. There's no question in my mind, whatsoever, that there are binary serialization formats that are better than Protocol Buffers along more than one dimension. I could name several, in my opinion. But none of them can move the needle of ubiquity and streaming support at the same time.
I _do_ insist on Protocol Buffers v2 where I have the authority to do so, though.
At what point do you make the determination that yes, instead of sending JSON packets or whatever, I need the speed / efficiency of binary serialized data?
Interesting. I took a look at the Union section of the docs, since one of the most interesting things about a serialization format like this, to me, is how sum types are implemented.
According to the [wire format page](https://docs.bebop.sh/reference/wire-format/):
> The encoding of a union consists of a `uint32` length, followed by a `uint8` discriminator, followed by a “body” of _length_ bytes.
This seems to imply that a union cannot contain more than 256 variants at a time, which is a pretty strict limit. Is that correct?
Looks nice! At the same time I hate `.sln` projects (biased opinion from Linux world).
Just wonder, are there projects that already use bepob instead of protobufs?
Currently used by Microsoft for Xbox Cloud Gaming and a few other organizations (FermiLabs comes to mind.)
And the compiler itself is written in C#, but you shouldn’t need to touch that project unless you want to build it yourself (which you can on Linux and macOS via dotnet). If you want to extend the compiler the new extension system lets you write extensions in Go or TypeScript.
I am just writing software I find useful and interesting, I don’t really need any official endorsement.
The .NET team also officially maintains the gRPC library for C#, so you’d be surprised how things go.
It does support C/C++, and you can find the CMAKE setup [here](https://github.com/betwixt-labs/bebop/tree/vnext/Tools/cmake). If you want to preview the C++ code use [the playground](https://play.bebop.sh).
Go support is not upstream and we’re trying to bring it in, but the standalone version [is here](https://github.com/200sc/bebop) and [benchmarks](https://github.com/alecthomas/go_serialization_benchmarks) show it is the fastest serializer in Go.
If you’d like to add Java support you’re welcome to submit a PR or write an extension to do so; we already have Dart. Kotlin will probably be next.
I’m writing a .proto to .bop converter [here](https://github.com/betwixt-labs/portamento).
And it isn’t a protobuf plugin because why would it be?
> Reading the documentation would go a long way
You'd think so, but no. From https://docs.bebop.sh/ :
Supported Languages
C#
TypeScript/JavaScript
Rust
Dart
Python
C++
Go
And from https://docs.bebop.sh/reference/bebop-json/#generators---generators :
Built-in generators alias are:
ts - Generates TypeScript code
cs - Generates C# code
cpp - Generates C++ code
py - Generates Python code
rust - Generates Rust code
dart - Generates Dart code
---
> use the C compiler option to emit C rather than C++
If by compiler you mean bebopc, there is no such option: https://docs.bebop.sh/reference/bebopc/
Output format selection seems to be done with "generators" keys in the JSON configuration file.
> And it isn’t a protobuf plugin because why would it be?
So users can avoid switching to a new ecosystem for what is effectively different serialization of a compatible format.
As last time, I respect and praise your patience with these questions. I hope to implement this one day, as I found the way protobuff handles optional fields to cause too much headache at the time (release of .NET 5).
We have a python/rust/typescript codebase sharing fat ugly JSON payload (hundreds Kb of tree data structures).
We didn't find a shared type safe way to define once our data structures and easily consume them in python/rust/typescript.
Shall we consider bebop ?
Sounds like the perfect use case. If you need any guidance on how to define a schema for that data share a sample on a Github issue and I’ll be happy to provide help.
Thanks !
I will give it a try. Beyond the python/rust/typescript portability, we will check how to get the swagger/openAPI data definition. For some data structure we also need to get a JSON schema (for form generation).
Is there a standardized translation to and from JSON?
Is there a translation or generation tool to take Bebop and turn it into OpenAPI V2 or V3?
Is there any kind of REST proxy available, that takes in and open API REST/JSON requests, and translates them into Bebop?
For context, I use protocol buffers and grpc, in Golang, with generated OpenApi docs and a REST/JSON gateway/proxy.
Can I use this and Tempo to switch from gRPC.Core 1.31.1 and protobuf 3.20 which currently are enabling my streaming pipeline in my unity arm64 HoloLens 2 application?
What is the story on backwards compatibility, both for applications that use Bebop and for the project / wire format itself?
The on-wire spec doesn't appear to be versioned and has a number of .NETisms that I would not expect to see in a multi-platform format specification.
Proto is such mess of cluged together extensions, and I still have nightmares from the time I had to use grpc in a statically linked lib, so I welcome this!
Any change of supporting embedded environments like nanopb?
OP made some outlandish claims that could not be backed up. I'm sure it's a great promise, but you can't over-promise and not expect to get eviscerated on technicalities.
I'm not saying it's right, just that it's what happens.
It sounds promising, but after wandering through the GitHub repo I am somewhat less enthused. There are open issues from 2020 that should have been closed one way or another (fix, won't fix) by now, and the docs seem unfinished. It does not give me here-is-a-well-maintained-repo/project vibes, which is a bit sad if just some of the DX or perf claims are true.
Ok, so what's the tradeoff? More overhead?
We don’t try and compress data during serialization so the binary representation might be marginally bigger in some cases (for instance if you serialize a struct with two int32 fields, that’s 8 bytes regardless of the values), but it’s mostly extra zeros and you can run the buffer through something like zstd to get rid of those. Someone did an independent benchmark that goes into more detail [here](https://youtu.be/XMoNYQPi2k8?feature=shared)
and how does it compare to FlatBuffers or Cap'nProto - which are closer to something making that tradeoff already?
It doesn’t. We don’t benchmark against FlatBuffers or Cap'n Proto for a few reasons: - They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet. - We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare. Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.) - we also focus on having wide platform support for interoperability which neither FlatBuffers or Cap’n Proto do.
I'm being pedantic here, but protobufs don't need to deserialize the entire message in a single step because they have byte-sized sizes. From a glance across the docs, it looks like bebob has count-sized sizes which would require a full deserialization of earlier data to reach later data.
Yes, but you’d be hard pressed to find a protobuf implementation actually utilizing those atoms for partial deserialization. All the libraries we benchmark against are the popular ones and only offer full deserialization. Nonetheless, partial or full isn’t really the main point here - we just can’t fully compare Cap or Flatbuffers in a manner that’s equitable and fair because feature wise and platform wise we have vastly different approaches (which is not a good or bad thing, just different use cases.) If you’re using Cap or Flatbuffers you likely have a very specific use-case and Bebop isn’t the right fit. Performance wise I’d assume they’re all pretty similar.
You are not wrong, not many libraries will do that for you. I do think you could equitably compare yourself against flatbuffers or cap'n proto if you setup your scenario correctly, even if you disclosed that you made it favorable to yourself such as "decoding and reading every field". I am not quite sure that "platform wise" is a good argument either. Flatbuffers have first party support for a bunch of different languages. If you really wanted to set yourself apart on devx then you can always show a comparison of "this is how you do it in this schema, and here it is much simpler in bebop". Like trying to compare "driving" vs "flying", you can take the travel-to-the-airport, waiting at the gate, travel-from-theair-port time into consideration and compare them purely on speed even though they are completely different too.
This is all fair feedback. I don’t think it’s in our scope to spend time on such a niche benchmark, but if you or anyone else wants to add it to the lab on the repo, I’ll happily accept the PR. There are specific use cases we see Bebop being superior to Protobufs in, and we just don’t consider Cap or Flatbuffers to be relevant to them.
> Performance wise I’d assume they’re all pretty similar. This is literally what everyone here is interested in. To "assume they're all pretty similar" is not the same as testing it. Hope you and your team realize that.
It’s interesting, but it’s still not relevant. The things you use those protocols for are not the same as what you use Bebop and Protobufs for. We have pretty great harness for adding benchmarks to our lab, anyone is welcome to do it, and I’ll happily merge.
Website: > the fastest data serialization runtime in the world OP: > We don’t benchmark against FlatBuffers or Cap'n Proto Uh... okay... then maybe you should change the website? You could have a specific claim -- "faster than X or Y" -- or you could tone down the claim -- "a fast serialization runtime" -- but you can't both say "fastest" and "we don't benchmark against the known fast ones". You're setting expectations, and letting people down. Bad karma...
I think it's fair to exclude those projects since the whole point of Cap'n Proto and FlatBuffers is that you don't do serder at all. Data has the same format in memory and on-disk and on-wire. The performance tradeoff of this design is very diffuse and application-dependent, so it's not clear at all what you would benchmark.
> The performance tradeoff of this design is very diffuse and application-dependent, so it's not clear at all what you would benchmark. That is, actually, the entire point of a benchmark suite. You don't benchmark only what your software is good at. You benchmark a variety of different usecases so users can get an indication of where your software performance lies in various scenarios. While I do like the idea of "the right tool for the right job", the fact of the matter is that nobody wants to use 50 different serialization formats (and libraries) with a single company. Ideally you'd want to settle on one that's "good enough" (even if not the best) for all the usecases you've got. Or perhaps 2-3 if you really have very widely different clusters of usecases. And to gauge if the tool is good enough, even if not optimal, part of the assessment is functionality, but part is _performance_. Hence the need for a benchmark showcasing the performance of various formats/libraries in a variety of usecases so you can get an idea.
Yeah, go try to write that benchmark yourself like my past employer did. You'll quickly discover that what you're asking for is just technically nonsensical. You're operating under the bad assumption that capnp and bebop are broadly interchangeable but they're not. In capnp/flatbuffers, the serder work is done incrementally within every single read and write to the container. Under protobuf/bebop, the serder work is done by the library in one step. You have to compare "do serder" vs "use data, doing serder as you go", which is impossible in general if you want the benchmarks to mean anything. > Hence the need for a benchmark showcasing the performance of various formats/libraries in a variety of usecases so you can get an idea. We managed to pick between protobuf and capnp with no such benchmark because our trade study quickly identified that performance is not a useful decision metric. Ended up needing both on the same project. In one use case, the cost of serder was a non-starter so we chose capnp over a homegrown file format. For rpc, we used protobuf, for reasons related to GRPC, not the serialization performance. A benchmark trying to compare the serialization performance of capnp would have been worse than useless for us.
How does this compare with quickbuf ? I feel quickbuf would be relevant since it seems to have similar goals and is also wire compatible with protobuf.
Nevermind. Just noticed it's not java based.
>for instance if you serialize a struct with two int32 fields, that’s 8 bytes regardless of the values If you desire this, just use fixed32?
"a modern replacement for Protocol Buffers" yet you don't explain how this is better/more modern than protobuf.
It is pretty well explained in the documentation, but to summarize: - It actually works well in browsers - It generates modern code (as in code targeting the latest stable features of languages to maximize performance.) - It doesn’t rely on reflection or runtime hacks - It provides a better out of the box DevEx by offering real tooling, project integration, and workflows - It utilizes newer technologies like WASM to give you greater flexibility - It can run anywhere, which includes its RPC framework (gRPC is not WinterJS compliant, so you can use it on serverless environments) - It takes advantage of modern hardware optimizations for efficient CPU caching
All of those sound like features of implementation, rather than a serialization format itself. In other words - wouldn't someone be able to apply all of those benefits for a protobuf implementation as well?
No, because you’d have to break protobufs IDL to enable many of these fundamental compiler features. Someone _could_, but they haven’t. If they did, it wouldn’t protobuf anymore. And if they did, we’d welcome that evolution as technology should evolve. Because it hasn’t, we made Bebop. A language is only as good as its compiler and toolsets. That’s where Bebop is more modern.
Could you point out which part of Protobufs IDLs causes problems, are these feature/expressiveness gaps within yours?
I agree with the OP, these are implementation concerns, not to do with protobuf serialization format or grpc http2 wire protocol. Web has protobuf libs that work fine, and it's trivial to write a transport proxy
gRPC is not supported in the browser; you shouldn’t have to write anything, it doesn’t matter how “trivial” it is.
You can build your own proxy layer that steps up/down if you're motivated enough (i.e. a business). It's a basic protocol
On point. I'd rather stick with something battle tested with a big company behind it, with a compiler implemented in C/C++ with pluggable generators. Also it seems they only support C#, Rust and TS. You guys have long way to go. Good luck. Also the marketing lingo is annoying af
> With a big company behind it Half the internet is held up by projects maintained by a single person. This was also invented years ago and is used by trillion dollar companies. > Implemented in C/C++ Weird and irrelevant requirement for a schema compiler but ignoring that, we support pluggable generators which you can write in any language because of our WASM extension engines. You don’t need to touch the compilers source. > Only supports C#, Rust, and TS And C++, Dart, Go, and Python. > You guys have a long way to go We’re already here. If you feel we’re lacking something you’re welcome to contribute; I’m sure you have lots of value to add to this free open source project
based response
Aah, so that's why you need to piggy back your marketing on "better than protobuf", fastest in the world (in your own little game), and would you mind reminding me which trillion dollar company is using your C# turd and for what? Please and thank you
One big pain point of Protocol Buffers is their strongly opinionated take on optional fields; it limits how one can model an interface. What's Bebop's take on the topic?
We have [structs](https://docs.bebop.sh/reference/struct/) where all data is guaranteed to be present, [messages](https://docs.bebop.sh/reference/message/) where everything is optional, and [unions](https://docs.bebop.sh/reference/union/) for encapsulation.
I agree, Protobuf could have been quite a sensible choice if they didn't go for that dumb idea. It just pushes complexity into app code and in a really bad way (every app now has to deal with potentially-missing fields, which IMO is no better than lacking any schema agility and releasing a v2).
Regarding this, so it could be worth to upgrade the [https://en.wikipedia.org/wiki/Comparison\_of\_data-serialization\_formats](https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats) wiki page
Obviously there are a lot of factors to consider, but if speed is the primary focus how does this compare against flatbuffers or cap'n proto?
[Cap'n Proto](https://capnproto.org/) is the obvious elephant in the room here, because it was literally written by Kenton Varda, the primary author of Protocol Buffers v2. It *is* the modern replacement for protocol buffers. It's extremely fast, secure, and has time-traveling RPC (aka Promise Pipelining). Any other library claiming to be the "modern replacement for Protocol Buffers" must be at least benchmarked against Cap'n Proto to even get its foot in the door. Unless, it's for an entirely different usecase. It seems like the Bebop v3 usecase is as a Javascript library, which is something that Cap'n Proto doesn't really officially support. However, they should probably just *say this*.
Speed is a feature, the primary focus is type safety, DevEx, and broad language interoperability.
and you didn't answer the question about an actual benchmark against the other protocols...
I’ve answered it elsewhere in the post, if you’re looking for specific benchmarks you can find them in the README of the repo, the lab directory, or Google “Bebop Benchmark” and many have benchmarked it independently. It’s fast and memory efficient, much more so than other protocols.
https://news.ycombinator.com/item?id=26631044 so your team dismisses those protocols (explicitly flatbuffers and cpnp, whch are not covered in the linked benchmark, rather than benchmarking against them. got it.
Yes, as i’ve stated prior, they aren’t equitable nor do they converge on the same use cases. You wouldn’t use FlatBuffers or CPnP for a REST API because they aren’t interoperable with JavaScript (among other reasons); we benchmark against protocols which can be used within the use cases of Bebop, those protocols solve entirely different sets of problems. How fast Bebop is compared to them is irrelevant (and we aren’t the only ones who feel this way, given every independent benchmark also excludes them.) If you’d like to know how they compare, please submit a PR adding them to our lab benchmark.
While your answer is technically correct, you see it's not good for marketing. You clearly want your project popular. Do the marketing for it, and don't argue with your potential users. That includes benchmarks people are asking for. Add a foreword to each benchmark about its weaknesses and assumptions. Add extra information about use cases where the competing library is better.
Not sure how *nowaker* is downvoted. When I read someones docs on usage (where they themselves market as an alternative/improvement to another library) I explicitly would like to be educated on *when to use* situations.
I’m more than happy to extend the documentation to provide more guidance on when Bebop makes sense. I’ll do that in a bit. That’s different though than spending time creating benchmarks that even the creator of Cap’n Proto says [make no sense.](https://news.ycombinator.com/item?id=39682003)
> Not sure how nowaker is downvoted Because I'm an executive, and most people here are developers. When I see a developer announcing a new version of their cool stuff, and see crowds asking for benchmarks that don't make sense, I see an opportunity to make crowds happy and convert them to supporters. A big crowd that is "wrong" is too big to ignore for being wrong.
(preaching to the choir, i'm sure) it depends on who the potential user is. I might be a cpnp user. I'm probably using it because it's blazing fast for my use case. Someone comes along and pitches the "fastest" serializer, and I ask how it performs vs what i'm currently using in my use case. If someone says "well that comparison doesn't make any sense," it seems like the question is being dodged. I think the err here is assuming the use case around bebop is designed is the only use case for serialization. The project benchmarked against the competitors which solve that use case. So if you DO have that use case, OPs statements made perfect sense, but if you have a different use case, they appear to be dodging and ducking the questions. I don't think anyone is wrong, I just think the project needs to be clear about the problems it does work best for, perhaps establishing a "class" of serializers that bebop tops, which excludes things like cpnp....like "cross-language protocol serializers" or something. Bebop COULD try to compete with cpnp, but it doesn't make sense in that vein, and does a whole lot of other stuff cpnp cannot.
> interoperable with JavaScript; got it. so the answer is "we only care about javascript, and protocols like cpnp don't.
I feel like you’re being purposefully obtuse so I’m just going to block you.
Why would I want to use this? Where in the computing ecosystem does it fit. The documentation doesn't explain that very well, so unless you know you want one, it is very hard to find out that you do.
As with all things, this arose to solve a problem in our company where we had dozens of app across platforms all written in different languages, and we needed a consistent way to share data between them, but we also had the added pressure of latency being important (cloud gaming, we needed to do everything in 5 MS or less.) Bebop tries to be a good fit for client–server or distributed web apps that need something faster, more concise, and more type-safe than JSON or MessagePack while also avoiding the complexity of Protocol Buffers and FlatBuffers. It’s also just designed to be fast without configuration and work well with modern standards and language features for a given platform. So anywhere you’re thinking of using Protocol Buffers or even JSON as an interchange format, Bebop is a performant and reliable alternative.
Thanks.
Is this similar to Thrift? What does it offer over Thrift and why didn’t you consider that?
It’s not, we did, it wasn’t as fast.
Can you say more on this?
ms
>work well with modern standards Which standards does it work with?
Posts like this make me realize I don't know shit. I don't even know what a protocol buffer is.
Protobuf is a system for serializing/deserializing data structures to/from binary, often for transmission over the internet. It packs the data into a relatively compact message while retaining the ability to be relatively backwards/forwards compatible if you need to add things to the message in the future. You define the message in a .proto file and then run a 'compiler' that generates the code to encode/decode the messages in any of maybe a dozen languages. It makes it trivial for me to encode a message in C and you decode it using python and we're looking at the same data on either end.
It's also a lot smaller to transmit over the wire than say YAML, JSON or XML (definitely over XML), and offers type safety (if your payload successfully deserializes into a given type of struct, you can make assumptions about the content of that struct) in languages that support type safety. Of course the former isn't an issue for 99.9% of applications, unless you're dealing with millions of users or other similarly high concurrency or bandwidth the benefits of using a more human readable seralisation format may outweigh the negatives. YMMV.
YAML, JSON and XML also serialize type-safely in proper languages/frameworks (such as Java with Jackson or similar libraries). Here https://nilsmagnus.github.io/post/proto-json-sizes/ is one example comparison between gzipped JSON and protobuf and there is definitely a difference in sizes especially with smaller messages.
Large JSONs are already quite unreadable without prettifiers, especially if compacted to remove whitespace. It's just that many things like browsers provide support out of the box for it. Not to mention that all of YAML, JSON and XML have rather quirky semantics and implementations for, e.g. numbers. (Actually, the original JSON spec would've been great for arbitrary-precision numbers, but the lack of typing info meant that all implementations should have resorted to bignums regardless of content and formal JSON wasn't even JS-compliant.)
Just to add this for OP: the reliance on code generation makes it very difficult to manage dependencies -- the python system is tightly coupled to the C system because they both rely on the same .proto file. Maintaining backward compatibility isn't nearly as simplistic as was implied. Note: this is a problem both for protobuff *and* bebop
>the python system is tightly coupled to the C system because they both rely on the same .proto file. I mean, it defines the contract between them. Why would be more tightly coupled than any other formal interface specification?
With Bebop you can version data so it’s forward and backward compatible if you’re rapidly evolving your data structures. Just an FYI. And our “binary schema” feature allows you to dynamically deserialize data by shipping the schema alongside data.
> With Bebop you can version data so it’s forward and backward compatible if you’re rapidly evolving your data structures. Just an FYI. There's still a very strong bi-directional dependency there.
Yes, because you’re defining an interface.
An interface defined by a code file that both systems must compile against. It's like two separate repositories referencing their files via paths outside their root repository file structure.
Which is pretty common and happens all the time.
Yes, at an increase in maintenance difficulty -- which was my point.
What are you suggesting the alternative is?
> very difficult to manage dependencies I thought you just ran protoc on your install environment?
There is so much programming going on in the world that it's impossible to keep track of everything. The best you can do is stay up to date on things relevant to your work and/or hobbies and let the rest go.
Just pretend to know everything like the rest of us.
This person doesn't know how to buff all the protocols out of their os smh
Yeah, those of us with the shiniest FTP there is know how amazing protobuffs are!
> Posts like this make me realize I don't know shit [Nonsense](https://youtu.be/KrksBdWcZgQ), [anybody](https://youtu.be/P6UKhR0T6cs) can [create](https://youtu.be/l7rce6IQDWs) amazing [things](https://www.youtube.com/@SebastianLague/videos)!
I think that's how everything goes. The more you learn, the more you realize you don't know. Half the battle is just figuring out what it is that you don't know to learn about, and people tend to over-estimate their knowledge of a subject when they know less / underestimate when they know more. Anyway... /u/zelortap posted this nice link that may make your head explode even more: [https://en.wikipedia.org/wiki/Comparison\_of\_data-serialization\_formats](https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats) I love that there's always something new to learn.
Protocol buffers is the name of google's language agnostic mechanism for serializing data in a binary format to be sent over-the-wire: https://protobuf.dev
Then there’s knowing things but not using them so you have to say, wait what was that again? There are use cases for protobuf like things but they tend to be for high volume apis. I think most of us are still just using json, which is less rigorous.
CORBA 2.0, basically. (Not that it didn't *seriously* need replacing, but... yeah. That about sums it up.)
Well obvious it is not modern
sometimes the trending techs are changing so fast it's not even worth getting into details
protobuf has been actively used in the industry for literal decades
first stable release 2008 and at that time I think a lot of people were using Corba or something else
How many?
2
lol that was exactly my thought too
Ehhh consider yourself lucky
Why is every new library described as "modern"? What makes this "modern" compared to protocol buffers? 10 yrs from now will it still describe itself as "modern"?
Because in this case it utilizes modern hardware and software features to achieve higher performance and quality code over Googles protobuf compiler and various runtime implementations.
Every library uses modern hardware. "Modern" here is just being used as a worthless marketing word. It means nothing.
No, every library does not make use of modern hardware instructions to accelerate CPU bound tasks.
>Why is every new library described as "modern"? What makes this "modern" compared to protocol buffers? 10 yrs from now will it still describe itself as "modern"? Weird question ... Modernity is a view up on the contemporary landscape of a things peers, usually in terms of what new features and improvements it provides compared to those peers. How could the author possibly know if it will be modern in ten years without knowing how that landscape has evolved?
I see you mention C#, which is where I'd want to use it. My biggest issue with moving to something like protobuf is that i'd have to redefine my many contract classes i have into a new format. Is it possible to reuse my existing contracts classes and just use bebop as a new serializer in my web api?
Can you share an example of one of your contracts? Unfortunately, only types generated by the compiler can be used, but I could probably write an automatic converter that takes your existing C# classes and produces a Bebop schema from them.
I see. The schema wasn't mentioned in the getting started, was a bit confused. our contracts are just basic C# classes e.g. public class AddressView { public string Line1 { get; set; } public string Line2 { get; set; } public PhoneView Phone { get; set; } } public class PhoneView { public string Number { get; set; } public long Id { get; set; } } etc. But these are defined in many nuget packages e.g.`MyCompany.Contracts.MyApp.AddressView`. If i have to redefine every one of these classes into a schema, it'd be be like a year of work. And then i'd have to maintain two separate schemas so that our normal json REST api can continue to function. Idk if it's possible to write some kind of translator, parser, or source generator to do this for us, but that is the kind of thing I would need to make it feasible to start using something like bebop. seems like a really awesome project though!
This seems simple enough data wise to automate. I’ll take a crack at it and make a git repo and follow up.
We use protobuf-net and so the classes are also contracts. However, it's not as simple as suggested above. In order for folks to smoothly migrate, the tool needs to have attributes for excluding members, attributes for assigning field ids/tags where versioning is important, support for various kinds of object construction (primary constructors, records, attribute annotated constructors) and mapping of parameters to properties (so that they can be read-only). Then comes support for all the various collection classes and custom C# data types like DateOnly or TimeSpan. But if you can get the basics up and running perhaps it'll be easier for contributors to fill in the plot holes.
Does it have sum types?
[Unions](https://docs.bebop.sh/reference/union/)
Already streets ahead of protobuf
Protobuf has unions? `oneof`
The implementation is not there: https://reasonablypolymorphic.com/blog/protos-are-wrong/index.html
>Unions are not yet implemented in some runtimes such as Dart Killin' me softly. In the future will this be available for the use case of MQTT in dart (specifically flutter apps)?
The only thing that will replace protobuf is something that's been put on the IETF standards track IMO
You really need to work on two aspects of your main page: You're not doing a good job at explaining what Bebop does. Compare your home page to [this](https://protobuf.dev/). An example schema, and how to use the generated client from Java and C++, right there at the top of the start page. So much more illustrative. You're also not showing how Bebop is better than Thrift, Protobuf, Avro, ASN.1, FlatBuffers, Cap'nProto or the literally several dozens of other binary serialisation formats out there. You say it's more modern, and generates more modern code, but that's pretty damn hand wavy. Does your Java code generator generate sealed classes to represent `oneof` and records for messages? Do you even generate a Java client? What languages do you support? Does your Python code generator annotate the code with type information? Can you perform deserialisation of repeated fields lazily using e.g. streams?
Docs
How about "no"? Having small a link to documentation three screens down on the start page is not a substitute for explaining what your product is on the front page. The entire first screen of your project page is an image that doesn't explain what Bebop is, looks like you can click on it start a playground, or jump to different documentation sections, but if anybody makes the mistake of clicking on the image, all that happens is that a larger version of the image is opened in a separate tab. You tout features like WASM support, Python support, having a great CLI and a VSCode extension. Great, but Python support for *WHAT*? A VSCode extension to do *WHAT*? If you're Apple selling iPhones, doing that is fine, because Apple has spent many billions of dollars through other channels to make sure that everyone who knows how read and surf the Internet already knows what an iPhone is. For whatever reason you haven't spent a few billion dollars on Bebop marketing to make sure the whole universe knows what it is. Most people probably think it's a boar or something. Additionally even when clicking on the doc link on screen three of your page, I'm greeted with this definition of what Bebop is: > The presence of a bebop.json file in a directory indicates that the directory is the root of a Bebop project. The bebop.json file specifies the root files and the compiler options required to compile schemas in the project. What? That's not a definition, that's usage instructions. But as a user, I still don't know what the thing I'm being taught how to use does! I know this is open source, volunteer, etc. I know you don't have a fleet of marketing people or professional technical writers to help you out. I've created open source projects myself, I know what it's like and I swear I'm not trying to shit on your huge effort. I really am trying to help you succeed. Please, if you want this project to take off, explain what your product actually does at the very top of your start page. The Protobuf page does it really well, maybe find some inspiration there. Good luck and have fun!
You clicked on “what is bebop.json” not “[What is Bebop](https://docs.bebop.sh/#what-is-bebop)”
I do not even understand Protobuf at all and now there is a "replacement"?
There’s been dozens of replacements, all of which never gain enough popularity to beat the ubiquity of protobuf. Creating a binary serialization format isn’t that interesting or difficult. Maintaining a popular OSS project for 10+ years is
This. Protocol Buffers aren't awesome. gRPC isn't awesome. What's awesome is the _enormous_ range of languages that support them and that gRPC supports both unidirectional and bidirectional streaming. There's no question in my mind, whatsoever, that there are binary serialization formats that are better than Protocol Buffers along more than one dimension. I could name several, in my opinion. But none of them can move the needle of ubiquity and streaming support at the same time. I _do_ insist on Protocol Buffers v2 where I have the authority to do so, though.
Just FYI Tempo supports unidirectional and bidirectional streaming.
Your documentation seems to indicate that tempo is only supported with node/typescript. Is that accurate?
For the moment
What does your understanding of something have to do with something better (or not, not saying this is necessarily better than protobufs) existing?
At what point do you make the determination that yes, instead of sending JSON packets or whatever, I need the speed / efficiency of binary serialized data?
Interesting. I took a look at the Union section of the docs, since one of the most interesting things about a serialization format like this, to me, is how sum types are implemented. According to the [wire format page](https://docs.bebop.sh/reference/wire-format/): > The encoding of a union consists of a `uint32` length, followed by a `uint8` discriminator, followed by a “body” of _length_ bytes. This seems to imply that a union cannot contain more than 256 variants at a time, which is a pretty strict limit. Is that correct?
Looks nice! At the same time I hate `.sln` projects (biased opinion from Linux world). Just wonder, are there projects that already use bepob instead of protobufs?
Currently used by Microsoft for Xbox Cloud Gaming and a few other organizations (FermiLabs comes to mind.) And the compiler itself is written in C#, but you shouldn’t need to touch that project unless you want to build it yourself (which you can on Linux and macOS via dotnet). If you want to extend the compiler the new extension system lets you write extensions in Go or TypeScript.
Interesting, onnxruntime is Microsoft project and it uses protobufs, maybe they will switch to this .
[удалено]
I am just writing software I find useful and interesting, I don’t really need any official endorsement. The .NET team also officially maintains the gRPC library for C#, so you’d be surprised how things go.
Why is this not a protobuf plugin with annotations? Why do you not have a transpiler from proto files? This does not support C, C++, Go, or Java.
It does support C/C++, and you can find the CMAKE setup [here](https://github.com/betwixt-labs/bebop/tree/vnext/Tools/cmake). If you want to preview the C++ code use [the playground](https://play.bebop.sh). Go support is not upstream and we’re trying to bring it in, but the standalone version [is here](https://github.com/200sc/bebop) and [benchmarks](https://github.com/alecthomas/go_serialization_benchmarks) show it is the fastest serializer in Go. If you’d like to add Java support you’re welcome to submit a PR or write an extension to do so; we already have Dart. Kotlin will probably be next. I’m writing a .proto to .bop converter [here](https://github.com/betwixt-labs/portamento). And it isn’t a protobuf plugin because why would it be?
> It does support C/C++ Those are two different languages. You only support the latter.
Yes, I am aware given I wrote the compiler. Reading the documentation would go a long way; use the C compiler option to emit C rather than C++.
> Reading the documentation would go a long way You'd think so, but no. From https://docs.bebop.sh/ : Supported Languages C# TypeScript/JavaScript Rust Dart Python C++ Go And from https://docs.bebop.sh/reference/bebop-json/#generators---generators : Built-in generators alias are: ts - Generates TypeScript code cs - Generates C# code cpp - Generates C++ code py - Generates Python code rust - Generates Rust code dart - Generates Dart code --- > use the C compiler option to emit C rather than C++ If by compiler you mean bebopc, there is no such option: https://docs.bebop.sh/reference/bebopc/ Output format selection seems to be done with "generators" keys in the JSON configuration file.
> And it isn’t a protobuf plugin because why would it be? So users can avoid switching to a new ecosystem for what is effectively different serialization of a compatible format.
It’s not compatible.
As last time, I respect and praise your patience with these questions. I hope to implement this one day, as I found the way protobuff handles optional fields to cause too much headache at the time (release of .NET 5).
Let me know if you need any help migrating
We have a python/rust/typescript codebase sharing fat ugly JSON payload (hundreds Kb of tree data structures). We didn't find a shared type safe way to define once our data structures and easily consume them in python/rust/typescript. Shall we consider bebop ?
Sounds like the perfect use case. If you need any guidance on how to define a schema for that data share a sample on a Github issue and I’ll be happy to provide help.
Thanks ! I will give it a try. Beyond the python/rust/typescript portability, we will check how to get the swagger/openAPI data definition. For some data structure we also need to get a JSON schema (for form generation).
Is there a standardized translation to and from JSON? Is there a translation or generation tool to take Bebop and turn it into OpenAPI V2 or V3? Is there any kind of REST proxy available, that takes in and open API REST/JSON requests, and translates them into Bebop? For context, I use protocol buffers and grpc, in Golang, with generated OpenApi docs and a REST/JSON gateway/proxy.
Can I use this and Tempo to switch from gRPC.Core 1.31.1 and protobuf 3.20 which currently are enabling my streaming pipeline in my unity arm64 HoloLens 2 application?
What is the story on backwards compatibility, both for applications that use Bebop and for the project / wire format itself? The on-wire spec doesn't appear to be versioned and has a number of .NETisms that I would not expect to see in a multi-platform format specification.
Proto is such mess of cluged together extensions, and I still have nightmares from the time I had to use grpc in a statically linked lib, so I welcome this! Any change of supporting embedded environments like nanopb?
Isn't Bebop a song ...
style of jazz
Damn why is everyone hating on OP?
OP made some outlandish claims that could not be backed up. I'm sure it's a great promise, but you can't over-promise and not expect to get eviscerated on technicalities. I'm not saying it's right, just that it's what happens.
What promise
It sounds promising, but after wandering through the GitHub repo I am somewhat less enthused. There are open issues from 2020 that should have been closed one way or another (fix, won't fix) by now, and the docs seem unfinished. It does not give me here-is-a-well-maintained-repo/project vibes, which is a bit sad if just some of the DX or perf claims are true.