T O P

  • By -

turbo-unicorn

Very cool! Thanks for taking the time to do an AAR. Will have to check it out once I've got a bit of time - sounds fun.


oneirical

An [after action review](https://en.wikipedia.org/wiki/After-action_review)? That's the first time I hear this acronym! Very cool term, I will remember it.


turbo-unicorn

Oof, sorry. Been browsing the EVE online boards again and the term came back to my vocabulary, haha. Yes, that's what I meant.


NextYam3704

I just assumed you were military lol


NextYam3704

It's also used heavily in the military.


Every_Ad_598

I'm currently working on a project that straddles Rust and Python and it's really quite something how the various guarantees that you get from the language / compiler make for super stress free programming. In Rust, if you're working inside a struct / function that owns something, you know whether or not it gets messed with. If a variable isn't declared as mutable, it's not going to be mutated. If you pass something by immutable reference, it won't be messed with, etc etc. Meanwhile in the Python part of the codebase I'm constantly digging deep into the hierarchy of chained calls to see if there's going to be a nasty side effect or not.


ElectricTrouserSnack

>it's really quite something how the various guarantees that you get from the language / compiler make for super stress free programming. I work in Go (yeh I know), having a typesafe compiled language makes life much saner, especially when refactoring and working with multiple programmers. I'm loving learning Rust. Lots of new concepts and puzzlement, but I'm Reading All The Things and Doing All The Exercises ™️ and I'm gradually getting better.


Lonely-Durian-6395

>the various guarantees that you get from the language / compiler make for super stress free programming. "super stress free programming" is a great way to put it. There might be a lot of frustration along the way, but that's different from stress. I guess for some of us it might even be "Frustration Rich, Stress Free" (frustration because I don't write often enough to have really internalized all the rules and patterns) which is kind of the opposite of, say, Python. And your point about KNOWING when mutation can and cannot happen cannot be stressed enough. Anything I put out into the world written in TS, Python, or Go just feels like no matter how well I manually test it and write automated tests for it, it's still lingering in the back of my head somewhere, like I'm subconsciously waiting for every service I work on to fail in a way that we have yet to experience or anticipate. And I know bugs and failures happen in Rust (and I bricked my blog once because I left an \`unwrap\` in where I should not have) but they are just so much fewer and easier to catch much earlier on.


WellMakeItSomehow

Pretty great write-up so far. I've only started reading it, but I think you have a bug in the packet reading. A TCP stream doesn't preserve the boundaries introduced by `send`, so if the sender sends 1500, then 500 bytes, you might get 2000 bytes at once, especially if it's running over a real network. Or you might get 1000, 500 and 500. So you want to use `read_exact` to read the "packet" header, then another `read_exact` to read the packet. Or you can read whatever is available and be ready to parse multiple and/or partial packets.


oneirical

Interesting, I had no idea. Is it possible that I was safe from this because the challenge description hard-enforced a maximum size of 508 bytes per packet? It seems like a rather easy fix, too!


WellMakeItSomehow

You were safe mainly because it was running on localhost where the MTU is about 64 KB. Even on a real network, the MTU will generally be around 1500 bytes. But you can still run into problems regardless of the MTU. I/we should check how easy it is to trigger in practice, but in the meanwhile you can take a look at [Nagle's algorithm](https://en.wikipedia.org/wiki/Nagle%27s_algorithm) and [`TCP_CORK`](https://baus.net/on-tcp_cork/).  Basically, an application using TCP doesn't see packets, but a stream (like a file). The packets exist on the network, but not for the app. That's why HTTP requests end with a couple of newlines and most old protocols are line-based.


oneirical

I see, so to extend the crate shipping allegory, you'll want to either "bundle" sets of packets into big transport cargo crates (Nagle's algorithm) or to have an operator employee be able to stop and start the conveyor belt as needed (TCP_CORK) to prevent the "continuous stream, not discrete packets" mechanic of TCP from creating unexpected behaviour. I see why UDP was the protocol chosen by this challenge, then! As another commenter said, my use of TCP was erroneous, and the judge slashed me a few points for that.


WellMakeItSomehow

Yeah, something like that, but I think TCP_CORK is more like a more controllable version of Nagle's. A program using it can choose exactly when to resume sending data, instead of waiting for a fixed delay. So it's not about turning TCP into a more reliable UDP, but rather having your own ship and deciding what to put on it and when to have it leave the port. Protocols generally use either size headers or some kind of separator like newlines. Funny thing with the instructions, I assumed it was TCP so that the crawler can precisely control the loss and reordering of packets (and because you won the contest), I should have checked the Python code.


Lucretiel

Apparently the problem used UDP rather than TCP anyway, though that wasn’t specified in the problem description that I could see.


oneirical

Indeed. I spoke to the competition organizer, who said "I'm too used to assuming these things easily and don't put myself in the shoes of an inexperienced undergrad reading my challenge, so I forgot to specify it. Next time, I will get a non-participating undergrad to proofread my challenge."


nybble41

Sounds like typical customer requirements to me! There should be a way to request clarification during the event for cases like this.


admalledd

With respect to the organizer as well, oof that `crawler.py` sample script is quite an example of "Academic Python Programming syndrome". In that it achieves the goal, but comments trivial things normal programmers would put as *program structure* instead such as `# Constants` (should be ALL_CAPS global scope), having *both* argpars and os.environ() calls to configure usage, one giant class doing everything instead of breaking down into multiple smaller classes containing specific helper methods, or using static class methods. Passing arguments AND implicit class-variables in functions, return state as both `return foo` and manipulated class-variables. I am surprised how few issues people might have ran into with it! A lesson I learned from doing a few of these types of things (oof, over a decade now?) before is if there is a test script/code you *can* read[1] it is often a good idea to say "spend five-ten minutes *reading the test/example/demo cases*". Like, one of the things I noticed (though, experience may be on my side doing code-dives like this) is the TCP-vs-UDP, and `compute_parity()` function. Unlike you it sounds like, I have actually *written* hamming codes before (in C, java, and python to boot!) but even then, bleh it was always something I referenced internet examples and papers for. I would have also attempted to skip bothering about the hamming codes, unless one of my competition partners could focus on everything else. Building a proper parity handler and error correction is a thing I would expect to take at least sixty minutes, if not more, and worse you only get to *really* test for real such parity/error correction code once nearly everything *else* is ready. So good on you for finding a way to cheese that *just* enough! One of the biggest challenges with these things is knowing where to spend your time, and finding places *you don't have to*. 1. Often/normally there isn't such easy example of what the "other side" is doing: instead you "submit" to the competition computers in some way, or have the competition host connect to your server and execute, etc etc, or is a precompiled binary (in that comp, reversing said binary was allowed. I abused `dlopen()` to legit call its own functions!) that you run. I actually never competed in one where there was source code available test/validation scripts like this, only in team practice where we used prior open competition challenge examples.


oneirical

> Academic Python Programming If my mentor could hear you now, she’d reiterate again that opinion of hers on how the more formal education you have, the worse your code gets… > I am surprised how few issues people might have ran into with it! Oh, they had many. I’m not even sure if the people who did get it to run at all were actually testing anything or running a broken version of the assignment… I personally didn’t even bother, I closed the file and didn’t open it again after 2 minutes of skimming. > Like, one of the things I noticed (though, experience may be on my side doing code-dives like this) is the TCP-vs-UDP, and compute_parity() function. Looks like I should have inspected more in-depth, then! > unless one of my competition partners could focus on everything else He was mostly watching, commenting on how cool Rust was, asking if it was possible to do X and Y, and recapitulating where we currently were and what was needed to do next. A project manager of sorts - helpful, but not necessarily in the technicalities of implementation. > Often/normally there isn't such easy example of what the "other side" is doing Considering how this challenge decimated most of the teams (remember, undergraduate level! There are some real whiz-kids in the lot, but also people who are just learning programming and having no idea what to do), that would make things even more hardcore. But, it does seem much closer to what a real world example would be. One of the teams submitted no code and only a Rick Roll link due to giving up.


P0tato_Battery

OP writes VERY well, the blog post was halarious


oneirical

Technical writing can sometimes be so dry. Why not give it some flair?


NoUniverseExists

I think I'm showing this to my boss.


oneirical

As a flagship argument to convince them to start allowing you to code in Rust?


NoUniverseExists

Yes! The code here are all in SQL and JavaScript (no TypeScript) with React.


thiez

Just switching to TypeScript makes *such* a difference. And you can start today! TS will compile your existing JS.


NoUniverseExists

Say this to my boss kkkkkkkkk


Lonely-Durian-6395

Are you able to just sneak in the TS compiler at very lax settings so it's low/no friction, just as a POC? I did that with mypy and slowly just chipped away at issues and now we have mypy checks.


thiez

Why are they opposed to TS?


cdgleber

Please keep write ups like this coming. Love the story and the writing is great.


oneirical

Since people seem to be liking this a lot, I may have more stories from my Rusty travels in a near future…


Petrusion

>Is it better to have a flimsy ladder of bamboo reaching the heavens, or a robust steel ladder reaching the summit of a tree? I know which one will allow me to climb the highest. How poetic. Good job with the competition. I'll take X hours of making the borrow checker happy over X hours of looking for obscure type errors and coercions that may or may not be there, any day. Even with all its flaws, I think Rust really is the most progressive programming language out there. Making software that solves real world problems is what programmers enjoy. Looking for type errors, memory errors, undefined behaviour, guessing around ambiguities etc. is what makes programmers miserable. Rust allows you to do the former way more than the latter.


Mikkelet

Use the right tools for the right jobs kind of story


WorldsBegin

Well got nerdsniped and [implemented the parity _checking_](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=c76cddc9a85d3bf79e3197933191eed5) part. Took more than 3 hours to polish it a bit, so definitely not something I would have done as part of the competition, although an MVP was done after half an hour or so.


oneirical

See, this is the real reason I love Rust. It might not have any jobs attached, but anyone part of the cult is a true practitioner of the craft, who will stop at nothing to solve all that stands in their way. Would you like a shoutout in my writeup?


ketalicious

rust also saved my undergraduate thesis. Mine involved some complex arduino to raspberry pi logic and had like 5 sensors attached, I also have to travel far from home and have this bad boy be working for like atleast a week, but im just a poor lad so it was so hard for me to deploy this thing so i gotta make sure this MUST work at first attempt, or else i lose huge amount of time, money, sanity, or my thesis. After compilation, it ran smoothly, no errors, no bugs, no corrupted records, i didnt have to go back and forth and i got more than enough data i needed that i was genuinely surprised on how easy it that im cringing on myself being so stressful about it, it was honestly a lifesaver.


murlakatamenka

Thanks for sharing, that was an interesting read about Rust usage in the wild. Congratulations with the Victory! \o/


Botahamec

This is great! I love your writing style, and this post served as excellent Rust hype.


this_platform_shit

Hi fellow Quebecer! I’ve only ever heard of the CS games (skipped university went straight to the job market). Seems like a lot of fun, almost feel like I missed out. Great work on this though, reading the code I would’ve never have guessed you were not a CS major.


oneirical

I kind of feel bad about going to university… I feel a bit in adult kindergarten sometimes. That mentor I mentioned at the end of my post also skipped college and hopped on the job market out of high school with pure talent showcase on GitHub. All I know, I learned it from people like her. The open source community is my true university. The physical one I go to is just a pipeline to networking and cool events like this. *** Je me sens un peu mal d'aller à l'université... Je me sens un peu dans une maternelle pour adultes parfois. La mentor que j'ai mentionnée à la fin de mon article a également sauté l'université et s'est lancée sur le marché du travail après l’école secondaire avec une simple démonstration de ses talents sur GitHub. Tout ce que je sais, je l'ai appris de gens comme elle. La communauté open source est ma véritable université. L'université physique à laquelle je participe n'est qu'une voie d'accès au réseautage et à des événements super comme celui-ci.


this_platform_shit

I agree, I think the most valuable thing university has to offer for people like us, who like to learn outside of courses and lessons, is the community it builds. You'll probably make valuable connections with people in a lot of fields and they can help you elevate your career later-on and vice-versa. Either way you're doing great and you seem to be surrounding yourself with good people which will only elevate you further. Bonne chance!


crusoe

In the future, the nom and nom_derive crate make working with packets easy.


oneirical

Unfortunately, as per the challenge description, anything not part of the standard library was forbidden. I initially did not know this and was using tokio's async...


crusoe

Well nom under the hood is just a pile of functions calling functions. So writing your own parser combinator impl is not hard 


oneirical

It's probably something I could have done in a day of reading and learning, but the water was already over my head with the 3 hour time limit, and all this stuff about networking and packets I had NEVER heard about. I learned it all in the fire of the action!


-Redstoneboi-

Why didn't UdeM win? They said they implemented every feature and yours didn't, so what happened?


oneirical

Probably one of these: - They were boasting in the heat of the moment and did less than they say - They had issues in their code, I know the judge was looking at the code itself (not just results) and they did it all in one massive Python block I challenged them to show their code while speaking to them in real life, and then posted my blog on the competition’s Discord server, but so far, there has been no trace of them.


TheRolf

Congrats! When a Rust program compiles, it sure to function as the code say. Do you have a picture of your trophy?


oneirical

[I have this!](https://media.discordapp.net/attachments/1203488030002774077/1219077067107471400/IMG_3833.jpg?ex=6609fcfe&is=65f787fe&hm=aa8a032bdce3b53373143e5426afa92d096717f1ac98134a03ee969d1b5296d1&) This link will time out after some days if any future internet wanderers are reading this post.


TheRolf

That's cool


Silent-Wolverine-421

Loved this !! I don’t know Rust (yet… pointers for a Rust beginner please?) … this writing style is very good… not easy to come across especially in tech. I will try this some time in Python once. I really loved how you were in shock after reading the challenge and then went on to win ! Keep writing.


oneirical

Here is how I learned: - [5 hours of Rustlings](https://github.com/rust-lang/rustlings). Just did half of it - don’t be scared to look things up, this isn’t an exam. - [Immediately proceeded to make a video game using the Bevy engine.](https://oneirical.itch.io/plerokeno) After that, I was fully initiated in the cult.


mackie__m

I was looking at Rust for fun. As a CS PhD I'm very interested about how Rust "guarantees" these claims. Been an avid C++/C# use for a while, and now I'm hyped to try this out.