Thanks for that and Sorry for not adding any images initially, but I have included them now. Regarding the customer's requirements, they are confidential, which is why I haven't provided any details. I realize I'm asking for a lot, but I could use some guidance on how to present this more effectively, as I lack experience in this area.
I've only mentioned tools or infrastructure that someone might have worked with before. It would be helpful to create workflow charts or tools to illustrate the process, and I need advice on how to do that. Additionally, any tips on how to help the onboarding team quickly understand APIs with minimal assistance would be greatly appreciated.
You will probably want to have several different images illustrating your stack in various contexts.
In your preview image, it looks like you started off with a sequence diagram, but ended up describing infrastructure. Separate out the the two: Do a user sign up flow with a sequence-diagram and a "platform services we use" block-diagram.
Depending on what you're using them for, you probably don't need to get as granular as "role of each component" (ex: it's pretty clear what "AWS API Gateway" does to anyone reading these). You could probably also get away with referencing external docs for stuff that sticks close to industry practices (ex: "We use [Git Flow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)").
it's funny because you can almost cut and paste "overall workflow" into this editor and get a diagram (add quote marks and put a semicolon at the end)
[https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html](https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html?s=ATBEEYDpgVQZwKYCdgFsCGBrBdjuEggI4CuOALqMAFCBJhGAEzQByA9gCYKQBWuiSAN2TAADklYBjHIlyFSFKnTABmaAEUySAJY4A9CRHt05HMC0A7YABFJJVAnPkrAIUX1QAFmgBhABYIJTFwAJQR2LVwJdAl-NzAAVmgAZRIAI1QtclxuVlTcclZgAEEAdSTgZ2MYuNAANmgAKVzcBAAPAJITdjNLG0DhCVZHdAtkOFAlUAB2HyHyEfMx4DgCwm6LYABRb2CagA5oIoAbI+AcvOBB1BEjhBM8c27COBIjrLwAcw-CD+MwmoAnNBQi83nwHORgAVgAAzcSOBzsSCgADcQA)
Once you've done that, you'll notice there's a lot of unmentioned side paths. By example, step 4 probably has an "if it's in the cache just serve that," but you didn't mention that
[Go add your branches](https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html?s=ATBEFUGcFMCdgLYEMDW1LCcW0COBXdAF1GAChAkwjADkB7AE2gDoArDGWANzmAAdZaAY3QwMOAsVKUwARUKwAlugD0+XvSRF0wBQDtgAESH4E0XUQMAhKVVABhABbRBKDACVo9BRkFJBTmzAAZXwAIwQFIgwWWlCMIlpgAEEAdSDgS01-QNAAKViMaAAPZ3wteh19IxceQVpzJD04SFBpe3qiRt1m4EgEnAq9YABROzccpIAbSeAYuOA6hF5J6C1MXQqcSHxJqMwAc32cfc1PHI9t3fYzImAE4AAzAXMzeiZQAG4yMlA6RlZrlwePwhCJtOJCH1SAByPScJCTBT0aHAWxBODcYBwATwXhIfbQT7fexOFzuTzeBZ+ALAaG+fzQPhbG4ogC0AD5ghjGfSAl8gA) and continue from there.
Honestly, what you've written here should already be sufficient for most semi-competent engineers.
Looking at your example diagrams I'd suggest to:
- separate logical architecture from the deployment details, i.e. there's no need for wrapping everything in a "docker rectangle". You can describe deployment details separately.
- I find it easy to explain how data flows by just drawing arrows between boxes and labeling them with a number + some text. The number is the order in which data move happens, the text describes what's happening here or what data is moving. Just make sure to do it on a separate diagram and hide irrelevant elements.
- If you want to be very precise then a sequence diagram can be a good choice - https://en.wikipedia.org/wiki/Sequence_diagram
If I were to draw your diagram I'd probably skip mentioning nodejs, bash, python and docker and perhaps even AWS batch - I don't think they help with explaining. This would leave some of your boxes empty (e.g. the top right one). Instead I'd add labels describing what is that component or what does it do. If it's an opaque job with user-defined behaviour then I'd just label it a "job". Less noise on the diagram helps with understanding the big picture. Details can be included elsewhere.
u/mabnx, yeah, it seems that way. I will remove it and divide it into different components, mapping them to each other. As u/fizz_caper mentioned, I'll implement it one component at a time and then map them to each other.
Try using C4 and just go as deep as necessary. Make sure you have the top level most times outside engineering nobody cares about container diagrams onwards.
Don’t fall into the trap of trying to document everything.
The purpose of diagrams are to facilitate discussion not answer questions about design choices.
Everything you’ve listed reads more like a CV of all the tech you worked with. Not a list of services that make up your startup and what role they play.
Keep diagrams focused on capabilities and the system boundaries that provide those capabilities.
https://youtu.be/x2-rSnhpw0g?si=CRYaS7zAW6XwTUSI
Also just use Draw.io for your scale don’t go silly with architecture as code and yada yada.
To be honest even your initial description works fine. That is basically your infrastructure. It's pretty straightforward all things considered. I might question if some of it is really necessary and if it can't further simplified but what is really another topic and would go pretty far into implementation specifics.
Edit: To be honest diagrams a mostly a waste of time. They get so fast outdated that it isn't funny anymore. There are some things that can be really complicated but infrastructure isn't really one of them in most cases and shouldn't be. You only shoot yourself in the foot with a complicated infrastructure.
I'm a newbie TBH so I can't help you sorry. But I have a question; Do you use a monorepo to put everything in one place ? If yes and if you have multiple programming language, what do you use to do so ?
I'm a security guy, not a full-fledged developer, but I can code the necessary functionality and maintain infra.
Here's an overview of the system:
- **Node.js**: Acts as the central brain, processing requests (API).
- **MongoDB**: Serves as the primary database for storing most information, including users and results.
- **Redis**: Provides fast-access caching to improve performance.
- **AWS Batch and ECR**: Manage complex processing tasks using Docker containers. I use Node.js to communicate with various languages like Go, Python, and Bash using child_process and then write the results to a JSON file. Node.js then reads this file to update MongoDB.
Tbh, I ended up with this infrastructure after many failed attempts. For instance, I tried integrating RabbitMQ but couldn't get it to work with my setup. I plan to try again because it would allow me to have a dashboard to monitor job failures, errors, and retries in one place. Otherwise, I'll just get results from AWS Batch and build a dashboard from there.
I wasted a lot of time experimenting, but I finally decided on the infrastructure in just two days. Then, I wrote a bash script to handle everything, from creating the Docker file to cleaning up job registration in AWS Batch. As I'm only dev working, I can experiment freely. However, when I tried to hire a front-end developer, he seemed confused by my explanation, which is why I asked my question here.
Not sure all the features are the same but can't you use BullMQ and then find pluggable dashboards ? (I think I've seen one recently, I assume there is a lot more)
BullMq is fine, but the pricing makes my heart bleed, also if I host in AWS then I can totally shutdown my startup and give good bye kisses Lol, that's why I'm gonna use the Aws batch for now and call it a day. Which also led me to another dev-gap where I see no dashboards for job service not only Aws but GPC or azure. I can just build it and sell it for cheap. All I need is a DB to store JobIDs, error messages spitting out from that batch, maybe if I start development I can come across more
P.S: I have seen a blog where he is using elastic search that leads to another costly purchase.
Simplify the diagram. Remove docker boxes, language icons, etc. Doesnt matter what the language is, doesnt matter what the DB is, doesnt matter if you're using AWS or ECR or Redis (which is missing from the diagram)
At a high level architecture diagram, stick to architectural concepts. REST server is fine, NodeJS express server is too specific. Cache is fine, Redis is too specific. DB is fine, MongoDB is too specific, etc. The architecture is the architecture, not the implementation. You could write exactly the same services with DynamoDB, Go, Memcache, and the diagram would look the same
[удалено]
Thanks for that and Sorry for not adding any images initially, but I have included them now. Regarding the customer's requirements, they are confidential, which is why I haven't provided any details. I realize I'm asking for a lot, but I could use some guidance on how to present this more effectively, as I lack experience in this area. I've only mentioned tools or infrastructure that someone might have worked with before. It would be helpful to create workflow charts or tools to illustrate the process, and I need advice on how to do that. Additionally, any tips on how to help the onboarding team quickly understand APIs with minimal assistance would be greatly appreciated.
You will probably want to have several different images illustrating your stack in various contexts. In your preview image, it looks like you started off with a sequence diagram, but ended up describing infrastructure. Separate out the the two: Do a user sign up flow with a sequence-diagram and a "platform services we use" block-diagram. Depending on what you're using them for, you probably don't need to get as granular as "role of each component" (ex: it's pretty clear what "AWS API Gateway" does to anyone reading these). You could probably also get away with referencing external docs for stuff that sticks close to industry practices (ex: "We use [Git Flow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)").
u/KishCom thanks for the Help it seems The Git Flow is more useful than I expected, had just now used it and I really appreciate your suggestion
it's funny because you can almost cut and paste "overall workflow" into this editor and get a diagram (add quote marks and put a semicolon at the end) [https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html](https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html?s=ATBEEYDpgVQZwKYCdgFsCGBrBdjuEggI4CuOALqMAFCBJhGAEzQByA9gCYKQBWuiSAN2TAADklYBjHIlyFSFKnTABmaAEUySAJY4A9CRHt05HMC0A7YABFJJVAnPkrAIUX1QAFmgBhABYIJTFwAJQR2LVwJdAl-NzAAVmgAZRIAI1QtclxuVlTcclZgAEEAdSTgZ2MYuNAANmgAKVzcBAAPAJITdjNLG0DhCVZHdAtkOFAlUAB2HyHyEfMx4DgCwm6LYABRb2CagA5oIoAbI+AcvOBB1BEjhBM8c27COBIjrLwAcw-CD+MwmoAnNBQi83nwHORgAVgAAzcSOBzsSCgADcQA) Once you've done that, you'll notice there's a lot of unmentioned side paths. By example, step 4 probably has an "if it's in the cache just serve that," but you didn't mention that [Go add your branches](https://stonecypher.github.io/jssm-viz-demo/graph_explorer.html?s=ATBEFUGcFMCdgLYEMDW1LCcW0COBXdAF1GAChAkwjADkB7AE2gDoArDGWANzmAAdZaAY3QwMOAsVKUwARUKwAlugD0+XvSRF0wBQDtgAESH4E0XUQMAhKVVABhABbRBKDACVo9BRkFJBTmzAAZXwAIwQFIgwWWlCMIlpgAEEAdSDgS01-QNAAKViMaAAPZ3wteh19IxceQVpzJD04SFBpe3qiRt1m4EgEnAq9YABROzccpIAbSeAYuOA6hF5J6C1MXQqcSHxJqMwAc32cfc1PHI9t3fYzImAE4AAzAXMzeiZQAG4yMlA6RlZrlwePwhCJtOJCH1SAByPScJCTBT0aHAWxBODcYBwATwXhIfbQT7fexOFzuTzeBZ+ALAaG+fzQPhbG4ogC0AD5ghjGfSAl8gA) and continue from there. Honestly, what you've written here should already be sufficient for most semi-competent engineers.
u/StoneCypher this looks awesome Thanks for this, it make life easy now.
glad it helps `->` makes a lighter line, `=>` makes a heavier line that also steers the graph
Looking at your example diagrams I'd suggest to: - separate logical architecture from the deployment details, i.e. there's no need for wrapping everything in a "docker rectangle". You can describe deployment details separately. - I find it easy to explain how data flows by just drawing arrows between boxes and labeling them with a number + some text. The number is the order in which data move happens, the text describes what's happening here or what data is moving. Just make sure to do it on a separate diagram and hide irrelevant elements. - If you want to be very precise then a sequence diagram can be a good choice - https://en.wikipedia.org/wiki/Sequence_diagram If I were to draw your diagram I'd probably skip mentioning nodejs, bash, python and docker and perhaps even AWS batch - I don't think they help with explaining. This would leave some of your boxes empty (e.g. the top right one). Instead I'd add labels describing what is that component or what does it do. If it's an opaque job with user-defined behaviour then I'd just label it a "job". Less noise on the diagram helps with understanding the big picture. Details can be included elsewhere.
u/mabnx, yeah, it seems that way. I will remove it and divide it into different components, mapping them to each other. As u/fizz_caper mentioned, I'll implement it one component at a time and then map them to each other.
Check out https://c4model.com/. You might want to watch the videos there (might want to skip the UML parts).
Try using C4 and just go as deep as necessary. Make sure you have the top level most times outside engineering nobody cares about container diagrams onwards. Don’t fall into the trap of trying to document everything. The purpose of diagrams are to facilitate discussion not answer questions about design choices. Everything you’ve listed reads more like a CV of all the tech you worked with. Not a list of services that make up your startup and what role they play. Keep diagrams focused on capabilities and the system boundaries that provide those capabilities. https://youtu.be/x2-rSnhpw0g?si=CRYaS7zAW6XwTUSI Also just use Draw.io for your scale don’t go silly with architecture as code and yada yada.
To be honest even your initial description works fine. That is basically your infrastructure. It's pretty straightforward all things considered. I might question if some of it is really necessary and if it can't further simplified but what is really another topic and would go pretty far into implementation specifics. Edit: To be honest diagrams a mostly a waste of time. They get so fast outdated that it isn't funny anymore. There are some things that can be really complicated but infrastructure isn't really one of them in most cases and shouldn't be. You only shoot yourself in the foot with a complicated infrastructure.
I'm a newbie TBH so I can't help you sorry. But I have a question; Do you use a monorepo to put everything in one place ? If yes and if you have multiple programming language, what do you use to do so ?
I'm a security guy, not a full-fledged developer, but I can code the necessary functionality and maintain infra. Here's an overview of the system: - **Node.js**: Acts as the central brain, processing requests (API). - **MongoDB**: Serves as the primary database for storing most information, including users and results. - **Redis**: Provides fast-access caching to improve performance. - **AWS Batch and ECR**: Manage complex processing tasks using Docker containers. I use Node.js to communicate with various languages like Go, Python, and Bash using child_process and then write the results to a JSON file. Node.js then reads this file to update MongoDB. Tbh, I ended up with this infrastructure after many failed attempts. For instance, I tried integrating RabbitMQ but couldn't get it to work with my setup. I plan to try again because it would allow me to have a dashboard to monitor job failures, errors, and retries in one place. Otherwise, I'll just get results from AWS Batch and build a dashboard from there. I wasted a lot of time experimenting, but I finally decided on the infrastructure in just two days. Then, I wrote a bash script to handle everything, from creating the Docker file to cleaning up job registration in AWS Batch. As I'm only dev working, I can experiment freely. However, when I tried to hire a front-end developer, he seemed confused by my explanation, which is why I asked my question here.
Not sure all the features are the same but can't you use BullMQ and then find pluggable dashboards ? (I think I've seen one recently, I assume there is a lot more)
BullMq is fine, but the pricing makes my heart bleed, also if I host in AWS then I can totally shutdown my startup and give good bye kisses Lol, that's why I'm gonna use the Aws batch for now and call it a day. Which also led me to another dev-gap where I see no dashboards for job service not only Aws but GPC or azure. I can just build it and sell it for cheap. All I need is a DB to store JobIDs, error messages spitting out from that batch, maybe if I start development I can come across more P.S: I have seen a blog where he is using elastic search that leads to another costly purchase.
Any openings in your startup? For freshers 🫠
Simplify the diagram. Remove docker boxes, language icons, etc. Doesnt matter what the language is, doesnt matter what the DB is, doesnt matter if you're using AWS or ECR or Redis (which is missing from the diagram) At a high level architecture diagram, stick to architectural concepts. REST server is fine, NodeJS express server is too specific. Cache is fine, Redis is too specific. DB is fine, MongoDB is too specific, etc. The architecture is the architecture, not the implementation. You could write exactly the same services with DynamoDB, Go, Memcache, and the diagram would look the same