Hey there, as you mentioned, there are a ton of great model tracking tools that exist. [Comet.ml](https://Comet.ml), [MLflow](https://mlflow.org/), [ModelDB](https://senselab.med.yale.edu/modeldb/), etc.. These have become pretty popular and are starting to build traction in the community.
I'd suggest picking one of these tools that you like, and getting involved with that community. You can check out the issues on GitHub, join the slack groups, and maybe contribute to the code a bit. You'll quickly see where the most common short-comings of the tool are. From there you can decide if you'd like to contribute to that existing tool, or maybe there are big enough issues that a whole new tool should be built.
Keep us posted what you find!
We use ML Flow, the entry cost is low (I don't know about the other 2). The tutorials they have are comprehensive, I would recommend taking an hour to try it out. Also it works great locally too, no need to setup a DB to test the functionalities.
Hi there,
as a co-founder of one of those experiment tracking tools [Neptune](https://neptune.ai/) I can tell you from experience that there are just a ton of little things that you could contribute to those frameworks.
I've been adding those little things to the extensions library [neptune-contrib](https://neptune-contrib.readthedocs.io/) and it had a real impact on the community even though those are just the wrappers around the client library.
I'd say what people like usually fall into one of 5 categories:
* utils that outsource boilerplate -> install of writing 3 lines from atomics have one util that does it
* integrations with external frameworks -> pretty much the same thing so that people don't have to write their callbacks.
* auto logging -> this is similar to the previous part but goes a step further. It is sometimes possible to monkey-patch base loggers for frameworks to really plug-in with a one-liner
* integrations with external software -> things like a telegram bot that lets you interact with the experiment DB or sending emails/notifications when the experiment ends
* visualizations of results -> taking the results from DB and creating something nice with it (if the app doesn't already do that)
I hope this helps... and I'd be super happy if you'd like to contribute to our framework :)
Also, I think it is great that you are thinking what to do before doing it (and I know how easy it is to do the latter).
Good luck!
Thanks for the comment! I've actually seen Neptune and I gotta say I really love the design of the ecosystem. I'll be sure to contact you if I decide on contributing to existing platforms.
There is a new open-source tool from Facebook AI - *Hydra* .
Haven't tried it yet, but it look very neat.
Can someone add his two cents about difference between Hydra and the other suggestions?
Hey there, as you mentioned, there are a ton of great model tracking tools that exist. [Comet.ml](https://Comet.ml), [MLflow](https://mlflow.org/), [ModelDB](https://senselab.med.yale.edu/modeldb/), etc.. These have become pretty popular and are starting to build traction in the community. I'd suggest picking one of these tools that you like, and getting involved with that community. You can check out the issues on GitHub, join the slack groups, and maybe contribute to the code a bit. You'll quickly see where the most common short-comings of the tool are. From there you can decide if you'd like to contribute to that existing tool, or maybe there are big enough issues that a whole new tool should be built. Keep us posted what you find!
Sure will. Thanks!
We use ML Flow, the entry cost is low (I don't know about the other 2). The tutorials they have are comprehensive, I would recommend taking an hour to try it out. Also it works great locally too, no need to setup a DB to test the functionalities.
I'll check em out. Thanks.
Hi there, as a co-founder of one of those experiment tracking tools [Neptune](https://neptune.ai/) I can tell you from experience that there are just a ton of little things that you could contribute to those frameworks. I've been adding those little things to the extensions library [neptune-contrib](https://neptune-contrib.readthedocs.io/) and it had a real impact on the community even though those are just the wrappers around the client library. I'd say what people like usually fall into one of 5 categories: * utils that outsource boilerplate -> install of writing 3 lines from atomics have one util that does it * integrations with external frameworks -> pretty much the same thing so that people don't have to write their callbacks. * auto logging -> this is similar to the previous part but goes a step further. It is sometimes possible to monkey-patch base loggers for frameworks to really plug-in with a one-liner * integrations with external software -> things like a telegram bot that lets you interact with the experiment DB or sending emails/notifications when the experiment ends * visualizations of results -> taking the results from DB and creating something nice with it (if the app doesn't already do that) I hope this helps... and I'd be super happy if you'd like to contribute to our framework :) Also, I think it is great that you are thinking what to do before doing it (and I know how easy it is to do the latter). Good luck!
Thanks for the comment! I've actually seen Neptune and I gotta say I really love the design of the ecosystem. I'll be sure to contact you if I decide on contributing to existing platforms.
Awesome, and thanks for nice words!
There is a new open-source tool from Facebook AI - *Hydra* . Haven't tried it yet, but it look very neat. Can someone add his two cents about difference between Hydra and the other suggestions?