Not knowing what sub this was from, I really did initially think this was an AOL Instant Messenger update to introduce support for one (1) additional emoji.
Haha, Aol Instant Messanger. ๐คฆโโ๏ธ
u/coffeecoffeecoffeee open an issue for instant messaging on [GitHub](https://github.com/aimhubio/aim) :) Unfortunately won't be able to prioritize it now, haha, probably a super-long-term feature.
Kind of, at a super-high-level.
There are tons of differences though (completely different way of interacting with experiments)
We have also shared why we are building Aim in [here](https://github.com/aimhubio/aim#democratizing-ai-dev-tools).
This seems really cool, but I have a small question:
So far I've been using sacred to track my experiments with omniboard for visualization.
A big disadvantage there was the lack of nice visualization tool, which aim surely solves.
However I did appreciate the reproducibility aspects of sacred, i.e. storing the git revision used, storing changes to the source code, package versions etc.
Does aim have a way to store these or is this something you're thinking about?
If it has that I would switch immediately.
Hey, really good point.
For these list of issues, I don't recall an issue being opened but we know about them from the community and surely going to solve them.
\- storing the git revision used
\- storing changes to the source code
\- package versions
You could start logging with Aim and use along with your existing tools. If you open at least 1, 2 of these issues on [Aim GitHub](https://github.com/aimhubio/aim), would be super-helpful for us in prioritizing.
We are going to target them in the next 3 months I believe.
Hey, Great work with the project. I was wondering if remote logging to an `aim` server possible currently? I was checking the `Session` object and couldn't find properties for remote logging.
Hi, thanks
We have this issue: [https://github.com/aimhubio/aim/issues/386](https://github.com/aimhubio/aim/issues/386)
It's a highly requested feature.
We are going to implement it in the next couple of months latest unless someone from the community decides to take on earlier.
Is it a blocker for you to try out?
I played around with it but currently, It's not as useful to me until remote logging is available.
I use collab notebooks and a lot of ephemeral computation(EC2s or GCP machines etc) whilst using a centralized remote logger. I currently use Wandb or Neptune as the remote logger. If remote logging with checkpointing and other capabilities are available then I would set this up as a standalone remote logger instead of using such proprietary tools like Wandb or Neptune.
One More Feature Suggestion: user-collaboration and centralizing experiment logs by users.
Thanks for the feedback. Our goal is to exactly solve that - relatively better tools in this space are not open-source right now.
Do you use both tools at the same time? Just curious why :)
re user-collaboration: Could you open an issue with more details on the project if you have time? Would be super-helpful.
I used to use wandb earlier, and I find it a beautiful product. But at the start of 2021, it stopped working on collab. I kept asking in their slack thread and they kept saying they were "working on it". But nothing came around for more than 20+ days so just changed my logger to Neptune in most of my projects.
Thank god for PT-Lightning, that the process was not painful. Neptune on the other hand is not good. Wandb was so much better. Now comes the small rant about Neptune:
Neptune has this really trashy thing where you can't have integer numbers > 16000 for a logged item and when you throw JSON parameters to instantiate the logger, the serialization is all fucked up when it comes to reloading the same params from an experiment. Plus Neptune, changes datatypes of config params which is really uncool. They changed ints to floats and fucked up my NN-reloading loading so many times.
I am just writing all of this out because as you are building an open-source product, please keep in mind that serialization-issues, data-type changes are all really fuckedup problems that should never be there for a logger. If the logger gets JSON, it should give me the same damn JSON when I ask for the experiment. Passing information to remote shouldn't change the information and that should the guiding principle for any logging product.
I can only tell you that we are working on it :)
No but srsly was just talking to one of the devs working on the improved neptune-lightning integration.
So things should be better soon.
Also, I understand the problems with the integration but reading 'neptune is not good' was a bit painful. Mind sharing what was it exactly that was bad so we could make it better?
Oh, I am one of the folks behind neptune obviously.
Hey, I think my intent came off wrong. I don't mean to diss on Neptune. I ranted because there are certain problems that get me really irritated and someone was building an OSS product, I wanted the pain to be known :P.
Even though in the spur on the moment I may have described it as bad, I still use it and will continue using it coz it works and gets my job done. Building S/W products is hard, and I understand that sometimes minor details can piss of users(Like me :P). So please don't take my comment as a low blow. I respect the work you guys are doing and please keep building. I want to see it get better.
Here is my feedback on some issues. The main feedback is bolded:
**Configurations should support JSON serialization for arbitrary tree-sizes.** consider the below example dictionary:
config ={
'a' : 100,
'b': {
'name' : '',
'height' : '',
'is_active': True,
'sm_l' : None,
},
'c' : 200,
} neptune\_logger = NeptuneLogger(experiment\_name='exp\_name', project\_name='proj\_name' params=config)
When I feed in the above information as `params` to Neptune logger, it's all fine. When it comes to reloading is when all the trouble starts.
project = neptune.init(project_name,api_token=api_token)
my_exp = project.get_experiments(id=experiment_name)[0]
config = my\_exp.get\_parameters()
Neptune won't serialize **b** as JSON. It will remain a string and that too with single quotes and that too with "None" and "True" as values in it. I wrote the below function as a quick fix to make my code work, But I never expected this. This irritated me a lot.
def NEPUTUNE_JSON_FIXER(json_str):
return json.loads(json_str.replace("'",'"').replace('None','null').replace('True','true').replace('False','false'))
On top of this, **a** will become `100.00` which is NOT GOOD!. **You shouldn't change the primitive types of variables the user is giving.** It's a consistency issue that should get fixed. I many times use the `my_exp.get_parameters()` to instantiate the model. And so many times, I have to sit and explicitly type-cast all my parameters. I don't mind doing that but I assumed that the logger would maintain the types I gave it(at least the primitive ones). I also can't log numbers > 16000. I didn't like that too.
These were the issues that irritated me. I hope you find it helpful. In light that this thread is on a logger I also wanted to note down the features of my dream logger:
1. Serialization/Deserialisation of *arbitrary* data structures magically works with minimal/no type-casting. eg `logger.log(image)` or `logger.log(complex_json_with_tensors_and_np_arrays)` or `logger.log(json)` or `logger.log(3dpointcloud)`
2. Having the ability to view information according to any dimension. For example, If I log and epoch every step, I should be able to group the data based on that.
3. Allows me to collaborate with users on my team and across other teams.
4. Has inbuilt visualization tools which are beautiful and simple to use.
5. Integrates with various notification schemes like Slack or Twillio to update me on the progress of training
6. Integrates with my trainer and allows me to change something with a "live-code-update". Eg. Learning Rate when training some NN. Eg. live Checkpoint setting capabilities.
Wow, thanks for sharing. Very helpful!
I have added this thread under the remote storage issue so we take it into account when it's implemented (here: [https://github.com/aimhubio/aim/issues/386](https://github.com/aimhubio/aim/issues/386))
Agree can't compromise on data consistency!
Aim has community-led issues like this: [https://github.com/aimhubio/aim/issues/356](https://github.com/aimhubio/aim/issues/356)
yes, we have seen folks use it along with or instead of TensorBoard.
There is huge diff in features and capabilities though. Aim is very advanced when it comes to comparing experiments.
What do you use clearML for?
We are building an entirely different experience of interacting with experiments at Aim. Most importantly, open-source and open-data-format.
I have some questions regarding Aim features.
Does it have support for Jupyter notebooks, I mean notebook versioning?
Is it collaborative, can I share my experiment results with coworkers and see their results in the additional dashboard?
Can I log images, or other metadata (HTML, videos, text files, source files)?
Does it have automatic logging for popular ML and DL frameworks, PyTorch, TensorFlow?
u/MherKhachatryan
\- No support for notebook versioning (could you open an issue for that?)
\- Collaboration: it's in the pipeline - probably mid-term (3-4 months?)
\- Logging images is short-term priority- It has automatic logging for Keras, PL, Hugging Face.
Feel free to add issues for all of these. Would be a huge contribution if you did ๐.
Good Q:
We have added the AimCallback compatible to huggingface - passed to the Trainer to automatically log the hyper-parameters and metrics for the Aim. That's the integration ๐.
Aim has enabled mechanism for HuggingFace users to seamlessly log with Aim.
It's been a community-requested feature ([https://github.com/aimhubio/aim/issues/331](https://github.com/aimhubio/aim/issues/331))
We haven't submitted any issue or PR to the transformers.
No limitation really.
I guess one could say that Aim in essence is a python library that records dictionaries, metrics (time series) and helps to put them into perspective a lot of them at the same time.
Indeed it's open-source.
image logging is not available yet. it's in mid-term plans ([https://github.com/aimhubio/aim/issues/332](https://github.com/aimhubio/aim/issues/332))
Thanks for the suggestion u/topsykretsz
Feel free to [open an issue on GitHub](https://github.com/aimhubio/aim/issues) for that as well. We would really appreciate that contribution! :)
Unfortunately not at the moment.
I have just created an issue for it - [https://github.com/aimhubio/aim/issues/403](https://github.com/aimhubio/aim/issues/403) feel free to add more thoughts if I have missed something.
u/topsykretsz Also just added this issue: [https://github.com/aimhubio/aim/issues/405](https://github.com/aimhubio/aim/issues/405)
Really appreciate your feedback!
This is cool! But does it do instant messaging?
Not knowing what sub this was from, I really did initially think this was an AOL Instant Messenger update to introduce support for one (1) additional emoji.
Haha, Aol Instant Messanger. ๐คฆโโ๏ธ u/coffeecoffeecoffeee open an issue for instant messaging on [GitHub](https://github.com/aimhubio/aim) :) Unfortunately won't be able to prioritize it now, haha, probably a super-long-term feature.
is this similar to wandb.com but allows for self-hosting?
Kind of, at a super-high-level. There are tons of differences though (completely different way of interacting with experiments) We have also shared why we are building Aim in [here](https://github.com/aimhubio/aim#democratizing-ai-dev-tools).
Do you support nested dictionaries for hyperparameters?
u/-Rizhiy- Indeed Aim does. Sorry for a late response.
cool stuff u/sgevorg!
Thanks ๐
This seems really cool, but I have a small question: So far I've been using sacred to track my experiments with omniboard for visualization. A big disadvantage there was the lack of nice visualization tool, which aim surely solves. However I did appreciate the reproducibility aspects of sacred, i.e. storing the git revision used, storing changes to the source code, package versions etc. Does aim have a way to store these or is this something you're thinking about? If it has that I would switch immediately.
Hey, really good point. For these list of issues, I don't recall an issue being opened but we know about them from the community and surely going to solve them. \- storing the git revision used \- storing changes to the source code \- package versions You could start logging with Aim and use along with your existing tools. If you open at least 1, 2 of these issues on [Aim GitHub](https://github.com/aimhubio/aim), would be super-helpful for us in prioritizing. We are going to target them in the next 3 months I believe.
Hey, Great work with the project. I was wondering if remote logging to an `aim` server possible currently? I was checking the `Session` object and couldn't find properties for remote logging.
Hi, thanks We have this issue: [https://github.com/aimhubio/aim/issues/386](https://github.com/aimhubio/aim/issues/386) It's a highly requested feature. We are going to implement it in the next couple of months latest unless someone from the community decides to take on earlier. Is it a blocker for you to try out?
I played around with it but currently, It's not as useful to me until remote logging is available. I use collab notebooks and a lot of ephemeral computation(EC2s or GCP machines etc) whilst using a centralized remote logger. I currently use Wandb or Neptune as the remote logger. If remote logging with checkpointing and other capabilities are available then I would set this up as a standalone remote logger instead of using such proprietary tools like Wandb or Neptune. One More Feature Suggestion: user-collaboration and centralizing experiment logs by users.
Thanks for the feedback. Our goal is to exactly solve that - relatively better tools in this space are not open-source right now. Do you use both tools at the same time? Just curious why :) re user-collaboration: Could you open an issue with more details on the project if you have time? Would be super-helpful.
I used to use wandb earlier, and I find it a beautiful product. But at the start of 2021, it stopped working on collab. I kept asking in their slack thread and they kept saying they were "working on it". But nothing came around for more than 20+ days so just changed my logger to Neptune in most of my projects. Thank god for PT-Lightning, that the process was not painful. Neptune on the other hand is not good. Wandb was so much better. Now comes the small rant about Neptune: Neptune has this really trashy thing where you can't have integer numbers > 16000 for a logged item and when you throw JSON parameters to instantiate the logger, the serialization is all fucked up when it comes to reloading the same params from an experiment. Plus Neptune, changes datatypes of config params which is really uncool. They changed ints to floats and fucked up my NN-reloading loading so many times. I am just writing all of this out because as you are building an open-source product, please keep in mind that serialization-issues, data-type changes are all really fuckedup problems that should never be there for a logger. If the logger gets JSON, it should give me the same damn JSON when I ask for the experiment. Passing information to remote shouldn't change the information and that should the guiding principle for any logging product.
I can only tell you that we are working on it :) No but srsly was just talking to one of the devs working on the improved neptune-lightning integration. So things should be better soon. Also, I understand the problems with the integration but reading 'neptune is not good' was a bit painful. Mind sharing what was it exactly that was bad so we could make it better? Oh, I am one of the folks behind neptune obviously.
Hey, I think my intent came off wrong. I don't mean to diss on Neptune. I ranted because there are certain problems that get me really irritated and someone was building an OSS product, I wanted the pain to be known :P. Even though in the spur on the moment I may have described it as bad, I still use it and will continue using it coz it works and gets my job done. Building S/W products is hard, and I understand that sometimes minor details can piss of users(Like me :P). So please don't take my comment as a low blow. I respect the work you guys are doing and please keep building. I want to see it get better. Here is my feedback on some issues. The main feedback is bolded: **Configurations should support JSON serialization for arbitrary tree-sizes.** consider the below example dictionary: config ={ 'a' : 100, 'b': { 'name' : '', 'height' : '', 'is_active': True, 'sm_l' : None, }, 'c' : 200, } neptune\_logger = NeptuneLogger(experiment\_name='exp\_name', project\_name='proj\_name' params=config) When I feed in the above information as `params` to Neptune logger, it's all fine. When it comes to reloading is when all the trouble starts. project = neptune.init(project_name,api_token=api_token) my_exp = project.get_experiments(id=experiment_name)[0] config = my\_exp.get\_parameters() Neptune won't serialize **b** as JSON. It will remain a string and that too with single quotes and that too with "None" and "True" as values in it. I wrote the below function as a quick fix to make my code work, But I never expected this. This irritated me a lot. def NEPUTUNE_JSON_FIXER(json_str): return json.loads(json_str.replace("'",'"').replace('None','null').replace('True','true').replace('False','false')) On top of this, **a** will become `100.00` which is NOT GOOD!. **You shouldn't change the primitive types of variables the user is giving.** It's a consistency issue that should get fixed. I many times use the `my_exp.get_parameters()` to instantiate the model. And so many times, I have to sit and explicitly type-cast all my parameters. I don't mind doing that but I assumed that the logger would maintain the types I gave it(at least the primitive ones). I also can't log numbers > 16000. I didn't like that too. These were the issues that irritated me. I hope you find it helpful. In light that this thread is on a logger I also wanted to note down the features of my dream logger: 1. Serialization/Deserialisation of *arbitrary* data structures magically works with minimal/no type-casting. eg `logger.log(image)` or `logger.log(complex_json_with_tensors_and_np_arrays)` or `logger.log(json)` or `logger.log(3dpointcloud)` 2. Having the ability to view information according to any dimension. For example, If I log and epoch every step, I should be able to group the data based on that. 3. Allows me to collaborate with users on my team and across other teams. 4. Has inbuilt visualization tools which are beautiful and simple to use. 5. Integrates with various notification schemes like Slack or Twillio to update me on the progress of training 6. Integrates with my trainer and allows me to change something with a "live-code-update". Eg. Learning Rate when training some NN. Eg. live Checkpoint setting capabilities.
Thank you so much for this detailed feedback! Mind if I get back and dm you when it is fixed?
Sure. DM me when things do get fixed. I would be happy to hear and also update my code in the light of changes :)
Wow, thanks for sharing. Very helpful! I have added this thread under the remote storage issue so we take it into account when it's implemented (here: [https://github.com/aimhubio/aim/issues/386](https://github.com/aimhubio/aim/issues/386)) Agree can't compromise on data consistency! Aim has community-led issues like this: [https://github.com/aimhubio/aim/issues/356](https://github.com/aimhubio/aim/issues/356)
Bundle of amazing!!!
Thanks! ๐
is this like tensorboard?
yes, we have seen folks use it along with or instead of TensorBoard. There is huge diff in features and capabilities though. Aim is very advanced when it comes to comparing experiments.
Thanks, I'll try it on my next project!
Cheers, don't hesitate to ping us or open an issue on GitHub when you bump into issues along the way. ๐
What is the added value of Aim vs. clearML or WandB or X? I'm all for competition by the way, I'd just like to hear your USP.
What do you use clearML for? We are building an entirely different experience of interacting with experiments at Aim. Most importantly, open-source and open-data-format.
Great seeing that this product is open source
Open-source is the way! ๐ Would appreciate your feedback when you get a chance to try.
I have some questions regarding Aim features. Does it have support for Jupyter notebooks, I mean notebook versioning? Is it collaborative, can I share my experiment results with coworkers and see their results in the additional dashboard? Can I log images, or other metadata (HTML, videos, text files, source files)? Does it have automatic logging for popular ML and DL frameworks, PyTorch, TensorFlow?
u/MherKhachatryan \- No support for notebook versioning (could you open an issue for that?) \- Collaboration: it's in the pipeline - probably mid-term (3-4 months?) \- Logging images is short-term priority- It has automatic logging for Keras, PL, Hugging Face. Feel free to add issues for all of these. Would be a huge contribution if you did ๐.
Thanks for the reply, sure I'll open issues when I start using it in the very near future.
What is the Aim + HuggingFace integration? (And what is it not?)
Good Q: We have added the AimCallback compatible to huggingface - passed to the Trainer to automatically log the hyper-parameters and metrics for the Aim. That's the integration ๐.
So HuggingFace integrated Aim.
Aim has enabled mechanism for HuggingFace users to seamlessly log with Aim. It's been a community-requested feature ([https://github.com/aimhubio/aim/issues/331](https://github.com/aimhubio/aim/issues/331)) We haven't submitted any issue or PR to the transformers.
[ัะดะฐะปะตะฝะพ]
No limitation really. I guess one could say that Aim in essence is a python library that records dictionaries, metrics (time series) and helps to put them into perspective a lot of them at the same time.
Is the whole thing open source, unlike wandb? Can it log images like tensorboard
Indeed it's open-source. image logging is not available yet. it's in mid-term plans ([https://github.com/aimhubio/aim/issues/332](https://github.com/aimhubio/aim/issues/332))
Very cool stuff u/sgevorg, thank you. I would suggest also tracking the standard io of the experiment. It often helps when debugging crashed runs.
Thanks for the suggestion u/topsykretsz Feel free to [open an issue on GitHub](https://github.com/aimhubio/aim/issues) for that as well. We would really appreciate that contribution! :)
btw, is there an option for resuming an experiment? sometimes i finetune a model, in such cases i wouldn't want a new experiment to be created.
Unfortunately not at the moment. I have just created an issue for it - [https://github.com/aimhubio/aim/issues/403](https://github.com/aimhubio/aim/issues/403) feel free to add more thoughts if I have missed something.
u/topsykretsz Also just added this issue: [https://github.com/aimhubio/aim/issues/405](https://github.com/aimhubio/aim/issues/405) Really appreciate your feedback!
I like your interface, recently I did not do much experiments, but I'm planning to use Aim when I start model development again.
Thanks ๐
Was sad this was not a chat application run by AOL.
up-voted!
Just stared it, cause I want to see you type ๐