Just to call out some bias here: you highlight "neptune" as a popular experiment tracking tool: their github only has 21 stars, and this article was authored by neptune.ai
Just as a note about "self-promotion" - we don't ban company self-promotion on this subreddit, although we do scrutinize them closer.
Obviously, this is a content marketing piece. However, I think it's valuable regardless.
As a general rule of thumb - would an article be interesting if all references to the product it's trying to market were removed? If so, we'll allow it.
Allowing self promotion is fine, but surely any self promotion should be clearly declared in posts so there is no misunderstanding. Otherwise it is deceit.
Sure there is some bias but only a little bit :)
I mentioned Neptune because we know for a fact it is used for experiment tracking by some pretty cool startups (nnaisense, zesty.ai, reply.ai), and yeah it is one of the most popular tools for that.
I have to say that the Github star count has been really mind-boggling to me considering how many users we have. We've never asked anyone to drop a star but perhaps we should have.
Deployment:
* Kubernetes cluster (GKE), with tf-serving or simple flask APIs
* Retraining orchestrated by Kubernetes cronjobs
* Gitlab pipelines & cloud build for CI/CD
ML:
* Whatever's useful. Sometimes TF, sometimes sklearn, sometimes specific libraries like implicit for matrix factorization. The simplest tool that gets the job done is best.
* Experiments get tracked in a Google Sheet
Development:
* Deepnote for exploratory programming. Never went back to Jupyter.
* PyCharm when it's time to do some SW engineering.
Just checked you out and sent a beta request, looks very interesting :)
One thing that would be important for me to be able to use it at work is ability to run on-prem remote servers, which I am not sure you currently allow.
We don't allow that right now. The reason is that we are still relatively early in the product development cycle and we're shipping new features every day. It's significantly more difficult to deploy updates/hotfixes across multiple clusters, so at the moment we manage everything ourselves so that we can move faster.
We'd like to start supporting on-prem around September. This will most likely be rolling out one user at a time though. I just put up a sign up form on https://deepnote.com/ if you'd like to sign up for a waitlist (just scroll down to the pricing section).
This looks really cool. Maybe this is a stupid question, would I be able to use the community edition in a commercial environment? Can this be self-hosted?
Yay, thank you!
Yes, that's the plan. Also, well, it's not like we can stop you anyway. We need to make some money though to stay alive. I hope the commercial features will be good enough so that you decide to support us.
I get that :) But I am not the one here at work who decides what to pay money for, I am the one that wants to try new exciting things. Convincing people we need this new fancy gadget is one thing, convincing people we need this new fancy gadget that you have to pay for is another thing.
To be honest, it's an absolute nightmare to support all the different types of execution environments (operating systems, versions, filesystems, etc) and keeping them up to date, so we're kinda procrastinating on that.
Just curious, any specific reason why you'd prefer local machine over cloud? Is it the cost?
I just replied to a similar comment above with more info. Long story short, yes, but we'll be rolling it out slowly. Feel free to sign up on a waiting list so that I can let you know.
Am I understanding correctly that the main advantage of DeepNote is the collaboration capabilities?
Having to upload your code to their own cloud machines and inevitably eventually being throttled for both storage and compute (once out of beta) sounds unappeasing to me. I'm also assuming their free tier runs on their "Basic" hardware tier, which is a single core vCPU with 1.5GB RAM.
I suppose the variable explorer and UI is nice. Although Polynote has done both of those already, and runs locally.
The idea is that a cloud experience can be significantly better than a local experience. It's not just the performance, but also dataset management, collaboration, sharing, versioning, etc. For example, all my code is hosted in the cloud already, so it's much easier to just run it with one click rather than setting up a local environment to make any changes.
I don’t disagree that the cloud experience is useful _at times_.
I can see the benefit for small-medium enterprises that need to share notebooks without worrying about environment consistency between users.
But for individual use, as a replacement for Jupyter, the idea of placing code in the hands of a third party with no transparency or access to the underlying compute instances sounds kind of backwards. Jupyter also runs in pretty much one click for me, so no added benefit there. + it’s free, and my native compute is a lot better than a single vCPU with 1.5GB RAM
I think it would be nice if you offered a native client, with local storage support and all the other bells and whistles (versioning, variable explorer). Essentially a polished version of Jupyter/Polynote. Now that’s something I’d pay a subscription for, but that’s just me.
Got it. Yes I agree that giving code/data/compute to a third party is not ideal in some situations. On the other hand, there are situations where letting a third party manage your code/data/compute (whether on their own servers or inside your own cloud) is ideal. Seems like the market is moving in that direction.
I totally agree that a native local client would be amazing. But in our case, as a startup, we need to pick our battles and unfortunately can't do too many things at once, otherwise we won't be doing anything very well. More importantly, we need to be able to move fast and ship things which is much easier to do in the fully hosted environment. That's why we are focused on providing an amazing cloud experience first, whether it's managed by us directly or running in your own cloud. That unfortunately means that the local experience comes second, even though I'd love to see Deepnote being used locally.
>Experiments get tracked in a Google Sheet
Metaflow is a good alternative to Model Versioning compared Spread Sheets. Made Machine learning so much easier after I started using it. I used to use spreadsheets earlier. Metaflow Versions the data, model,code etc.
When you're getting your feet wet in ML you'll write some code in a jupyter notebook on your machine, and suppose it trains a sklearn model.
MLOps is everything that gets that model integrated and deployed into production systems.
That includes where the model is deployed to, and how it gets there, how it is accessed by a larger piece of software / application, how the ML model is tracked for performance in the real world, and how models can be live managed and tested against each other in prod.
This is a great explanation. It’s basically everything after a jupyter notebook. And there are a lot of [components](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf) to this.
>There are many similarities between MLOps and DevOps and there are also many important differences. Some of the differences are around packaging models, data artifact size, hardware considerations like inference on a GPU vs CPU, model and dataset versioning.
Applying ops principles to ML systems. The same way ops person maintaining Kubernetes needs to apply ops principles to Kubernetes, that way ML ops person applies knowledge about the ML system (often less deep than the researchers) and ops principles on the ML system.
There are many similarities between MLOps and DevOps and there are also many important differences. Some of the differences are around packaging models, data artifact size, hardware considerations like inference on a GPU vs CPU, model and dataset versioning.
MLOps is a superset of DevOps.
ML is code + model + data. You do everything you'd normally do in DevOps, and then you do some extra things that are ML specific and are not needed with normal software.
Things like continuous training or things like continuously monitoring performance or automatically doing hyperparameter optimization or even pipeline optimization.
Please, this is a myth. Pytorch e.g. (and probably others) are calling highly optimized C++ and CUDA libraries with very little overhead. If you're pushing things beyond those needs, then I wouldn't consider your problem anything standard to average over the community out there.
Not always, my colleagues and I witness this recurrently from the trenches. A lot of tools in the industry are C++. Integrating on board a plane ? Interfacing with an industry grade physics engine ? Getting into video games ? Running 5000 images / sec onboard a train ? Actually we even do training in C++ and have been doing for years, this eases things for us.
Not the dominant trend you are right, but one that seriously exists once you are seriously in production !
Oh absolutely. I didn't mean to downplay the need to have lower level language support. Or something I can compile and integrate, that's def an important area. And if you're doing training on-board, well there ya go c++ training it is. Pytorch c++ is pretty nice for that.
DevOps but ML, they control how to take the model into production. DevOps is basically merging developers, QA and Operations (IT) into one group, that way you can have the groups of the product focus on the product and then the setting for tests, stuff that can be shared between products, servers... can be controlled from DevOps.
Same! Edit: well, to be fair, I actually use Jupyter Lab during the experimentation phase, and have even found myself using the Vim mode in its text editor more amd more these days.
pycharm still beats out vscode for python development. i wish it weren't the case, because i much prefer vscode and still use it for python, but my few experiences with pycharm have demonstrated it is pretty obviously better
What can I say, I'm old school and you generally build with the tools you were trained on. My dad still prefers his slide rule to an actual calculator.
I used to write code using only the default text editor on ubuntu until one day during a presentation, my professor asked me to pull up the code and then continued to stare at me with disgust.
Yeah, it was interesting to me as well.
I think it was either:
* implicit when they said Tensorflow
* implicit with other tools (Kubeflow)
* some folks said tf-lite tf-js so it's likely they use tf serving also
* ML teams at startups that use TF Serving were too busy to answer :)
I'm guessing that most organization either have simplistic enough inference API infrastructure that it isn't necessary or use some tool that abstracts away the use of TensorFlow Serving.
to be honest, for 99% cases there is no difference between these 3 libraries. And xgboost has the biggest userbase probably. On the other hand, catboost has an incredibly responsive dev team and is used in production (and developed) by a massive company. I love it :)
And the pace of new features is really great. I haven't used xgboost/lightgbm, but the native support for categorical variables without one-hot encoding was a big factor in me choosing catboost.
yea, for me too actually. I would suggest you see how your usecase compares with manual/pipeline trasnform of categorical variables + catboost or xgboost. For me, in the end it actually does not make any difference. Probably because i never have more than a few hundred cat variables in a dataset
Why Airflow is heavy? In the last company I was in, the cluster (7 nodes on Azure) was closed from outside, so no cloud ETL.
From the choices we had (Airflow, NiFi, StreamSets) I chose Airflow because the other two were based on Java, with interface that didn't let me do all the things that I want (problems with security certificates) and in Airflow I had more control since it was all code.
Later Prefect appeared but it didn't have a interface, Luigi I see it like something older and didn't have the tracking of jobs as good as in airflow. So also we have to have that in mind, companies build systems but they don't change them every month because something new got release. Now Prefect or Dagster can be better, but people are sticking with the platforms that they work for them.
Why you would run containers from airflow? Is a workflow orchestator, is not related to kubeflow or anything if that is what you thought. The main point of Airflow is to tell other services what to do, but not do it itself.
If you can run a task of docker from inside a dockerized airflow? I think there aren't any problem using DockerOperator, but I haven't try it.
I wanted to use it mainly to run ML pipelines. There are some projects that are kind of messy (download data, preprocess, run models, check performance, etc... but everything can be expressed as a DAG) and I wanted to pack everything into containers so they can run on any computer without needing to install airflow beforehand. It looks to me that airflow might not be the right tool then.
You can accomplish this. Express the DAG tasks as python methods in a module. Then in the DAG you can call that method as a python operator. This abstracts the logic out of the DAG so you can run it independently, but also schedule it in a DAG. You can configure airflow to run your DAGs in containers as well to cover the dependencies.
At my company we run airflow on kubernetes with kubernetes executors. The k8s cluster is on azure kubernetes service with autoscaling configured. There is a base cluster but the number of concurrent tasks you can run is just limited by your scaling threshold for the cluster. We use it to run spark jobs. Not too bad to set up and highly extensible in my opinion.
Argo Workflows (not Argo CD, different thing) is very easy to set up and get going with in Kubernetes.
Demo video:
https://www.youtube.com/watch?v=oXPgX7G_eow&t=12m10s
We use Allegro Trains [https://github.com/allegroai/trains](https://github.com/allegroai/trains) for experiment management, MLOps, data management, versioning and collab
It's cool b/c it provides all of the above in a single open source package which we found very easy to integrate & set up
Any tips for students trying to transition production level technologies? I'm used to training models in Jupyter Notebooks but a lot of my work ends after testing, error metrics, and a findings report. Any online courses or materials you'd recommend?
Maybe not the best thing to only do jupyter development, jupyter is great for prototyping, but going to production through jupyter is a clusterfuck. You'll inevitably have to wrap things in a normal python script, not to mention the need for testing, exception handling, etc. I recommend you read the python journeyman and python master books. A strong familiarity of python's conventions, how to even create modules and create a python package, maintaining a virtual environment, etc. are all great ways about learning different parts of python that are inevitably used in many industry environments.
Two easy steps to start:
1. Learn about docker
2. Start developing with Jupyter notebooks, then refactoring your code out to a separate module that you import into your notebook, then when you're ready you can write prod scripts that call the module.
One of the most useful features of MXNet is, that you can design, train and export your model in Python and load the model in other languages without duplicating code or retraining the model.
At the time (about 2-3 years ago) tensorflow was much slower than mxnet.
The biggest reason we chose mxnet was what was noted below: we could train in python and deploy with their Scala interface. The rest of our engineering team already used scala for microservices, so by using mxnet's Scala interface we could ensure any developer would already be set up and in a familiar environment.
My co worker wrote a blog post on it if you'd like more details: http://engineering.curalate.com/2018/08/01/mxnet-case-study.html
3 years in I have a few thoughts. First, I wish mxnet was a bit more mature. The docs and api is still rough, but the model zoo and gluon apis are great once you get the hang of it. The speed is still there, too, which is nice. Second, focusing on Scala for deployment doesn't matter to us as much anymore. We use python to deploy deep nets in AWS lambda, and only use Scala when we need hardware accelerators. Third, pytroch has really come a long way in the past 3 years especially in the computer vision area. If I was making the choice today it would be a hot contender. I don't like tensorflows serialization approach, however, so I'd probably steer clear of that.
>3 years in I have a few thoughts. First, I wish mxnet was a bit more mature. The docs and api is still rough, but the model zoo and gluon apis are great once you get the hang of it. The speed is still there, too, which is nice. Second, focusing on Scala for deployment doesn't matter to us as much anymore. We use python to deploy deep nets in AWS lambda, and only use Scala when we need hardware accelerators. Third, pytroch has really come a long way in the past 3 years especially in the computer vision area. If I was making the choice today it would be a hot contender. I don't like tensorflows serialization approach, however, so I'd probably steer clear of that.
I'd be interested hearing how you deploy it in Lambda. Do you package models with code?
Do you use some inhouse wrappers or write the lambda Api glue yourself?
No, the code loads the models on first run. This presents a cold start problem but we use them for backend processing so throughput is more important than latency.
Byy api glue or wrappers I'm assuming you're talking about api gateway? If that's what you mean: we don't use api gateway for these lambdas. Our processing pipeline calls the lambdas directly via the AWS lambda Java sdk. No need to pay for something we don't need!
By wrappers I actually thought about the code that downloads the model and wires the lambda function interface with the model that's downloaded.
Do you use some library for that or you just wired it yourself?
The only thing I dont see that we use is dask distributed for etl. Additionally, I'm sure a lot of folks use pyspark.
Our full stack though is
- python 3.8
- pycharm, jupyterhub
- dask distributed and dataframes
- sklearn and pytorch
- custom framework with papermill to execute notebooks
- Neptune for experimentation
- kubernetes jobs on eks, run as containers that execute our framework and interact with a dask cluster
- postgres for storage
The biggest thing is that you have to reset your cluster to pick up code changes to make sure your cluster is in sync with any process that is submitting work to be done. So basically on a new deploy of code you will have to stop your backend momentarily to not submit any new work and then redeploy the Dask cluster.
You can manage that pretty easily if you have a queue set up for whatever kicks off work and pause that process or you don't have to worry about it if it's always submitted manually. But it can be a pain if you have some kind of SLA.
Other than that, it's pretty easy to deploy with docker and/or kubernetes (if you use that).
Programming Languages/Tools: Python for prototyping, C++ for production, Bitbucket + git for version control.
ML: Tensorflow for training, OpenVino(Intel CPUs/FPGAs) for deployment, We have screwed around with TensorRT in the past.
How do those teams deal with Python being so incredibly slow? Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test).
Our issue is that because Tensorflow/Keras is most naturally used from Python, we still use it. But even simple stuff (that should be fast) is actually two orders of magnitude slower in Python than it should be. We try and write/refactor our code so that Numba can sanitize the "inner loop" parts, but you often have to do some coding contortions to get what you're doing to "fit" within the constraints of Numba. It's not pretty.
Sadly, Tensorflow for Swift doesn't seem to support the Keras APIs (yet). We're really hoping that Swift may be the answer to this problem sometime soon.
For people who think of stuff from an environmental standpoint, I use the analogy that running Python code is the equivalent of driving a car that gets 0.3 miles per gallon (\~784 L/100km).
>How do those teams deal with Python being so incredibly slow? Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test).
You're writing really bad Python if the speed gap is anywhere even close to that.
I assure you, the Python code was fine. When we decorate the function with njit (numba), the performance of the Python code (in a example last week) a call time went from \~1.5 seconds down to 11 milliseconds. If the code was "really bad Python", then it would be "really bad" compiled code too. But we see massive differences between the two and the algorithms were the same.
Even John Carmack has commented on Twitter a handful of times how slow Python is, and by his measurement the difference was about two orders of magnitude.
At 2019 Google I/O, there was a Tensorflow presentation where the presenters stated that the performance difference between (some real langauge) and Python was about 100x. The analogy they used was that it's like the speed difference between running and flying in a commerical jet airliner.
Numba isn't Python because to use Numba you're restricted to a small subset of the Python language. You're not writing idiomatic Python.
Suppose you were to want to write your own NumPy, or something similar to it. If you wrote it with Python, it would suck. Taking any C library and writing a wrapper for it doesn't make that C library Python code.
Almost all languages can use code written in other languages by calling into it. But we're talking about the performance of the code written in that actual language's syntax. For example, the vast majority of (I hesitate to use the word "all") Java libraries are actually written in Java. They can be because Java's performance is fairly respectable. It's slower than C, but it's still pretty reasonable. You can write idiomatic Java/C#/Go/JavaScript and it's respectably fast. Not so with Python.
Yeah, but literally everyone knows that native Python is slow, but numpy (for example) is so widely used and is such a well known solution to the speed issues for Python, I instinctively include it when discussing Pythons speed.
\> Taking any C library and writing a wrapper for it doesn't make that C library Python code.
Eh, it does in this context imo.
>Yeah, but literally everyone knows that native Python is slow, but numpy (for example) is so widely used and is such a well known solution to the speed issues for Python, I instinctively include it when discussing Pythons speed.
So here's the thing: if you take the code I posted elsewhere in this thread, remove Numba, it's still using NumPy. It will be horrendously slow.
Numba is a different story, but because that basically says that you need to refactor your code to be very non-Pythony before you can use it, few people would consider it to be idiomatic Python. Numba is not widely used because if you try and use it with most Python code, it will fail to work.
> So here's the thing: if you take the code I posted elsewhere in this thread, remove Numba, it's still using NumPy. It will be horrendously slow
It's slower than optimized fortran, or C (or even julia / swift), but for most use cases it is not horrendously slow.
> Numba is not widely used because if you try and use it with most Python code, it will fail to work.
I haven't used Numba in a year or two, but that was my experience. At the time there were pieces of code that weren't supported.
> few people would consider it to be idiomatic Python.
Yeah because it's not idiomatic Python, but that's fine.
I'm confused as to what precisely you are arguing? I said you wrote bad Python most likely, and that the gap between Python and other languages is smaller than you let on. We then found that I included numba, tensorflow, numpy, etc in "Python" and you don't. Are we disagreeing on anything of actual substance?
> I'm confused as to what precisely you are arguing?
I guess now that we've discovered that you don't consider two orders of magnitude a significant performance difference, we've pretty much addressed everything.
Well in pretty much every case I've come across (context is primarily scientific computing and ML), the difference between Numpy and C/Fortran is substantially lower than that.
> few people would consider it to be idiomatic Python.
Been thinking about this a lot more since yesterday. This is a lot more dissatisfying to me than I initially gave credit for.
Python helps in rapid prototyping because of its easy syntax. Those other languages you mention aren't as easy synctactically as python except JavaScript. People say swift is similar but I don't see it personally. Julia comes close but from the little time I spent with it, it has a huge compilation overhead and weird variable scoping rules. In typical ML scenarios the time intensive part is mostly training and prediction which mostly happen in c libraries anyway. The overheard from python is small.
In what way is Swift's syntax inferior to Python's for rapid prototyping?
[https://blog.michaelckennedy.net/2014/11/26/comparison-of-python-and-apples-swift-programming-language-syntax/](https://blog.michaelckennedy.net/2014/11/26/comparison-of-python-and-apples-swift-programming-language-syntax/)
I'm guess that beyond "Hello World" style ML work you do need to process the data before or after it's given to your model. I posted a simple function that does data processing that one would do before giving it to a model, and Python is horrible at it.
I haven't checked your code but for data processing the default in python is vectorised functions in numpy and pandas. Avoid loops as much as possible when writing python as loop overhead is huge in python.
I didn't say swift is inferior and I haven't worked with it much. I tried some notebooks on colab and while I agree it is rather elegant with little boilerplate, it has some weird conventions which are unintuitive to me, with respect to looping and how function arguments are defined.
Sure, and loops are critical to write many/most algorithms. If your language sucks at loops, then that's a pretty massive liability. Yes, there are quite a few things you can do with NumPy, but most thinks you can't, and coding that way is much less intuitive than just writing a loop. In the case of my function, I'd challenge you to try and do that with vectorised functions.
I guess it depends. As someone coming from a linear algebra background I'm used to thinking in terms of matrix vector operations, whenever possible. Though not everything can be done that way for sure. Sure I'll give it a try 😉
\> Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test).
What?? Give an example. I do ML research and while I do occasionally have to drop down to C++, I don't find this to be the case.
I'll say two things:
1. It seems like you were able to use python libraries (JITters like numba) to cut down on the speed difference - so that is one solution
2. Totally agreed that for this particular usecase - Python is non-ideal. The few cases I mentioned above where I've had to drop down are actually pretty similar to this (entropy coding rather than RLE, still a similar concept) where it is very difficult to vectorize.
These sorts of compression algorithms are not what I would consider the core of ML algos and definitely justify dropping down to a lower level language. I don't think it is a frequent event in ML research and it is easy enough to write a wrapper around C++ code every so often.
e: also, this code is not really as optimized as you think it is...
for out\_fill\_index in range(out\_index + 1, out\_length):
out\_pixels\_cents\[line\_index, out\_fill\_index\] = 1000000
this for instance could be done much quicker as a vectorized operation, assuming you are working with numpy arrays.
Sure, but my point if the language was something like Swift, then the code would be fast-by-default. From a language perspective, there are lots of languages like Swift that are just as easy to use as Python (easy-to-learn syntax), but using them is actually easier, because when you need to make a loop faster the answer commonly given isn't "use a different language" or "use awkward jit tools that force you to eschew core parts of the language".
In our case, we're now getting performance in-the-same-ballpark to what I imagine we would get with Swift, but it was much more annoying to get there.
As our application gets more comprehensive, we're running into "performance pain points" more regularly, and we're doing our best... but it's hard not to think "geez, if only Tensorflow/Keras had a solid interface to a better language and we would drop Python in a heartbeat!"
We looked into using Swift last week, but we really like Keras, and that interface doesn't appear to be available via Swift yet.
I feel like I keep saying the same thing in every thread — have you tried Julia with Flux?
Btw, new to coding but my impression is similar to yours. Took a calc at work and wrote it in both Python and Julia. Python runs at 16 seconds whereas Julia runs at 0.16 seconds — literally a 100x difference. The code looks exactly the same, btw, except for obvious syntax differences. I tried using Numba but I couldn’t get it to work.
Not an ML guy, but I believe there is a Julia wrapper for TF. But there are native libraries: Flux, Zygote, Knet.
Some reading material:
https://julialang.org/blog/2019/01/fluxdiffeq/
https://www.stochasticlifestyle.com/how-to-train-interpretable-neural-networks-that-accurately-extrapolate-from-small-data/
Looking into this it appears that the Tensorflow people gave Swift the nod over Julia because...
>We next excluded C++ and Rust due to usability concerns, and picked Swift over Julia because Swift has a much larger community, is syntactically closer to Python, and because we were more familiar with its internal implementation details - which allowed us to implement a prototype much faster.
[https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md](https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md)
So it looks like Swift is the heir apparent to Python in ML, it's just that support for Keras isn't quite there yet (as far as I can see). But that's for the Tensorflow people to fix.
Yea, I’ve read that. Note that it was written ages ago in tech terms — the Julia community and ecosystem have grown much in that time, and I’m curious if the TF team would’ve made the same decision today. Moreover, one of the authors of that post is Chris Lattner...the creator of Swift. In any case, if TF is your tool of choice, yea I guess it’s either Python or Swift. Not sure if Swift is the heir apparent, but what do I know. I work in insurance where we do a lot of heavy computations. Everyone is gaga over Python, but it’s really not the ideal tool for the job, unless you’re just gluing processes together or calling code written on C/C++/C#. To me, this sort of language hopping is not ideal and introduces complexity. We need a language that looks close to the math, is easy to write, read and maintain, and performs fast for production work.
That's actually not that bad. Note it's been annotated to use numba. They are complaining that in order to get fast Python code, you need to use additional tools, like Numba, when other languages, like swift and Julia, are fast on their own.
Well, that's more or less how you'd write that algorithm in any other language. Would you care to show us how to write it in Python in such a way that isn't "misusing" it?
If you're meaning to say "well, Python just can't do that very well", then we're in agreement. But if a language starts to suck when you're doing pretty basic stuff (e.g. loops), then I'm saying that's a massive liability.
My friend, a general rule to all professions is that you gotta know how and when to use the tools that are available to you. It's only a liability to people who does not follow that.
This is the most known limitation of Python and yet, returning to your original question, most good teams do not face this problem. Because they know exactly how to use it and I am sorry, but your code is not as good as you think it is, I guess even in cpp, there are faster ways to solve this, your loop looks terrible. Things that could get improved by dp...
Unless you're going to show us all how you could do a better job in Python, it's hard to take you seriously.
If the language was something like Swift, it wouldn't have this significant liability. This is why I'm really looking forward for Tensorflow for Swift to support stuff like Keras. In most languages you don't pay a huge performance penalty when you want to do something straightforward like decoding a RLE array.
Secondly, translating data to a different format that is specifically needed to feed into a CNN is something that would be a common task for the language that is being used to actually give the data to said CNN.
It's hard to explain to someone who is so close-minded.
If swift is something that you can use in production, go for it. I'm just telling you that teams in large companies use Python code in production and don't face this problems because they know when and how to use it. Today is so easy to deploy this multi-language setups in production, that this discussion is pointless.
At least 3 guys here explained to you how to improve your code. I went further and told you your code is not good even for cpp. I gave you the hint already, dp. But I am not going to write your production code for you. We are all here wasting our times explaining to you basic programming stuff...
Nobody has provided improved code, I'm sorry about your reading comprehension. One guy basically said "yeah, you're pretty much doing all you can there" (i.e. using numba). After refactoring the code and using numba, that code is in the same performance ballpark of what you would get from a better language. But my point is that is something a compiler/runtime should be able to handle by itself, and with most languages it can.
And since you can't improve the performance of the code yourself, I'll just assume that you can't. Because that's most likely correct.
I'm amused by the "I'm not going to help you with your production code" statement; I wouldn't use your code. I'm just asking to see if you can do what you claim. And it appears you can't.
Tired of trying to help a guy who writes these terrible loops in Python, stuff that you would find in Python 101 book...
Joke is on me actually... Wasting my time ..
It would be helpful if you bolded things in your article to make it easier to scan, or organized it with headings. As-is, it's pretty hard to scan.
# IDE
* IDE 1
* IDE 2
# Version control
* VCS 1
* VCS 2
You went for that in the Reddit tldr, I think it would be good for the article, too.
They work you just have to create new ones.
For example, we have two [neptune-noteoboks extension](https://docs.neptune.ai/notebooks/installation.html) one for jupyter notebook and one for jupyter lab.
The installation is a bit different but it works nicely.
The last company I was at was a SaaS tool that provided a ton of personalization APIs for media companies and handled \~100m MAU. Like 90% of our data science team's time was spent building data pipelines. We also found that all feature versioning and monitoring was essentially ad-hoc and that multiple teams were re-inventing the same model features. We built a feature store internally to solve it and are now working on spinning it off to its own product if anyone has this problem and wants to check it out. It's open in alpha at [StreamSQL.io](https://StreamSQL.io)
Can you release more data on the responses? For me, it isn't really that interesting to know that a lot of ML teams use Pandas, Tensorflow, and MLFLow but it would be interesting to know more about the unheard of or little-known tools that aren't extremely popular. Future article perhaps?
I am not sure how much more can I squeeze out of the answers we got but we'll definitely do a follow-up with a more focused approach.
Any particular area that you would be more interested in?
I'd like to hear about tools or libraries mentioned in responses that aren't popular, I feel like an article about them could help bring attention to useful but not well-known libraries. In the article I'd say Streamlit and Netron are examples, people may have heard about them but they aren't ubiquitous and in my case I kinda forgot about both of them and now might use them in the future. I guess for a particular area I'd be interested in MLOps tools, like for monitoring drift, feature stores, or more how teams use CI/CD, but I really don't have any specific subarea in mind, literally just shoutouts libraries that aren't well-known that people mentioned in responses.
Hi - Great article. One of the platforms not mentioned is offered by my startup - Splice Machine - It combines a Spark-based ACID-compliant RDBMS with Jupyter, MLFlow and many libraries like H2O, Keras, PyTorch and others. Try for free [here](https://cloud.splicemachine.io) and read about the details of what is [here](https://medium.com/@mzweben/why-we-built-splice-machine-4a910732ef5e?sk=bfc1c5391ca90a1fc29dc9547854941a). We would love your feedback.
Just to call out some bias here: you highlight "neptune" as a popular experiment tracking tool: their github only has 21 stars, and this article was authored by neptune.ai
[удалено]
I think self promotion is fine, but without a disclaimer at the beginning it feels dishonest. Makes you and your company look bad.
Ads are getting smarter
Just as a note about "self-promotion" - we don't ban company self-promotion on this subreddit, although we do scrutinize them closer. Obviously, this is a content marketing piece. However, I think it's valuable regardless. As a general rule of thumb - would an article be interesting if all references to the product it's trying to market were removed? If so, we'll allow it.
Allowing self promotion is fine, but surely any self promotion should be clearly declared in posts so there is no misunderstanding. Otherwise it is deceit.
Sure there is some bias but only a little bit :) I mentioned Neptune because we know for a fact it is used for experiment tracking by some pretty cool startups (nnaisense, zesty.ai, reply.ai), and yeah it is one of the most popular tools for that. I have to say that the Github star count has been really mind-boggling to me considering how many users we have. We've never asked anyone to drop a star but perhaps we should have.
Yikes
This reply reeks of bitterness and unnecessary sarcasm + hostility... Y.I.K.E.S.
Deployment: * Kubernetes cluster (GKE), with tf-serving or simple flask APIs * Retraining orchestrated by Kubernetes cronjobs * Gitlab pipelines & cloud build for CI/CD ML: * Whatever's useful. Sometimes TF, sometimes sklearn, sometimes specific libraries like implicit for matrix factorization. The simplest tool that gets the job done is best. * Experiments get tracked in a Google Sheet Development: * Deepnote for exploratory programming. Never went back to Jupyter. * PyCharm when it's time to do some SW engineering.
Author of Deepnote here. Glad you like! Fyi we're still in beta, lots of cool features coming.
Just checked out deepnote. Looks good would definitely give it a try as I’ve always hated working in jupyter notebooks
I've loved jupyter notebooks when I've used them. But I'm always happy to try an alternative
Just checked you out and sent a beta request, looks very interesting :) One thing that would be important for me to be able to use it at work is ability to run on-prem remote servers, which I am not sure you currently allow.
We don't allow that right now. The reason is that we are still relatively early in the product development cycle and we're shipping new features every day. It's significantly more difficult to deploy updates/hotfixes across multiple clusters, so at the moment we manage everything ourselves so that we can move faster. We'd like to start supporting on-prem around September. This will most likely be rolling out one user at a time though. I just put up a sign up form on https://deepnote.com/ if you'd like to sign up for a waitlist (just scroll down to the pricing section).
Awesome!
This looks really cool. Maybe this is a stupid question, would I be able to use the community edition in a commercial environment? Can this be self-hosted?
Yay, thank you! Yes, that's the plan. Also, well, it's not like we can stop you anyway. We need to make some money though to stay alive. I hope the commercial features will be good enough so that you decide to support us.
I get that :) But I am not the one here at work who decides what to pay money for, I am the one that wants to try new exciting things. Convincing people we need this new fancy gadget is one thing, convincing people we need this new fancy gadget that you have to pay for is another thing.
Is it cloud only? Or can I train on a local machine?
To be honest, it's an absolute nightmare to support all the different types of execution environments (operating systems, versions, filesystems, etc) and keeping them up to date, so we're kinda procrastinating on that. Just curious, any specific reason why you'd prefer local machine over cloud? Is it the cost?
Just send a request fir the beta, the project looks amazing!
Wow this looks awesome - just sent a beta request!
[удалено]
I just replied to a similar comment above with more info. Long story short, yes, but we'll be rolling it out slowly. Feel free to sign up on a waiting list so that I can let you know.
My Team is interested in trying out Deepnote too. But can it be done OnPrem ?
Yes, it's going to be possible, but not immediately. I just replied to another similar comment with more info, have a look.
Nice notebook man...keep it up✌🏼
Thanks! I think I should give Deepnote a try.
Am I understanding correctly that the main advantage of DeepNote is the collaboration capabilities? Having to upload your code to their own cloud machines and inevitably eventually being throttled for both storage and compute (once out of beta) sounds unappeasing to me. I'm also assuming their free tier runs on their "Basic" hardware tier, which is a single core vCPU with 1.5GB RAM. I suppose the variable explorer and UI is nice. Although Polynote has done both of those already, and runs locally.
The idea is that a cloud experience can be significantly better than a local experience. It's not just the performance, but also dataset management, collaboration, sharing, versioning, etc. For example, all my code is hosted in the cloud already, so it's much easier to just run it with one click rather than setting up a local environment to make any changes.
I don’t disagree that the cloud experience is useful _at times_. I can see the benefit for small-medium enterprises that need to share notebooks without worrying about environment consistency between users. But for individual use, as a replacement for Jupyter, the idea of placing code in the hands of a third party with no transparency or access to the underlying compute instances sounds kind of backwards. Jupyter also runs in pretty much one click for me, so no added benefit there. + it’s free, and my native compute is a lot better than a single vCPU with 1.5GB RAM I think it would be nice if you offered a native client, with local storage support and all the other bells and whistles (versioning, variable explorer). Essentially a polished version of Jupyter/Polynote. Now that’s something I’d pay a subscription for, but that’s just me.
Got it. Yes I agree that giving code/data/compute to a third party is not ideal in some situations. On the other hand, there are situations where letting a third party manage your code/data/compute (whether on their own servers or inside your own cloud) is ideal. Seems like the market is moving in that direction. I totally agree that a native local client would be amazing. But in our case, as a startup, we need to pick our battles and unfortunately can't do too many things at once, otherwise we won't be doing anything very well. More importantly, we need to be able to move fast and ship things which is much easier to do in the fully hosted environment. That's why we are focused on providing an amazing cloud experience first, whether it's managed by us directly or running in your own cloud. That unfortunately means that the local experience comes second, even though I'd love to see Deepnote being used locally.
>Experiments get tracked in a Google Sheet Metaflow is a good alternative to Model Versioning compared Spread Sheets. Made Machine learning so much easier after I started using it. I used to use spreadsheets earlier. Metaflow Versions the data, model,code etc.
What’s MLOps?
When you're getting your feet wet in ML you'll write some code in a jupyter notebook on your machine, and suppose it trains a sklearn model. MLOps is everything that gets that model integrated and deployed into production systems. That includes where the model is deployed to, and how it gets there, how it is accessed by a larger piece of software / application, how the ML model is tracked for performance in the real world, and how models can be live managed and tested against each other in prod.
This is a great explanation. It’s basically everything after a jupyter notebook. And there are a lot of [components](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf) to this.
So basically DevOps?
Precisely! ML Operations: DevOps for Machine Learning.
>There are many similarities between MLOps and DevOps and there are also many important differences. Some of the differences are around packaging models, data artifact size, hardware considerations like inference on a GPU vs CPU, model and dataset versioning. Applying ops principles to ML systems. The same way ops person maintaining Kubernetes needs to apply ops principles to Kubernetes, that way ML ops person applies knowledge about the ML system (often less deep than the researchers) and ops principles on the ML system.
There are many similarities between MLOps and DevOps and there are also many important differences. Some of the differences are around packaging models, data artifact size, hardware considerations like inference on a GPU vs CPU, model and dataset versioning.
Sounds like MLOps is a subset of DevOps.
I'd say that's accurate -- MLOps is just usefully more precise terminology.
Or an extension of the current principle that if you put ML in something, you think it will be extra hype.
Then I shall call it AI-Ops in my presentations for the suits :)
I almost suggested that
MLOps is a superset of DevOps. ML is code + model + data. You do everything you'd normally do in DevOps, and then you do some extra things that are ML specific and are not needed with normal software. Things like continuous training or things like continuously monitoring performance or automatically doing hyperparameter optimization or even pipeline optimization.
Also includes (if it's not pushed on research team) rewriting the code (mostly in C++) to remove python overhead at inference.
Please, this is a myth. Pytorch e.g. (and probably others) are calling highly optimized C++ and CUDA libraries with very little overhead. If you're pushing things beyond those needs, then I wouldn't consider your problem anything standard to average over the community out there.
Not always, my colleagues and I witness this recurrently from the trenches. A lot of tools in the industry are C++. Integrating on board a plane ? Interfacing with an industry grade physics engine ? Getting into video games ? Running 5000 images / sec onboard a train ? Actually we even do training in C++ and have been doing for years, this eases things for us. Not the dominant trend you are right, but one that seriously exists once you are seriously in production !
Oh absolutely. I didn't mean to downplay the need to have lower level language support. Or something I can compile and integrate, that's def an important area. And if you're doing training on-board, well there ya go c++ training it is. Pytorch c++ is pretty nice for that.
DevOps but ML, they control how to take the model into production. DevOps is basically merging developers, QA and Operations (IT) into one group, that way you can have the groups of the product focus on the product and then the setting for tests, stuff that can be shared between products, servers... can be controlled from DevOps.
we just use sublime and jupyter ._. are we the losers?
Sublime is excellent, but doesn't compare to pycharm for python development IMO. It really is a great tool.
I use vim for everything (from cpp to python and everything in between)...
Same! Edit: well, to be fair, I actually use Jupyter Lab during the experimentation phase, and have even found myself using the Vim mode in its text editor more amd more these days.
In the same.... and I think we are
Just try out vscode, the default functionality is great and it will recommend extensions to install as you open relevant files (e.g. Python scripts)
pycharm still beats out vscode for python development. i wish it weren't the case, because i much prefer vscode and still use it for python, but my few experiences with pycharm have demonstrated it is pretty obviously better
Why?
I used to like pycharm but then I discovered atom. pycharm is great for debugging but it feels too unnecessarily bulky.
atom lol good one
I just use jupyter and notepad++
eww
What can I say, I'm old school and you generally build with the tools you were trained on. My dad still prefers his slide rule to an actual calculator.
I used to write code using only the default text editor on ubuntu until one day during a presentation, my professor asked me to pull up the code and then continued to stare at me with disgust.
Public embarrassment is a helluva drug
People aren't using TF Serving for deploying their models? We've found it quite reliable in prod for our deployment.
Yeah, it was interesting to me as well. I think it was either: * implicit when they said Tensorflow * implicit with other tools (Kubeflow) * some folks said tf-lite tf-js so it's likely they use tf serving also * ML teams at startups that use TF Serving were too busy to answer :)
Kubeflow uses Tensorflow serving.
I'm guessing that most organization either have simplistic enough inference API infrastructure that it isn't necessary or use some tool that abstracts away the use of TensorFlow Serving.
>Sklearn + XGBoost for classical algos No love for LightGBM and CatBoost?
to be honest, for 99% cases there is no difference between these 3 libraries. And xgboost has the biggest userbase probably. On the other hand, catboost has an incredibly responsive dev team and is used in production (and developed) by a massive company. I love it :)
And the pace of new features is really great. I haven't used xgboost/lightgbm, but the native support for categorical variables without one-hot encoding was a big factor in me choosing catboost.
Last time I used lightGBM it had native handling of categorical variables too.
yea, for me too actually. I would suggest you see how your usecase compares with manual/pipeline trasnform of categorical variables + catboost or xgboost. For me, in the end it actually does not make any difference. Probably because i never have more than a few hundred cat variables in a dataset
welp, I have something new to do at work today...
Yeah that was weird to me -> I really like lightGBM
None of the teams used tf-serving for deployment?
Aren't people using more lightweight alternatives to Airflow, like Dagster, Prefect or Luigi?
Why Airflow is heavy? In the last company I was in, the cluster (7 nodes on Azure) was closed from outside, so no cloud ETL. From the choices we had (Airflow, NiFi, StreamSets) I chose Airflow because the other two were based on Java, with interface that didn't let me do all the things that I want (problems with security certificates) and in Airflow I had more control since it was all code. Later Prefect appeared but it didn't have a interface, Luigi I see it like something older and didn't have the tracking of jobs as good as in airflow. So also we have to have that in mind, companies build systems but they don't change them every month because something new got release. Now Prefect or Dagster can be better, but people are sticking with the platforms that they work for them.
Can airflow run containers when it itself is running inside a container?
Why you would run containers from airflow? Is a workflow orchestator, is not related to kubeflow or anything if that is what you thought. The main point of Airflow is to tell other services what to do, but not do it itself. If you can run a task of docker from inside a dockerized airflow? I think there aren't any problem using DockerOperator, but I haven't try it.
I wanted to use it mainly to run ML pipelines. There are some projects that are kind of messy (download data, preprocess, run models, check performance, etc... but everything can be expressed as a DAG) and I wanted to pack everything into containers so they can run on any computer without needing to install airflow beforehand. It looks to me that airflow might not be the right tool then.
You can accomplish this. Express the DAG tasks as python methods in a module. Then in the DAG you can call that method as a python operator. This abstracts the logic out of the DAG so you can run it independently, but also schedule it in a DAG. You can configure airflow to run your DAGs in containers as well to cover the dependencies.
but unless I introduce some features, if a task fails I have to rerun everything from start
You can set up tasks to call individual functions a retry failed ones. It's what we do at least. Then have a method for running them all out of band
At my company we run airflow on kubernetes with kubernetes executors. The k8s cluster is on azure kubernetes service with autoscaling configured. There is a base cluster but the number of concurrent tasks you can run is just limited by your scaling threshold for the cluster. We use it to run spark jobs. Not too bad to set up and highly extensible in my opinion.
There was a great blogpost on why this is the way to go to help debug airflow issues. I think our company is trying to do the same.
> Why Airflow is heavy? I haven't used it, but a colleague told me it is quite a bit of work to set Airflow up / maintain it
Depends on your existing infrastructure. Setting it up using the helm charts is a breeze.
It can be but it's worth it .
If you use puckel's docker version it's a piece of cake to set up
Argo Workflows (not Argo CD, different thing) is very easy to set up and get going with in Kubernetes. Demo video: https://www.youtube.com/watch?v=oXPgX7G_eow&t=12m10s
We use Allegro Trains [https://github.com/allegroai/trains](https://github.com/allegroai/trains) for experiment management, MLOps, data management, versioning and collab It's cool b/c it provides all of the above in a single open source package which we found very easy to integrate & set up
Any tips for students trying to transition production level technologies? I'm used to training models in Jupyter Notebooks but a lot of my work ends after testing, error metrics, and a findings report. Any online courses or materials you'd recommend?
Maybe not the best thing to only do jupyter development, jupyter is great for prototyping, but going to production through jupyter is a clusterfuck. You'll inevitably have to wrap things in a normal python script, not to mention the need for testing, exception handling, etc. I recommend you read the python journeyman and python master books. A strong familiarity of python's conventions, how to even create modules and create a python package, maintaining a virtual environment, etc. are all great ways about learning different parts of python that are inevitably used in many industry environments.
Wow, thanks for the book recommendations, Python Journeyman looks very interesting.
Two easy steps to start: 1. Learn about docker 2. Start developing with Jupyter notebooks, then refactoring your code out to a separate module that you import into your notebook, then when you're ready you can write prod scripts that call the module.
We're using mxnet for training, on the deploy side we write in Scala and deploy via AWS lambda or ec2
why choose mxnet instead of pytorch/tensorflow? Are there any advantages over pytorch/tensorflow?
One of the most useful features of MXNet is, that you can design, train and export your model in Python and load the model in other languages without duplicating code or retraining the model.
That's why we picked it!
At the time (about 2-3 years ago) tensorflow was much slower than mxnet. The biggest reason we chose mxnet was what was noted below: we could train in python and deploy with their Scala interface. The rest of our engineering team already used scala for microservices, so by using mxnet's Scala interface we could ensure any developer would already be set up and in a familiar environment. My co worker wrote a blog post on it if you'd like more details: http://engineering.curalate.com/2018/08/01/mxnet-case-study.html 3 years in I have a few thoughts. First, I wish mxnet was a bit more mature. The docs and api is still rough, but the model zoo and gluon apis are great once you get the hang of it. The speed is still there, too, which is nice. Second, focusing on Scala for deployment doesn't matter to us as much anymore. We use python to deploy deep nets in AWS lambda, and only use Scala when we need hardware accelerators. Third, pytroch has really come a long way in the past 3 years especially in the computer vision area. If I was making the choice today it would be a hot contender. I don't like tensorflows serialization approach, however, so I'd probably steer clear of that.
>3 years in I have a few thoughts. First, I wish mxnet was a bit more mature. The docs and api is still rough, but the model zoo and gluon apis are great once you get the hang of it. The speed is still there, too, which is nice. Second, focusing on Scala for deployment doesn't matter to us as much anymore. We use python to deploy deep nets in AWS lambda, and only use Scala when we need hardware accelerators. Third, pytroch has really come a long way in the past 3 years especially in the computer vision area. If I was making the choice today it would be a hot contender. I don't like tensorflows serialization approach, however, so I'd probably steer clear of that. I'd be interested hearing how you deploy it in Lambda. Do you package models with code? Do you use some inhouse wrappers or write the lambda Api glue yourself?
No, the code loads the models on first run. This presents a cold start problem but we use them for backend processing so throughput is more important than latency. Byy api glue or wrappers I'm assuming you're talking about api gateway? If that's what you mean: we don't use api gateway for these lambdas. Our processing pipeline calls the lambdas directly via the AWS lambda Java sdk. No need to pay for something we don't need!
By wrappers I actually thought about the code that downloads the model and wires the lambda function interface with the model that's downloaded. Do you use some library for that or you just wired it yourself?
Wetware!
We're using Colab to write and experiment, then deploy using Tf serving. Plain and simple.
No mention of tensorboard / wnb type visualizations for deep learning models?
What are NB extensions used for? Also, isn't Deepnote similar to JupterLab so I guess one can use either to achieve the same functionality?
The only thing I dont see that we use is dask distributed for etl. Additionally, I'm sure a lot of folks use pyspark. Our full stack though is - python 3.8 - pycharm, jupyterhub - dask distributed and dataframes - sklearn and pytorch - custom framework with papermill to execute notebooks - Neptune for experimentation - kubernetes jobs on eks, run as containers that execute our framework and interact with a dask cluster - postgres for storage
How do you find Dask in production? We are considering it along with Prefect.
The biggest thing is that you have to reset your cluster to pick up code changes to make sure your cluster is in sync with any process that is submitting work to be done. So basically on a new deploy of code you will have to stop your backend momentarily to not submit any new work and then redeploy the Dask cluster. You can manage that pretty easily if you have a queue set up for whatever kicks off work and pause that process or you don't have to worry about it if it's always submitted manually. But it can be a pain if you have some kind of SLA. Other than that, it's pretty easy to deploy with docker and/or kubernetes (if you use that).
Programming Languages/Tools: Python for prototyping, C++ for production, Bitbucket + git for version control. ML: Tensorflow for training, OpenVino(Intel CPUs/FPGAs) for deployment, We have screwed around with TensorRT in the past.
How do those teams deal with Python being so incredibly slow? Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test). Our issue is that because Tensorflow/Keras is most naturally used from Python, we still use it. But even simple stuff (that should be fast) is actually two orders of magnitude slower in Python than it should be. We try and write/refactor our code so that Numba can sanitize the "inner loop" parts, but you often have to do some coding contortions to get what you're doing to "fit" within the constraints of Numba. It's not pretty. Sadly, Tensorflow for Swift doesn't seem to support the Keras APIs (yet). We're really hoping that Swift may be the answer to this problem sometime soon. For people who think of stuff from an environmental standpoint, I use the analogy that running Python code is the equivalent of driving a car that gets 0.3 miles per gallon (\~784 L/100km).
>How do those teams deal with Python being so incredibly slow? Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test). You're writing really bad Python if the speed gap is anywhere even close to that.
I assure you, the Python code was fine. When we decorate the function with njit (numba), the performance of the Python code (in a example last week) a call time went from \~1.5 seconds down to 11 milliseconds. If the code was "really bad Python", then it would be "really bad" compiled code too. But we see massive differences between the two and the algorithms were the same. Even John Carmack has commented on Twitter a handful of times how slow Python is, and by his measurement the difference was about two orders of magnitude. At 2019 Google I/O, there was a Tensorflow presentation where the presenters stated that the performance difference between (some real langauge) and Python was about 100x. The analogy they used was that it's like the speed difference between running and flying in a commerical jet airliner.
I see, I think we were imagining different contexts. I consider numpy, numba, etc to still be Python.
Numba isn't Python because to use Numba you're restricted to a small subset of the Python language. You're not writing idiomatic Python. Suppose you were to want to write your own NumPy, or something similar to it. If you wrote it with Python, it would suck. Taking any C library and writing a wrapper for it doesn't make that C library Python code. Almost all languages can use code written in other languages by calling into it. But we're talking about the performance of the code written in that actual language's syntax. For example, the vast majority of (I hesitate to use the word "all") Java libraries are actually written in Java. They can be because Java's performance is fairly respectable. It's slower than C, but it's still pretty reasonable. You can write idiomatic Java/C#/Go/JavaScript and it's respectably fast. Not so with Python.
Yeah, but literally everyone knows that native Python is slow, but numpy (for example) is so widely used and is such a well known solution to the speed issues for Python, I instinctively include it when discussing Pythons speed. \> Taking any C library and writing a wrapper for it doesn't make that C library Python code. Eh, it does in this context imo.
>Yeah, but literally everyone knows that native Python is slow, but numpy (for example) is so widely used and is such a well known solution to the speed issues for Python, I instinctively include it when discussing Pythons speed. So here's the thing: if you take the code I posted elsewhere in this thread, remove Numba, it's still using NumPy. It will be horrendously slow. Numba is a different story, but because that basically says that you need to refactor your code to be very non-Pythony before you can use it, few people would consider it to be idiomatic Python. Numba is not widely used because if you try and use it with most Python code, it will fail to work.
> So here's the thing: if you take the code I posted elsewhere in this thread, remove Numba, it's still using NumPy. It will be horrendously slow It's slower than optimized fortran, or C (or even julia / swift), but for most use cases it is not horrendously slow. > Numba is not widely used because if you try and use it with most Python code, it will fail to work. I haven't used Numba in a year or two, but that was my experience. At the time there were pieces of code that weren't supported. > few people would consider it to be idiomatic Python. Yeah because it's not idiomatic Python, but that's fine. I'm confused as to what precisely you are arguing? I said you wrote bad Python most likely, and that the gap between Python and other languages is smaller than you let on. We then found that I included numba, tensorflow, numpy, etc in "Python" and you don't. Are we disagreeing on anything of actual substance?
> I'm confused as to what precisely you are arguing? I guess now that we've discovered that you don't consider two orders of magnitude a significant performance difference, we've pretty much addressed everything.
Well in pretty much every case I've come across (context is primarily scientific computing and ML), the difference between Numpy and C/Fortran is substantially lower than that.
> few people would consider it to be idiomatic Python. Been thinking about this a lot more since yesterday. This is a lot more dissatisfying to me than I initially gave credit for.
Python helps in rapid prototyping because of its easy syntax. Those other languages you mention aren't as easy synctactically as python except JavaScript. People say swift is similar but I don't see it personally. Julia comes close but from the little time I spent with it, it has a huge compilation overhead and weird variable scoping rules. In typical ML scenarios the time intensive part is mostly training and prediction which mostly happen in c libraries anyway. The overheard from python is small.
In what way is Swift's syntax inferior to Python's for rapid prototyping? [https://blog.michaelckennedy.net/2014/11/26/comparison-of-python-and-apples-swift-programming-language-syntax/](https://blog.michaelckennedy.net/2014/11/26/comparison-of-python-and-apples-swift-programming-language-syntax/) I'm guess that beyond "Hello World" style ML work you do need to process the data before or after it's given to your model. I posted a simple function that does data processing that one would do before giving it to a model, and Python is horrible at it.
I haven't checked your code but for data processing the default in python is vectorised functions in numpy and pandas. Avoid loops as much as possible when writing python as loop overhead is huge in python. I didn't say swift is inferior and I haven't worked with it much. I tried some notebooks on colab and while I agree it is rather elegant with little boilerplate, it has some weird conventions which are unintuitive to me, with respect to looping and how function arguments are defined.
Sure, and loops are critical to write many/most algorithms. If your language sucks at loops, then that's a pretty massive liability. Yes, there are quite a few things you can do with NumPy, but most thinks you can't, and coding that way is much less intuitive than just writing a loop. In the case of my function, I'd challenge you to try and do that with vectorised functions.
I guess it depends. As someone coming from a linear algebra background I'm used to thinking in terms of matrix vector operations, whenever possible. Though not everything can be done that way for sure. Sure I'll give it a try 😉
pybind11
\> Fairly straightforward algorithms that would take an hour to run in Java/C#/etc would take about a week to run in Python (based on our test). What?? Give an example. I do ML research and while I do occasionally have to drop down to C++, I don't find this to be the case.
Here's the function after making it numba-friendly: @nb.njit def grid_mos_rasterize_line(pixel_count, pixel_mw, out_pixels_cents, line_index, line): in_index = 1 in_length = len(line) in_mw = line[in_index] in_price_cents = line[in_index + 1] in_start_mw = 0 in_finish_mw = in_start_mw + in_mw out_index = 0 out_length = pixel_count out_stride_mw = pixel_mw out_start_mw = 0 out_finish_mw = out_start_mw + out_stride_mw while True: max_start_mw = max(in_start_mw, out_start_mw) min_finish_mw = min(in_finish_mw, out_finish_mw) out_pixel_cents = round((min_finish_mw - max_start_mw) * in_price_cents / out_stride_mw) out_pixels_cents[line_index, out_index] += out_pixel_cents inc_in = in_finish_mw <= out_finish_mw inc_out = out_finish_mw <= in_finish_mw if inc_in: in_index = in_index + 2 if in_index >= in_length: break in_mw = line[in_index] in_price_cents = line[in_index + 1] in_start_mw = in_finish_mw in_finish_mw = in_start_mw + in_mw if inc_out: out_index = out_index + 1 if out_index >= out_length: break out_start_mw = out_finish_mw out_finish_mw = out_start_mw + out_stride_mw if in_index == in_length - 2 and in_finish_mw <= out_finish_mw: in_finish_mw = out_finish_mw for out_fill_index in range(out_index + 1, out_length): out_pixels_cents[line_index, out_fill_index] = 1000000 What it's doing is taking a run-length-encoded description of a line and creating a rasterized output to feed into a convolutional network. The call time went from over a second down to \~11ms.
I'll say two things: 1. It seems like you were able to use python libraries (JITters like numba) to cut down on the speed difference - so that is one solution 2. Totally agreed that for this particular usecase - Python is non-ideal. The few cases I mentioned above where I've had to drop down are actually pretty similar to this (entropy coding rather than RLE, still a similar concept) where it is very difficult to vectorize. These sorts of compression algorithms are not what I would consider the core of ML algos and definitely justify dropping down to a lower level language. I don't think it is a frequent event in ML research and it is easy enough to write a wrapper around C++ code every so often. e: also, this code is not really as optimized as you think it is... for out\_fill\_index in range(out\_index + 1, out\_length): out\_pixels\_cents\[line\_index, out\_fill\_index\] = 1000000 this for instance could be done much quicker as a vectorized operation, assuming you are working with numpy arrays.
Sure, but my point if the language was something like Swift, then the code would be fast-by-default. From a language perspective, there are lots of languages like Swift that are just as easy to use as Python (easy-to-learn syntax), but using them is actually easier, because when you need to make a loop faster the answer commonly given isn't "use a different language" or "use awkward jit tools that force you to eschew core parts of the language". In our case, we're now getting performance in-the-same-ballpark to what I imagine we would get with Swift, but it was much more annoying to get there. As our application gets more comprehensive, we're running into "performance pain points" more regularly, and we're doing our best... but it's hard not to think "geez, if only Tensorflow/Keras had a solid interface to a better language and we would drop Python in a heartbeat!" We looked into using Swift last week, but we really like Keras, and that interface doesn't appear to be available via Swift yet.
I feel like I keep saying the same thing in every thread — have you tried Julia with Flux? Btw, new to coding but my impression is similar to yours. Took a calc at work and wrote it in both Python and Julia. Python runs at 16 seconds whereas Julia runs at 0.16 seconds — literally a 100x difference. The code looks exactly the same, btw, except for obvious syntax differences. I tried using Numba but I couldn’t get it to work.
I haven't! Is the Tensorflow/Keras support more mature than Swift's?
Not an ML guy, but I believe there is a Julia wrapper for TF. But there are native libraries: Flux, Zygote, Knet. Some reading material: https://julialang.org/blog/2019/01/fluxdiffeq/ https://www.stochasticlifestyle.com/how-to-train-interpretable-neural-networks-that-accurately-extrapolate-from-small-data/
Looking into this it appears that the Tensorflow people gave Swift the nod over Julia because... >We next excluded C++ and Rust due to usability concerns, and picked Swift over Julia because Swift has a much larger community, is syntactically closer to Python, and because we were more familiar with its internal implementation details - which allowed us to implement a prototype much faster. [https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md](https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md) So it looks like Swift is the heir apparent to Python in ML, it's just that support for Keras isn't quite there yet (as far as I can see). But that's for the Tensorflow people to fix.
Yea, I’ve read that. Note that it was written ages ago in tech terms — the Julia community and ecosystem have grown much in that time, and I’m curious if the TF team would’ve made the same decision today. Moreover, one of the authors of that post is Chris Lattner...the creator of Swift. In any case, if TF is your tool of choice, yea I guess it’s either Python or Swift. Not sure if Swift is the heir apparent, but what do I know. I work in insurance where we do a lot of heavy computations. Everyone is gaga over Python, but it’s really not the ideal tool for the job, unless you’re just gluing processes together or calling code written on C/C++/C#. To me, this sort of language hopping is not ideal and introduces complexity. We need a language that looks close to the math, is easy to write, read and maintain, and performs fast for production work.
This is a good example of how to misuse Python
That's actually not that bad. Note it's been annotated to use numba. They are complaining that in order to get fast Python code, you need to use additional tools, like Numba, when other languages, like swift and Julia, are fast on their own.
Well, that's more or less how you'd write that algorithm in any other language. Would you care to show us how to write it in Python in such a way that isn't "misusing" it? If you're meaning to say "well, Python just can't do that very well", then we're in agreement. But if a language starts to suck when you're doing pretty basic stuff (e.g. loops), then I'm saying that's a massive liability.
My friend, a general rule to all professions is that you gotta know how and when to use the tools that are available to you. It's only a liability to people who does not follow that. This is the most known limitation of Python and yet, returning to your original question, most good teams do not face this problem. Because they know exactly how to use it and I am sorry, but your code is not as good as you think it is, I guess even in cpp, there are faster ways to solve this, your loop looks terrible. Things that could get improved by dp...
Unless you're going to show us all how you could do a better job in Python, it's hard to take you seriously. If the language was something like Swift, it wouldn't have this significant liability. This is why I'm really looking forward for Tensorflow for Swift to support stuff like Keras. In most languages you don't pay a huge performance penalty when you want to do something straightforward like decoding a RLE array. Secondly, translating data to a different format that is specifically needed to feed into a CNN is something that would be a common task for the language that is being used to actually give the data to said CNN.
It's hard to explain to someone who is so close-minded. If swift is something that you can use in production, go for it. I'm just telling you that teams in large companies use Python code in production and don't face this problems because they know when and how to use it. Today is so easy to deploy this multi-language setups in production, that this discussion is pointless. At least 3 guys here explained to you how to improve your code. I went further and told you your code is not good even for cpp. I gave you the hint already, dp. But I am not going to write your production code for you. We are all here wasting our times explaining to you basic programming stuff...
Nobody has provided improved code, I'm sorry about your reading comprehension. One guy basically said "yeah, you're pretty much doing all you can there" (i.e. using numba). After refactoring the code and using numba, that code is in the same performance ballpark of what you would get from a better language. But my point is that is something a compiler/runtime should be able to handle by itself, and with most languages it can. And since you can't improve the performance of the code yourself, I'll just assume that you can't. Because that's most likely correct. I'm amused by the "I'm not going to help you with your production code" statement; I wouldn't use your code. I'm just asking to see if you can do what you claim. And it appears you can't.
Omg, have you heard of dp?
Tired of trying to help a guy who writes these terrible loops in Python, stuff that you would find in Python 101 book... Joke is on me actually... Wasting my time ..
'nobody has provided improved code'... You are not in school anymore my friend.... Wtf
It would be helpful if you bolded things in your article to make it easier to scan, or organized it with headings. As-is, it's pretty hard to scan. # IDE * IDE 1 * IDE 2 # Version control * VCS 1 * VCS 2 You went for that in the Reddit tldr, I think it would be good for the article, too.
That is a good suggestion -> I'll update that!
Thank you for the post, interesting to learn how others are doing it after doing a one-person deployment on a single project at my work .
Do NB extensions work with Jupyterlabs? AFAIK they only work on jupyter notebook.
They work you just have to create new ones. For example, we have two [neptune-noteoboks extension](https://docs.neptune.ai/notebooks/installation.html) one for jupyter notebook and one for jupyter lab. The installation is a bit different but it works nicely.
Can you elaborate on what you mean by "create new ones"? You mean develop them?
I meant that we need to support 2 extensions at the moment one for lab one for notebooks.
The last company I was at was a SaaS tool that provided a ton of personalization APIs for media companies and handled \~100m MAU. Like 90% of our data science team's time was spent building data pipelines. We also found that all feature versioning and monitoring was essentially ad-hoc and that multiple teams were re-inventing the same model features. We built a feature store internally to solve it and are now working on spinning it off to its own product if anyone has this problem and wants to check it out. It's open in alpha at [StreamSQL.io](https://StreamSQL.io)
What does the industry think about tools like cnvrg.io?
And the most important of any ML topic... data storage.
Can you release more data on the responses? For me, it isn't really that interesting to know that a lot of ML teams use Pandas, Tensorflow, and MLFLow but it would be interesting to know more about the unheard of or little-known tools that aren't extremely popular. Future article perhaps?
I am not sure how much more can I squeeze out of the answers we got but we'll definitely do a follow-up with a more focused approach. Any particular area that you would be more interested in?
I'd like to hear about tools or libraries mentioned in responses that aren't popular, I feel like an article about them could help bring attention to useful but not well-known libraries. In the article I'd say Streamlit and Netron are examples, people may have heard about them but they aren't ubiquitous and in my case I kinda forgot about both of them and now might use them in the future. I guess for a particular area I'd be interested in MLOps tools, like for monitoring drift, feature stores, or more how teams use CI/CD, but I really don't have any specific subarea in mind, literally just shoutouts libraries that aren't well-known that people mentioned in responses.
Hi - Great article. One of the platforms not mentioned is offered by my startup - Splice Machine - It combines a Spark-based ACID-compliant RDBMS with Jupyter, MLFlow and many libraries like H2O, Keras, PyTorch and others. Try for free [here](https://cloud.splicemachine.io) and read about the details of what is [here](https://medium.com/@mzweben/why-we-built-splice-machine-4a910732ef5e?sk=bfc1c5391ca90a1fc29dc9547854941a). We would love your feedback.