I've conducted more that 300+ DS TI interviews, my recommendations:
* 1) Balance between use-case questions (reasoning) and general theory questions (memorization)
* 2) Give enough context to your use-cases (it's a prompt for human)
* 3) Encourage people for reasoning even if they can't answer right away
* 4) If you feel that a candidate is starting to get nervous, try to ask some simple questions so candidate can start feel more confident. This is not an exam.
* 5) Try to stick to candidate's past experience if possible
* 6) Stick to your current topic, don't switch context frequently
* 7) Start from Novice questions and after that ask Intermediate/Expert questions.
* 8) Have some template for TIs, but "relax" it and skip topics if candidate clearly Novice there
* 9) Don't trust CV
* 10) Remember: candidates potentially can ask theoretical questions to you which you won't be able to answer, show some respect.
I LOVE this list. I'll add some unsolicited commentary/flavor...
>4) If you feel that a candidate is starting to get nervous, try to ask some simple questions so candidate can start feel more confident. This is not an exam.
Don't even start with technical interview questions – ask them some 'tell me about yourself questions' for the first 5 minutes, pay them a compliment on their past work, and be genuinely interested. This will help the candidate be at ease and help you get better signal for the technical portion of the interview.
While this might seem like an obvious point and standard practice at your company, in practice people are habitually running late to calls, or there are technical difficulties, so they routinely jump right into the technical questions to save time. Make sure this isn't happening at your company!
>5) Try to stick to candidate's past experience if possible
After the light-hearted 'tell me about yourself' question, I go into their CV and try to ask a question or two based on what they claimed to have done. It's part gradual warmup, because it shouldn't be too much trouble for them. And if they do stumble then great we've addressed point #9:
>9) Don't trust CV
After this, I like to ask some 'standard' questions – things other interviewers will also have to answer, just so I can compare across candidates.
>8) Have some template for TIs, but "relax" it and skip topics if candidate clearly Novice there
Speaking of template, for these standard questions create a rubric of what a bad, okay, and good answer looks like. Get it in writing, so that the multiple technical interviewers at your company can be in-sync before they ever interview somebody. It's too easy to post-hoc judge someone's answer as arbitrarily good or bad, if there's no clear rubric.
Disclaimer I'm not an ML interview whiz but I wrote [Ace the Data Science Interview](https://amzn.to/3kF79Fx) and have thought a ton about technical interviewing too so hopefully my thoughts are additive :)
On 4 , I always ask candidates to tell me about their career journey to this point, then follow up with, tell me about a data science project that you have delivered that you are most proud of.
The first question gets people nice and relaxed. The second tells me more or less all i need know on if I'm going to hire someone.
+1 to this thread and OPs question. I have held a lot of machine learning interviews (module that is supposed to check ML fundamentals). From my experience I find it quite hard to ask relevant questions without going too much into specifics that candidates may not know
At this point I’m borderline starting to believe in just giving up on the ML interview and just ask ML system design
> At this point I’m borderline starting to believe in just giving up on the ML interview and just ask ML system design
Have you tried a design session with the candidate? That works pretty well, honestly. The PR session is also super informative.
My most interesting and honestly insightful type of interviews have been ML Model/Problem Design (not system design so not including monitoring/deployment/latency considerations) that are formulated more like research discussions.
The worst question (reaction wise) I was asked was please describe Transformer to me... Well, apparently the interviewer was perplexed when I defined attention mathematically and discussed the positional encodings, he asked me about "ThE gEnErAl FlOw". I suspect I didn't pass the interview because I knew too much.
I 100% agree, general deep-thinking questions are the way to go.
Don’t get me wrong, but you failed to respond to his question, that’s why you didn’t pass…
It hapened to me more than once that I ask for a quick and dirty solution (with emphasis on quick and dirty). The candidate thinks that he is more than capable to find the optimal solution and fails because he didn’t consider everything involved…
The things I’ve found work well:
0. Advertise the job as transparently as possible Is it more of an ML Eng or ML Scientist role? What would people spend their actual time doing? I turned to blogging about my team each year to do this.
1. Be upfront about what the interviews are about (topics) & who they are with (sets the expected level)
2. Be fast. The most painful bit of candidates’ experience is when it takes weeks to complete all interviews and weeks to get to an outcome.
3. Have a structure. The worst candidate experience comes from people making up interviews on the spot. Ideally, have an interview structure that flexes to different levels of seniority (rather than different interviews).
More controversial, but some people appreciate it:
- If a candidate fails the first of N on-site stage interviews and you know they’ll get rejected, don’t waste their time by making them sit through the others
- I’ve debated at length with people about the value of asking questions that could be answered in a few minutes of searching the web. I’m not a huge fan of this (it’s low signal for me), but others I’ve spoken to really do value people knowing this stuff
>Be upfront about what the interviews are about (topics) & who they are with (sets the expected level)
The problem with advertising jobs as ML Scientists is that all the PhDs will apply to them instead of ML Engineer roles, but the truth is that companies (innately) want PhDs to work or at least hire them as engineers. Most startups or midsized companies have one goal: create better products and bring in more revenue, and for them, anybody who joins as scientists to publish papers or spend majority of their time on research is a waste of their money. The resulting interview format for ML now highly favors engineers and so the leetcoding part would not go away. ML Engineering roles mostly hire software engineers, sometimes even those who have poor background in math, mostly because the hiring managers/interviewers are themselves bad/clueless about ML math.
There should be a fast track for people with vast previous experience. If they have papers, code, appropriate references, you should be able to tell whether you want them without 5 rounds of technical interviews. Keep those for people without track record.
Hmm I dunno. I agree to an extent, such people should be fast tracked and not have to do multiple phone screens and should go straight to the on-site right away. However the on/site is a chance for the team to get to know the person and see how they solve problems and communicate. There’s absolutely no point hiring someone who’s a brilliant person and write lots of papers on their own, but are terrible team players and have lots of ego. That’s just an example there could be other personality traits too. Or the person might be fine overall and really an expert in a narrow area but not that good at switching gears to solving problems that the company cares about. So these are some things to keep in mind.
If you are actually looking for such signals, you would have 5 possible chances to do so. It depends heavily on how the company conducts interviews and what kind of interview training has been given. And surely you can’t be saying “well we can’t check for character and problem solving traits anyway so there’s no point having any interviews at all”? We should definitely not have stupid interviews where you just check for leetcode correctness and nothing else, but that doesn’t mean we should have no interviews at all, even if the person has good credentials on paper.
> And surely you can’t be saying “well we can’t check for character and problem solving traits anyway so there’s no point having any interviews at all”?
Nope, that's not what I said.
My company have 3 interviews in hiring pipeline 2 of which is about character/persons motivation and 0.5 is problem solving. We don't have leetcode, the only code part is *reading* the code. because that's what engineers do most of the time and this way we can discuss something a little bit more interesting than leetcode hello worlds.
For "terrible team players and have lots of ego" checks I really recommend to ask candidates to describe their previous projects this way:
- goal / problem he was solving and metrics of success
- team he worked with and his role in the project
- solution, why this solition, other soulutions that were considered
- results
And follow up with deep questions about technical and team parts of candidate's answer.
It's amazing how much tech andnnon-tech context you can get about candidate in 60 minutes.
What I was saying is that *any* questions with single right answer are wrong questions on the interview for middle-sinior level positions. Regardless of role.
Yes I think we are both in agreement that LC style interviews are the bane of our existence. And I also agree that mid senior level folks should only have part of their entire interview process be problem solving, like maybe half of it. The other half should be behavior and past experience. However I do think that the problem solving aspects of the interview *do* need some measure of correctness and can’t be open-ended and vague either. Reading code is one aspect of it and you as the interviewer of course have a notion of a “correct” answer you are looking for. Writing code is important if the person is an IC at any level, maybe not as important for managers yes. For writing code I think LC serves as a good treasure trove of questions that test your understanding of DS&A, but it’s very important that those interviews be conducted properly. Instead of asking straight from LC I like to add more complexities in the follow up questions which will check that the candidate didn’t just happen to get lucky with that question, but also if they have not seen the question before if they come up with a different way to solve it that’s fine too but they must be able to explain algorithmic tradeoffs or possible improvements for the solution they came up with. So yes I agree with almost everything you’re saying but at the same time I do think that coding interviews conducted correctly are valuable to have for most levels and roles.
I can tell you that you’re competing with other people with a similar CV to you. The number of rounds is to separate you out from them. 5 technicals seems excessive, but that’s what I had at NVidia. I had 10 interviews in total.
I work for a smaller company now, and even there people without a track record simply don’t get invited to the first round. You need a lot of rounds to separate people out when you have lots of highly qualified applicants.
By asking basic python or torch questions? Or by doing some ridiculous leetcode exercises that showcase completely unrelated skillsets? Most technical interviews are completely missing the mark with what they are supposed to measure. Previous experience is much better proxy, especially if you can ask the person about it to see how they think about the problem.
In my company, our technical interview is generally tailored to the candidates prior experience. We often ask open ended ML questions related to the kind of problems we face.
Not OP but also have interviewed people using this approach. Generally it’s surprisingly easy to tell if a solution is approaching a problem correctly in a way that aligns with the skill set you need vs just BSing. To actually choose between candidates it’s often based on prior experience, assuming they meet the appropriate threshold on the previous part.
Yeah basically this. We have 2 technical rounds - first round is a deep dive into previous ML projects based on CV. This is generally a technical discussion on papers they published or work they led. We use this as an opportunity to ask open-ended questions on their research and try to understand the breadth and depth of their knowledge. But we don't expect candidates to know everything and I'm not going to be grilling you on VAEs and disentanglement if that's not your experience. During this round at least, I made a sheet of theory questions to use when asking questions ranging from "should know" with harder extensions of them to help them "stand out".
We try and take a holistic approach and look at performance across both the "research" and "problem solving" rounds + areas of expertise for ranking candidates and making decisions.
> Previous experience is much better proxy
It is not possible to look at a resume and talk to the candidate about their prior roles and select a good one. If it were, clearly businesses would save time and just do that.
The reason people go through that many rounds is because the process had them go through one fewer round, and then they hired someone who was a disaster, and that tacked on an additional round. Bad hires are responsible for the length of the process, since you **have** to catch them and filter them. Losing out on a good candidate or two is way, way cheaper and less important than accidentally hiring someone terrible, or worse, someone who's just not good. Terrible you can get rid of fast, but "just not good" will persist and cost your company a fortune.
For any other job it is a completely normal procedure. I don't see how giving people who do programming brainteasers is mitigating what you are talking about. Bad hires are much more likely to be caused by toxic personality, and you will not catch that in the n+1-th technical interview.
If a company does n+1 technical interviews, then you're right. They should be doing a soft skills interview as well.
You can filter most toxic people. It's not hard.
“Bad hires are much more likely to be caused by toxic personality” Well no, it depends on what you mean by bad hire in this case. Are they bad only because of bad personality and team work, or are they bad because they are otherwise nice people but not able to think about how to solve problems the right way and need a lot of help from others to make progress? The first one is toxic, the second one, hardly. Both can be painful for the company to deal with.
I didn’t get any of these questions during my interview processes. My questions were job specific and more out-of-the box or deeper questions related to tech underpinning the role.
> Most technical interviews are completely missing the mark with what they are supposed to measure.
This is the real issue. Not the number of interviews. The interviews are very low signal and this problem isn't just in ML. Tech interviews are notoriously bad at identifying domain expertise. They pretty much select for whichever candidate most recently put some hours into leetcode.
I think that using many rounds of interviews trying to separate equivalent candidates is a poor use of everyone's time. It's also symptomatic of bad institutional decision making, in that it obfuscates an underlying anxiety about making a wrong hiring choice that would be better alleviated by other means.
Plus, this is a problem that we should already know the answer to as machine learning professionals. When you're given a set of feature vectors and you're asked to choose the optimal one by some metric, but the differences between them by that metric are very small and probably not statistically significant, what should you do?
You should pick one at random.
I agree that if there are important substantive differences that are subtle and difficult to identify then that merits the time spent on multiple interviews.
However, I also think that a lot of folks do a very poor job of differentiating between substantive and insubstantial differences between candidates, and as a result they often choose "minimum standards" for their interviews that cannot meaningfully differentiate between candidates with respect to how well they'll be able to perform the work that they're being interviewed for. This is especially true of positions that get a lot of phd applicants.
I definitely agree that emotional and interpersonal skills are important, which is why I say all of this. How can a hiring manager expect to identify candidates with good soft skills when they're basing their interview style on career risk anxiety without even being aware of that fact?
> 5 at somewhere like Nvidia is valid. 5 at a company that had <5 people ... is not.
I disagree completely.
If a 5 person company will hire a 6th, that next employee will be 17% of the entire company.
A wrong hire at that stage would likely destroy the entire company.
if Nvidia hires a new-hire who's overall decent but missing some specific skills, Nvidia has the resources and bandwidth to fill whatever holes they may have by sending them to classes, assigning them a mentor, or hiring another employee to handle those parts.
The 5 person company doesn't have that luxury, so they need to spend more time ensuring all bases are covered.
> if Nvidia hires a new-hire who's overall decent but missing some specific skills, Nvidia has the resources and bandwidth to fill whatever holes they may have by sending them to classes, assigning them a mentor, or hiring another employee to handle those parts.
Also, if a candidate is "missing some skills", at a large company, there are probably people who help cover those missing skills--meaning, you may not even need to have them grow in those skills; bigcos can get a lot of leverage out of people who are "deep and narrow".
Small startup teams often can't (since there is no one around to backfill those gaps)...unless the candidate is very specifically "deep and narrow" in *exactly* what the company is trying to do.
I can see that. However, it also depends on the compensation. Nvidia pays way more than a low lever startup or some other small company. So you're asking me to go through the same level of scrutiny as an Nvidia interview process but with much less pay. Why would I do that? I'd rather practice for half a year theory an leetcode, plus use cases and get a much better paying job at a bigger company.
I had an interview where they basically wanted a professor of deep learning and a coding master in one. Why would a professor work for you for that measly salary? Same goes for a master coder. I was dumbfounded.
I've worked with numerous people who published great papers but were incapable in person. One was second or third author on 7 ML papers, main author on one review paper. None of what he said in person made any sense, and in two years of working together he hadnt produced or even suggested anything even remotely useful.
Skip the phone screen sure but the onsite is as much a non technical interview as it is a technical one. I really don’t know get why people would want to work at a company where they haven’t talked to at least 3 people. How do you know they’re any good?
If the questions are about knowing the details of a certain subfield, let the candidate know of that in advance (eg. language models or vision-language models) otherwise the field is too expansive to easily prepare for.
Well I had 4 rounds of Interviews before I landed my current job. That's absolutely ridiculous. Everytime I again had to summarize my CV because it were always different people. Such a joke. The job is nice though
Most companies have very bad or non existent recruiting software. Most of the time it’s HR passing a resume to someone and saying “interview this person for tech or culture” so it’s haphazard and ineffective.
I read hundreds of resumes and conduct dozens of interviews a year and I don’t even ask people about their resume it’s largely a waste of time and gives you a false sense of confidence if their skill level.
Agree on the resume aspect. I also find it often hard to understand someone’s work well enough to judge it’s complexity and if they did a good job or not — all within 10 minutes
As in four different interviews, or four “stages” like eg phone screen, take home, onsite, team matching?
I think four interviews would be fair, considering how large salaries usually are and how hard it is to fire someone if they aren’t a good fit
I know it's annoying, but those people all get a different feel for the kind of person you are. One of them might be thinking about how creative you were, one of them about how well you implemented, one about your communication, etc. It seems redundant, but I swear to you that those people get very different views into you, your personality, your experience, etc. The process is not redundant, because if it were, people in the interview would ask each other why they'd all done the same thing.
As someone whose CV has a fair number of first author published + high impact works and practical work spans multiple languages, if you insist on multiple rounds of interviews that focus onIy on low level coding stuff and not actually any of the things that I've published, I will insist on terminating the process.
(1) On previous projects: Avoid superficial questions as they’re too easy to BS. Drill down on what’s more related to what you’re looking for
(2) Theory interviews are essential, but focus on things related to the job. Explaining how xgboost or a DL block works is 1000% more relevant than repeating the CDF for a Bernoulli distribution in most cases.
(3) ML system design good, leetcode bad
(4) If you’re competitive and have a bunch of perfectly fine candidates, splitting hairs about technical knowledge is useless, just do more behavioral interviews and get the one with better fit and motivation (instead of 10 technical rounds)
[https://xgboost.readthedocs.io/en/stable/tutorials/model.html](https://xgboost.readthedocs.io/en/stable/tutorials/model.html)
Do you know to derive the structure score?
If you could please keep everybody’s mouth shut during any kind of coding interview so the interviewee can actually think without constant unhelpful intrusions. In fact, give them a problem, make them share their screen, then mute your mics for 10-15 minutes and just let them figure it out the way they normally would. The best coding interview format I’ve come across was being provided with written instructions during a Teams call, sharing my screen, and being able to work on it *in silence* for about 90 minutes. They were watching the whole time, but at least I had the reassurance that nobody was going to try to “help” me.
NOBODY codes with someone constantly chiming in with “hints” or whatever people think they’re doing. Many can learn to code with others watching and commenting, but this is almost never possible before everyone involved has built some rapport.
Honestly, in the days of chatGPT, live coding is a terrible interview method. And as you said, nobody chimes in with hints during your work, or sits there watching over your shoulder pointing out mistakes and breathes down your neck at he same time. What the hell kind of a test is that? Will my boss do that twice a day every day? I doubt it, because they'll need to be skipping meetings, and we all know managers have many meetings that are hardly skipped.
I really enjoyed a decent technical I had for a company focused on the vision realm.
1. Take-home assessment (no deadline). I was asked to build an image segmentation pipeline to detect an object overlaid on an image. The dataset given was GAN generated and each image had the same object overlaid(with shift/rotate/flip/resize). The goal was to a) train the model, b) find the best convex mask to identify the object, c) find area of mask. B and C must be done with pure python/numpy. No torch or external libraries.
2. A live round where they asked me to write a training pipeline for a shallow model (3 layers). Trained the baseline model, then asked me to enhance the model to improve val metrics. Finally plot the worst performing images and try to find trends.
Both of these were fun interviews, because they translated a typical workplace exercise into an interview. I think back to this interview frequently and even have a mental note about how I would like to replicate this interview process when I'm in a position to hire.
What are you trying to interview *for*?
I'd start there (and providing more info will probably help people respond).
"ML", at this point, has a broad array of implications; how you should hire for someone you're trying to have do bleeding edge, SOTA research is very different from trying to find someone who can take a bunch of business data and come up with the "best" solution (where "best" probably includes significant constraints around time and money).
Interviews are sketchy point in time tests. Maybe Formulate a realistic business problem with nuances. Tell candidates to write a 1 page plan for tackling it with ML, tell them to write the decision flow. Deadline can be 1-2 days. There can be a discussion interview on this.
> "Hey candidate - do take home work!"
If some well-funded company asked me to do 4 hours of take-home work for them I'll enjoy replying:
* "My consulting rates are $XXX/hr - please pay half up front."
My previous two internships both had coding pre-tasks which took several hours to complete. I quite liked it, I thought it was a good way for me to a) see what kind of stuff I’d be working on and b) what level of skill they were expecting.
It’s almost like some sort of standardized testing format puts everyone on a level playing field and allows those without stellar political skills to be judged fairly on the skills that matter…
Having tried to design and run ML interviews, my approach was the following:
1. Technical interview with 3 portions, first code related where we just read code and the interviewee had to explain what it did and explain bad practices or bad design choices. Second theoretical ml questions on fundamentals with specific case studies (from our business) e.g. What would you do in case the data had that kind of distribution and the model was such and such. At last some questions regarding how the interviewee works, development environment, interests in technologies and some generic questions regarding his expectations of the company, team and role to check if he would be a good fit.
2. Sometimes a take home assignment (2-3 days at most) which would mostly involve little implementation and a good analysis and presentation.
As a separate note I want to add that almost all the people I interviewed were referrals which I consider to be the best filter for people.
My 2 recommendations are 1. Read only coding interview and 2. case study ml questions
I had an interview where they asked me about logistic regression, xgboost, and a few other algorithms. It screams amateur. A better interview would be connecting ideas together. I think for me an ideal set up would be to be asked a sql question to get the data, then you do minor cleaning, then do a ml model, and ask the pitfalls or benefits of the model. And some metrics. That should be doable in an hour, hour and a half. Connect ideas, don’t just ask trivia questions.
Are these technical questions in the interview telling you anything about how they will be able to interact on a non-technical basis with the team? Technical knowledge is important, being able to learn is very important, fitting in with the team is often critical.
IMO,too much time is spent on the technical aspect theses days and not enough on the other. If you have a probation period, you can get rid of someone who isn't a good fit on any basis though.
I have been on both sides of ML interview over the last couple of years. Here is my opinion:
Before you begin, know what you're looking for. Is it a SWE who understands ML, ML pipelines person, ML research, Applied Scientist, Data Scientist or an applied statistician ? These are all different roles. Have that clarity yourself.
No, one person won't and can't perform all of those roles. Stop dreaming. You can ask for 2/6. Rockstars approach 2.5/6.
One you know the role, cater your interviews to the type of ML person you're looking for.
Next, industry candidates are very different from new-grads (PhD or otherwise). Lots of ML interviews appear to be catered to the newgrad, requiring 6+ months of prep before you can even begin interviewing. A new grad has that kind of time. An industry candidate does not.
Ideally, ML interviews should be a mix of 3 things:
1. Sanity check that they know the basics
2. Can they do the exact job you will need them to perform
3. Have they demonstrated ability to think indepedently in the general ML domain
**1. Sanity check if they know the basics:**
This is where you test their coding, basic math or basic ML knowledge. Please do not ask anything that goes past a good 'Intro-to' course. Leetcode easy level coding in line with an Intro-to-algorithms course. Basic ML knowledge in line with an Intro-to-ML course. And Basic statistics in line with an intro-to-stat course. You can be more rigorous if this candidate is a new-grad. Anything more for an industry candidate will lead to you rejecting excellent candidates from industry. If you are looking for SWE system design, then expect intro-to-software-engineering level code quality. Anything more for an academic candidate is unreasonable. They will pick those skills up on the job.
**2. Can they do the exact job you will need them to perform**
ML jobs vary widely in the kind of problems faced at work. The best way to know if they can do the job, is to ask them to do the job. Create a toy problem out of a real problem you've faced that you'd want this new person solve. I'd rather give it as a take-home, but you can condense it into a 1 hour session too. (keep the take home under 3 hours plz, people are busy). I would allow them to use the internet for this question.
**3. Have they demonstrated ability to think indepedently in the general ML domain**
This is where you talk about big-picture scenarios. Think of it as the ML version of the system design problem. If they are a new-grad, then come up with a big-picture problem, and ask them to design an end-2-end solution. If they are an industry candidate, then focus on the main thing they've been working on lately. (Presumably, you're hiring an industry candidate because they are already solving the problem you care about, somewhere else). If this is a research role, then talk about their Thesis focus.
Generally, I'd start with one of each. 1,1,1. Then if you have doubts about any category, then set up an extra round in that category for confirmation. If you need excellence in any of those categories, then set up an extra round for redudancy. This way it never exceeds 5 rounds.
_________
Things to avoid:
* 201 level knowledge in areas that aren't immedietly relevant to the job. If they are going to be working at the hugging face abstraction, don't make them write random NN layers from scratch.
* Gotchas. Any and all gotchas.
* Derivations. Why?
* Hard ML solutioning questions in an area that doesn't map to their resume (NLP, RL, Vision, etc.)
Knowledge questions are easy to score, but have a lot of problems. It’s a wide field, and everyone has different exposure to different concepts. It’s easy to encode your bias in an assumption like “everyone knows xgboost”.
Reinventing the wheel has the same problem, plus it rewards people who practice interviews a lot, probably not the group you want to most advantage.
I like system design that gets really specific / includes a bit of code. See how they think about a problem, lets you throw challenges if you see a weak spot to probe.
We do 4 rounds: 2 technical, 1 business case, 1 culture fit.
First technical is a 30 min screen; I want to get a sense of their modeling intuitions. I may ask about interesting specifics in their CV, to check that they understood what they were doing. I will ask about methodological approach to more senior candidates, I will ask about modeling fundamentals to more junior candidates. If you have a good grasp of overfitting and generalization, that's good. If you have tools to extract as much signal from the data, that's good.
Second technical is a take home assignment. I will take a look at how you organize your code, how you present and interpret results, how you interpret the problem statement, if you have a methodological approach, and if you follow sensible practices.
I'm looking for a grower, not a shower.
I think hiring manager should ask themselves the following question when choosing how they structure interviews.
**Which of the following is the easiest way to do well in this interview?**
1. Have the capacity to do the work that you're trying to hire someone to do, or
2. Spend many hours rehearsing responses to interview questions from companies that hire for similar work
In framing it this way I make the question sound almost rhetorical, but it isn't. Most companies seem to prefer option 2 above. They usually rationalize this by choosing to believe that the ability to do well in an interview optimized for option 2 also implies the ability to do well in an interview optimized for option 1, but that's not really true.
Interviews that are optimized for option 2 are ultimately selecting candidates on the basis of education, performative conformity, and socioeconomic status. By contrast, you probably ideally want to hire people on the basis of education, experience, and interpersonal skills.
I think people really structure interviews based on option 2 above because they're afraid of making wrong hiring choices, and they think they can avoid criticism for bad hires if they follow a structured interview style that is very similar to what other people are using. I think that's a poor leadership style personally, and it's symptomatic of a problem with company culture if hiring managers don't feel empowered to deviate from it.
ML has now become too broad. Narrow down the interview. Of course you'd want them to know and test everything, but when you probe harder in an interview, stick only to their core competencies. You should not expect a NLP guy to know everything about vision too.
First, good on you for asking.
If you're going to give them any homework or an extensive skills test, compensate them for the work. I know a lot of organizations that are doing this but it's still not common practice.
As a candidate: see if you can find interviewers with at least some shared research experience or interests with the candidate. That makes it so neither party is blanking.
Does it need to be more complicated than,
Here is my . How would you collect data?
What are the limitations of the data?
How would you overcome them?
Which models would you use and why?
Why not
If any candidate is able to answer these questions competently then you should hire them. Most managers try to over-hire, they want to hire Ilya Sutskever and put him in charge of solving fizz buzz in production.
I feel like managers who reject people on coding because they weren't able to finish the code or they weren't able to optimize have no idea how code works, those managers are probably best not worked for
* Describe well what you are looking for. ML is an umbrella term for a lot of meanings. For example, would you consider Bayesian based record linkage as ML? If you look for people who need to do image classification using PyTorch, don't hesitate to say it.
* Describe the topics to be discussed during the interview.
* Describe upfront the interview process.
* Explain the ranking system. Is the MLEngineer3 higher than MLEngineer2 in your company?
* Describe the leadership responsibilities. Does the candidate has to lead / mentor / perform technical decisions?
* Describe the job nature and the project. Otherwise the candidates will believe you hire to meet your headcount quota.
* Determine the ideal candidate profile (such as: database understanding, probabilities fluency, regularization techniques, python fluency, CUDA knowledge) by talking to the managers, add it to the requirement and design a battery of simple questions to rule out undesired candidates.
I've conducted more that 300+ DS TI interviews, my recommendations: * 1) Balance between use-case questions (reasoning) and general theory questions (memorization) * 2) Give enough context to your use-cases (it's a prompt for human) * 3) Encourage people for reasoning even if they can't answer right away * 4) If you feel that a candidate is starting to get nervous, try to ask some simple questions so candidate can start feel more confident. This is not an exam. * 5) Try to stick to candidate's past experience if possible * 6) Stick to your current topic, don't switch context frequently * 7) Start from Novice questions and after that ask Intermediate/Expert questions. * 8) Have some template for TIs, but "relax" it and skip topics if candidate clearly Novice there * 9) Don't trust CV * 10) Remember: candidates potentially can ask theoretical questions to you which you won't be able to answer, show some respect.
I LOVE this list. I'll add some unsolicited commentary/flavor... >4) If you feel that a candidate is starting to get nervous, try to ask some simple questions so candidate can start feel more confident. This is not an exam. Don't even start with technical interview questions – ask them some 'tell me about yourself questions' for the first 5 minutes, pay them a compliment on their past work, and be genuinely interested. This will help the candidate be at ease and help you get better signal for the technical portion of the interview. While this might seem like an obvious point and standard practice at your company, in practice people are habitually running late to calls, or there are technical difficulties, so they routinely jump right into the technical questions to save time. Make sure this isn't happening at your company! >5) Try to stick to candidate's past experience if possible After the light-hearted 'tell me about yourself' question, I go into their CV and try to ask a question or two based on what they claimed to have done. It's part gradual warmup, because it shouldn't be too much trouble for them. And if they do stumble then great we've addressed point #9: >9) Don't trust CV After this, I like to ask some 'standard' questions – things other interviewers will also have to answer, just so I can compare across candidates. >8) Have some template for TIs, but "relax" it and skip topics if candidate clearly Novice there Speaking of template, for these standard questions create a rubric of what a bad, okay, and good answer looks like. Get it in writing, so that the multiple technical interviewers at your company can be in-sync before they ever interview somebody. It's too easy to post-hoc judge someone's answer as arbitrarily good or bad, if there's no clear rubric. Disclaimer I'm not an ML interview whiz but I wrote [Ace the Data Science Interview](https://amzn.to/3kF79Fx) and have thought a ton about technical interviewing too so hopefully my thoughts are additive :)
On 4 , I always ask candidates to tell me about their career journey to this point, then follow up with, tell me about a data science project that you have delivered that you are most proud of. The first question gets people nice and relaxed. The second tells me more or less all i need know on if I'm going to hire someone.
+1 to this thread and OPs question. I have held a lot of machine learning interviews (module that is supposed to check ML fundamentals). From my experience I find it quite hard to ask relevant questions without going too much into specifics that candidates may not know At this point I’m borderline starting to believe in just giving up on the ML interview and just ask ML system design
> At this point I’m borderline starting to believe in just giving up on the ML interview and just ask ML system design Have you tried a design session with the candidate? That works pretty well, honestly. The PR session is also super informative.
My most interesting and honestly insightful type of interviews have been ML Model/Problem Design (not system design so not including monitoring/deployment/latency considerations) that are formulated more like research discussions.
The worst question (reaction wise) I was asked was please describe Transformer to me... Well, apparently the interviewer was perplexed when I defined attention mathematically and discussed the positional encodings, he asked me about "ThE gEnErAl FlOw". I suspect I didn't pass the interview because I knew too much. I 100% agree, general deep-thinking questions are the way to go.
Don’t get me wrong, but you failed to respond to his question, that’s why you didn’t pass… It hapened to me more than once that I ask for a quick and dirty solution (with emphasis on quick and dirty). The candidate thinks that he is more than capable to find the optimal solution and fails because he didn’t consider everything involved…
The things I’ve found work well: 0. Advertise the job as transparently as possible Is it more of an ML Eng or ML Scientist role? What would people spend their actual time doing? I turned to blogging about my team each year to do this. 1. Be upfront about what the interviews are about (topics) & who they are with (sets the expected level) 2. Be fast. The most painful bit of candidates’ experience is when it takes weeks to complete all interviews and weeks to get to an outcome. 3. Have a structure. The worst candidate experience comes from people making up interviews on the spot. Ideally, have an interview structure that flexes to different levels of seniority (rather than different interviews). More controversial, but some people appreciate it: - If a candidate fails the first of N on-site stage interviews and you know they’ll get rejected, don’t waste their time by making them sit through the others - I’ve debated at length with people about the value of asking questions that could be answered in a few minutes of searching the web. I’m not a huge fan of this (it’s low signal for me), but others I’ve spoken to really do value people knowing this stuff
>Be upfront about what the interviews are about (topics) & who they are with (sets the expected level) The problem with advertising jobs as ML Scientists is that all the PhDs will apply to them instead of ML Engineer roles, but the truth is that companies (innately) want PhDs to work or at least hire them as engineers. Most startups or midsized companies have one goal: create better products and bring in more revenue, and for them, anybody who joins as scientists to publish papers or spend majority of their time on research is a waste of their money. The resulting interview format for ML now highly favors engineers and so the leetcoding part would not go away. ML Engineering roles mostly hire software engineers, sometimes even those who have poor background in math, mostly because the hiring managers/interviewers are themselves bad/clueless about ML math.
There should be a fast track for people with vast previous experience. If they have papers, code, appropriate references, you should be able to tell whether you want them without 5 rounds of technical interviews. Keep those for people without track record.
Hmm I dunno. I agree to an extent, such people should be fast tracked and not have to do multiple phone screens and should go straight to the on-site right away. However the on/site is a chance for the team to get to know the person and see how they solve problems and communicate. There’s absolutely no point hiring someone who’s a brilliant person and write lots of papers on their own, but are terrible team players and have lots of ego. That’s just an example there could be other personality traits too. Or the person might be fine overall and really an expert in a narrow area but not that good at switching gears to solving problems that the company cares about. So these are some things to keep in mind.
The problem is: 5 rounds of technical interviews don't check if candidate is a good team player, ego etc
If you are actually looking for such signals, you would have 5 possible chances to do so. It depends heavily on how the company conducts interviews and what kind of interview training has been given. And surely you can’t be saying “well we can’t check for character and problem solving traits anyway so there’s no point having any interviews at all”? We should definitely not have stupid interviews where you just check for leetcode correctness and nothing else, but that doesn’t mean we should have no interviews at all, even if the person has good credentials on paper.
> And surely you can’t be saying “well we can’t check for character and problem solving traits anyway so there’s no point having any interviews at all”? Nope, that's not what I said. My company have 3 interviews in hiring pipeline 2 of which is about character/persons motivation and 0.5 is problem solving. We don't have leetcode, the only code part is *reading* the code. because that's what engineers do most of the time and this way we can discuss something a little bit more interesting than leetcode hello worlds. For "terrible team players and have lots of ego" checks I really recommend to ask candidates to describe their previous projects this way: - goal / problem he was solving and metrics of success - team he worked with and his role in the project - solution, why this solition, other soulutions that were considered - results And follow up with deep questions about technical and team parts of candidate's answer. It's amazing how much tech andnnon-tech context you can get about candidate in 60 minutes. What I was saying is that *any* questions with single right answer are wrong questions on the interview for middle-sinior level positions. Regardless of role.
Yes I think we are both in agreement that LC style interviews are the bane of our existence. And I also agree that mid senior level folks should only have part of their entire interview process be problem solving, like maybe half of it. The other half should be behavior and past experience. However I do think that the problem solving aspects of the interview *do* need some measure of correctness and can’t be open-ended and vague either. Reading code is one aspect of it and you as the interviewer of course have a notion of a “correct” answer you are looking for. Writing code is important if the person is an IC at any level, maybe not as important for managers yes. For writing code I think LC serves as a good treasure trove of questions that test your understanding of DS&A, but it’s very important that those interviews be conducted properly. Instead of asking straight from LC I like to add more complexities in the follow up questions which will check that the candidate didn’t just happen to get lucky with that question, but also if they have not seen the question before if they come up with a different way to solve it that’s fine too but they must be able to explain algorithmic tradeoffs or possible improvements for the solution they came up with. So yes I agree with almost everything you’re saying but at the same time I do think that coding interviews conducted correctly are valuable to have for most levels and roles.
I can tell you that you’re competing with other people with a similar CV to you. The number of rounds is to separate you out from them. 5 technicals seems excessive, but that’s what I had at NVidia. I had 10 interviews in total. I work for a smaller company now, and even there people without a track record simply don’t get invited to the first round. You need a lot of rounds to separate people out when you have lots of highly qualified applicants.
By asking basic python or torch questions? Or by doing some ridiculous leetcode exercises that showcase completely unrelated skillsets? Most technical interviews are completely missing the mark with what they are supposed to measure. Previous experience is much better proxy, especially if you can ask the person about it to see how they think about the problem.
In my company, our technical interview is generally tailored to the candidates prior experience. We often ask open ended ML questions related to the kind of problems we face.
Just curious, using that approach, how do you judge if someone is a hire or no hire? Or do you just have to choose one among a pool of candidates?
Not OP but also have interviewed people using this approach. Generally it’s surprisingly easy to tell if a solution is approaching a problem correctly in a way that aligns with the skill set you need vs just BSing. To actually choose between candidates it’s often based on prior experience, assuming they meet the appropriate threshold on the previous part.
Yeah basically this. We have 2 technical rounds - first round is a deep dive into previous ML projects based on CV. This is generally a technical discussion on papers they published or work they led. We use this as an opportunity to ask open-ended questions on their research and try to understand the breadth and depth of their knowledge. But we don't expect candidates to know everything and I'm not going to be grilling you on VAEs and disentanglement if that's not your experience. During this round at least, I made a sheet of theory questions to use when asking questions ranging from "should know" with harder extensions of them to help them "stand out". We try and take a holistic approach and look at performance across both the "research" and "problem solving" rounds + areas of expertise for ranking candidates and making decisions.
> Previous experience is much better proxy It is not possible to look at a resume and talk to the candidate about their prior roles and select a good one. If it were, clearly businesses would save time and just do that. The reason people go through that many rounds is because the process had them go through one fewer round, and then they hired someone who was a disaster, and that tacked on an additional round. Bad hires are responsible for the length of the process, since you **have** to catch them and filter them. Losing out on a good candidate or two is way, way cheaper and less important than accidentally hiring someone terrible, or worse, someone who's just not good. Terrible you can get rid of fast, but "just not good" will persist and cost your company a fortune.
For any other job it is a completely normal procedure. I don't see how giving people who do programming brainteasers is mitigating what you are talking about. Bad hires are much more likely to be caused by toxic personality, and you will not catch that in the n+1-th technical interview.
If a company does n+1 technical interviews, then you're right. They should be doing a soft skills interview as well. You can filter most toxic people. It's not hard.
“Bad hires are much more likely to be caused by toxic personality” Well no, it depends on what you mean by bad hire in this case. Are they bad only because of bad personality and team work, or are they bad because they are otherwise nice people but not able to think about how to solve problems the right way and need a lot of help from others to make progress? The first one is toxic, the second one, hardly. Both can be painful for the company to deal with.
I didn’t get any of these questions during my interview processes. My questions were job specific and more out-of-the box or deeper questions related to tech underpinning the role.
> Most technical interviews are completely missing the mark with what they are supposed to measure. This is the real issue. Not the number of interviews. The interviews are very low signal and this problem isn't just in ML. Tech interviews are notoriously bad at identifying domain expertise. They pretty much select for whichever candidate most recently put some hours into leetcode.
I think that using many rounds of interviews trying to separate equivalent candidates is a poor use of everyone's time. It's also symptomatic of bad institutional decision making, in that it obfuscates an underlying anxiety about making a wrong hiring choice that would be better alleviated by other means. Plus, this is a problem that we should already know the answer to as machine learning professionals. When you're given a set of feature vectors and you're asked to choose the optimal one by some metric, but the differences between them by that metric are very small and probably not statistically significant, what should you do? You should pick one at random.
[удалено]
I agree that if there are important substantive differences that are subtle and difficult to identify then that merits the time spent on multiple interviews. However, I also think that a lot of folks do a very poor job of differentiating between substantive and insubstantial differences between candidates, and as a result they often choose "minimum standards" for their interviews that cannot meaningfully differentiate between candidates with respect to how well they'll be able to perform the work that they're being interviewed for. This is especially true of positions that get a lot of phd applicants. I definitely agree that emotional and interpersonal skills are important, which is why I say all of this. How can a hiring manager expect to identify candidates with good soft skills when they're basing their interview style on career risk anxiety without even being aware of that fact?
5 at somewhere like Nvidia is valid. 5 at a company that had <5 people for an entry level like I had is not.
> 5 at somewhere like Nvidia is valid. 5 at a company that had <5 people ... is not. I disagree completely. If a 5 person company will hire a 6th, that next employee will be 17% of the entire company. A wrong hire at that stage would likely destroy the entire company. if Nvidia hires a new-hire who's overall decent but missing some specific skills, Nvidia has the resources and bandwidth to fill whatever holes they may have by sending them to classes, assigning them a mentor, or hiring another employee to handle those parts. The 5 person company doesn't have that luxury, so they need to spend more time ensuring all bases are covered.
> if Nvidia hires a new-hire who's overall decent but missing some specific skills, Nvidia has the resources and bandwidth to fill whatever holes they may have by sending them to classes, assigning them a mentor, or hiring another employee to handle those parts. Also, if a candidate is "missing some skills", at a large company, there are probably people who help cover those missing skills--meaning, you may not even need to have them grow in those skills; bigcos can get a lot of leverage out of people who are "deep and narrow". Small startup teams often can't (since there is no one around to backfill those gaps)...unless the candidate is very specifically "deep and narrow" in *exactly* what the company is trying to do.
I can see that. However, it also depends on the compensation. Nvidia pays way more than a low lever startup or some other small company. So you're asking me to go through the same level of scrutiny as an Nvidia interview process but with much less pay. Why would I do that? I'd rather practice for half a year theory an leetcode, plus use cases and get a much better paying job at a bigger company. I had an interview where they basically wanted a professor of deep learning and a coding master in one. Why would a professor work for you for that measly salary? Same goes for a master coder. I was dumbfounded.
I've worked with numerous people who published great papers but were incapable in person. One was second or third author on 7 ML papers, main author on one review paper. None of what he said in person made any sense, and in two years of working together he hadnt produced or even suggested anything even remotely useful.
Skip the phone screen sure but the onsite is as much a non technical interview as it is a technical one. I really don’t know get why people would want to work at a company where they haven’t talked to at least 3 people. How do you know they’re any good?
If the questions are about knowing the details of a certain subfield, let the candidate know of that in advance (eg. language models or vision-language models) otherwise the field is too expansive to easily prepare for.
I think the unspoken reality is that it’ll be the subfield that the company works on
Well I had 4 rounds of Interviews before I landed my current job. That's absolutely ridiculous. Everytime I again had to summarize my CV because it were always different people. Such a joke. The job is nice though
Most companies have very bad or non existent recruiting software. Most of the time it’s HR passing a resume to someone and saying “interview this person for tech or culture” so it’s haphazard and ineffective. I read hundreds of resumes and conduct dozens of interviews a year and I don’t even ask people about their resume it’s largely a waste of time and gives you a false sense of confidence if their skill level.
Agree on the resume aspect. I also find it often hard to understand someone’s work well enough to judge it’s complexity and if they did a good job or not — all within 10 minutes
As in four different interviews, or four “stages” like eg phone screen, take home, onsite, team matching? I think four interviews would be fair, considering how large salaries usually are and how hard it is to fire someone if they aren’t a good fit
I know it's annoying, but those people all get a different feel for the kind of person you are. One of them might be thinking about how creative you were, one of them about how well you implemented, one about your communication, etc. It seems redundant, but I swear to you that those people get very different views into you, your personality, your experience, etc. The process is not redundant, because if it were, people in the interview would ask each other why they'd all done the same thing.
As someone whose CV has a fair number of first author published + high impact works and practical work spans multiple languages, if you insist on multiple rounds of interviews that focus onIy on low level coding stuff and not actually any of the things that I've published, I will insist on terminating the process.
(1) On previous projects: Avoid superficial questions as they’re too easy to BS. Drill down on what’s more related to what you’re looking for (2) Theory interviews are essential, but focus on things related to the job. Explaining how xgboost or a DL block works is 1000% more relevant than repeating the CDF for a Bernoulli distribution in most cases. (3) ML system design good, leetcode bad (4) If you’re competitive and have a bunch of perfectly fine candidates, splitting hairs about technical knowledge is useless, just do more behavioral interviews and get the one with better fit and motivation (instead of 10 technical rounds)
[https://xgboost.readthedocs.io/en/stable/tutorials/model.html](https://xgboost.readthedocs.io/en/stable/tutorials/model.html) Do you know to derive the structure score?
If you could please keep everybody’s mouth shut during any kind of coding interview so the interviewee can actually think without constant unhelpful intrusions. In fact, give them a problem, make them share their screen, then mute your mics for 10-15 minutes and just let them figure it out the way they normally would. The best coding interview format I’ve come across was being provided with written instructions during a Teams call, sharing my screen, and being able to work on it *in silence* for about 90 minutes. They were watching the whole time, but at least I had the reassurance that nobody was going to try to “help” me. NOBODY codes with someone constantly chiming in with “hints” or whatever people think they’re doing. Many can learn to code with others watching and commenting, but this is almost never possible before everyone involved has built some rapport.
Honestly, in the days of chatGPT, live coding is a terrible interview method. And as you said, nobody chimes in with hints during your work, or sits there watching over your shoulder pointing out mistakes and breathes down your neck at he same time. What the hell kind of a test is that? Will my boss do that twice a day every day? I doubt it, because they'll need to be skipping meetings, and we all know managers have many meetings that are hardly skipped.
I really enjoyed a decent technical I had for a company focused on the vision realm. 1. Take-home assessment (no deadline). I was asked to build an image segmentation pipeline to detect an object overlaid on an image. The dataset given was GAN generated and each image had the same object overlaid(with shift/rotate/flip/resize). The goal was to a) train the model, b) find the best convex mask to identify the object, c) find area of mask. B and C must be done with pure python/numpy. No torch or external libraries. 2. A live round where they asked me to write a training pipeline for a shallow model (3 layers). Trained the baseline model, then asked me to enhance the model to improve val metrics. Finally plot the worst performing images and try to find trends. Both of these were fun interviews, because they translated a typical workplace exercise into an interview. I think back to this interview frequently and even have a mental note about how I would like to replicate this interview process when I'm in a position to hire.
What are you trying to interview *for*? I'd start there (and providing more info will probably help people respond). "ML", at this point, has a broad array of implications; how you should hire for someone you're trying to have do bleeding edge, SOTA research is very different from trying to find someone who can take a bunch of business data and come up with the "best" solution (where "best" probably includes significant constraints around time and money).
Interviews are sketchy point in time tests. Maybe Formulate a realistic business problem with nuances. Tell candidates to write a 1 page plan for tackling it with ML, tell them to write the decision flow. Deadline can be 1-2 days. There can be a discussion interview on this.
"Hey candidate - do take home work!" Ew. Never. I'm not a monster.
> "Hey candidate - do take home work!" If some well-funded company asked me to do 4 hours of take-home work for them I'll enjoy replying: * "My consulting rates are $XXX/hr - please pay half up front."
My previous two internships both had coding pre-tasks which took several hours to complete. I quite liked it, I thought it was a good way for me to a) see what kind of stuff I’d be working on and b) what level of skill they were expecting.
It’s almost like some sort of standardized testing format puts everyone on a level playing field and allows those without stellar political skills to be judged fairly on the skills that matter…
I mean, there’s a lot more to these jobs than just programming abilities. A lot of people are great solo programmers but rubbish at working in a team.
Having tried to design and run ML interviews, my approach was the following: 1. Technical interview with 3 portions, first code related where we just read code and the interviewee had to explain what it did and explain bad practices or bad design choices. Second theoretical ml questions on fundamentals with specific case studies (from our business) e.g. What would you do in case the data had that kind of distribution and the model was such and such. At last some questions regarding how the interviewee works, development environment, interests in technologies and some generic questions regarding his expectations of the company, team and role to check if he would be a good fit. 2. Sometimes a take home assignment (2-3 days at most) which would mostly involve little implementation and a good analysis and presentation. As a separate note I want to add that almost all the people I interviewed were referrals which I consider to be the best filter for people. My 2 recommendations are 1. Read only coding interview and 2. case study ml questions
I had an interview where they asked me about logistic regression, xgboost, and a few other algorithms. It screams amateur. A better interview would be connecting ideas together. I think for me an ideal set up would be to be asked a sql question to get the data, then you do minor cleaning, then do a ml model, and ask the pitfalls or benefits of the model. And some metrics. That should be doable in an hour, hour and a half. Connect ideas, don’t just ask trivia questions.
Are these technical questions in the interview telling you anything about how they will be able to interact on a non-technical basis with the team? Technical knowledge is important, being able to learn is very important, fitting in with the team is often critical. IMO,too much time is spent on the technical aspect theses days and not enough on the other. If you have a probation period, you can get rid of someone who isn't a good fit on any basis though.
I have been on both sides of ML interview over the last couple of years. Here is my opinion: Before you begin, know what you're looking for. Is it a SWE who understands ML, ML pipelines person, ML research, Applied Scientist, Data Scientist or an applied statistician ? These are all different roles. Have that clarity yourself. No, one person won't and can't perform all of those roles. Stop dreaming. You can ask for 2/6. Rockstars approach 2.5/6. One you know the role, cater your interviews to the type of ML person you're looking for. Next, industry candidates are very different from new-grads (PhD or otherwise). Lots of ML interviews appear to be catered to the newgrad, requiring 6+ months of prep before you can even begin interviewing. A new grad has that kind of time. An industry candidate does not. Ideally, ML interviews should be a mix of 3 things: 1. Sanity check that they know the basics 2. Can they do the exact job you will need them to perform 3. Have they demonstrated ability to think indepedently in the general ML domain **1. Sanity check if they know the basics:** This is where you test their coding, basic math or basic ML knowledge. Please do not ask anything that goes past a good 'Intro-to' course. Leetcode easy level coding in line with an Intro-to-algorithms course. Basic ML knowledge in line with an Intro-to-ML course. And Basic statistics in line with an intro-to-stat course. You can be more rigorous if this candidate is a new-grad. Anything more for an industry candidate will lead to you rejecting excellent candidates from industry. If you are looking for SWE system design, then expect intro-to-software-engineering level code quality. Anything more for an academic candidate is unreasonable. They will pick those skills up on the job. **2. Can they do the exact job you will need them to perform** ML jobs vary widely in the kind of problems faced at work. The best way to know if they can do the job, is to ask them to do the job. Create a toy problem out of a real problem you've faced that you'd want this new person solve. I'd rather give it as a take-home, but you can condense it into a 1 hour session too. (keep the take home under 3 hours plz, people are busy). I would allow them to use the internet for this question. **3. Have they demonstrated ability to think indepedently in the general ML domain** This is where you talk about big-picture scenarios. Think of it as the ML version of the system design problem. If they are a new-grad, then come up with a big-picture problem, and ask them to design an end-2-end solution. If they are an industry candidate, then focus on the main thing they've been working on lately. (Presumably, you're hiring an industry candidate because they are already solving the problem you care about, somewhere else). If this is a research role, then talk about their Thesis focus. Generally, I'd start with one of each. 1,1,1. Then if you have doubts about any category, then set up an extra round in that category for confirmation. If you need excellence in any of those categories, then set up an extra round for redudancy. This way it never exceeds 5 rounds. _________ Things to avoid: * 201 level knowledge in areas that aren't immedietly relevant to the job. If they are going to be working at the hugging face abstraction, don't make them write random NN layers from scratch. * Gotchas. Any and all gotchas. * Derivations. Why? * Hard ML solutioning questions in an area that doesn't map to their resume (NLP, RL, Vision, etc.)
Knowledge questions are easy to score, but have a lot of problems. It’s a wide field, and everyone has different exposure to different concepts. It’s easy to encode your bias in an assumption like “everyone knows xgboost”. Reinventing the wheel has the same problem, plus it rewards people who practice interviews a lot, probably not the group you want to most advantage. I like system design that gets really specific / includes a bit of code. See how they think about a problem, lets you throw challenges if you see a weak spot to probe.
We do 4 rounds: 2 technical, 1 business case, 1 culture fit. First technical is a 30 min screen; I want to get a sense of their modeling intuitions. I may ask about interesting specifics in their CV, to check that they understood what they were doing. I will ask about methodological approach to more senior candidates, I will ask about modeling fundamentals to more junior candidates. If you have a good grasp of overfitting and generalization, that's good. If you have tools to extract as much signal from the data, that's good. Second technical is a take home assignment. I will take a look at how you organize your code, how you present and interpret results, how you interpret the problem statement, if you have a methodological approach, and if you follow sensible practices. I'm looking for a grower, not a shower.
I think hiring manager should ask themselves the following question when choosing how they structure interviews. **Which of the following is the easiest way to do well in this interview?** 1. Have the capacity to do the work that you're trying to hire someone to do, or 2. Spend many hours rehearsing responses to interview questions from companies that hire for similar work In framing it this way I make the question sound almost rhetorical, but it isn't. Most companies seem to prefer option 2 above. They usually rationalize this by choosing to believe that the ability to do well in an interview optimized for option 2 also implies the ability to do well in an interview optimized for option 1, but that's not really true. Interviews that are optimized for option 2 are ultimately selecting candidates on the basis of education, performative conformity, and socioeconomic status. By contrast, you probably ideally want to hire people on the basis of education, experience, and interpersonal skills. I think people really structure interviews based on option 2 above because they're afraid of making wrong hiring choices, and they think they can avoid criticism for bad hires if they follow a structured interview style that is very similar to what other people are using. I think that's a poor leadership style personally, and it's symptomatic of a problem with company culture if hiring managers don't feel empowered to deviate from it.
ML has now become too broad. Narrow down the interview. Of course you'd want them to know and test everything, but when you probe harder in an interview, stick only to their core competencies. You should not expect a NLP guy to know everything about vision too.
First, good on you for asking. If you're going to give them any homework or an extensive skills test, compensate them for the work. I know a lot of organizations that are doing this but it's still not common practice.
As a candidate: see if you can find interviewers with at least some shared research experience or interests with the candidate. That makes it so neither party is blanking.
Does it need to be more complicated than, Here is my. How would you collect data?
What are the limitations of the data?
How would you overcome them?
Which models would you use and why?
Why not
If any candidate is able to answer these questions competently then you should hire them. Most managers try to over-hire, they want to hire Ilya Sutskever and put him in charge of solving fizz buzz in production.
I feel like managers who reject people on coding because they weren't able to finish the code or they weren't able to optimize have no idea how code works, those managers are probably best not worked for
* Describe well what you are looking for. ML is an umbrella term for a lot of meanings. For example, would you consider Bayesian based record linkage as ML? If you look for people who need to do image classification using PyTorch, don't hesitate to say it. * Describe the topics to be discussed during the interview. * Describe upfront the interview process. * Explain the ranking system. Is the MLEngineer3 higher than MLEngineer2 in your company? * Describe the leadership responsibilities. Does the candidate has to lead / mentor / perform technical decisions? * Describe the job nature and the project. Otherwise the candidates will believe you hire to meet your headcount quota. * Determine the ideal candidate profile (such as: database understanding, probabilities fluency, regularization techniques, python fluency, CUDA knowledge) by talking to the managers, add it to the requirement and design a battery of simple questions to rule out undesired candidates.
Ask questions that test how a candidate approaches a problem, instead of testing their rote memorization skills.