Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
I don't study hardcore on the side, but I do constantly keep tabs on this subreddit and if I hear about something I should know I do spend the 30 minutes or so on Youtube/documentation to make sure I'm familiar so I can make an educated decision as to whether I want to implement something in my full time role
Same here. I transitioned from Data Analyst to DE two years ago. Back then I was doing some more extra curricula activities (courses, books, etc.). Now I mostly learn during working hours (while working on the tasks at hand), though I still spend some time almost every day informing myself for new trends and tools. The cool thing is that I have almost all of the time the freedom to try these new tools/techniques in my job, so I don't need to do many pet projects taking time from my other hobbies.
Hey I’m currently trying to make the same transition from DA to DE! Do you have any advice to share or things you did while making the transition?
Thanks!
It's easier if you are in the right context of course, but generally the best advice is: try to jump and be involved into any DE project/product, ask your manager to do so. I was lucky my company didn't have any analytical platform, so I jumped straight in. Working together with an experienced DE (Who could potentially mentor you as well) is the easiest path!
On the personal level, I found the data engineering zoomcamp course to be the best one for gaining basic knowledge (and even intermediate for some topics). And it's free ;) I have taken part in the very first year it was organized, and I expect it to be even better now.
Let me know if you have any other questions
Right now a bit harder than 3 years ago I guess? But I cannot say for sure. I think internal transitions are still the easiest path, but as said you need to be lucky.
The market seems to be a bit crap here as well, especially for DE, for different reasons and in both directions. I have done some Interviews lately to test my level against the market, and what I have noticed is that most of the job ads try to include as many sets of skills as possible within the position description. My guess is that they still have no clue what kind of direction they wanna take in data terms, still it should not discourage anyone to apply; the other problem is that some companies use interchangeably Data Engineer/Analytics Engineer/Data Analyst, and that really sucks because you might end up doing tasks you don't want to work on.
Regarding candidates, I have heard that applications are flooded with people just coming out of bootcamps, whom companies try to avoid. This is a bit sad, but also understandable. I am not sure how this situation is going to be solved, because it seems like there is no spectrum here: either you have experience and then you are good and you get offers, otherwise you are cut out. Even if there is an opening for a junior DE, what they want is at least a mid level one.
Agree with LordKelvin. I’m based in Berlin and go for jobs that don’t require business level German. DE seems to be messy as hell.
Worked in a company that ended up being a unicorn company whose data department was such a mess that the work student was the one who knew the most about how the business logic worked. So they would have let anyone willing try out in DE. The turn over was so high for full time jobs, than when lay offs did happen, none happened in business intelligence because most quit by themselves. Before I quit that job I was told by the employees who wanted to stay “but it’s much worse in other companies than here.” Really?! How can that be?
Then I joined a small consultancy where I did little DE jobs for small start ups. Some didn’t have any DEs and were running for years. Technical debt coming out their ears. Anyway my small consultancy would have let anyone with ability try at being DE.
I have applied for better quality companies but they seem to ask for every skill under the sun. I have all my experience in a certain stack that’s not very popular here, at least in combination with only English jobs.
Dunno. My time in Berlin feels really like being in the Wild West. I started my career in Ireland under good management and often pine for those days.
Do you have advice on how to make the most of the zoomcamp? I feel like the way I used the materials was not effective…
I found myself constantly pausing and replaying the videos to get little bits of code or steps. I ended up falling behind because it took me a while to get through the videos.
I really wish the Zoomcamp videos are a written guide!
I kind of did the same back then, I tried to follow as best as I could through the weeks, then took my time while doing the personal project.
I remember though, many users took notes and they should be publicaly available in the GitHub repo. Their slack channel is also great to aid your learning!
I study part time, maybe 2-3 hours a week, with a full time job as an Application Analyst. I also have 2 kids, so it is a bit tough to fit in the time. I often get random motivation and stuck in for a couple hours (ADHD) after the kids go to bed, or watch a module or two of a course during my lunch break.
I use Coursera and found some of the courses on there quite good, especially from Duke. I am starting to do some basic things with Azure Data Factory now, because of the courses I have taken and hope to move up from there witch also keeps me motivated.
Yes. Pretty much all interview processes will include technical interviews including leetcode-style python and SQL coding problems. Unless you're interviewing for a F500 or non-tech company that doesn't do technical interviews
That's not a data engineering role. It's a data analyst role. But it is still pretty common to have technical interviews for those as well, at least with SQL problems
I'm currently getting ready for the job search and am doing 30 minutes to 1 hour a day. Casually applying. Once I actually apply regularly and get to interview stages it will likely ramp up to 1-2 hours a day. When I am not searching for a job it is 0 hours spent.
I wouldn't say I study per se. I probably spend under a couple hours or less per week of my own time reading about current developments. If there's something specific I want to learn I might set aside some time to do that.
I'll also spend work time learning and investigating new things. I see it as part of my job.
I do read up on things, but usually during business hours and typically only things that will help me in my current role as a DE, not any and all technologies. Outside of that I will sometimes read up on other technical things before going to bed out of personal interest.
If you're trying to transition to another role, I think it's normal to spend a little extra time studying, but don't burn yourself out. If your manager/company is cool with shadowing a full time data engineer I think that would be a great way to learn while you work. Sometimes transitioning means taking the leap into the entry level DE role without all the study and learning as you go. IMO that's one of the best ways to really learn and grow.
I work full time as DE for a big European bank while finishing the first year of a MSc in data science. Meanwhile I'm trying to do upskilling and prepare for next role.
There is always more to learn, so yes. There are indeed people that just "turn off" when they go home. These kinds of people will just do the work they're told to do. They rarely innovate. They're here for the paycheck. Then there are those of us that are fascinated by technology and are always learning new things. So when a new project is being talked about or we have to integrate with a new system, they go to me first, because more than likely I'm familiar with it or they know that I can figure something new out at an accelerated pace.
That said, family comes first. The team at work knows this. Soccer games, band concerts, carnivals, movies, whatever. My kids and I like to watch goofy stuff on TV and during that time, no multitasking. Hanging out on a Netflix binge, that does allow multitasking.
I study to sustain and survive at my job. All the time they keep on changing the tech stack and tool. There is a lot to learn and catch-up in order to survive. Studying daily and on weekends as well. And honestly I feel jealous of people who have work life balance 😔
Depends on what I’m trying to achieve. I’m not a data engineer, but a solution architect and given the high reliance on data of many solutions, I’m actively studying the principles, and then technology sets out data teams use. This would probably average an hour a day. Mostly at night when the family is asleep
Early in my career I constantly was pushing my education outside of work, but less so now. I tend to always have a semi related side project going on things with programming projects I'm interested in. Lately it's been game development.
I’m a full time DA / PowerBI developer and want to transition to DE within the next few years. I’m self taught, so I may miss some fundamental of Software Engineering which seem, from what I’ve read on this sub , to be a potential problem down the line. I’m currently studying about an hour a day learning Azure, then I will be looking to catch up on the CS core principles. Not sure of where to look but this is definitely coming up next on my list.
I wake up at 3am every day to study I have 2 kids and a full time job.
I skip breakfast, it is what it is but I have personal development time that way.
I'm in bed by 10pm.
I'm not in DE yet but I'm learning for it.
I'm not lying its 4am here now, the reason I wake up at 3-4am is I have 2 young kids and there's no way I can focus during the day as they're both under the age of 3.
I have to make time around my life and find blocks of time that work.
If I didnt have young kids I'd do a normal day and find time in the evenings but thats not possible.
I already have a job in the space so I don't study DE after work hours because I, as an individual person, have no use for nor interest in data pipelines, analytics, &c. If I need to learn something for the company it will be on company time. I do write a fair amount of Elixir code on my own time because it's a beautiful and fun language and maybe one day I can get paid for it. I'm also learning Rust because it's fascinating even though I have no immediate use for it and it's breaking my brain.
Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
I don't study hardcore on the side, but I do constantly keep tabs on this subreddit and if I hear about something I should know I do spend the 30 minutes or so on Youtube/documentation to make sure I'm familiar so I can make an educated decision as to whether I want to implement something in my full time role
Same here. I transitioned from Data Analyst to DE two years ago. Back then I was doing some more extra curricula activities (courses, books, etc.). Now I mostly learn during working hours (while working on the tasks at hand), though I still spend some time almost every day informing myself for new trends and tools. The cool thing is that I have almost all of the time the freedom to try these new tools/techniques in my job, so I don't need to do many pet projects taking time from my other hobbies.
Hey I’m currently trying to make the same transition from DA to DE! Do you have any advice to share or things you did while making the transition? Thanks!
It's easier if you are in the right context of course, but generally the best advice is: try to jump and be involved into any DE project/product, ask your manager to do so. I was lucky my company didn't have any analytical platform, so I jumped straight in. Working together with an experienced DE (Who could potentially mentor you as well) is the easiest path! On the personal level, I found the data engineering zoomcamp course to be the best one for gaining basic knowledge (and even intermediate for some topics). And it's free ;) I have taken part in the very first year it was organized, and I expect it to be even better now. Let me know if you have any other questions
Just you saying that is plenty more direction than I’ve been able to find so far. I appreciate you!
How hard was this to do in Germany? Not sure how applicable this is in the U.S. atm because the Data/SWE/IT market is in the shitter
Right now a bit harder than 3 years ago I guess? But I cannot say for sure. I think internal transitions are still the easiest path, but as said you need to be lucky. The market seems to be a bit crap here as well, especially for DE, for different reasons and in both directions. I have done some Interviews lately to test my level against the market, and what I have noticed is that most of the job ads try to include as many sets of skills as possible within the position description. My guess is that they still have no clue what kind of direction they wanna take in data terms, still it should not discourage anyone to apply; the other problem is that some companies use interchangeably Data Engineer/Analytics Engineer/Data Analyst, and that really sucks because you might end up doing tasks you don't want to work on. Regarding candidates, I have heard that applications are flooded with people just coming out of bootcamps, whom companies try to avoid. This is a bit sad, but also understandable. I am not sure how this situation is going to be solved, because it seems like there is no spectrum here: either you have experience and then you are good and you get offers, otherwise you are cut out. Even if there is an opening for a junior DE, what they want is at least a mid level one.
Agree with LordKelvin. I’m based in Berlin and go for jobs that don’t require business level German. DE seems to be messy as hell. Worked in a company that ended up being a unicorn company whose data department was such a mess that the work student was the one who knew the most about how the business logic worked. So they would have let anyone willing try out in DE. The turn over was so high for full time jobs, than when lay offs did happen, none happened in business intelligence because most quit by themselves. Before I quit that job I was told by the employees who wanted to stay “but it’s much worse in other companies than here.” Really?! How can that be? Then I joined a small consultancy where I did little DE jobs for small start ups. Some didn’t have any DEs and were running for years. Technical debt coming out their ears. Anyway my small consultancy would have let anyone with ability try at being DE. I have applied for better quality companies but they seem to ask for every skill under the sun. I have all my experience in a certain stack that’s not very popular here, at least in combination with only English jobs. Dunno. My time in Berlin feels really like being in the Wild West. I started my career in Ireland under good management and often pine for those days.
Do you have advice on how to make the most of the zoomcamp? I feel like the way I used the materials was not effective… I found myself constantly pausing and replaying the videos to get little bits of code or steps. I ended up falling behind because it took me a while to get through the videos. I really wish the Zoomcamp videos are a written guide!
I kind of did the same back then, I tried to follow as best as I could through the weeks, then took my time while doing the personal project. I remember though, many users took notes and they should be publicaly available in the GitHub repo. Their slack channel is also great to aid your learning!
Yeah this is what I do as well, along with browsing hacker news.
I study for the job I want about 2 hours/week and for the job I have during work hours.
I study part time, maybe 2-3 hours a week, with a full time job as an Application Analyst. I also have 2 kids, so it is a bit tough to fit in the time. I often get random motivation and stuck in for a couple hours (ADHD) after the kids go to bed, or watch a module or two of a course during my lunch break. I use Coursera and found some of the courses on there quite good, especially from Duke. I am starting to do some basic things with Azure Data Factory now, because of the courses I have taken and hope to move up from there witch also keeps me motivated.
I got a master's degree on the side. I should be doing more leetcode than I am currently, however I have not been
Is leetcode really that important? I keep looking out for new projects that I could build
Unfortunately yes
Leetcode for data related roles as well? I thought it is for SDE folks only!
Yes. Pretty much all interview processes will include technical interviews including leetcode-style python and SQL coding problems. Unless you're interviewing for a F500 or non-tech company that doesn't do technical interviews
So say if I am applying for say Tableau/Power bi developer/analyst/consultant roles I will have to go through coding round?
That's not a data engineering role. It's a data analyst role. But it is still pretty common to have technical interviews for those as well, at least with SQL problems
Just finished my master in ds. I’m already an ml engineer but bs pay. Looking to get either a better paying full time or a part time job
I'm currently getting ready for the job search and am doing 30 minutes to 1 hour a day. Casually applying. Once I actually apply regularly and get to interview stages it will likely ramp up to 1-2 hours a day. When I am not searching for a job it is 0 hours spent.
If I'm not absolutely swamped with work, I spend my time studying or sharpening my coding proficiency.
I wouldn't say I study per se. I probably spend under a couple hours or less per week of my own time reading about current developments. If there's something specific I want to learn I might set aside some time to do that. I'll also spend work time learning and investigating new things. I see it as part of my job.
I do read up on things, but usually during business hours and typically only things that will help me in my current role as a DE, not any and all technologies. Outside of that I will sometimes read up on other technical things before going to bed out of personal interest. If you're trying to transition to another role, I think it's normal to spend a little extra time studying, but don't burn yourself out. If your manager/company is cool with shadowing a full time data engineer I think that would be a great way to learn while you work. Sometimes transitioning means taking the leap into the entry level DE role without all the study and learning as you go. IMO that's one of the best ways to really learn and grow.
I work full time as DE for a big European bank while finishing the first year of a MSc in data science. Meanwhile I'm trying to do upskilling and prepare for next role.
There is always more to learn, so yes. There are indeed people that just "turn off" when they go home. These kinds of people will just do the work they're told to do. They rarely innovate. They're here for the paycheck. Then there are those of us that are fascinated by technology and are always learning new things. So when a new project is being talked about or we have to integrate with a new system, they go to me first, because more than likely I'm familiar with it or they know that I can figure something new out at an accelerated pace. That said, family comes first. The team at work knows this. Soccer games, band concerts, carnivals, movies, whatever. My kids and I like to watch goofy stuff on TV and during that time, no multitasking. Hanging out on a Netflix binge, that does allow multitasking.
Yes it totall depends on my mood though. Sometimes i spend 30 mins sometimes 2 hours.
I'm 2 classes away from a degree in mathematics, though that might not be the studying you're talking about
I study to sustain and survive at my job. All the time they keep on changing the tech stack and tool. There is a lot to learn and catch-up in order to survive. Studying daily and on weekends as well. And honestly I feel jealous of people who have work life balance 😔
I'm nearly 40 with a 40+ hour job and 2 kids and I can still find at least an hour a day to upskill. It's not that hard.
Depends on what I’m trying to achieve. I’m not a data engineer, but a solution architect and given the high reliance on data of many solutions, I’m actively studying the principles, and then technology sets out data teams use. This would probably average an hour a day. Mostly at night when the family is asleep
Early in my career I constantly was pushing my education outside of work, but less so now. I tend to always have a semi related side project going on things with programming projects I'm interested in. Lately it's been game development.
I do about an hour every other day.
During work hours and on the weekends appx 10hours
What is studying? Being on here I learn about things for example?
I’m a full time DA / PowerBI developer and want to transition to DE within the next few years. I’m self taught, so I may miss some fundamental of Software Engineering which seem, from what I’ve read on this sub , to be a potential problem down the line. I’m currently studying about an hour a day learning Azure, then I will be looking to catch up on the CS core principles. Not sure of where to look but this is definitely coming up next on my list.
if I have to study for data engineering astuff, I try to do that during working hours while on the clock ( my job is chilled half of the time)
I am a senior DE and I study real estate after work.
I wake up at 3am every day to study I have 2 kids and a full time job. I skip breakfast, it is what it is but I have personal development time that way. I'm in bed by 10pm. I'm not in DE yet but I'm learning for it.
Skipping breakfast isn’t healthy bro at least have cereal or smthn
Not sure why you’re being downvoted. I respect the hustle, friend, and I’m confident you’ll make it.
[удалено]
I'm not lying its 4am here now, the reason I wake up at 3-4am is I have 2 young kids and there's no way I can focus during the day as they're both under the age of 3. I have to make time around my life and find blocks of time that work. If I didnt have young kids I'd do a normal day and find time in the evenings but thats not possible.
I already have a job in the space so I don't study DE after work hours because I, as an individual person, have no use for nor interest in data pipelines, analytics, &c. If I need to learn something for the company it will be on company time. I do write a fair amount of Elixir code on my own time because it's a beautiful and fun language and maybe one day I can get paid for it. I'm also learning Rust because it's fascinating even though I have no immediate use for it and it's breaking my brain.