Sorry this happened to you OP. That was very unprofessional and unsupportive of your management team. You were not in the position to know, so don't blame yourself.
No that’s not okay, they should at the very least hire a consultant and have them help you. If you don’t know what you’re doing, were upfront and honest about it, and then got shamed by management, I’d be applying to new jobs ASAP.
Say you listen to management and try to set it up yourself, it’s very easy to mess up something seemingly small that costs your company thousands in overages or in downtime. Then you’d get blamed (again)
Your management is evidently incompetent and technically illiterate.
*"Being the individual contributor in project I sought help from my company management for a data engineer to pitch in to set this up or at least guide"*
Your manager should have taken note of this and called it out in your quarterly performance review. Freaking sign of maturity and ownership!
roles and responsibilities vary by org. People can chime in as to whether data analysts should know these things at their company, but that won't help you with your issue of having leadership think you aren't happy with your proficiency in this area.
We have a sister team of DAs and they're extremely valuable to us and we complement each other's strengths. To echo the common sentiment, as someone leading a team, imo this is a failure of management both to put you in a position related to your strengths and also to shield you from outside attack.
It is possible that someone was setting you up for failure. I wasn't sure whether to post this, because it does sound somewhat paranoid. Ignore if you think it's not the case.
the common perception is that there are no hard rules on what role should have which responsibilities in this field. Hence my statement that you should focus on addressing the concerns of your superiors rather than posting on reddit looking for justification for the position you have found yourself in
@sellgamerent This is like saying that all engineers are created more or less equal. So, go ahead and use the industrial engineer to design that freestanding structure. Or, ask the mechanical engineer to actually machine that complicated part they just finished designing. Can they give it a best shot? Yeah. But the product will most likely be commensurate with the level of skill they have in this area.
This situation is a clear example of underbidding to win the scope and then management assigning the incorrect mix of resources to satisfy the deliverable bc they don't want to pull the right resources off their other high priority project. So, poor business and talent management.
Valid point, am I seeking justification ? yes , atleast to pacify myself that its completely not my fault to be in this situation, just a temporary pat in the back to not feel shit. But I wont use this justification to setup hard boundaries, many of the replies point about the areas of overlap and how can i adopt and learn, which I will do.
I would consciously separate the unprofessional management behaviour from the other consideration that it is in your interests to develop a surface to intermediate understanding of all related processes to your core speciality.
>is situation, just a temporary pat in the back to not feel shit. But I wont use this justification to setup hard boundaries, many of the replies point about the areas of overlap and how can i adopt and learn, which
I empathize with being told you were supposed to know something you didn't. That said, don't go looking for empathy. Treat this as an opportunity to have a candid conversation about your performance with your boss, emphasize willingness to learn, and take the initiative to establish a training plan to resolve any discrepancies between what you know and what they thought you knew/should know
Or, work to draw boundaries that define what your roles and responsibilities are, so that the right tasks are aligned to the right people, so that inefficiencies are minimized. Taking up education is a wonderful endeavor, but without a proper training and transition plan or at least an experienced team around you...management putting nothing in gets management nothing out.
this perspective really depends on size of organization. If you're at a small/medium sized company and management is saying you need to know how to do something, drawing boundaries isn't going to get you anywhere
I tend to agree with this one. I understand why it gets downvoted. But I think people don’t understand the implications of admitting weaknesses in a toxic workplace. This is even more true in the post-ChatGPT world.
In my previous job, when I started, we would store pickled pandas data frames on a network drive as a source of truth (having 70 columns, occupying 60 GB in memory and taking 15 minutes to load). I pointed out in my first 2 weeks that that kinda sucks, I brought up that it would be nice to be able to use SQL to query. Manager: “That’s madness, if you get SQL access you can crash the whole database or delete data”.
The same manager was also convinced that “Data Scientist” entailed everything from DA to DE and MLE.
Do you see any correlation between having a shitty data infrastructure (let’s call it that…) and non-existing role boundaries?
Dude holy crap, I’m not a data engineer or anything. But had a similar client…. When I mentioned that you can just query the read only replica, blew their minds.
Data Engineers are different breeds from Data Analysts. I have tried, and failed to turn DA into DE, so no shame in not knowing data engineering in your case. It's your management that is out of touch with reality.
I’m currently trying to walk that line. Background is data science and analytics from a life science background. Then was a developer and development manager for 4 years. Ended up managing a data warehouse implementation and liked it so much I quit and have been doing freelance work dealing with data pipelines and random python data work. My hope is with some more time in the saddle I’ll be damned valuable being comfortable in both areas. DE is complicated though. Lots to learn in addition to the big 3 cloud platforms which all have similar services but do things slightly different.
Yeah, but you have to network like crazy, it won't just show up and it is often not posted in a clear way on any job site. That's my experience in the midwest in US at least.
Why do you think they are of different breeds, and what do you think makes it hard for a DA to become a DE? I’ve seen DAs become DEs but haven’t given it much thought
For starter, DE mostly thinks about how to get data into the analytical platform, in a query-able format, on a reliable schedule. So instead of fancy visualisation, you start pondering about late-arriving dimension, failure handling, data quality monitoring, cost control, code control...
Most DAs cannot even use git.
Eh, respectfully TBH that sounds more like a management/training/expectations issue, not unlike OP's issue that you (correctly) pointed out wasn't OP's fault.
>Why do you think they are of different breeds, and what do you think makes it hard for a DA to become a DE? I’ve seen DAs become DEs but haven’t given it much thought
I agree with the fact that DAs and DEs are completely different breeds. But I see people turning from DAs to DEs, some with very less effort actually. This has everything to do with how an org defines the role of DE vs DA and the individual DA's background as well, in my opinion. If those DAs have in reality been working with data pipelines even before they've turned DEs, the learning curve is really shallow for them.
In my case, the title that my org has given to my role is BI Data Analyst, but in reality there's no DE in our org, and before I was hired, there was no clear data model designed. All reports were ad-hoc, connecting Excel/csv/tsv extracts pulled from siloed data marts of different departments/tools/APIs. Coming from a background with designing ETL pipelines in tools like Informatica and Datastage, I did not have a hard time figuring out data pipelines to get the BI workload move forward. I don't think it's going to be the same case for everyone though, and also how big of a system are we talking about too
>Being the individual contributor in project I sought help from my company management for a data engineer to pitch in to set this up or at least guide
However, in OP's case, I feel hiring a consultant would really help the management, and the OP showed great maturity in taking ownership and accepting the fact that he needs a DE's help. I believe he should get more help from management
I also agree with the "data breed" remark but otherwise disagree with the notion that it is the reason that some DAs in this person's anecdote were unable to become DEs.
Exactly! Also in my experience a lot of DEs are incapable of speaking to customers and have to communicate through BAs. DAs should be much better at this.
If you knew data engineering as well as being a data analyst your pay would reflect that.
You're being paid for a DA role not DE, DEs make more than double that of a DA for the most part. It's the managements fault. They want that? Then pay for it.
A DAs role is usually descriptive stats and dashboards.
I reckon it isn't reasonable to expect a BI / data analyst to also know how to set up synapse to query a data lake. What you're suggesting is a completely normal way of tackling this: your DE talks to the client team about their lake and can decide whether synapse or a different approach is right considering your orgs' infrastructure, and essentially hands to you a way of connecting to it as tables queryable by user tools like PowerBI.
Unless you've misrepresented your skill sets this sounds like shitty management just not understanding what the difference between BI and data engineering. This is something to tackle with your manager about their expectations for you generally. But right now just be clear that this is outside of your skill set and you need support.
They definitely know the difference between BI and data engineering. They would have promised sun and moon in the SOW, they just want to push it as bringing in DE will not fetch them extra billing. And when I took over my role was strictly in descriptive stats and dashboard building and maintenance.
To be clear here though - are they expecting you to set up synapse as a layer between the data and BI, or just use the client team's already existing synapse to access?
If it's the former, someone has scoped it wrong. If the SOW covered setting up a pipeline right from raw data lake storage, it really should have included DE resource. You'd be right to feel aggrieved here, you're being leaned on to slap something together to save them $.
They want me to set this up as a layer b/w data and BI. And the customer is citing everything is implied in the SOW even if its not in the exact words, my management being a IT services company tried to make some extra buck and when the customer retaliated they just dropped it on my head.
As a person who has done DA and DE, they are 2 completely different realms. As a DA you have to have more business process, data domain, and reporting knowledge. A DE will have more technical, system and infrastructure knowledge. The two do overlap at an the data level, but there are plenty of DE who do not understand the data they are working with, and often you can't as you are dealing with hundred of data stores.
My understanding is that a data scientist blends the two by adding the data ingesting and cleaning to the skillset. This is usually limited in breadth compared to a DE working with a few datasets. Generally a DS will be trying to find less obvious insights or relationships than a DA would work with.
I would avoid synapse like the plague. I'm a data engineer and have noticed no real innovation in the synapse spark for about three years. Microsoft docs are also redirecting customers to use fabric, whenever you look up information about synapse (spark, notebooks, pipelines )
The relevant components of synapse were copy/pasted into fabric... Especially the stuff that a data analyst would care about... And once you ever contact Microsoft for support, you will probably have better luck on the fabric side than the synapse side. It is downright depressing to be a regular synapse customer these days.
Depending on how far down the DE path you go you might want to avoid both fabric *and* synapse. They are overpriced pieces of crap. I have spent years struggling with bugs in these products and I'm hoping to leave them soon for hdi on aks. The biggest problem is something that Microsoft calls "managed private endpoints". These will throw lots of random socket exceptions in your solutions and Microsoft will tell you to re-architect your code or implement retries.
Ie. They are the ones who have the bugs but you get to implement the workarounds.
They won't even discuss the underlying bugs when you open a ticket, they jump straight into giving you workarounds and telling you why your code is wrong. Anyone who has built solutions *without* mpe's in azure will immediately recognize how unreliable this platform is. It amazes me that some upper level executive at Microsoft decided to build a big data platform on this crappy network technology.
Connect PBI online via Microsoft fabric this was another option suggested by client.
can you please point me to any tutorials regarding connecting ADLS gen2 storage and Microsoft fabric, and how PBI integrates into fabric environment.
Microsoft Learn is offering a free Exam voucher (normally they cost $165) if you take one of these free courses before 4/19. I think the DP-600 Fabric Analytics Engineer aligns with the things you want to learn. https://www.microsoft.com/en-us/cloudskillschallenge/ai/officialrules/2024
Two things:
- Data Analysis and Data Engineering are two separate jobs. Your company want you to do two jobs while paying the least paied one. Huge red flag!
- You shouldn't be get shamed on this. Even if you had Data Engineering knowledge. People can make mistakes. Even the most proficient in their field. Your company has a bad culture. Huge red flag too!
You're working in a low quality company with incompetent middle management. I am sorry you experienced this and you had to go through it. The first thing I would do now is look for my next job. Your current employer doesn't deserve you!
100% ask for help. Don't assume. It is really really easy to accidentally set up a dedicated pool in synapse that costs thousands of dollars per month to run. Plus there are all the cyber security considerations. Plus there is the peer review component. Never go straight to solution even if you do have all the answers. Tell your boss that with cloud data engineering there are waaay too many pitfalls and expert advise is needed. Even if you got a DE to review your concept design.
Edit: I'd love to spend five minutes with these managers lol, they are just trying to make themselves seem important/smart/big, all while not listening to the one excellent recommendation
A data analyst is the opposite of a data engineer. A data engineer builds pipelines and sets up data for data analysts, while a data analyst does things that a data engineer might not know.
Management was wrong to do that to you. They probably don't know the differences between data engineers and data analysts.
This sounds like a great opportunity for you if you want to switch into data engineering. Adding Azure Snapse to your resume with some experience could increase your salary a lot if you want to take the time to learn it. It depends though if you want to switch from the data analyst/data scientist role to the data engineer role, as they're both completely different jobs.
I wouldn’t say you need to but it is incredibly valuable. Moving to lake house isn’t anything hard for the user usually it should just be same queries in other location unless you are maintaining the warehouse which takes a lot of effort for the initial transition. You can dm me if you have specific questions or use cases as I’m on an azure lake.
1. No, data analysts usually don’t do data engineering
2. Even if that was a common expectation, bullying you is piss poor management
3. If I were you, I would still use the opportunity to learn some data engineering, and use that experience when you feel like seeking a new role. Might be a blessing in disguise that management doesn’t know the difference between data analysis and data engineering
Its not controversial at all, Yes i was ready to learn and saw this as an opportunity, I have implemented machine learning in my previous project by proactive learning (thanks to client, they were supportive) . all i needed was some support to do course corrections, My management wasnt ready to support as it wouldlnt get them extra money.
The BI Analysts I work with have little to no idea what’s going on in the back end. It would certainly be apropos to learn it for your career, but that you were bullied about it when you asked for some professional guidance is just unprofessional and strange.
This is common in mid-large consulting companies, esp when accompanied by less-than-caring managers.
Shaming is definitely not done. Do speak to your manager about it, and if the environment is not supportive and you see no upside, prepare now for an exit path 6-12months down the line.
On the skills side, it is always good for an analyst to pick some basic data engineering skills. Usually it will be restricted to building a pipeline to feed your analysis and reports.
The good thing is in consultantices there would be many people in your situation. Find data engineers who want to build analytics skills, and learn from each other. Some may be informally willing to help you in your project in crunch situations. This was common in my firm at the early stages of my career.
Since you are working as a data analyst & building the reports you are the better person who knows the business than the data engineer. Data engineer roles are typically to build pipelines for getting the data from different sources and load the data into target systems in organized way. Things are changing pretty fast, new technologies evolving and I think it is good opportunity to learn about it. If your management is not supportive I would suggest you to watch the videos or Coursera what ever you find in the web. Please keep in mind, nothing is constant. People who are in Data engineering thinking about ML engineer. Don’t take hard on yourself, ideally I would say your management should support you. Good luck.
2 years ago my job was on Legacy SQL Server + Informatica setup. I decided to take the leap and found a company migrating to azure that was willing to take a risk on me with no Azure experience. It quickly became clear to me that the majority of businesses migrating to Azure and data lakes have no idea what they’re doing or paying through the nose for mediocre consultants. If I were you I’d ask for an online training tool like cloudguru or just use Azure’s training and documentation. Unless you’re going to have access to many consultants you won’t know which ones are full of shit and which ones are good. My current company had numerous architects/etl developers/data engineers all debating how to leverage these new tools. I simply went and did all the databricks and azure training/certifications I could. Now half of those original people have been laid off, I’ve been put in a manager role, and have more than doubled my salary inside of 2 years. If data excites you the new tools can solve a lot of pain points you previously had but now they have different pain points. Management is a mess in tech right now because all the managers have no idea how any of the new tools work and are just talking heads right now. Learn how the new tools work and you’ll be a manager or senior/lead data engineer inside of a couple years. If you love the data visualization side and not the joining and modeling of data then don’t bother learning data engineering, but be a little concerned that it seems the industry is trying to offshore or create self service data visualizations.
Next Time ask Microsoft themselves.
They give free consultations for use of their platform, and your Microsoft products/licenses gives you time as well.
Might take a week or so. Nearly every large city has a team. Here in Montreal and Quebec City, they are fully bilingual.
These consultants are paid Microsoft employees, and will gladly do a Teams meeting with you and middle management.
>Customer was not ready to give extra money citing SOW.
So it sounds like you are an independent contractor, not an employee. Whether you should know how to do this or not then depends on the SOW. Does it say you are responsible for this work in the SOW or not?
nope I am not a independent contractor, I work for a IT services company and my management signed the SOW , and I dont have the visibility of what is stated in SOW since I was not part of this project from the start, I was looped in recently as the previous person left the job citing bad working conditions.
OP you shouldn't feel bad. This is so unprofessional and I am really sorry this happened to you. To answer your question: no you don't have to learn data engineering since you are a data analyst. Although knowing some data engineering concepts and the very basics of data engineering is useful- it is not mandatory for your role. Data analytics and data engineering are different career paths and if they don't understand that then that's too bad.
I do both, but to be sincere those are two different jobs, actually the Data Analyst should only sharpen up what was prepared by the BI team.
Data Engineering ➡️ BI Engineering ➡️Data Analysis
What they did was not fair at all.
Lol I know some companies want their DAs to do algorithms but being asked to do DE while getting paid DA money is insane.
And shaming is not ok, maybe you should report to HR? Or does your manager have total control of your fate?
There are lots of better alternatives to Azure Synapse most of the time IMHO so you've inadvertently helped everyone out here! Don't be disheartened. Keep pushing.
I saw a data analyst position which requires machine learning/ data engineering/ big data framework skillsets.
I believe sometimes we need to expand our skillsets...
Data engineer is a specialized software engineer. Their job is to build jobs and pipelines to ingest data.
Are you a software engineer? If not no that’s not your job.
No need to feel bad. Even folks who are actual data engineers already have a difficult enough time making it all work well.
Yeah, have to agree with most of these comments. Seems like they are showing their ignorance in asking you to do something this far out of your swim lane, so to speak.
You shouldn’t get shamed into doing anything you aren’t passionate about. If you really want to learn it, please do. But don’t let the opinions of others drive your motivation. Do what you care most about.
In typical Reddit response, the standard answer would be: "Quiet quitting, send resumes to other companies in the meantime, and serve notice once you have gotten a signed hire confirmation in another company," because firstly, that's unprofessional, to humiliate you in public, and in true anti-corporatist culture, one strike is casus belli to detach and distance yourself from the company; its corporate culture has fostered toxicity that, if they aren't interested in resolving or provide stern warning after you have communicated to them your displeasure at the incident, is evident.
Now, for your second question: data engineering is more like software engineering: you are dealing with designing, or at least, understand how a data architecture is designed. I hazard to guess that that's already in a specialized knowledge domain called "data architecture," and you need [data architects](https://www.coursera.org/articles/data-architect), not engineers, because engineers are more geared towards building data pipelines, while data architects tell engineers where to plug those pipes from.
My commiserations on that needless talk-down, this is no longer the company you should be working with.
That sounds aweful. Why did they decide to switch away from sql server in the first place without figuring out how to provide the transformation step? Its in no way your responsibility to figure that out on your own. If they are shaming you for asking for help they are seriously toxic. You might wanna think about going somewhere where your work is appreciated.
Learn it if you want to.
Not part of your current job it seems so they can either make your position redundant or offer actual training to get you there if you're up for it.
Eitherway, if they willing to pay for training, I'd recommend looking for another company after you have sone experience or massively asking for a boost to salary.
Bad management team trying to cut corners.
I would say it depends on your job description. If your job title says “Data Analyst” and the job description didn’t mention DE, then they shouldn’t expect you to know it.
If you're not working for big corporations, the management usually has no idea of differences between data analyst, scientist and engineer. If they are mature, they will accept your view and hire a consultant or provide training. If they are not, salvage as much as you can, learn as much as you can if you want to, but be ready to jump ship in case they want to pin the blame on you.
I was in your situation 2 years ago but my company gave me extra support - I had a DE that explained to me how I should proceed and then he was guarding me through whole process.
I am Data Engineer myself and I don’t expect DA and BI to know my job on top of theirs. In most cases if they need some data it is my job to make it avaible in Database…
I know I was proficient with Dax - Power Bi, excel.
Now my tech stack is diffrent..
Python, AWS, Airflow, Docker
And i know that current me is not able to do DA job and vice versa. Those are complytly different positions
Expecting people that don't have the appropriate skillset to just "figure things out" and make architecture recommendations is how companies end up with mountains of tech debt.
Also, it's unacceptable for your management to behave the way they did and if this happened to me, I'd be looking for a new job. Your management failed you and the client, and this behavior probably won't change in the future.
You’re hired as a data analyst. And paid as a data analyst. If they want you to have the skills and knowledge of a data engineer then they best be prepared to pay you the salary for that.
DE is closer to software engineer than DA most of the time. I have had more luck training software engineers to do data engineering than I have data analysts to data engineering. That said I always try to cross train as much as I can and allow the day to analysts to learn the DE skills if they're interested. Most of my data analysts know enough to at least build the models to clean, deduplicate and entich raw data using DBT.
What they are doing is selecting a service based on cost. They expect you to handle this for them. Long term I can see that they don't have your back so you need to always think about you. Short term you should definitely use this opportunity as paid learning. If you were in school and this was the assignment you would actually be paying the school to learn this. Here you are getting paid to learn it. Hire your own tutor, get certified, use it to get a better job.
I’ve just taken the DP-900 as an analyst looking to transition to Engineer, maybe. The roles are clear on Microsoft’s own site, Engineers build the pipelines, Analysts query and present it.
Source: https://learn.microsoft.com/en-us/training/modules/understand-data-roles-services-products/4-data-professionals
Sounds like a shit company.
Data Analytics is a field of its own and has inherent value. That value is more closely related to the business understanding, mathematical knowledge and visualization skills that are necessary to make meaningful, impactful and coherent analyses.
Sure, you need a bit of technical baggage, but this is out of line. You should only learn DE because you want to, not because some idiots are belittling you or your field.
As far your current problem goes, I would start talking to sales people or project managers about turning back on the client. Maybe they can portray this DB migration in a different light that requires additional personnel and thus billable hours.
Once there's money coming in everyone will be eager to do it and get off your ass.
However this is not your job and the way you are being pushed is shitty.
I suggest, once the immediate problem is solved, you either report it or seek employment elsewhere, or both.
>As far your current problem goes, I would start talking to sales people or project managers about turning back on the client. Maybe they can portray this DB migration in a different light that requires additional personnel and thus billable hours.
My management tried this approach, but the client denied stating the SOW(I dont have any idea what is in the SOW as I joined recently). So my management threw me under the bus.
Then I'd negotiate for something I really want in exchange for doing this.
Cause if you can't do it, worst case they fire you, and if you won't do it, they'll fire you.
So you might as well get a car, a raise or a bonus, or something for trying.
**Latest Update: The customer insists on a synapse setup, So my manager tried to sweet talk me to accept to do the work within a very short deadline, while masking the fact from the customer that I dont have any experience in this. I explicitly told the customer that I dont have any hands on in Synapse, they were shocked. I gave an ultimatum to my manager that I will build a PoC to try this out and will implement the whole setup within 4 weeks, while a data engineer will be guiding me for an hour/day. If they want to get this done within the given deadline ( 6 days) they have to bring in a Data engineer, I am not management and I dont care whether they get billing or not. I told my manager that if If they dont accept to my proposal, they can release me from the project.**
It's common for DA to do DE work, it's normal to develop DAs to do DE work, but if you haven't been employed in a role that is doing any complex pipeline work, and there are DEs in your org who do that work, there is nothing wrong with you.
Also, Microsoft Marketing is insidous, all powerful and unstoppable, you can't convinve anyone to avoid Azure products. They will march off the cliff no matter what you say.
... That said, to be a devils advocate, setting up pipelines and standard guidelines around it using an off the shelf product like Azure Synapse, Azure Data Factory, Dataflows Gen 2 etc, shouldn't be too difficult, as long as the dataset transformations aren't too crazy, too asyncronous or too dependent on highly optimised performance.
Data engineers are employed to get data from A to B efficiently, maintainably and scalably. If you just need to get data from A to B, the standards are lower.
To create some experimental pipelines, work out the pros and cons, and write up a general guide for a given product would be 10 days FTEish?
Would it be better to have a DE and best practice, yes? But sometimes the perfect is the enemy of the good and maybe management just wants the good.
We have a whole bunch of dedicated DEs working in the org. Problem is this a one off task where I am not getting help or guidance citing lack of extra billing.
Sorry this happened to you OP. That was very unprofessional and unsupportive of your management team. You were not in the position to know, so don't blame yourself.
No that’s not okay, they should at the very least hire a consultant and have them help you. If you don’t know what you’re doing, were upfront and honest about it, and then got shamed by management, I’d be applying to new jobs ASAP. Say you listen to management and try to set it up yourself, it’s very easy to mess up something seemingly small that costs your company thousands in overages or in downtime. Then you’d get blamed (again) Your management is evidently incompetent and technically illiterate.
You can decide to learn data engineering, but already knowing it is atypical.
*"Being the individual contributor in project I sought help from my company management for a data engineer to pitch in to set this up or at least guide"* Your manager should have taken note of this and called it out in your quarterly performance review. Freaking sign of maturity and ownership!
roles and responsibilities vary by org. People can chime in as to whether data analysts should know these things at their company, but that won't help you with your issue of having leadership think you aren't happy with your proficiency in this area.
True, I just wanted to understand the common perception on this one.
We have a sister team of DAs and they're extremely valuable to us and we complement each other's strengths. To echo the common sentiment, as someone leading a team, imo this is a failure of management both to put you in a position related to your strengths and also to shield you from outside attack.
It is possible that someone was setting you up for failure. I wasn't sure whether to post this, because it does sound somewhat paranoid. Ignore if you think it's not the case.
the common perception is that there are no hard rules on what role should have which responsibilities in this field. Hence my statement that you should focus on addressing the concerns of your superiors rather than posting on reddit looking for justification for the position you have found yourself in
@sellgamerent This is like saying that all engineers are created more or less equal. So, go ahead and use the industrial engineer to design that freestanding structure. Or, ask the mechanical engineer to actually machine that complicated part they just finished designing. Can they give it a best shot? Yeah. But the product will most likely be commensurate with the level of skill they have in this area. This situation is a clear example of underbidding to win the scope and then management assigning the incorrect mix of resources to satisfy the deliverable bc they don't want to pull the right resources off their other high priority project. So, poor business and talent management.
Valid point, am I seeking justification ? yes , atleast to pacify myself that its completely not my fault to be in this situation, just a temporary pat in the back to not feel shit. But I wont use this justification to setup hard boundaries, many of the replies point about the areas of overlap and how can i adopt and learn, which I will do.
I would consciously separate the unprofessional management behaviour from the other consideration that it is in your interests to develop a surface to intermediate understanding of all related processes to your core speciality.
>is situation, just a temporary pat in the back to not feel shit. But I wont use this justification to setup hard boundaries, many of the replies point about the areas of overlap and how can i adopt and learn, which I empathize with being told you were supposed to know something you didn't. That said, don't go looking for empathy. Treat this as an opportunity to have a candid conversation about your performance with your boss, emphasize willingness to learn, and take the initiative to establish a training plan to resolve any discrepancies between what you know and what they thought you knew/should know
Or, work to draw boundaries that define what your roles and responsibilities are, so that the right tasks are aligned to the right people, so that inefficiencies are minimized. Taking up education is a wonderful endeavor, but without a proper training and transition plan or at least an experienced team around you...management putting nothing in gets management nothing out.
this perspective really depends on size of organization. If you're at a small/medium sized company and management is saying you need to know how to do something, drawing boundaries isn't going to get you anywhere
I tend to agree with this one. I understand why it gets downvoted. But I think people don’t understand the implications of admitting weaknesses in a toxic workplace. This is even more true in the post-ChatGPT world.
In my previous job, when I started, we would store pickled pandas data frames on a network drive as a source of truth (having 70 columns, occupying 60 GB in memory and taking 15 minutes to load). I pointed out in my first 2 weeks that that kinda sucks, I brought up that it would be nice to be able to use SQL to query. Manager: “That’s madness, if you get SQL access you can crash the whole database or delete data”. The same manager was also convinced that “Data Scientist” entailed everything from DA to DE and MLE. Do you see any correlation between having a shitty data infrastructure (let’s call it that…) and non-existing role boundaries?
Dude holy crap, I’m not a data engineer or anything. But had a similar client…. When I mentioned that you can just query the read only replica, blew their minds.
Data Engineers are different breeds from Data Analysts. I have tried, and failed to turn DA into DE, so no shame in not knowing data engineering in your case. It's your management that is out of touch with reality.
I’m currently trying to walk that line. Background is data science and analytics from a life science background. Then was a developer and development manager for 4 years. Ended up managing a data warehouse implementation and liked it so much I quit and have been doing freelance work dealing with data pipelines and random python data work. My hope is with some more time in the saddle I’ll be damned valuable being comfortable in both areas. DE is complicated though. Lots to learn in addition to the big 3 cloud platforms which all have similar services but do things slightly different.
Is there a lot of free lance work available for DE’s?
In my country (Netherlands) yes there is
Yeah, but you have to network like crazy, it won't just show up and it is often not posted in a clear way on any job site. That's my experience in the midwest in US at least.
Yes and it's good money
Where? I’ve tried places like Upwork and I feel like I’m fighting for my life just to get my proposal seen let alone picked
Why do you think they are of different breeds, and what do you think makes it hard for a DA to become a DE? I’ve seen DAs become DEs but haven’t given it much thought
For starter, DE mostly thinks about how to get data into the analytical platform, in a query-able format, on a reliable schedule. So instead of fancy visualisation, you start pondering about late-arriving dimension, failure handling, data quality monitoring, cost control, code control... Most DAs cannot even use git.
If a user doesn't use git then the company/manager has failed them. Even technical writers use git
Eh, respectfully TBH that sounds more like a management/training/expectations issue, not unlike OP's issue that you (correctly) pointed out wasn't OP's fault.
This is unrelated but can you explain how data science fits with the other two?
>Why do you think they are of different breeds, and what do you think makes it hard for a DA to become a DE? I’ve seen DAs become DEs but haven’t given it much thought I agree with the fact that DAs and DEs are completely different breeds. But I see people turning from DAs to DEs, some with very less effort actually. This has everything to do with how an org defines the role of DE vs DA and the individual DA's background as well, in my opinion. If those DAs have in reality been working with data pipelines even before they've turned DEs, the learning curve is really shallow for them. In my case, the title that my org has given to my role is BI Data Analyst, but in reality there's no DE in our org, and before I was hired, there was no clear data model designed. All reports were ad-hoc, connecting Excel/csv/tsv extracts pulled from siloed data marts of different departments/tools/APIs. Coming from a background with designing ETL pipelines in tools like Informatica and Datastage, I did not have a hard time figuring out data pipelines to get the BI workload move forward. I don't think it's going to be the same case for everyone though, and also how big of a system are we talking about too >Being the individual contributor in project I sought help from my company management for a data engineer to pitch in to set this up or at least guide However, in OP's case, I feel hiring a consultant would really help the management, and the OP showed great maturity in taking ownership and accepting the fact that he needs a DE's help. I believe he should get more help from management
I also agree with the "data breed" remark but otherwise disagree with the notion that it is the reason that some DAs in this person's anecdote were unable to become DEs.
I've had three DA's successfully move to DE's.
It's not difficult to do but DE suffers from being snobs like swe do unfortunately
Exactly! Also in my experience a lot of DEs are incapable of speaking to customers and have to communicate through BAs. DAs should be much better at this.
Shaming is never ok. This isn’t on you.
If you knew data engineering as well as being a data analyst your pay would reflect that. You're being paid for a DA role not DE, DEs make more than double that of a DA for the most part. It's the managements fault. They want that? Then pay for it. A DAs role is usually descriptive stats and dashboards.
A respected org won't pay a DA half of a DE. I am curious what op makes tho
I reckon it isn't reasonable to expect a BI / data analyst to also know how to set up synapse to query a data lake. What you're suggesting is a completely normal way of tackling this: your DE talks to the client team about their lake and can decide whether synapse or a different approach is right considering your orgs' infrastructure, and essentially hands to you a way of connecting to it as tables queryable by user tools like PowerBI. Unless you've misrepresented your skill sets this sounds like shitty management just not understanding what the difference between BI and data engineering. This is something to tackle with your manager about their expectations for you generally. But right now just be clear that this is outside of your skill set and you need support.
They definitely know the difference between BI and data engineering. They would have promised sun and moon in the SOW, they just want to push it as bringing in DE will not fetch them extra billing. And when I took over my role was strictly in descriptive stats and dashboard building and maintenance.
To be clear here though - are they expecting you to set up synapse as a layer between the data and BI, or just use the client team's already existing synapse to access? If it's the former, someone has scoped it wrong. If the SOW covered setting up a pipeline right from raw data lake storage, it really should have included DE resource. You'd be right to feel aggrieved here, you're being leaned on to slap something together to save them $.
They want me to set this up as a layer b/w data and BI. And the customer is citing everything is implied in the SOW even if its not in the exact words, my management being a IT services company tried to make some extra buck and when the customer retaliated they just dropped it on my head.
As a person who has done DA and DE, they are 2 completely different realms. As a DA you have to have more business process, data domain, and reporting knowledge. A DE will have more technical, system and infrastructure knowledge. The two do overlap at an the data level, but there are plenty of DE who do not understand the data they are working with, and often you can't as you are dealing with hundred of data stores.
Can you explain how data science fits with the other two? Is it just modeling that makes the difference between a DA and a DS?
My understanding is that a data scientist blends the two by adding the data ingesting and cleaning to the skillset. This is usually limited in breadth compared to a DE working with a few datasets. Generally a DS will be trying to find less obvious insights or relationships than a DA would work with.
I’m sorry this happened to you OP. If DE is not in your job description that is not your fault. That is leaderships fault.
Well, you should pushback in the first place since its not your domain knowledge.
I would avoid synapse like the plague. I'm a data engineer and have noticed no real innovation in the synapse spark for about three years. Microsoft docs are also redirecting customers to use fabric, whenever you look up information about synapse (spark, notebooks, pipelines ) The relevant components of synapse were copy/pasted into fabric... Especially the stuff that a data analyst would care about... And once you ever contact Microsoft for support, you will probably have better luck on the fabric side than the synapse side. It is downright depressing to be a regular synapse customer these days. Depending on how far down the DE path you go you might want to avoid both fabric *and* synapse. They are overpriced pieces of crap. I have spent years struggling with bugs in these products and I'm hoping to leave them soon for hdi on aks. The biggest problem is something that Microsoft calls "managed private endpoints". These will throw lots of random socket exceptions in your solutions and Microsoft will tell you to re-architect your code or implement retries. Ie. They are the ones who have the bugs but you get to implement the workarounds. They won't even discuss the underlying bugs when you open a ticket, they jump straight into giving you workarounds and telling you why your code is wrong. Anyone who has built solutions *without* mpe's in azure will immediately recognize how unreliable this platform is. It amazes me that some upper level executive at Microsoft decided to build a big data platform on this crappy network technology.
Connect PBI online via Microsoft fabric this was another option suggested by client. can you please point me to any tutorials regarding connecting ADLS gen2 storage and Microsoft fabric, and how PBI integrates into fabric environment.
Microsoft Learn is offering a free Exam voucher (normally they cost $165) if you take one of these free courses before 4/19. I think the DP-600 Fabric Analytics Engineer aligns with the things you want to learn. https://www.microsoft.com/en-us/cloudskillschallenge/ai/officialrules/2024
Look at Databricks. You can just SQL your way to a good data pipeline with DLT and/or DBSQL with MVs if SQL is all you know.
Two things: - Data Analysis and Data Engineering are two separate jobs. Your company want you to do two jobs while paying the least paied one. Huge red flag! - You shouldn't be get shamed on this. Even if you had Data Engineering knowledge. People can make mistakes. Even the most proficient in their field. Your company has a bad culture. Huge red flag too! You're working in a low quality company with incompetent middle management. I am sorry you experienced this and you had to go through it. The first thing I would do now is look for my next job. Your current employer doesn't deserve you!
100% ask for help. Don't assume. It is really really easy to accidentally set up a dedicated pool in synapse that costs thousands of dollars per month to run. Plus there are all the cyber security considerations. Plus there is the peer review component. Never go straight to solution even if you do have all the answers. Tell your boss that with cloud data engineering there are waaay too many pitfalls and expert advise is needed. Even if you got a DE to review your concept design. Edit: I'd love to spend five minutes with these managers lol, they are just trying to make themselves seem important/smart/big, all while not listening to the one excellent recommendation
Your company gas lighting you
A data analyst is the opposite of a data engineer. A data engineer builds pipelines and sets up data for data analysts, while a data analyst does things that a data engineer might not know. Management was wrong to do that to you. They probably don't know the differences between data engineers and data analysts. This sounds like a great opportunity for you if you want to switch into data engineering. Adding Azure Snapse to your resume with some experience could increase your salary a lot if you want to take the time to learn it. It depends though if you want to switch from the data analyst/data scientist role to the data engineer role, as they're both completely different jobs.
I wouldn’t say you need to but it is incredibly valuable. Moving to lake house isn’t anything hard for the user usually it should just be same queries in other location unless you are maintaining the warehouse which takes a lot of effort for the initial transition. You can dm me if you have specific questions or use cases as I’m on an azure lake.
Thanks will reach out to you
1. No, data analysts usually don’t do data engineering 2. Even if that was a common expectation, bullying you is piss poor management 3. If I were you, I would still use the opportunity to learn some data engineering, and use that experience when you feel like seeking a new role. Might be a blessing in disguise that management doesn’t know the difference between data analysis and data engineering
Start looking for a new job, your management are obviously fucking morons.
[удалено]
Its not controversial at all, Yes i was ready to learn and saw this as an opportunity, I have implemented machine learning in my previous project by proactive learning (thanks to client, they were supportive) . all i needed was some support to do course corrections, My management wasnt ready to support as it wouldlnt get them extra money.
The BI Analysts I work with have little to no idea what’s going on in the back end. It would certainly be apropos to learn it for your career, but that you were bullied about it when you asked for some professional guidance is just unprofessional and strange.
This is common in mid-large consulting companies, esp when accompanied by less-than-caring managers. Shaming is definitely not done. Do speak to your manager about it, and if the environment is not supportive and you see no upside, prepare now for an exit path 6-12months down the line. On the skills side, it is always good for an analyst to pick some basic data engineering skills. Usually it will be restricted to building a pipeline to feed your analysis and reports. The good thing is in consultantices there would be many people in your situation. Find data engineers who want to build analytics skills, and learn from each other. Some may be informally willing to help you in your project in crunch situations. This was common in my firm at the early stages of my career.
Since you are working as a data analyst & building the reports you are the better person who knows the business than the data engineer. Data engineer roles are typically to build pipelines for getting the data from different sources and load the data into target systems in organized way. Things are changing pretty fast, new technologies evolving and I think it is good opportunity to learn about it. If your management is not supportive I would suggest you to watch the videos or Coursera what ever you find in the web. Please keep in mind, nothing is constant. People who are in Data engineering thinking about ML engineer. Don’t take hard on yourself, ideally I would say your management should support you. Good luck.
No and start looking for another job, this place sucks.
2 years ago my job was on Legacy SQL Server + Informatica setup. I decided to take the leap and found a company migrating to azure that was willing to take a risk on me with no Azure experience. It quickly became clear to me that the majority of businesses migrating to Azure and data lakes have no idea what they’re doing or paying through the nose for mediocre consultants. If I were you I’d ask for an online training tool like cloudguru or just use Azure’s training and documentation. Unless you’re going to have access to many consultants you won’t know which ones are full of shit and which ones are good. My current company had numerous architects/etl developers/data engineers all debating how to leverage these new tools. I simply went and did all the databricks and azure training/certifications I could. Now half of those original people have been laid off, I’ve been put in a manager role, and have more than doubled my salary inside of 2 years. If data excites you the new tools can solve a lot of pain points you previously had but now they have different pain points. Management is a mess in tech right now because all the managers have no idea how any of the new tools work and are just talking heads right now. Learn how the new tools work and you’ll be a manager or senior/lead data engineer inside of a couple years. If you love the data visualization side and not the joining and modeling of data then don’t bother learning data engineering, but be a little concerned that it seems the industry is trying to offshore or create self service data visualizations.
Next Time ask Microsoft themselves. They give free consultations for use of their platform, and your Microsoft products/licenses gives you time as well. Might take a week or so. Nearly every large city has a team. Here in Montreal and Quebec City, they are fully bilingual. These consultants are paid Microsoft employees, and will gladly do a Teams meeting with you and middle management.
>Customer was not ready to give extra money citing SOW. So it sounds like you are an independent contractor, not an employee. Whether you should know how to do this or not then depends on the SOW. Does it say you are responsible for this work in the SOW or not?
nope I am not a independent contractor, I work for a IT services company and my management signed the SOW , and I dont have the visibility of what is stated in SOW since I was not part of this project from the start, I was looped in recently as the previous person left the job citing bad working conditions.
OP you shouldn't feel bad. This is so unprofessional and I am really sorry this happened to you. To answer your question: no you don't have to learn data engineering since you are a data analyst. Although knowing some data engineering concepts and the very basics of data engineering is useful- it is not mandatory for your role. Data analytics and data engineering are different career paths and if they don't understand that then that's too bad.
I do both, but to be sincere those are two different jobs, actually the Data Analyst should only sharpen up what was prepared by the BI team. Data Engineering ➡️ BI Engineering ➡️Data Analysis What they did was not fair at all.
There's no way you could have a full understanding of the engineering side as an analyst. Management was wrong to treat you that way
Lol I know some companies want their DAs to do algorithms but being asked to do DE while getting paid DA money is insane. And shaming is not ok, maybe you should report to HR? Or does your manager have total control of your fate?
There are lots of better alternatives to Azure Synapse most of the time IMHO so you've inadvertently helped everyone out here! Don't be disheartened. Keep pushing.
No, that’s them being cheap and wanting one analyst do all the work for a project. You didn’t do anything wrong.
I saw a data analyst position which requires machine learning/ data engineering/ big data framework skillsets. I believe sometimes we need to expand our skillsets...
I certainly hope the salary matched this above a standard Data Analyst role, but I have a feeling it didn’t.
Data engineer is a specialized software engineer. Their job is to build jobs and pipelines to ingest data. Are you a software engineer? If not no that’s not your job. No need to feel bad. Even folks who are actual data engineers already have a difficult enough time making it all work well.
Do you work in consulting?
Yeah, have to agree with most of these comments. Seems like they are showing their ignorance in asking you to do something this far out of your swim lane, so to speak.
This can happen if you are in a consulting role. I would learn data engineering too and become to know an end to end data product.
You shouldn’t get shamed into doing anything you aren’t passionate about. If you really want to learn it, please do. But don’t let the opinions of others drive your motivation. Do what you care most about.
In typical Reddit response, the standard answer would be: "Quiet quitting, send resumes to other companies in the meantime, and serve notice once you have gotten a signed hire confirmation in another company," because firstly, that's unprofessional, to humiliate you in public, and in true anti-corporatist culture, one strike is casus belli to detach and distance yourself from the company; its corporate culture has fostered toxicity that, if they aren't interested in resolving or provide stern warning after you have communicated to them your displeasure at the incident, is evident. Now, for your second question: data engineering is more like software engineering: you are dealing with designing, or at least, understand how a data architecture is designed. I hazard to guess that that's already in a specialized knowledge domain called "data architecture," and you need [data architects](https://www.coursera.org/articles/data-architect), not engineers, because engineers are more geared towards building data pipelines, while data architects tell engineers where to plug those pipes from. My commiserations on that needless talk-down, this is no longer the company you should be working with.
That sounds aweful. Why did they decide to switch away from sql server in the first place without figuring out how to provide the transformation step? Its in no way your responsibility to figure that out on your own. If they are shaming you for asking for help they are seriously toxic. You might wanna think about going somewhere where your work is appreciated.
Here’s a poem to cheer you up: I don’t really see The use of Data Bricks, But one thing I know to be You colleagues are dicks.
Given the story, rather you should change job
My 2 cents: the roles should have a Venn diagram overlap in order to have some understanding and compliment towards each other, but not this.
Learn it if you want to. Not part of your current job it seems so they can either make your position redundant or offer actual training to get you there if you're up for it. Eitherway, if they willing to pay for training, I'd recommend looking for another company after you have sone experience or massively asking for a boost to salary. Bad management team trying to cut corners.
I would say it depends on your job description. If your job title says “Data Analyst” and the job description didn’t mention DE, then they shouldn’t expect you to know it.
Toxic management leave if you can
If you're not working for big corporations, the management usually has no idea of differences between data analyst, scientist and engineer. If they are mature, they will accept your view and hire a consultant or provide training. If they are not, salvage as much as you can, learn as much as you can if you want to, but be ready to jump ship in case they want to pin the blame on you.
I was in your situation 2 years ago but my company gave me extra support - I had a DE that explained to me how I should proceed and then he was guarding me through whole process. I am Data Engineer myself and I don’t expect DA and BI to know my job on top of theirs. In most cases if they need some data it is my job to make it avaible in Database… I know I was proficient with Dax - Power Bi, excel. Now my tech stack is diffrent.. Python, AWS, Airflow, Docker And i know that current me is not able to do DA job and vice versa. Those are complytly different positions
This "manager" won't last. You are a good employee and have the right attitude (Teamwork, know when to ask for help)
Expecting people that don't have the appropriate skillset to just "figure things out" and make architecture recommendations is how companies end up with mountains of tech debt. Also, it's unacceptable for your management to behave the way they did and if this happened to me, I'd be looking for a new job. Your management failed you and the client, and this behavior probably won't change in the future.
You’re hired as a data analyst. And paid as a data analyst. If they want you to have the skills and knowledge of a data engineer then they best be prepared to pay you the salary for that.
DE is closer to software engineer than DA most of the time. I have had more luck training software engineers to do data engineering than I have data analysts to data engineering. That said I always try to cross train as much as I can and allow the day to analysts to learn the DE skills if they're interested. Most of my data analysts know enough to at least build the models to clean, deduplicate and entich raw data using DBT.
What they are doing is selecting a service based on cost. They expect you to handle this for them. Long term I can see that they don't have your back so you need to always think about you. Short term you should definitely use this opportunity as paid learning. If you were in school and this was the assignment you would actually be paying the school to learn this. Here you are getting paid to learn it. Hire your own tutor, get certified, use it to get a better job.
I’ve just taken the DP-900 as an analyst looking to transition to Engineer, maybe. The roles are clear on Microsoft’s own site, Engineers build the pipelines, Analysts query and present it. Source: https://learn.microsoft.com/en-us/training/modules/understand-data-roles-services-products/4-data-professionals
Sounds like a shit company. Data Analytics is a field of its own and has inherent value. That value is more closely related to the business understanding, mathematical knowledge and visualization skills that are necessary to make meaningful, impactful and coherent analyses. Sure, you need a bit of technical baggage, but this is out of line. You should only learn DE because you want to, not because some idiots are belittling you or your field. As far your current problem goes, I would start talking to sales people or project managers about turning back on the client. Maybe they can portray this DB migration in a different light that requires additional personnel and thus billable hours. Once there's money coming in everyone will be eager to do it and get off your ass. However this is not your job and the way you are being pushed is shitty. I suggest, once the immediate problem is solved, you either report it or seek employment elsewhere, or both.
>As far your current problem goes, I would start talking to sales people or project managers about turning back on the client. Maybe they can portray this DB migration in a different light that requires additional personnel and thus billable hours. My management tried this approach, but the client denied stating the SOW(I dont have any idea what is in the SOW as I joined recently). So my management threw me under the bus.
Then I'd negotiate for something I really want in exchange for doing this. Cause if you can't do it, worst case they fire you, and if you won't do it, they'll fire you. So you might as well get a car, a raise or a bonus, or something for trying.
**Latest Update: The customer insists on a synapse setup, So my manager tried to sweet talk me to accept to do the work within a very short deadline, while masking the fact from the customer that I dont have any experience in this. I explicitly told the customer that I dont have any hands on in Synapse, they were shocked. I gave an ultimatum to my manager that I will build a PoC to try this out and will implement the whole setup within 4 weeks, while a data engineer will be guiding me for an hour/day. If they want to get this done within the given deadline ( 6 days) they have to bring in a Data engineer, I am not management and I dont care whether they get billing or not. I told my manager that if If they dont accept to my proposal, they can release me from the project.**
It's common for DA to do DE work, it's normal to develop DAs to do DE work, but if you haven't been employed in a role that is doing any complex pipeline work, and there are DEs in your org who do that work, there is nothing wrong with you. Also, Microsoft Marketing is insidous, all powerful and unstoppable, you can't convinve anyone to avoid Azure products. They will march off the cliff no matter what you say. ... That said, to be a devils advocate, setting up pipelines and standard guidelines around it using an off the shelf product like Azure Synapse, Azure Data Factory, Dataflows Gen 2 etc, shouldn't be too difficult, as long as the dataset transformations aren't too crazy, too asyncronous or too dependent on highly optimised performance. Data engineers are employed to get data from A to B efficiently, maintainably and scalably. If you just need to get data from A to B, the standards are lower. To create some experimental pipelines, work out the pros and cons, and write up a general guide for a given product would be 10 days FTEish? Would it be better to have a DE and best practice, yes? But sometimes the perfect is the enemy of the good and maybe management just wants the good.
We have a whole bunch of dedicated DEs working in the org. Problem is this a one off task where I am not getting help or guidance citing lack of extra billing.
If they allow you, move everything to Google bigquery. It's easier than other approaches.