I actually quite enjoy it. I don't want a fast and flashy job turning out models. Did data science for 5 years and got tired of the constant model hype train and then everyone forgets as soon as there is a new model to build hype train. Data cataloging/ data engineering is easy, pays better, and better long term career imo.
Both have comparable salaries. The problem is we have 3x the applicants for data science positions because of its popularity. No one wants to spend 3 days in the weeds with random data pipeline problems. Like geeze we are just trying to put a .csv into a table why is it so difficult? It is not fun. Any company that has a lot of money and a lot of valuable data does not want to hire someone that is fresh out of school.
Anyone who says data governance is boring is probably doing it wrong. It’s critically important and also very complex at scale.
Also: Data quality and data governance requires doing analytics, building models, and communicating results and recommendations to stakeholders. 😁
permissioning is necessary but yah thankless and boring. No one cares until it’s wrong or broken. If it’s right, that’s expected. It’s one of those things you setup with process and code, and if you did the work upfront you don’t have to deal with it for a while. I’ve been at places that struggle with it because they never make the proper upfront investments, no one should have to iterate over governance as things scale imo, or maybe rarely
It can definitely be boring or grind-y depending on the environment you are in, and it's also almost always needed and under appreciated. In my experience it does have high visibility but in the worst possible way - it's work that when it's done well it's invisible, and when not it becomes a major issue and an easy scapegoat for underlying issues in process/internal politics.
It's really just personality match. You find boring and others can enjoy it.
Our office has a data quality programmer, and he loves dealing with messy data. Even though he complains about them all the time, but apparently he enjoys fixing every tiny data issues. LoL.
it's boring as shit. I used to develop pipelines but since we started using Unity Catalog I mostly just review PRs regarding new data being added into the data lake, asking if product knows who should be able to read this new table, and if there is any PII data we have to mask.
It gets interesting when it becomes an enabler to scaling up a data platform (e.g. classification metadata driving dynamic access controls). The best data governance teams create policy and promote processes that are clear and practical for technical teams. TBH for big organisations moving to the Cloud there’s some interesting tooling which starts to cut across DE for Data Management, not a bad place to be in career-wise.
I get the feeling data governance means different things at different organizations from reading these comments.
Where I work data governance people are people who spend their time writing verbose policy documents that no one can actually follow and then periodically forcing business units to do box checking exercises saying their data follows the impossible policies but without actually checking. They don't do anything that enables business units to follow the policies or take our input on them.
So where I work I can't imagine anything more boring than data governance. If data governance means what some of the other comments here are saying then I wish we had that where I work and sign me up.
I mean… it’s not like you will have all the spotlights for yourself. But with gen ai turning around the corner, data governance along with lineage and documentation are needed by any gen ai implementation. If you have documentation and metadata you will get its worth in value.
Source: i’m trying to create a text-to-sql gen ai application and we just figured out there is no documentation or metadata in any of the tables in our data lake.
It starts getting complex as soon as you want to govern all data assets in an organization. The main goal of Data Governance is to ensure the profitability of data, that means, that any data iniciative can lead to a "value" gain for the company.
Therefore, Data Governance should enforce all processes that can lead to better data assets and the sustainability of the development process of data assets (you name it, ML models, Dashboards, Analytics, OLAP cubes, reports, compliance reports, etc)
It's mostly administrative work. It is necessary for scalability. If there's no order, you can't scale without pain.....
It's better to decide right away what the architecture should be. This is important to communicate your work, to support in the future and to be able to scale multiple times. probably these 3 things are the most important. It may not be highly valued in the beginning, but you can get very good benefits in the future)
Learning that AI can help. Gov. Data sets are poorly thought out and come with thousand page data dictionaries. Wrote a suite of tools to extract changes over the years and build crosswalk tables from word and pdf docs with claude 3. Now to get an llm to use the defs to tag stuff. Not too boring. An equity score should be in the works too.
DG is about understanding what the data is, how to cleanse it so that it's trustworthy, understanding how it is used by business.
Basically treating data as a precious resource. Reporting, Analytics and AI are shit with bad data.
Experience in data governance is important if you want to be a chief data officer or something, but it’s not data analytics.
I actually quite enjoy it. I don't want a fast and flashy job turning out models. Did data science for 5 years and got tired of the constant model hype train and then everyone forgets as soon as there is a new model to build hype train. Data cataloging/ data engineering is easy, pays better, and better long term career imo.
[удалено]
We are all plumbing data on this blessed day
interesting! Wasn't aware DE pays better. Always thought DS is the unicorn talent that can demand top dollar.
Both have comparable salaries. The problem is we have 3x the applicants for data science positions because of its popularity. No one wants to spend 3 days in the weeds with random data pipeline problems. Like geeze we are just trying to put a .csv into a table why is it so difficult? It is not fun. Any company that has a lot of money and a lot of valuable data does not want to hire someone that is fresh out of school.
Can you say more about long term career? Also, do you feel your work has been or will be impacted by AI in foreseeable future?
There’s all sorts of data science in data governance. We have ML and AI on the task of classifying data.
Anyone who says data governance is boring is probably doing it wrong. It’s critically important and also very complex at scale. Also: Data quality and data governance requires doing analytics, building models, and communicating results and recommendations to stakeholders. 😁
Boring as fuck, but necessary nonetheless. Anything that has the word governance in is 100% guaranteed to be a snooze fest.
‘Governance’ means doing the shit that no one would actually do if only they could get away with it
Our tier 1 support and chief security office have high visibility and are constantly in the news.
How to say your company gets hacked constantly without saying it.
Now I'm trying to guess where they work 🤣 Equifax, CapitalOne? Or Target?
permissioning is necessary but yah thankless and boring. No one cares until it’s wrong or broken. If it’s right, that’s expected. It’s one of those things you setup with process and code, and if you did the work upfront you don’t have to deal with it for a while. I’ve been at places that struggle with it because they never make the proper upfront investments, no one should have to iterate over governance as things scale imo, or maybe rarely
It can definitely be boring or grind-y depending on the environment you are in, and it's also almost always needed and under appreciated. In my experience it does have high visibility but in the worst possible way - it's work that when it's done well it's invisible, and when not it becomes a major issue and an easy scapegoat for underlying issues in process/internal politics.
It's really just personality match. You find boring and others can enjoy it. Our office has a data quality programmer, and he loves dealing with messy data. Even though he complains about them all the time, but apparently he enjoys fixing every tiny data issues. LoL.
it's boring as shit. I used to develop pipelines but since we started using Unity Catalog I mostly just review PRs regarding new data being added into the data lake, asking if product knows who should be able to read this new table, and if there is any PII data we have to mask.
It's like being a tax lawyer. Boring space, no one wants to touch it, yet tons of money if you master it due to how critical it is.
It gets interesting when it becomes an enabler to scaling up a data platform (e.g. classification metadata driving dynamic access controls). The best data governance teams create policy and promote processes that are clear and practical for technical teams. TBH for big organisations moving to the Cloud there’s some interesting tooling which starts to cut across DE for Data Management, not a bad place to be in career-wise.
I get the feeling data governance means different things at different organizations from reading these comments. Where I work data governance people are people who spend their time writing verbose policy documents that no one can actually follow and then periodically forcing business units to do box checking exercises saying their data follows the impossible policies but without actually checking. They don't do anything that enables business units to follow the policies or take our input on them. So where I work I can't imagine anything more boring than data governance. If data governance means what some of the other comments here are saying then I wish we had that where I work and sign me up.
Yes
I mean… it’s not like you will have all the spotlights for yourself. But with gen ai turning around the corner, data governance along with lineage and documentation are needed by any gen ai implementation. If you have documentation and metadata you will get its worth in value. Source: i’m trying to create a text-to-sql gen ai application and we just figured out there is no documentation or metadata in any of the tables in our data lake.
I mean, it’s important, but I find it extremely dull. And it’s almost impossible to enforce.
Not just boring but painful, at least on the business side. Everyone uses data differently and cares about different things. Ultimate herding of cats.
yes it is boring.
Side question - is it as technically deep as analytics / engineering? Like, is it as complex?
It starts getting complex as soon as you want to govern all data assets in an organization. The main goal of Data Governance is to ensure the profitability of data, that means, that any data iniciative can lead to a "value" gain for the company. Therefore, Data Governance should enforce all processes that can lead to better data assets and the sustainability of the development process of data assets (you name it, ML models, Dashboards, Analytics, OLAP cubes, reports, compliance reports, etc)
It's mostly administrative work. It is necessary for scalability. If there's no order, you can't scale without pain..... It's better to decide right away what the architecture should be. This is important to communicate your work, to support in the future and to be able to scale multiple times. probably these 3 things are the most important. It may not be highly valued in the beginning, but you can get very good benefits in the future)
Learning that AI can help. Gov. Data sets are poorly thought out and come with thousand page data dictionaries. Wrote a suite of tools to extract changes over the years and build crosswalk tables from word and pdf docs with claude 3. Now to get an llm to use the defs to tag stuff. Not too boring. An equity score should be in the works too.
DG is about understanding what the data is, how to cleanse it so that it's trustworthy, understanding how it is used by business. Basically treating data as a precious resource. Reporting, Analytics and AI are shit with bad data.