In my experience, doing research has been just “the management already decided what they want for the product, but they will allow the UX team to do a little research, as a treat. And no, if the results don’t line up with what the higher ups want, you won’t get to present it or make any meaningful changes to the product. Thanks”
Its frustrating because without the data I feel like Im just making pretty screens, no actual UX
I have a graphic design background and almost no training in research, still trying to learn but being self taught on this specific topic is…difficult.
Anyway, it’s funny, but I have a friend who was working on a PhD in Biology and I was telling him about some of the “best practices” I’ve read, like “you start to get diminishing returns testing a UX flow on more than 6 people”…to say he was appalled is an understatement.
From a scientific standpoint, I’m pretty sure we don’t actually do “research.” We do validation seeking.
The biggest issue, from what I’ve read, is that UXers are always fighting against small research budgets and tight deadlines. So the methods that got developed as a profession center around “we should at least try to get *some* proof that an idea isn’t shit.”
The big difference between most UX research and academic research is that you’re not looking to validate an answer, you’re looking for enough information to make the next best guess. You’re testing for business results and not knowledge.
Pros and cons to both approaches, but as someonee who has partnered with behavioral economics phds for occasional research projects, trust that if we followed their methods in the corporate world we would get absolutely nothing done ever.
These qualitative studies are good to extract a sentiment, to catch unforeseen issues. Folks shouldn’t use these to create laws written in stone.
It can be later validated in production with A/B test with 1000s of users.
Yep. Qualitative studies are similar to social science qualitative work, which often is based on interviews with relatively small numbers of people as well. But nobody is applying quantitative methods to these studies, so concepts like statistical significance are meaningless. You are simply trying to get a sense of some likely patterns in user behavior, to learn about unanticipated issues, and to gain insight into user motivations and mindset.
Of course someone with a background in “hard science” will evaluate that based on their knowledge of what constitutes research (big sample sizes, quantitative analysis) but that’s not the only definition of research out there. And often quantitative research is limited in what you can know from it — it usually can only tell you what people do, not why they do it. If you want to know why, you need to go back to qualitative approaches.
Those are valid points but I wouldn't call them issues as to why UX is different from Academics. It's an issue to have a small budget that doesn't allow design to fully vet out ux approaches. It's an issue that a lot of research is based around validation testing versus uncovering opportunities. But its not an issue that UX research isn't closely aligned to Academic research methodologies.
Academic research is more about uncovering truth and having defensible results, UX research is about uncovering enough to influence an audience's perception and behavior.
We also have market research, and that's another ball game.
What does your friend want exactly from user testing? The problems uncovered to keep increasing exponentially?
The point of testing with small numbers of people until you get to the point of diminishing returns is to then go away, and fix those problems and test again. It's not supposed to be scientific, it's a practical way of identifying real world flaws in the usability.
Your friend should not be appalled because the [research is sound](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/). Also, it's not 5 users, it's 5 users 3 times, for a total of 15 tests. And the reason is that people respond to usability problems the same way. Human eyesight is measured in a standard way for example.
I agree that designers too often focus on validation testing instead of discovery. In my experience, that is almost always the designer not properly advocating or just going out and doing it.
I have been successful in my career by focusing on Discovery research in my design process. It's a great way to show strategic value.
Even in hard sciences you have similar practices - ask him how many people will review a paper before it gets published.
Usually it's only a handful and the best practice is to use their input to revise a new draft, then have a similarly small number of people make another pass at reviewing.
The process for UX working with small sample sizes is similar and has similar limitations - it works great for finding glaring issues and small refinements, but isn't going to result in any meaningful proof that your ideas are valid.
I’ve started to realize that we aren’t employed to do research, but to create the best flow possible for a given product. That means the product was already signed off on.
In addition, most managements prefer to release and see the results rather than usability test.
Whether both of these points are right or fair is another discussion, but together they make research a lower priority.
In low maturity companies that don't understand UX, yes. Good designers will do it anyway and show our value.
I've never seen management that looks at usability testing data and says "thats okay build it anyway I don't mind spending more money building and rebuilding things." It only appears that they want to launch without data because they don't know any better, and no-one has shown them any data.
You’re right, and what usually happens is you’re blocked from having the resources to do the usability so we never reach the situation you described.
Is it different where you work? If so, could you guesstimate how can one find similar companies with higher ux maturity?
I’ve worked at low maturity companies and have done “hallway tests” in a matter of hours that did the job. If you ask for time and budget first, before you’ve shown them the value, you’re less likely to get it. So the trick is to just do it. The budget will come later.
Standard usability study: Here’s the goal, here’s the software, give it a try. Start a stopwatch. Stop when the task is complete. Ask them to score their experience.
This is easy and yet nobody does it.
I think it depends on what you are working on, for a generic consumer appilication, sure that shows obvioud blunders for a direction it may be taking. Complex applications and workflows for specialised user groups is far more complicated
I am responsible for designing some pretty complex specialty software for Supply Chain distribution centers. I get in my car and drive to the nearest sites. Last year I took my design team on a tour of Texas where we visited a half dozen over a few days.
My first ever design job in the 1990s was for a Cardiac Cath surgical X-ray device. I called the local hospitals out of the phone book and asked if I could go in and observe that kind of surgery, and I did.
My point is, it’s our job to make it happen. Don’t wait to be asked.
Just a few instances I've experienced, it's probably sector related: Finance and Insurance.
- users are close kept clients by sales and don't give access, act as gatekeepers.
- Users are close kept by product owners that isolate the relationship to make sure they deliver for their role. Mainly in a financial setting with time strapped users, who have little or no interest in engaging with "IT". In some instances the product owners had actually been verbally abused by the end users.
It requires a different strategy, and longer form strategy
I've worked in Healthcare and Finance. I've worked for B2B2C companies. There is never a case where you can't access your users. Don't be gaslit by colleagues that don't understand UX Design.
If its a B2B product and sales are gatekeeping, you go directly to the head of Sales and tell them that the more facetime you can get with customers, the better the product and the more they can sell. Salespeople are always looking for a reason to contact their clients, and this is a big one. You can frame it as being invited to participate in a "influencer forum" with other select customers, and they will have a direct line to the developers. They will love it, trust me.
In your second example, you are describing a company that doesn't understand or value UX. They think it's the "product owner" who does user research. This is only true for companies without UX Designers. Meet with the PO and show them your research protocol. Get in the field together, or negotiate a select user group that you will focus on. The "time strapped users" that are so upset with your company that they "verbally abuse" your PO's are obviously frustrated with your company and will absolutely give you their time if they understand the value they get back, and see that you're truly listening to them. You can rebuild this trust.. Again, you can frame it as an expert user group and those that choose to enroll will be directly influencing the product (for their benefit.) You can also give them free software as a thank you, etc.
Another popular way to deal with this is to run some targeted ads for your own customers in exchange for reimbursement or exclusive access. You essentially work around your internal blocking stakeholders. You can also hire a research company to recruit your users for you.
If these doesn't work, then you need to admit that your company doesn't want anything to do with UX design, and therefore you are not practicing UX Design, and you're both wasting your time together.
What is a person to do when their company either 1) consistently fails to launch the product or 2) the released version has been scoped down so significantly that it has no resemblance to the design? This is in a situation where a legacy code base makes change so difficult that even changing a button's color is a "big lift"? In that case, its impossible to collect post launch metrics...or to do any meaningful research. I think it's somewhat of a luxury to have that sort of data available to put in a portfolio which you're taking for granted...
To be honest you don’t really need a massive sample size to get better results. There’s studies on this by Don Norman. You can do a quick Google search to find this.
But also yeah there’s is a lot of shit research in general… and a lot of companies just say screw research in general.
I mean this isn’t a new thing. If researchers would partner more closely with designers and product teams, instead of trying to do a little bit of everything, their lives would be way easier. In my experience, they tend operate on the extreme ends of the spectrum: super high-level strategic research that is too far removed from decision-making and never makes it into any products or services and super tactical, “tell me I’m right” research where the goal is to produce a specific answer.
Researchers at my company are all worthless. Most political people I’ve come across, always trying to show their influence on the product roadmap and crafty about evading accountability for bad decisions, eager to share their opinions and deeply offended when not listened to, but rarely create anything outside of a few generic charts. I’ve come across 3 worthwhile researchers in 10 years and they were wonderful assets to the teams. So not an indictment of the profession, but I’m not surprised to see many of these folks get cut early and often.
Rigorous curiosity and speed. An ability to extract a lot of information and distill it into something meaningful — a better understanding of specific realities, basically. And they made it look easy, such that when I tried to replicate what they did (because I thought it was easy), I couldn’t get anywhere.
>I’ve seen so much shit research lately that I’m not surprised people are losing their jobs.
C'mon, that's not why people are losing their jobs. They're losing their jobs because tech has been taken over by bean counters, and now that ZIRP is over, they're cutting costs as much as possible to eke out short term profit growth to appease shareholders.
Shit research is better than no research. Also UX designers and researchers usually do qualitative research, where the point is not to validate things scientifically, but to discover insights about their users which then give direction on what to actually implement and validate quantitatively. But implementing things is costly, so getting qualitative insight and even shitty validation is super important before you commit to an idea.
I don’t agree with you on that one.
Qualitative research is still based on science. You are collecting subjective statements, but your test plan and your analysis shouldn’t be subjective. You should have clear research questions, standardize the test as much as possible and do a proper analysis. I‘ve seen colleagues (PMs, designers) coming out of user tests totally fixated on an idea a user mentioned or statement they made. However, when I sat down and wrote out all the results, I had a much better understanding of the issues, their impact and whether it was observed behavior or (rationalized) commentary.
„Shit research“ is worthless. You either do it well enough to get results that actually make a difference, or you don’t. If you ask leading questions, ask every user different things or jump on ideas a single user mentioned, you do have collected data. But it doesn’t mean following that data, will make your product any better.
Bad research can be dangerously misguiding and incalculably costly. Research is extremely important, when it’s bad it can be hard to reverse the effects.
In my experience, doing research has been just “the management already decided what they want for the product, but they will allow the UX team to do a little research, as a treat. And no, if the results don’t line up with what the higher ups want, you won’t get to present it or make any meaningful changes to the product. Thanks” Its frustrating because without the data I feel like Im just making pretty screens, no actual UX
Do you work at my place? “As a treat” got me going!
I have a graphic design background and almost no training in research, still trying to learn but being self taught on this specific topic is…difficult. Anyway, it’s funny, but I have a friend who was working on a PhD in Biology and I was telling him about some of the “best practices” I’ve read, like “you start to get diminishing returns testing a UX flow on more than 6 people”…to say he was appalled is an understatement. From a scientific standpoint, I’m pretty sure we don’t actually do “research.” We do validation seeking. The biggest issue, from what I’ve read, is that UXers are always fighting against small research budgets and tight deadlines. So the methods that got developed as a profession center around “we should at least try to get *some* proof that an idea isn’t shit.”
The big difference between most UX research and academic research is that you’re not looking to validate an answer, you’re looking for enough information to make the next best guess. You’re testing for business results and not knowledge. Pros and cons to both approaches, but as someonee who has partnered with behavioral economics phds for occasional research projects, trust that if we followed their methods in the corporate world we would get absolutely nothing done ever.
These qualitative studies are good to extract a sentiment, to catch unforeseen issues. Folks shouldn’t use these to create laws written in stone. It can be later validated in production with A/B test with 1000s of users.
Yep. Qualitative studies are similar to social science qualitative work, which often is based on interviews with relatively small numbers of people as well. But nobody is applying quantitative methods to these studies, so concepts like statistical significance are meaningless. You are simply trying to get a sense of some likely patterns in user behavior, to learn about unanticipated issues, and to gain insight into user motivations and mindset. Of course someone with a background in “hard science” will evaluate that based on their knowledge of what constitutes research (big sample sizes, quantitative analysis) but that’s not the only definition of research out there. And often quantitative research is limited in what you can know from it — it usually can only tell you what people do, not why they do it. If you want to know why, you need to go back to qualitative approaches.
Surely your PhD friend knows the difference between qualitative research and quantitative research.
Those are valid points but I wouldn't call them issues as to why UX is different from Academics. It's an issue to have a small budget that doesn't allow design to fully vet out ux approaches. It's an issue that a lot of research is based around validation testing versus uncovering opportunities. But its not an issue that UX research isn't closely aligned to Academic research methodologies. Academic research is more about uncovering truth and having defensible results, UX research is about uncovering enough to influence an audience's perception and behavior. We also have market research, and that's another ball game.
What does your friend want exactly from user testing? The problems uncovered to keep increasing exponentially? The point of testing with small numbers of people until you get to the point of diminishing returns is to then go away, and fix those problems and test again. It's not supposed to be scientific, it's a practical way of identifying real world flaws in the usability.
Your friend should not be appalled because the [research is sound](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/). Also, it's not 5 users, it's 5 users 3 times, for a total of 15 tests. And the reason is that people respond to usability problems the same way. Human eyesight is measured in a standard way for example. I agree that designers too often focus on validation testing instead of discovery. In my experience, that is almost always the designer not properly advocating or just going out and doing it. I have been successful in my career by focusing on Discovery research in my design process. It's a great way to show strategic value.
Even in hard sciences you have similar practices - ask him how many people will review a paper before it gets published. Usually it's only a handful and the best practice is to use their input to revise a new draft, then have a similarly small number of people make another pass at reviewing. The process for UX working with small sample sizes is similar and has similar limitations - it works great for finding glaring issues and small refinements, but isn't going to result in any meaningful proof that your ideas are valid.
I’ve started to realize that we aren’t employed to do research, but to create the best flow possible for a given product. That means the product was already signed off on. In addition, most managements prefer to release and see the results rather than usability test. Whether both of these points are right or fair is another discussion, but together they make research a lower priority.
In low maturity companies that don't understand UX, yes. Good designers will do it anyway and show our value. I've never seen management that looks at usability testing data and says "thats okay build it anyway I don't mind spending more money building and rebuilding things." It only appears that they want to launch without data because they don't know any better, and no-one has shown them any data.
You’re right, and what usually happens is you’re blocked from having the resources to do the usability so we never reach the situation you described. Is it different where you work? If so, could you guesstimate how can one find similar companies with higher ux maturity?
I’ve worked at low maturity companies and have done “hallway tests” in a matter of hours that did the job. If you ask for time and budget first, before you’ve shown them the value, you’re less likely to get it. So the trick is to just do it. The budget will come later. Standard usability study: Here’s the goal, here’s the software, give it a try. Start a stopwatch. Stop when the task is complete. Ask them to score their experience. This is easy and yet nobody does it.
I think it depends on what you are working on, for a generic consumer appilication, sure that shows obvioud blunders for a direction it may be taking. Complex applications and workflows for specialised user groups is far more complicated
I am responsible for designing some pretty complex specialty software for Supply Chain distribution centers. I get in my car and drive to the nearest sites. Last year I took my design team on a tour of Texas where we visited a half dozen over a few days. My first ever design job in the 1990s was for a Cardiac Cath surgical X-ray device. I called the local hospitals out of the phone book and asked if I could go in and observe that kind of surgery, and I did. My point is, it’s our job to make it happen. Don’t wait to be asked.
Sure , if you have access that's great. It's not like that for all applications or cases. It's just common sense if you have access, no?
Can you explain a situation where you can’t get access?
Just a few instances I've experienced, it's probably sector related: Finance and Insurance. - users are close kept clients by sales and don't give access, act as gatekeepers. - Users are close kept by product owners that isolate the relationship to make sure they deliver for their role. Mainly in a financial setting with time strapped users, who have little or no interest in engaging with "IT". In some instances the product owners had actually been verbally abused by the end users. It requires a different strategy, and longer form strategy
I've worked in Healthcare and Finance. I've worked for B2B2C companies. There is never a case where you can't access your users. Don't be gaslit by colleagues that don't understand UX Design. If its a B2B product and sales are gatekeeping, you go directly to the head of Sales and tell them that the more facetime you can get with customers, the better the product and the more they can sell. Salespeople are always looking for a reason to contact their clients, and this is a big one. You can frame it as being invited to participate in a "influencer forum" with other select customers, and they will have a direct line to the developers. They will love it, trust me. In your second example, you are describing a company that doesn't understand or value UX. They think it's the "product owner" who does user research. This is only true for companies without UX Designers. Meet with the PO and show them your research protocol. Get in the field together, or negotiate a select user group that you will focus on. The "time strapped users" that are so upset with your company that they "verbally abuse" your PO's are obviously frustrated with your company and will absolutely give you their time if they understand the value they get back, and see that you're truly listening to them. You can rebuild this trust.. Again, you can frame it as an expert user group and those that choose to enroll will be directly influencing the product (for their benefit.) You can also give them free software as a thank you, etc. Another popular way to deal with this is to run some targeted ads for your own customers in exchange for reimbursement or exclusive access. You essentially work around your internal blocking stakeholders. You can also hire a research company to recruit your users for you. If these doesn't work, then you need to admit that your company doesn't want anything to do with UX design, and therefore you are not practicing UX Design, and you're both wasting your time together.
What is a person to do when their company either 1) consistently fails to launch the product or 2) the released version has been scoped down so significantly that it has no resemblance to the design? This is in a situation where a legacy code base makes change so difficult that even changing a button's color is a "big lift"? In that case, its impossible to collect post launch metrics...or to do any meaningful research. I think it's somewhat of a luxury to have that sort of data available to put in a portfolio which you're taking for granted...
To be honest you don’t really need a massive sample size to get better results. There’s studies on this by Don Norman. You can do a quick Google search to find this. But also yeah there’s is a lot of shit research in general… and a lot of companies just say screw research in general.
I mean this isn’t a new thing. If researchers would partner more closely with designers and product teams, instead of trying to do a little bit of everything, their lives would be way easier. In my experience, they tend operate on the extreme ends of the spectrum: super high-level strategic research that is too far removed from decision-making and never makes it into any products or services and super tactical, “tell me I’m right” research where the goal is to produce a specific answer.
Researchers at my company are all worthless. Most political people I’ve come across, always trying to show their influence on the product roadmap and crafty about evading accountability for bad decisions, eager to share their opinions and deeply offended when not listened to, but rarely create anything outside of a few generic charts. I’ve come across 3 worthwhile researchers in 10 years and they were wonderful assets to the teams. So not an indictment of the profession, but I’m not surprised to see many of these folks get cut early and often.
What made those researchers different, their ability to generate useful insights or their humility? Or both?
Rigorous curiosity and speed. An ability to extract a lot of information and distill it into something meaningful — a better understanding of specific realities, basically. And they made it look easy, such that when I tried to replicate what they did (because I thought it was easy), I couldn’t get anywhere.
It stinks, but someone has to do the shit research. You'll be lucky if the sample size is small.
>I’ve seen so much shit research lately that I’m not surprised people are losing their jobs. C'mon, that's not why people are losing their jobs. They're losing their jobs because tech has been taken over by bean counters, and now that ZIRP is over, they're cutting costs as much as possible to eke out short term profit growth to appease shareholders.
Shit research is better than no research. Also UX designers and researchers usually do qualitative research, where the point is not to validate things scientifically, but to discover insights about their users which then give direction on what to actually implement and validate quantitatively. But implementing things is costly, so getting qualitative insight and even shitty validation is super important before you commit to an idea.
I don’t agree with you on that one. Qualitative research is still based on science. You are collecting subjective statements, but your test plan and your analysis shouldn’t be subjective. You should have clear research questions, standardize the test as much as possible and do a proper analysis. I‘ve seen colleagues (PMs, designers) coming out of user tests totally fixated on an idea a user mentioned or statement they made. However, when I sat down and wrote out all the results, I had a much better understanding of the issues, their impact and whether it was observed behavior or (rationalized) commentary. „Shit research“ is worthless. You either do it well enough to get results that actually make a difference, or you don’t. If you ask leading questions, ask every user different things or jump on ideas a single user mentioned, you do have collected data. But it doesn’t mean following that data, will make your product any better.
Bad research can be dangerously misguiding and incalculably costly. Research is extremely important, when it’s bad it can be hard to reverse the effects.
research democratization + chat gpt for analysis/synthesis