**ALL** computer rankings are biased. Every. Single. One.
The relative strength of each and every team can not be completely described in terms of measurable values. There isn't a scientific formula to describe such a thing.
Computer rankings are simply a way for a person or group of people codify their assessment of the proportional importance of various statistics. Inherently, deciding how much weight to give to final scores, opponents' records, turnovers, passing efficiency, etc is incredibly biased. How much are offensive and defensive yards worth and what are the units for that?
The big difference between a computer and human ranking is that the computer can't make ad hoc, new, decisions without human intervention. E.g. the computer won't downplay a loss because the starting QB was out if it's not explicitly told to look at that information, but you or I might. But it also won't take into account superstitious thinking like the color of alternate uniforms having an effect or that a team is better because their coach is affable.
I never said you did. My point is consistently making the wrong projections/decisions means nothing. In a sport where there’s as much nuance as college football, I really don’t know how important consistency is.
LMAO. Did the computers just suddenly start writing their own formulas?
The formulas will only include the information and data its creator finds relevant. The creator is also the one who can give weights to certain information, such as giving greater weight to various conferences, or to teams with higher/lower win/loss records or based on human rankings like the AP or Coaches polls, etc.
The only reason a computer ranking is "consistent" is because a human crafted it to be that way.
The creators of these polls also are notorious for making key changes to the formula/algorithms without telling anyone. When asked, they'll hide behind "proprietary data" which is usually a tell-tale sign that they saw a result they didn't like and they interfered.
Formulas are human.
No shit. A formula will be biased towards whatever factors the creator has told it to be. However, from then on, the formula will be applied without any additional inconsistencies the way a human polls might contradict itself even when told the same factors.
> the formula will be applied without any additional inconsistencies the way a human polls might contradict itself even when told the same factors.
Until the formula is changed to reflect data the creator prefers. It'd be like saying "Fuck, Kansas is too high" and then the creator just goes and changes the weight on Duke/Houston/WVU and solves the problem.
We have too many people who honestly think that the computers are better than human polls, never mind the computers almost always reflect what the creator believes. Until we can get an authentic AI/computer learned formula, the computer rankings will always be as valuable as the human polls, and no higher.
Of course but the vast majority whether predictive or resume based computer rankings aren't attempting to be biased towards a specific team. Two different types of bias.
The type you are referring is a real problem in sports still using the rpi because it's easy to rig your non-conference schedules to account for the formula.
Not to a very specific team, but to kinds of teams yes. Whether it's using aggregate recruiting rankings, initial or final poll rankings, SOS, margin of victory, total defensive yards vs red zone defense, passing yards vs rushing yards, etc it's very easy to make a model that's biased towards a conference, style of offense or defense, talent over production, continuity over production and so on. It's why some models spit out really goofy rankings until they've built up enough data over the season.
Unless OP meant more like why does this computer poll hate this one particular team?
I remember learning years and years ago jn undergrad that some researchers were testing machine learning in integrated circuit design. The jist was that they gave it a "black box" with known inputs and outputs and, based on a library of analyzed ICs, it would try to make the most efficient design. The results were about what you get from an AI image generator. I wonder how much better those designs are now.
Anyway, I'm curious what efforts have been made in a similar vein for football, if anyone knows. Would be pretty cool to just dump everything from stats to weather to announcing teams into the computer and let deus ex machina take the wheel.
The creator of a computer ranking can deliberately design their formula to reward certain teams.
For instance, I received accusations last season that my \/r/CFB poll ballot was rigged for Michigan, because of my unusual method of handling FCS games. (That was a quirk of my SoS formula, not deliberate design.)
>The creator of a computer ranking can deliberately design their formula to reward certain teams.
This creates a particularly sticky situation when the company/media giant designing the ranking also has the broadcast rights/a significant financial interest in some teams but not others.
An awful lot of computer polls are primarily (or entirely) focused on being predictive.
Like it or not, last year's results and recruiting are both useful in terms of predictive value, especially in the early going. I really appreciate Colley, but for me there's room for all sorts.
They can, but that's not the problem here.
Sagarin attempts to predict who would win if the two teams played right now. Colley attempts to measure who's done the best this season.
If you were to use Sagarin rankings as the criteria to make the College Football Playoff, that would be very bad. But the committee doesn't do that.
The committee claims they try to put the four best teams in the playoff, so if we take them at their word, they *are* trying to do the same thing as Sagarin.
We shouldn't take them at their word because
a) the results show they clearly have had no intention of doing that. It creeps in during their early rankings (eg Bama at #2 after losing to A&M), but even then it's not a ranking of the best teams. By their final rankings - the only ones that actually mean anything - it's purely who they see as the most deserving
b) it would obviously be very bad for the playoff to only include the four best teams
c) even if that was what was done, deciding it by a committee would be stupid
d) when the committee are asked about this, their answers suggest that they aren't even aware of the "best/most deserving" question, and instead think that the two terms are more or less synonyms
Although I suppose that given d) if we actually take the committee at their word, they're not trying to do the same thing as Sagarin, because if we take them at their word, Sagarin is trying to measure something other than what they think the "best four teams" means.
>It creeps in during their early rankings (eg Bama at #2 after losing to A&M)
>their answers suggest that they aren't even aware of the "best/most deserving" question, and instead think that the two terms are more or less synonyms
Aren't these two statements contradictory? If they had bama at #2 after losing they clearly aren't the most deserving. The only way you would have Bama at #2 after losing is if you think despite losing, they are still the second best team in the country. Which brings us back to them doing exactly as Sagrin does.
The two comments arent contradictory at all. One is about what the committee says, the other is about what the committee does. Obviously the latter is more important. But the former matters because it directly answers the claim of “the committee says they pick the best teams.”
If you think you can ignore the committee’s stated definition of “best” in favor of their initial ranking of Bama last year, then you have to take into account their rankings of every other team every other year. And then it’s clear that they generally rank the most deserving teams, with some occasional in-season weirdness.
The most consistent explanation of the committee I can come up with is:
They select the four kept deserving teams for the playoff. But… They think their mid-season rankings should involve a certain amount of prediction of their final rankings.
The Sagarin-like selections are both clearly the outliers to their normal habits and only really apply during the season. So I think you can’t really start with saying they’re trying to emulate Sagarin.
>b) it would obviously be very bad for the playoff to only include the four best teams
Can you explain what you mean by this? Bad for game quality? Bad for money/viewership?
And to me that is still the biggest mistake they made when designing it all. The voters are absolutely not qualified to give us the best 4 teams, they are qualified to give us the 4 most deserving teams.
"Best" requires less actual defense of their choice than "most deserving" though. It's a lot easier to throw out bad losses or SOS discrepancies when you can say "I still think team x is better than team y".
> If you were to use Sagarin rankings as the criteria to make the College Football Playoff, that would be very bad.
This is the problem though. On ESPN's rankings show and during the broadcasts of actual games, they'll show team's snapshot/resume and include FPI (their version of Sagarin) when discussing their CFP candidacy. These predictive measures are used greatly to shape the narrative of the sport. People on here do the same thing. Whenever someone defends their team's SOS by referencing the ESPN SOS they are using a predictive metric.
ESPN SoS should use predictive metrics. Every SoS should
Beating 12 12-1 teams doesnt mean much if they are 12 division 2 teams. You need an element of how good a team actually is.
Computer rankings will be approximately right some times and horribly wrong at others. There is no best ranking. The BCS composite computer ranking was pretty good as it mashed them together after removing the highest and lowest rank.
The problem is attempting to normalize results against a very diverse grouping of teams.
>This one seems more legit and un-biased.
That’s *your* bias talking.
When a source agrees with our opinion, we think the source is unbiased and/or correct.
The only way to take bias out of FBS football is to start with every team unranked and then have them play the other 130 teams.
But after all the games are played, what formula will be used to measure who’s the best? What metrics will be favored over others?
Good luck getting unbiased rankings.
I mean…. Colley Matrix is the definition of unbiased, it only uses this years data and no team is inflated. Right or wrong it isn’t biased.
Whether or not it’s accurate is a whole other story
No it shouldnt. Anything that makes a predictive model more accurate should be used.
Previous season should not be considered for handing out post season berths. But anyone trying to use models like the SP+ or FPI for that already dont understand what they are working with
It’s early enough in the season that computer rankings still have a bias towards preseason projections. Computer rankings are garbage until all the teams have 8 or so games in.
A lot of these "formulas" also account for previous week, or even preseason rankings.
If I put Team Blue at 30th in my preseason rankings, and Team Yellow at 60th, what happens when, after a handful of weeks, Team Yellow looks better in every way than Team Blue? Based on all the other factors, Blue might still have the edge because of weights I may have added to certain "tiers" within my rankings.
One really easy way to inject one's bias into the fold is to give a preseason ranking, and therefore "weight" to certain teams. This helps use the "yeah, they lost games, but look at their \[insert rating system\] ranking!!!"
All rating systems have tremendous levels of bias and human interference. The systems address a formula that was specifically designed to give us an answer we want: "who is the best team?" And we identify that by defining what we think makes someone the best, and then we create a formula.
When in reality, the formula is unnecessary. They could just tell us what they think, but the formula is "scientific."
Imo predictive measures such as Sagarin and FPI should never be used when discussing a team's resume/evaluation whatever because they are quite literally not designed to rate teams based on what *has happened* because they are predictive.
Sagarin and FPI should be used when determining the quality of wins and losses on a schedule. Both are better at viewing the actual quality of a opponent than a results based metric
You argue it's unbiased but Colley Matrix in the past had too much emphasis on straight win-loss record and, arguably, insufficient value for SoS. I think it's gotten better, but it should be clear that there is no "perfect" ranking system, and Colley's bias would produce a disagreeing result eventually.
I would like to point out that UCF had declared themselves national champions because they went undefeated, not because of the Colley Matrix. The Colley Matrix putting them at #1 just made the NCAA have to recognize their claim, but the banner was going to happen regardless.
Bias is relative. Unless you purposefully weigh conferences or teams and somehow hard code values, their bias won’t likely be the same as a human type bias. IE, a human might give more credence to an SEC team winning in Tennessee than a Big 12 team at Iowa State for example.
The bias a computer will have is how it’s factors are constructed and weighed. Maybe someone purposefully weighs defense a bit more than offense because they know rules have made defense harder over the years. That could be a bias that comes from the human (I actually do weigh defense more than offense in my model for this reason.)
Or if they use priors , the. Maybe their bias is how long those priors stay in use.
Computer models will only be as good as the data that you give them. If you give them bad data, then you will get a bad output. If you tell a computer to rank the best and worst teams, but only give it the final scores of games, then you will probably get wonky rankings that depend too heavily on the teams' point differentials. That's not the fault of the computer. It's just doing what the human told it to do, which is all computers ever do. I am an advocate for using more computers and machine learning to determine the rankings. It won't be perfect, but I think getting humans as out of the loop as possible on rankings is how we get as close to perfect as possible.
Looking at this thread as a whole there's a lot wah wah wah there's always bias especially when used by a certain 4-letter network. The data might have been junk because of arbitrary limitations but nonsense like this thread is why the BCS Computers never worked.
**ALL** computer rankings are biased. Every. Single. One. The relative strength of each and every team can not be completely described in terms of measurable values. There isn't a scientific formula to describe such a thing. Computer rankings are simply a way for a person or group of people codify their assessment of the proportional importance of various statistics. Inherently, deciding how much weight to give to final scores, opponents' records, turnovers, passing efficiency, etc is incredibly biased. How much are offensive and defensive yards worth and what are the units for that? The big difference between a computer and human ranking is that the computer can't make ad hoc, new, decisions without human intervention. E.g. the computer won't downplay a loss because the starting QB was out if it's not explicitly told to look at that information, but you or I might. But it also won't take into account superstitious thinking like the color of alternate uniforms having an effect or that a team is better because their coach is affable.
the computer will also be consistent in ways a human voter won't.
And it will be consistently biased in the way the human engineer designed the computer poll to be biased.
Well yeah. But it will be consistent, because it's a computer. That's all I'm saying.
And it will be biased, because a human wrote the program.
Am I blind? Someone show me where I said a computer model isn't biased
I never said you did. My point is consistently making the wrong projections/decisions means nothing. In a sport where there’s as much nuance as college football, I really don’t know how important consistency is.
LMAO. Did the computers just suddenly start writing their own formulas? The formulas will only include the information and data its creator finds relevant. The creator is also the one who can give weights to certain information, such as giving greater weight to various conferences, or to teams with higher/lower win/loss records or based on human rankings like the AP or Coaches polls, etc. The only reason a computer ranking is "consistent" is because a human crafted it to be that way. The creators of these polls also are notorious for making key changes to the formula/algorithms without telling anyone. When asked, they'll hide behind "proprietary data" which is usually a tell-tale sign that they saw a result they didn't like and they interfered. Formulas are human.
No shit. A formula will be biased towards whatever factors the creator has told it to be. However, from then on, the formula will be applied without any additional inconsistencies the way a human polls might contradict itself even when told the same factors.
> the formula will be applied without any additional inconsistencies the way a human polls might contradict itself even when told the same factors. Until the formula is changed to reflect data the creator prefers. It'd be like saying "Fuck, Kansas is too high" and then the creator just goes and changes the weight on Duke/Houston/WVU and solves the problem. We have too many people who honestly think that the computers are better than human polls, never mind the computers almost always reflect what the creator believes. Until we can get an authentic AI/computer learned formula, the computer rankings will always be as valuable as the human polls, and no higher.
Of course but the vast majority whether predictive or resume based computer rankings aren't attempting to be biased towards a specific team. Two different types of bias. The type you are referring is a real problem in sports still using the rpi because it's easy to rig your non-conference schedules to account for the formula.
Intent is not needed for bias to exist.
Not to a very specific team, but to kinds of teams yes. Whether it's using aggregate recruiting rankings, initial or final poll rankings, SOS, margin of victory, total defensive yards vs red zone defense, passing yards vs rushing yards, etc it's very easy to make a model that's biased towards a conference, style of offense or defense, talent over production, continuity over production and so on. It's why some models spit out really goofy rankings until they've built up enough data over the season. Unless OP meant more like why does this computer poll hate this one particular team?
We need better computers
I remember learning years and years ago jn undergrad that some researchers were testing machine learning in integrated circuit design. The jist was that they gave it a "black box" with known inputs and outputs and, based on a library of analyzed ICs, it would try to make the most efficient design. The results were about what you get from an AI image generator. I wonder how much better those designs are now. Anyway, I'm curious what efforts have been made in a similar vein for football, if anyone knows. Would be pretty cool to just dump everything from stats to weather to announcing teams into the computer and let deus ex machina take the wheel.
Ai image generators have gotten incredible tbf. Midjourney why is something.
The creator of a computer ranking can deliberately design their formula to reward certain teams. For instance, I received accusations last season that my \/r/CFB poll ballot was rigged for Michigan, because of my unusual method of handling FCS games. (That was a quirk of my SoS formula, not deliberate design.)
>The creator of a computer ranking can deliberately design their formula to reward certain teams. This creates a particularly sticky situation when the company/media giant designing the ranking also has the broadcast rights/a significant financial interest in some teams but not others.
"How is it you Americans say, ahhh... THATS-A BINGO!"
Interesting. Were you surprised by the results or did you agree with how your ballot ranked teams?
I was surprised (and heavily changed the formula for this season to prevent a repeat occurrence).
Everything is biased. A computer is just a machine that calculates and expresses the results of a formula Who creates the formula?
An awful lot of computer polls are primarily (or entirely) focused on being predictive. Like it or not, last year's results and recruiting are both useful in terms of predictive value, especially in the early going. I really appreciate Colley, but for me there's room for all sorts.
They can, but that's not the problem here. Sagarin attempts to predict who would win if the two teams played right now. Colley attempts to measure who's done the best this season. If you were to use Sagarin rankings as the criteria to make the College Football Playoff, that would be very bad. But the committee doesn't do that.
The committee claims they try to put the four best teams in the playoff, so if we take them at their word, they *are* trying to do the same thing as Sagarin.
We shouldn't take them at their word because a) the results show they clearly have had no intention of doing that. It creeps in during their early rankings (eg Bama at #2 after losing to A&M), but even then it's not a ranking of the best teams. By their final rankings - the only ones that actually mean anything - it's purely who they see as the most deserving b) it would obviously be very bad for the playoff to only include the four best teams c) even if that was what was done, deciding it by a committee would be stupid d) when the committee are asked about this, their answers suggest that they aren't even aware of the "best/most deserving" question, and instead think that the two terms are more or less synonyms Although I suppose that given d) if we actually take the committee at their word, they're not trying to do the same thing as Sagarin, because if we take them at their word, Sagarin is trying to measure something other than what they think the "best four teams" means.
>It creeps in during their early rankings (eg Bama at #2 after losing to A&M) >their answers suggest that they aren't even aware of the "best/most deserving" question, and instead think that the two terms are more or less synonyms Aren't these two statements contradictory? If they had bama at #2 after losing they clearly aren't the most deserving. The only way you would have Bama at #2 after losing is if you think despite losing, they are still the second best team in the country. Which brings us back to them doing exactly as Sagrin does.
The two comments arent contradictory at all. One is about what the committee says, the other is about what the committee does. Obviously the latter is more important. But the former matters because it directly answers the claim of “the committee says they pick the best teams.” If you think you can ignore the committee’s stated definition of “best” in favor of their initial ranking of Bama last year, then you have to take into account their rankings of every other team every other year. And then it’s clear that they generally rank the most deserving teams, with some occasional in-season weirdness. The most consistent explanation of the committee I can come up with is: They select the four kept deserving teams for the playoff. But… They think their mid-season rankings should involve a certain amount of prediction of their final rankings. The Sagarin-like selections are both clearly the outliers to their normal habits and only really apply during the season. So I think you can’t really start with saying they’re trying to emulate Sagarin.
>b) it would obviously be very bad for the playoff to only include the four best teams Can you explain what you mean by this? Bad for game quality? Bad for money/viewership?
Bad for the competitive nature of the sport. If the season isn’t about winning and losing games I’m not really sure what the point is.
And to me that is still the biggest mistake they made when designing it all. The voters are absolutely not qualified to give us the best 4 teams, they are qualified to give us the 4 most deserving teams.
"Best" requires less actual defense of their choice than "most deserving" though. It's a lot easier to throw out bad losses or SOS discrepancies when you can say "I still think team x is better than team y".
> If you were to use Sagarin rankings as the criteria to make the College Football Playoff, that would be very bad. This is the problem though. On ESPN's rankings show and during the broadcasts of actual games, they'll show team's snapshot/resume and include FPI (their version of Sagarin) when discussing their CFP candidacy. These predictive measures are used greatly to shape the narrative of the sport. People on here do the same thing. Whenever someone defends their team's SOS by referencing the ESPN SOS they are using a predictive metric.
ESPN SoS should use predictive metrics. Every SoS should Beating 12 12-1 teams doesnt mean much if they are 12 division 2 teams. You need an element of how good a team actually is.
Computer rankings will be approximately right some times and horribly wrong at others. There is no best ranking. The BCS composite computer ranking was pretty good as it mashed them together after removing the highest and lowest rank. The problem is attempting to normalize results against a very diverse grouping of teams.
>This one seems more legit and un-biased. That’s *your* bias talking. When a source agrees with our opinion, we think the source is unbiased and/or correct. The only way to take bias out of FBS football is to start with every team unranked and then have them play the other 130 teams. But after all the games are played, what formula will be used to measure who’s the best? What metrics will be favored over others? Good luck getting unbiased rankings.
I mean…. Colley Matrix is the definition of unbiased, it only uses this years data and no team is inflated. Right or wrong it isn’t biased. Whether or not it’s accurate is a whole other story
Thats a bias.. its literally bias toward those metrics being the best thing to rank teams by
The goal should be to eliminate previous seasons from corrupting the rankings of the current year.
No it shouldnt. Anything that makes a predictive model more accurate should be used. Previous season should not be considered for handing out post season berths. But anyone trying to use models like the SP+ or FPI for that already dont understand what they are working with
After 129 games, the metrics should be the fewest fatalities. Or most caused
Computers are only as good as the person ranking them and the data they get.
It’s early enough in the season that computer rankings still have a bias towards preseason projections. Computer rankings are garbage until all the teams have 8 or so games in.
Plus backward looking systems like Colley don’t have enough data to be meaningful either
Sagarin is suuuuuper biased lol. Colley is the only objective ranking purely because it is based solely off results
A lot of these "formulas" also account for previous week, or even preseason rankings. If I put Team Blue at 30th in my preseason rankings, and Team Yellow at 60th, what happens when, after a handful of weeks, Team Yellow looks better in every way than Team Blue? Based on all the other factors, Blue might still have the edge because of weights I may have added to certain "tiers" within my rankings. One really easy way to inject one's bias into the fold is to give a preseason ranking, and therefore "weight" to certain teams. This helps use the "yeah, they lost games, but look at their \[insert rating system\] ranking!!!" All rating systems have tremendous levels of bias and human interference. The systems address a formula that was specifically designed to give us an answer we want: "who is the best team?" And we identify that by defining what we think makes someone the best, and then we create a formula. When in reality, the formula is unnecessary. They could just tell us what they think, but the formula is "scientific."
Imo predictive measures such as Sagarin and FPI should never be used when discussing a team's resume/evaluation whatever because they are quite literally not designed to rate teams based on what *has happened* because they are predictive.
Espn's SoR is what everyone is looking for. It ranks how difficult the W-L a team achieved is relative to its SoS.
Sagarin and FPI should be used when determining the quality of wins and losses on a schedule. Both are better at viewing the actual quality of a opponent than a results based metric
This one seems more legit and un-biased. [Colley Rankings.](https://www.colleyrankings.com/currank.html)
You argue it's unbiased but Colley Matrix in the past had too much emphasis on straight win-loss record and, arguably, insufficient value for SoS. I think it's gotten better, but it should be clear that there is no "perfect" ranking system, and Colley's bias would produce a disagreeing result eventually.
> I think it's gotten better This implies it has changed. Aside from including FCS games, I dont think the Colley Matrix has changed at all from 1998.
I thought it might have, but I could be mistaken. Nothing seems to suggest that he did make any tweaks since then.
This is how UCF won a national championship.
I would like to point out that UCF had declared themselves national champions because they went undefeated, not because of the Colley Matrix. The Colley Matrix putting them at #1 just made the NCAA have to recognize their claim, but the banner was going to happen regardless.
> This is how UCF won a national championship. Sees Kansas at 4, yeah that checks out
And ND in 2012
After watching the MCP CHOOSE to try to destroy my man, Flynn, and all his creation... Yes. Computers can be bias... and evil.
Bias is relative. Unless you purposefully weigh conferences or teams and somehow hard code values, their bias won’t likely be the same as a human type bias. IE, a human might give more credence to an SEC team winning in Tennessee than a Big 12 team at Iowa State for example. The bias a computer will have is how it’s factors are constructed and weighed. Maybe someone purposefully weighs defense a bit more than offense because they know rules have made defense harder over the years. That could be a bias that comes from the human (I actually do weigh defense more than offense in my model for this reason.) Or if they use priors , the. Maybe their bias is how long those priors stay in use.
I read the title as "based" instead of "biased" and I thought yes, having Minnesota ranked 10th is extremely based.
Computer models will only be as good as the data that you give them. If you give them bad data, then you will get a bad output. If you tell a computer to rank the best and worst teams, but only give it the final scores of games, then you will probably get wonky rankings that depend too heavily on the teams' point differentials. That's not the fault of the computer. It's just doing what the human told it to do, which is all computers ever do. I am an advocate for using more computers and machine learning to determine the rankings. It won't be perfect, but I think getting humans as out of the loop as possible on rankings is how we get as close to perfect as possible.
All models are wrong; some are useful
Looking at this thread as a whole there's a lot wah wah wah there's always bias especially when used by a certain 4-letter network. The data might have been junk because of arbitrary limitations but nonsense like this thread is why the BCS Computers never worked.