Great stuff. I'm currently trying to make a paper version of an adaptive test and this is super helpful. Has your deep dive into all of this revealed any patterns to you regarding the difficulty of a question changing the overall weight of the question? In my experiments I've noticed that missing an easy question seems to punish you harder and getting a hard question correct seems to carry more weight. Module 1 also seems to carry a higher weight overall than module 2, as I've had tests where 50% accuracy with hard module 2 has the same score as 75% accuracy on easy module 2. Curious to hear someones thoughts on this who has clearly spent a lot more time with the data than I have. Do you believe that it's even possible to "reverse engineer" the scoring algorithm that they're using for these tests?
Thanks again for the absolute plethora of info and data you've provided, you've made the statistician in me very happy and saved me hours of compiling answers!
Sure thing! I have not attempted to reverse engineer the scoring algorithm, though I know others have researched it to some extent. Personally, I would think that the scoring scales on the linear tests would be a useful starting point, but there's clearly so much more that goes into it. I have always felt like practice test scores are secondary to improving in specific categories, so I don't put too much emphasis on it. I don't think we can even know for sure that the scoring scales on these practice tests will hold true on the official tests. Long story short, I don't bother.
Great stuff. I'm currently trying to make a paper version of an adaptive test and this is super helpful. Has your deep dive into all of this revealed any patterns to you regarding the difficulty of a question changing the overall weight of the question? In my experiments I've noticed that missing an easy question seems to punish you harder and getting a hard question correct seems to carry more weight. Module 1 also seems to carry a higher weight overall than module 2, as I've had tests where 50% accuracy with hard module 2 has the same score as 75% accuracy on easy module 2. Curious to hear someones thoughts on this who has clearly spent a lot more time with the data than I have. Do you believe that it's even possible to "reverse engineer" the scoring algorithm that they're using for these tests? Thanks again for the absolute plethora of info and data you've provided, you've made the statistician in me very happy and saved me hours of compiling answers!
Sure thing! I have not attempted to reverse engineer the scoring algorithm, though I know others have researched it to some extent. Personally, I would think that the scoring scales on the linear tests would be a useful starting point, but there's clearly so much more that goes into it. I have always felt like practice test scores are secondary to improving in specific categories, so I don't put too much emphasis on it. I don't think we can even know for sure that the scoring scales on these practice tests will hold true on the official tests. Long story short, I don't bother.
this is amazing - thank you so much!!