Inquiry on Question Difficulty

Edmond.Dantes

I have a brief question regarding the question difficulty label (i.e. the Question Level from 1 -5) on 7sage. How are these determined? Are they relative difficulty compared to all LSAT questions, or relative to questions of the same set?

I searched the discussion archives and couldn't find anything. Some suggested that it's related to how people responded to the question on 7sage. But that doesn't seem to be right, for some questions (e.g. P38, Sect 1, #21, difficulty level 4), most people seemed to have gotten it right (based on the answer choice distribution).

I would love some insights about this. Thanks!

runiggyrun

It is based on the responses logged by the 7sage community to that question. I don't think it's quite a perfect correlation, as I've seen questions with the same percentage of correct answers in adjacent difficulty "bands", but it's roughly based on that. Maybe the distribution of wrong answers counts, or something.
For LR for example, a 1star question was answered correctly by >90% of people, a 2 star appears to be 85-90%, a 3 star is ~73-85%, a 4 star is 60-73% and a 5 star is <60% correct.
So, even a 5 star question could have been answered correctly by the majority of the people. Looks like PT38, LR1, Q21 is a level 5, with just over half correct answers.
I find that for me, personally, the posted difficulty of the questions correlates pretty well with how they "feel", except for logical games.
Logical games seem to have a systematic skew where earlier games in the section are often rated easier than they feel, and the last game is rated more difficult. That's likely because everyone is going to have enough time to tackle most of say, game 2, but they might run out of time/feel rushed on game 4 and miss more questions than the difficulty of the game might warrant. For games, the way they "feel", at least to me, seems to correlate better with JY's comments than with the star ratings.

Pacifico

It is based on analytics of 7Sagers. If you analyze the dynamics a bit more closely you'll see there is a rough threshold for each difficulty level based on the percentage of people that get each question right. Even for level 5 questions the threshold is still above 50% so technically most people are getting those right as well, though there are some I've seen where ~30% of people answer a question correctly. I'm on my phone so I don't have all the analytics in front of me but it is roughly something along the lines of >90% correct for level 1, 85-90 for level 2, 80-85 for level 3, 75-80 for level 4 and <75 for level 5. Of course that's just a rough approximation off the top of my head and I'm sure the algorithm is slightly more complicated but that should give you a general idea.

Nicole Hopkins

The algorithms for calculating difficulty are rather complex but I have found that, on the whole, they are representative of the relative difficulty for the vast majority of questions. I very rarely disagree with that the analytics indicate.

Edmond.Dantes

Awesome. Many thanks for the clarification!