I’ve made a few posts discussing California’s First Year Law Students’ Exam (the “FYLSX”), noting that I failed my first attempt by a fraction of a percent when the Bar applied a grading formula different from that which was advertised, and then, using the knowledge gained of their altered grading scheme, passed on my second try so spectacularly that the Bar published one of my essays as an example of how to write their exam. I also briefly mentioned that I — as I’m known to do — filed suit to ask a judge to effectively require them to grade like they say they will and be more transparent about the exam and how they grade it.
The particular dispute was that they advertised that the multiple-choice section and the essay section of the exam would be “converted to the same 400-point scale” in order to “give these sections equal weight.” When I received my score report from the first exam, I noticed that the scaling formula used by the bar resulted in it being impossible to score more than about 362 points even if every question was answered correctly, and likewise, it was possible to score far more than 400 points on the essays.
The attorney for the Bar assigned to the case disclosed additional documents, formerly considered to be secret and until now never disclosed to law students, to me yesterday that all but flatly admit that not only were the sections not weighted equally, but they also didn’t use a 400-point scale! See if you can follow this tortured grading system they describe, because it took me several reads to figure it out:
Multiple-choice raw scores (i.e., number of items answered correctly) were equated to the June 1998, 2011, and October 2013 exams using 21 items that were common to each of these exams. The equating formula was as follows:
Multiple Choice Scale = (3.4092 x raw multiple-choice score) + 21.6267
The candidates’ raw total essay scores were scaled to a score distribution that had the same mean and standard deviation as their multiple choice scores using the following formula:
Essay Scale = (2.3536 x raw essay total score) + -442.389
A candidate’s total score was the sum of that candidates’ multiple choice and essay scores.
What I gather from this is that 21 of the 100 multiple choice questions were repeats of previous years, and based on how well students did on those 21 questions, their grade on the entire 100 questions was curved. Then, they calculated the average multiple choice score and the standard deviation, and curved the essay scores such that the average student gets the same score on that section as the multiple choice, and the score distribution was normalized to the same standard deviation.
From this it is clear:
- This has nothing to do with a 400-point scale per section. The number 400 does not even appear in this internal document regarding their grading scheme. The scale is created simply by comparing this group’s scores to previous scores and trying to curve it accordingly. You’ll notice that if you plug in “100” as your raw multiple-choice score (a perfect 100 out of 100 questions) into the formula provided, the maximum score attainable is 362.5467. Not 400.
- The sections do not have “equal weight.” Saying that the average test taker got the same score on both their multiple-choice and their essay questions is not the same thing as saying “half your grade comes from multiple-choice, and half from the essays.” Whether the essays count more or less than the multiple-choice section depends entirely on how well the other students do on their exams. This is not equality, it is normalization.
So, for those taking the June 2017 exam, know this: on both exams I sat for, the essays counted far more than the multiple-choice. You should therefore be spending far more time studying how to write a good essay than how to select A, B, C, or D correctly, unless your multiple-choice practice exams are turning out abysmally (for more study tips, see my previous post). Hopefully by the October 2017 exam, a judge will have “persuaded” them to abandon this system.
Hot. Damn. Well done sir. This is madness. Really appreciate you doing this work and sharing your experience!! I’m slotted for June 2018 so if not by October, hopefully by then!! Thank you!!
Thanks Liam! It is almost certain that the court case will be resolved before June 2018… let’s hope it gets resolved favorably! 🙂
I just started reading this blog because of the other incident, but wanted to say congrats.
I think it’s more likely they’ll change how they say it’s scored rather than change the scoring.
That’s ok with me, tbh. Having a stupid scoring scheme is annoying, but having a stupid scoring scheme and pretending it’s something else is just wrong. Although improving how the exam is graded would be nice, transparency is the higher goal.
The fact of the matter is if they had said before the exam, “MCQ and essay scores will be normalized, and this tends to result in the essays counting significantly more than the MCQs,” I would have thought it was weird and then proceeded to study my essay writing harder. But, since they didn’t, I focused more on the MCQs, bringing my performance up to an extremely high level, which mathematically would have carried the day. The result was incredibly frustrating.
I agree Jonathan. I sat for the June exam and for preparation I was advised by my professors, as well as Flemings (for $600) to spend a few hours daily doing MBE’s. They instructed me to do atleast 3,000 before the test. The theory was if you score high on the MBE’s your writing could be mediocore or below and you could still attain the passing score. So I studied several hours a day, dedicating most of my time to MBE’s and the rest to rule memorization, outline writing, and I’d write a couple of essays a week. If I had this info prior to testing I would’ve done things completely different.
Some of my classmates found out yesterday they passed and I’m sitting her like :-(. This is frustrating. Im hoping to get news of passing so that I can put this nightmare behind me.
Congrats on passing the bar! Thank you for taking this flawed, manipulative, and hopefully illegal grading method and process of the Bar to task. Let me know if there is anything I can do to support.
It sucks. BTW, if grades are out, you should be able to see your status change if you log into your attorney status page. If it still says the FYLSX requirement is outstanding, you didn’t pass this one.
She Dixon, I am curious, did you pass?
Thank you for this insight .. as a first -time taker ( hopefully last time as well) of the exam this past June and awaiting my August results I find this information incredibly helpful. BTW good job on the complaint form, very impressed with your writing ability. I am also astounded at the high MBE score 94% almost perfect and with that you didn’t receive a passing grade, this is astonishing. I also believed that each exam portion was weighted equally, I guess not. Although, I studied equally but in talking to many other law students including past takers most people were in full agreement to get the MBE scores higher since this is where many people fail the exam due to lower MBE scores. A friend of mine would have passed the Oct 2016 exam if he would have had 6 more points in the MBE area and his essay scores were average and some even low. None of this makes any sense to me.. I am with you in the fact that transparency in the grading is tantamount for success and preparation of the exam. Good luck on the case and I am anxious to hear the results. Hopefully in your favor …. M
Reblogged this on Attorney Caree Harper and commented:
The Truth about the First Year Law Student’s Exam
Thank you for sharing this insightful information! How does lying to the test takers about how they are being graded benefit anyone or anything? I hope the judge rules in your favor.
It doesn’t benefit anyone. They made a mistake and are now being obtuse and stubborn.
Who writes the MCQs/MBEs for the First year law exam? I am curious
Lawyers who weren’t able to get a real job. 😉
I’m not a math guy so I find this grading formula to be confusing. For the essay scaling, where does the -442.389 number come from? Does this number change with every examination, or is it the same number (442.389) for every test?
I believe that number does change with every new exam.
Yep. The formula they come up with aims to have the same mean and standard deviation as previous exams, and to adjust for the difficulty of the exam compared to other sittings..
Thanks for the clarification guys!
That 442.347 number seems to be quite high so I decided to investigate a little bit. I was reviewing Ray Hayden’s six previous FYLSE exams from 2012-2014 and plugged in his raw essay scores for each examination, took the difference between the (2.3536 * raw essay score) and his scaled essay score, to obtain that number they use. I ended up with numbers between 310-335 as opposed to something in the 400s. The June 2016 exam being at 442.347 seems abnormally high in comparison and I believe that is because everyone was given a score of 100 on the criminal law essay. I could be completely wrong though haha.
One more thing. The lawsuit to challenge this system can’t come from a student. It needs to come from the general public, which is damaged by the grading system. You see, it’s a form of price fixing. If you only let 20% pass the test, then there will be fewer lawyers to choose from. This increases price, and diminishes quality due to reduced competition for service contracts. Worse, the Bar has been accused of gross negligence in managing their backlog of disciplinary cases: http://www.calbar.ca.gov/Portals/0/documents/reports/2017-Annual-Attorney-Discipline-Report.pdf According to statistics, there are 219 people for every lawyer in the state. We are honestly about average compared to other states, BUT there were over 6.2 million court cases filed from 2015 to 2016. How many people walk into a court without a lawyer, when they really need one, because they can’t find a good one that they can afford?
Jon i read somewhere you calculated MBE is only around 35% of total grade? I posted various charts on FB of the curve-just curious how you got the amount-I’m not totally surprised. If someone gets over 90’s on the essays they will get over 800. BTW way to go 703! Good luck next week!
What ever became of your lawsuit against the bar? Was their agreeing to rewrite the online description of their FYLSX grading methodology part of a settlement?
Just curious. I’m in my first year at NWCU and will take the baby bar soon.