August 27, 2013

The testing industry's Golden Age

When you are thinking about a career or a part-time job, don't overlook the testing industry. You might think that nothing much is going on in the cognitive testing racket since the implementation of item response theory awhile back as computing power became adequate, but you'd be mistaken. There are lots of new revenue opportunities in testing (and in it's evil/nice twin, tutoring). From the WSJ:
Are You Ready for the Post-College SAT? 
Employers Say They Don't Trust Grade-Point Averages 
By DOUGLAS BELKIN  
Next spring, seniors at about 200 U.S. colleges will take a new test that could prove more important to their future than final exams: an SAT-like assessment that aims to cut through grade-point averages and judge students' real value to employers. 
The test, called the Collegiate Learning Assessment, "provides an objective, benchmarked report card for critical thinking skills," said David Pate, dean of the School of Arts and Sciences at St. John Fisher College, a small liberal-arts school near Rochester, N.Y. "The students will be able to use it to go out and market themselves." 
The test is part of a movement to find new ways to assess the skills of graduates. Employers say grades can be misleading and that they have grown skeptical of college credentials. 
"For too long, colleges and universities have said to the American public, to students and their parents, 'Trust us, we're professional. If we say that you're learning and we give you a diploma it means you're prepared,' " said Michael Poliakoff, vice president of policy for the American Council of Trustees and Alumni. "But that's not true." 
The new voluntary test, which the nonprofit behind it calls CLA +, represents the latest threat to the fraying monopoly that traditional four-year colleges have enjoyed in defining what it means to be well educated.

Here's a sample question: You look at some graphs on cell phone usage while driving, then write a several hundred word essay on whether it would be a good idea to pass a law banning phoning while driving.

Reasoning from quantitative data doesn't sound like a bad thing to test at all -- I do it all day long -- but I suspect it will just widen The Gap, with Moneyball fans benefiting the most. We all have these stereotypes about the reason that Hispanic girls don't do as well on tests overall as upper middle class white males is because tests don't test critical thinking skills and synthesizing inferences from data and so forth. But if you look around an airport book store, the frequent fliers, the people whose employers find it profitable to send around the country to deal with problems, seem to be mostly white and Asian guys who like Moneyball, Freakonomics, Nate Silver, Malcolm Gladwell and the like. So, making tests more like reading a Bill James essay is probably not going to close the gap, but you are a racist if you predict that, so, sure, go ahead.

After awhile, it will be noted that this latest panacea test hasn't closed The Gap, so new tests will be demanded.

Regarding job opportunities: My first thought is that they are going to have to hire a whole bunch of people to read these essays and grade them. It's boring work, but you get to do it indoors while sitting down. My second thought is that there would be a market if anybody could come up with a way to computerize grading. My third thought is that if this takes off, there will be a huge market for tutoring. And think how much more effective your tutoring will be if you could, say, know what the questions will be ahead of time.

Much the same thinking applies to the huge opportunities opened up by all the new tests demanded by the Common Core.

And then after awhile, there will be new buzzwords and then new tests. Rinse and repeat, forever.

39 comments:

Anonymous said...

What exactly is "critical thinking"? I've never seen a non BS explanation for it.

It seems to derive from the "critical theory" of Cultural Marxism, which has heavily influenced the humanities and education theory over the past half century.

Victor said...

OT: In the New York local news, there are two news stories about transgendered people; one about a transgendered woman killed in Harlem and one about American Apparel looking for transgendered models for its ads.

I expect these stories to become more common. You called it.

Jokah Macpherson said...

So what happens when some kid decides to take the test without going to college? Does the whole system crumble?

Anonymous said...

There was a time when having a college degree was a reliable signal to potential employers that you had some level of intelligence beyond a certain threshold and were considered employable, conditional on other factors.

Since every politician, every teacher, and every public figure is expected to tell young people that they must all go to college, and parents have been hoodwinked into believing that a college degree would undoubtedly be beneficial for their children, admissions thresholds have been lowered. The result is that more incompetent people are admitted. A large portion fail out, but enough make it to the finish line. Couple that with declining academic standards, the predictive value of the college degree has declined considerably.

So now we need a new variable for employers to consider. Here's how it will play out: After the first round of testing, we will notice average race difference in test scores. Employers will find that this test has better predictive power in a prospective employee's intelligence than simply having a college degree, and will give it greater weight in assessing who to interview and hire. Some black civil rights group will have a meltdown and demand that the government ban employers from using the test results in making employment decisions. Just like with IQ tests, they will whine that it has a racially disparate impact and doesn't really measure what's truly important in a prospective employee.

Anonymous said...

"So what happens when some kid decides to take the test without going to college? Does the whole system crumble?"

Can a kid take the test without a degree from a member institution? I don't know but probably not. However a smart kid could stay local at an inexpensive member school, e.g. Cal Poly, and blow the lid off the exam. Would employers notice?

countenance said...

Okay, so we have a new post college test to expose weak majors and easy grades? What if someone files a Griggs vs Duke Power lawsuit against that test for disparate impact?

Anonymous said...

The test will instantly be labeled racist. Test success will follow the usual pattern of Asian, white, Hispanic, black. Use of the test for hiring will be illegal. Disparate impact dontcha know.

Anonymous said...

"So what happens when some kid decides to take the test without going to college? Does the whole system crumble?
"

Widespread usage of MOOCs will require expansion of standardized testing. Hopefully, MOOCs will make Grutter moot.



Anonymous said...

Actually Steve the market for entry exams is saturated, so if you want to get into the biz exit exams are the only way. Nah nah I figured it out before you did.

Anonymous said...

Related:

http://www.slate.com/blogs/the_slatest/2013/08/26/all_25_000_test_takers_fail_the_university_of_liberia_entrance_exam.html

Anonymous said...

The scores on this new test are likely to be so highly correllated with the SAT/ACT that the former is entirely superfluous. Plus, the disparate impact on less-intelligent minority groups will guarantee that it is DOA.

jody said...

"My second thought is that there would be a market if anybody could come up with a way to computerize grading."

maybe you could have IBM watson scan the essays and grade them.

it will learn over time what the test takers are looking for, in the way it absorbs unrelated texts on other subjects today. when optimized, maybe it could grade 100 essays per second.

Antioco Dascalon said...

As a member of the tutoring/test prep teaching profession, I am very happy for this development. I am expecting a large drop in college applicants after the bubble bursts but this may be a way to pick up the slack. The government has all sorts of entrance exams, the most well known being the Foreign Service Exam (not counting the military entrance exams) but there are many others. If the government can discriminate based on an exam, why can't the private sector?

MKP said...

"It's boring work, but you get to do it indoors while sitting down."

Yeah, I can think of a couple other things wrong with it, too.

http://www.forbes.com/sites/robertglatter/2013/02/09/why-sitting-increases-your-risk-of-dying-sooner/

Auntie Analogue said...


You know it's coming.

Yes, you know it's coming.

Wait for it....

Wait for it....:

DISPARATE IMPACT!

All the bosses forcing these new tests on applicants should repair to their tailors, to be pre-fitted for a complete wardrobe of law suits.

paleopaleo said...

You look at some graphs on cell phone usage while driving, then write a several hundred word essay on whether it would be a good idea to pass a law banning phoning while driving.

Ha, you have to look at the graphs while driving?

Anonymous said...

Maybe "Closing the Gap" is no longer a primary objective, or of any interest at all.

Signal recognition is a primary objective in all cultures, including our own. The more complicated the signal, the more expensive will be the algorithm for recognition, and the more interesting will be the game.

Neil Templeton

Mr. Anon said...

"countenance said...

Okay, so we have a new post college test to expose weak majors and easy grades? What if someone files a Griggs vs Duke Power lawsuit against that test for disparate impact?"

Would not the Griggs vs. Duke Power ruling already apply to this?

vandelay said...

OT, but I think you'd enjoy this Onion review of the "very important" film, The Butler.
http://www.theonion.com/video/the-onion-reviews-lee-daniels-the-butler,33517/

Al said...

"My second thought is that there would be a market if anybody could come up with a way to computerize grading"

I work in Machine Learning, and I don't think that we're close to quality computerized grading. Most methods rely on having quality training and testing data sets. In this case the domain is adversarial (i.e. students are giving answers to trick the computerized grading system) so it's much harder.

Jahn Ghalt said...

Anonymous said...

What exactly is "critical thinking"? I've never seen a non BS explanation for it.


I've always considered the term to mean ability to apply reason to an argument to assess validity.

The critical thinker can distinguish facts from opinions, will disregard common fallacies (ad hominem and others), properly apply syllogisms, etc. When she's really cooking, she can enumerate, estimate, accurately assess risk, etc.

I think one can test in a valid fashion for critical thinking skills.

Anonymous said...

I've always considered the term to mean ability to apply reason to an argument to assess validity.

The critical thinker can distinguish facts from opinions, will disregard common fallacies (ad hominem and others), properly apply syllogisms, etc. When she's really cooking, she can enumerate, estimate, accurately assess risk, etc.

I think one can test in a valid fashion for critical thinking skills.


But it's not clear at all that "critical thinking" refers to basic informal logic skills. It's never defined clearly as such. It's some sort of ideological term.

Informal logic existed before the term "critical thinking" ever became in vogue. And students used to be more educated in it, since elementary and secondary education involved rigorous study in grammar, rhetoric, logic, etc., all of which train informal logical ability. "Critical thinking" as a buzzword became used after these kinds of courses were deemphasized or thrown out all together.

Discard said...

"Critical thinking" in common practice means criticizing the White Racist Patriarchy. It does not mean thinking about the potential failings of DieVerSitY.

Anonymous said...

Life should be illegal. Disparate impact!!!

Anonymous said...

I personally like what Liberia did here and their method should be copied here. It would cut down on the cost and fluff of higher education. You either meet a minimum standard on a test or no college for you.

Saltine said...

You're looking at this all wrong, Steve. What essays provide is another way for subjective evaluation. Subjective evaluation is what allows schools to discriminate against white men as it is through essays on 'life experience' and other nonsense under the cover of a legitimate test.

I suspect a similar motive here. Subjectivity is bad. Objective tests where there is a clear right and wrong answer, like answering a typical math problem is what we should push for. The education industry has exhibited too much anti-white man bias for us to trust that they will be fair when handed the reins for subjective evaluation.

Anonymous said...

1. In Southern California, private SAT tutoring centers pay anywhere between $15 and $30 per hour, and during summer 'crash courses,' those hours can certainly rack up. It's not a full time job, but it comes close.

2. Machine-grading for written essays is a major area of research. A few MOOCs are implementing early products, but they aren't perfect yet. Essay-grading software is one of those computational linguistics products that nearly requires full AI because most essays need to be graded for more than surface-level grammatical mistakes.

Grotto Boy said...

""What exactly is "critical thinking"? I've never seen a non BS explanation for it.""

"I've always considered the term to mean ability to apply reason to an argument to assess validity.

The critical thinker can distinguish facts from opinions, will disregard common fallacies (ad hominem and others), properly apply syllogisms, etc. When she's really cooking, she can enumerate, estimate, accurately assess risk, etc."


No. You are right in what critical thinking should mean. And this is what the left wants everyone to feel in their hearts that it does mean. However 'critical thinking' is another one of those mischief phrases that the left use to, on the surface, appear to be an objective, important straightforward concept like the one you describe, yet underneath, what they are really concerned with is that you are thinking like they are.

Genius IQ, seeing through fallacies, etc - you fail at 'critical thinking' in their book unless you recognize that a dry stalk of asparagus in a black grocery store is a sign of insidious institutionalized racism and genocide, and that high heeled shoes show that the same level of malice exists in the West today towards women as the old practice of foot binding in Asia.

candid_observer said...

I'd be curious to know how most students who take this sample test actually fare on it.

It's not exactly demanding by any reasonable stretch. But I suspect that the number of students who can actually spot the fallacy and articulate it is pretty small.

It would be interesting to know too how performance is distributed across majors.

Cail Corishev said...

What exactly is "critical thinking"?

It seems to mean, "thinking of ways to be critical of white European civilization."

Anonymous said...

A few years ago the SAT added an essay section. In order to keep the cost of grading down they hire semi-literate HS teachers for cheap in their spare time and they give them something like a minute or two to "read" (skim) each essay and issue a grade from 1 to 6. A researcher showed that he could accurately predict the score of an essay from across the room by eyeballing the # of words written and the length of the words. So don't look for any great new employment opportunities. Either the testing organizations will do the same for this test or they will get a machine to do it, which will probably do a better job.

Anonymous said...

" In this case the domain is adversarial (i.e. students are giving answers to trick the computerized grading system) so it's much harder."

periodic retraining of the model by handscoring false positives and false negatives?

Pat Boyle said...

Most people today do not understand testing. That was also true twenty some years ago when I was teaching Novell.

Unlike Microsoft which viewed education in its technologies as a supplement to its efforts to monopolize the high tech market, Novell looked on Novell Education courses as a profit center. That means that Microsoft just wanted the student to learn Windows (and avoid UNIX). It wasn't particularly greedy about its training materials. If an independent author published a book on Windows, that was fine with Microsoft.

But Novell wanted to make as much money as possible on its training courses. So it was that it would cost a Novell student many thousands of dollars to earn a CNE and most of that money went to Novell. The commercial books and courses that weren't from Novell weren't effective in getting the student certified. Novell worked hard to assure that the only way to a CNE was through purchasing official Novell training materials and courses.

So it might cost a student ten or twenty thousand dollars just to get ready to step into a testing center and face the questions on the computer screen.

But Novell also offered 'adaptive" testing. That meant that if you missed a question in one area the next question would be in that same area. The machine probed for weaknesses.

So after spending years of study and thousands of dollars a student might be presented with only three questions and summarily flunk out.

I always found it difficult to explain to the outraged student why this was so.

Albertosaurus

Anonymous said...

"Just like with IQ tests, they will whine that it has a racially disparate impact and doesn't really measure what's truly important in a prospective employee." - If the tests were administered by some sort of College though, then they could do an end run around disparate impact. then GOTO LINE 1.

Anonymous said...

China to send an unmanned rover to the moon soon

Maybe they will seek out some of the US landing sites and put an end to those conspiracy theories forever.

Anonymous said...


The critical thinker can distinguish facts from opinions, will disregard common fallacies (ad hominem and others), properly apply syllogisms, etc. When she's really cooking, she can enumerate, estimate, accurately assess risk, etc.

I think one can test in a valid fashion for critical thinking skills.


That would appear hard, since so much of what passes for fact these days is fantasy.

See here for an example

The same author points out that most "critical thinking proponents" actually employ all of the logical fallacies with gay abandon.

Unpersoned at iSteve said...

Industry?

beowulf said...

The cost effectiveness of signaling by standardized testing vs. signaling by credentials is so overwhelming, it'd be worth going to racial quotas (to get around Griggs v. Duke Power issue). Limit the quota pool to enrolled members of Indian tribes and descendants of African American slaves (sorry Mr. President).

So 10% (or whatever the quota pool's share of Census is) will be in a group graded on its own curve for highest scoring Indians and blacks, the other 90% or so would be highest scores of everyone else. That's the only way to capture the value of testing without tripping over disparate impact.

Anonymous said...

"A few years ago the SAT added an essay section."

one study found that it correlated the most with grades, another that it didn't have to be factually accurate. Sounds about right.