Monday, August 24, 2015

On Registered Clinical Trials

A new PLoS ONE study looks at the effect of mandatory pre-registration of medical study methods and outcome measures, starting in 2000. Major findings:
  • Studies finding positive effects fell from 57% prior to the registry to just 8% afterward.
  • "...focused on human randomized controlled trials that were funded by the US National Heart, Lung, and Blood Institute (NHLBI) [and so required advanced registration by a 1997 U.S. law]. The authors conclude that registration of trials seemed to be the dominant driver of the drastic change in study results."
  • "Steven Novella of Yale University in New Haven, Connecticut, called the study 'encouraging' but also 'a bit frightening' because it casts doubt on previous positive results...”
  • "Many online observers applauded the evident power of registration and transparency, including Novella, who wrote on his blog that all research involving humans should be registered before any data are collected. However, he says, this means that at least half of older, published clinical trials could be false positives. 'Loose scientific methods are leading to a massive false positive bias in the literature,' he writes."
Reported in Nature here.

Sunday, August 16, 2015

Is Cohabitation Good for You?

Last week, Ars Technica (and I'm sure other news sites) posted an article on a large-scale survey of health outcomes in Britain, under the headline, "Good news for unmarried couples — cohabitation is good for you" (subtitle: "Married partners tend to be healthy, but living with someone works just as well"). Link.

I'm actually hyper-critical about people who sling around the phrase "correlation does not imply causation" too much in improper cases, but here's a golden example where it does apply; the headline "cohabitation is good for you", is totally unwarranted. Now, the findings do say that married & cohabiting people are healthier than people who live alone. But this could be either X causes Y, or Y causes X, or other more complicated interactions. One hypothesis is that "cohabitation is good for you [by improving health]"; another hypothesis is that "being healthy is good for your prospects of getting a partner", i.e., healthy people make for more attractive marriage/cohabitation partners. If you think about it, I'd say that the latter is actually the more common-sense direction of the causation here.

How could the direction of this effect be formally disentangled? Well, you could be on the lookout for a "natural experiment" where someone who did manage to get married/cohabited breaks up or gets divorced, and see if their health degrades during the later period in which they lack a partner. Of course, the researchers here were smart enough to do exactly that, and an entire paragraph of the Ars Technica article is in fact devoted to these findings:

"The study found that changes in status had no obvious impact—the transitions from/to marriage and nonmarital cohabitation did not have a detrimental effect on health. There wasn’t an obvious difference in these biomarkers when participants divorced and then remarried or cohabitated; they looked the same as participants who remained married. For men who divorced in their late 30s and didn’t remarry, the risk of metabolic syndromes in midlife was reduced."

In other words, for anyone in the category of at least being healthy and attractive enough to get married/cohabited once, being married or cohabited made no difference to their health. Which to my eye is overwhelming evidence that the causation is in the other direction, i.e., these headlines of "cohabitation is good for you" are flat-out wrong.

Might be a good example to include in my fall statistics course.

Monday, July 20, 2015

The Difference a Teacher Makes

I had an interesting natural experiment this past spring semester: I was teaching two remedial algebra courses, one in the afternoon, and one in the evening. Same calendar days, same class sizes, identical lectures from day to day, exact same tests, exact same numbers taking the final exam. In one class, only 23% of the registrants passed the final exam, while in the other class 60% passed. (Median scores on the final were 48% in one class and 80% in the other.)

This got me to wondering: How much difference does the teacher make in these classes? And the honest answer is: not very much. To be humble about it, I could do everything humanly possible both inside and outside the classroom as a teacher, work at maximal effort all the time (and that is generally my goal), and have it make very little difference in the overall classroom result. The example here of enormous variation between two sections, identically treated by me as an instructor, really highlights this fact.

On this point, I found a 2013 paper from ETS by Edward H. Haertel -- principally about the unreliability of teacher VAM scores -- that summarizes several studies as finding that the difference in test scores attributable to teacher proficiency is only about 10% (see p. 5). That actually seems about right based on my recent experiences.

Monday, June 8, 2015

Why Technology Won't Fix Schools

Kentaro Toyama is a professor at U. Michigan, a fellow at MIT, and a former researcher for Microsoft. He's just written a book titled "Geek Heresy: Rescuing Social Change from the Cult of Technology" (although I'd quibble with the title in one way: practically all geeks I know consider the following to be obvious and common-sense). He writes:
But no matter how good the design, and despite rigorous tests of impact, I have never seen technology systematically overcome the socio-economic divides that exist in education. Children who are behind need high-quality adult guidance more than anything else. Many people believe that technology “levels the playing field” of learning, but what I’ve discovered is that it does no such thing.

And, oh, how much do I agree with the following!:
... what I’ve arrived at is something I think of as technology’s Law of Amplification: Technology’s primary effect is to amplify human forces. In education, technologies amplify whatever pedagogical capacity is already there.

 More at the Washington Post (link).

Monday, June 1, 2015

Noam Chomsky on Corporate Colleges

Noam Chomsky speaks on issues of non-teaching administrators taking over America's colleges, the use of part-time and non-governing faculty, and related issues:
The university is probably the social institution in our society that comes closest to democratic worker control. Within a department, for example, it’s pretty normal for at least the tenured faculty to be able to determine a substantial amount of what their work is like: what they’re going to teach, when they’re going to teach, what the curriculum will be. And most of the decisions about the actual work that the faculty is doing are pretty much under tenured faculty control.

Now, of course, there is a higher level of administrators that you can’t overrule or control. The faculty can recommend somebody for tenure, let’s say, and be turned down by the deans, or the president, or even the trustees or legislators. It doesn’t happen all that often, but it can happen and it does. And that’s always a part of the background structure, which, although it always existed, was much less of a problem in the days when the administration was drawn from the faculty and in principle recallable.

Under representative systems, you have to have someone doing administrative work, but they should be recallable at some point under the authority of the people they administer. That’s less and less true. There are more and more professional administrators, layer after layer of them, with more and more positions being taken remote from the faculty controls.
 More at

Monday, May 25, 2015

Online Courses Fail at Community Colleges

More evidence for one of the most uniformly-verified findings I've seen in education: online courses strike out for community college students. From a paper by researchers at U.C.-Davis, presented at the American Educational Research Association's conference in April:
“In every subject, students are doing better face-to-face,” said Cassandra Hart, one of the paper’s authors. “Other studies have found the same thing. There’s a strong body of evidence building up that students are not doing quite as well in online courses, at least as the courses are being designed now in the community college sector.”
More at

Monday, May 18, 2015

Teaching Evolution in Kentucky

A professor discusses teaching evolution in Kentucky. "Every time a student stomps out of my auditorium slamming the door on the way, I can’t help but question my abilities." (Link.)

(Thanks to Jonathan Scott Miller for the link.)