Bringing the Evidence-Based Turn Full Circle: Critical Thinking About Disciplinary Practices

The professions are rapidly evolving into an “evidence-based” world. And yet, there is strangely little accountability about disciplinary practices in psychology. There are likely many reasons for this disparity, including power differentials favoring scientists, a rhetoric of objectivity that clouds over the interpretive and practical demands of the discipline, and the positivist assumption that discrete empirical findings somehow magically put themselves together in a coherent fashion.

Although many theoretical psychologists (me included) have bemoaned the evidence-based turn, I believe that the evidence-based turn provides an important opportunity to demand greater accountability of disciplinary practices that are in great need of the diagnostic care and prescription that theoretical psychologists are especially well-equipped to provide. I encourage us to take the evidence-based turn so seriously, as to become evidence-based nuisances in uncovering absurd claims by our colleagues. For example, whenever anyone tells me that one should always first try to use an evidence-based practice in treating any client, even if that client is from a demographic group not represented in clinical trials (and there are many in our discipline who believe this), I find myself saying something like: “That’s an interesting idea. What evidence do you have that this is the best practice?” I’ve never received a satisfactory answer to this question because, as far as I know, there is no evidence to justify it. Such a claim is, rather, an expression of one’s ideological commitment to a treatment philosophy; it may have convincing reasons but it is not obviously evidence-based. Thus, by taking the evidence-based turn very seriously, we can expose problems as being driven by ideology, politics, guilt interests, etc., rather than (merely) by intellectual and empirical merit. If scientists and other academicians are for some reason above the fray of such scrutiny (unlike applied professionals), the question is why.

Here are several kinds of disciplinary practices that I would advocate theoretical psychologists taking a lead on, and doing so through publishing in journals that will have a maximum impact on the field:

1. Research methods training and textbooks. William Hartmann and I did a presentation about this at the last mid-winter meeting. Research methods texts have been updated very little since the 1950s and reflect an uncritical positivist methodology, in spite of all kinds of theoretical and methodological advances. They are in great need of reform, and the time is ripe for this.

2. Grant funding mechanisms. The vast bulk of major research funding in psychology in the U.S. comes from the federal government (primarily NSF, NIH, and Dept. of Defense in psychology). Increasingly, being able to receive such funding is a prerequisite for research faculty positions. The largest factor in getting funded is whether research is aligned with agency priorities. Currently, NIMH has nearly exclusive priorities towards biological and neuroscientific interventions. But what is the evidence that these priorities are the right ones? Furthermore, what about the processes by which these precious grant dollars are examined and awarded? Have they themselves been examined with scrutiny and transparency?

3. Departmental hiring, tenure, and promotion decisions. Probably few things shape the discipline more than these decisions, and yet their practices (from what I hear) are often quite questionable, often coming down to voting and arbitrary metrics. Is there evidence that such procedures are in the best interest of the discipline? How are unfair biases and uninformed whims protected against? Lots of questions here.

4. Journal operations. The peer-reviewed journal system is extolled in the cover story of the virtues of science. But the actual processes of the peer review process are opaque to most. Editors have much leeway in terms of rejecting papers outright, and sending papers to reviewers he or she knows will be sympathetic or antagonistic. What kind of evidence is there that these processes are the best? Again, how are unfair biases protected against?

I’m curious about others thoughts about taking on these disciplinary practices. What are the best strategies for doing so? What other ideas do you have?