The problem with confirming a scientific finding

I was at lunch in Washington yesterday and my guest (from the policy world) brought up a very interesting point: namely the problem inherent in the request from a policy-maker to an analyst (or a senior professor to a graduate student) to confirm a finding by conducting some other observation.

Of course, from the scientific standpoint, we really should be asking the graduate student to attempt to disprove the finding, right? That is they should be attempting to experimentally falsify the hypothesis/theory/conventional wisdom.

Now think about the last figure in so very many papers–the cartoon diagram for how it is speculated that “everything works” in vivo. The picture I call the Just So Story after Kipling. How many times are we in danger of asking our trainees to confirm the Just So Story instead of falsifying it? How are our rewards structures set up in science vis a vis this problem?

Jim