Story boarding

When you "storyboard" the way a figure or figures for a scientific manuscript should look, or need to look, to make your point, you are on a very slippery slope.

It sets up a situation where you need the data to come out a particular way to fit the story you want to tell.

This leads to all kinds of bad shenanigans. From outright fakery to re-running experiments until you get it to look the way you want. 

Story boarding is for telling fictional stories. 

Science is for telling non-fiction stories. 

These are created after the fact. After the data are collected. With no need for storyboarding the narrative in advance.

32 responses so far

  • lurker says:

    Whaaaat, DM? Since when did you sound like a righteus noob? Every MPU I put together began with storyboarding. Not just glamour-mags, even society journals. It's the story's novelty, not the quality of data, that means whether a "professional" editor will send your paper out for just review or dismiss it. Storyboarding is the only way to compile MPU to get past this "new normal" of scientific publishing if you're not already a BSD.

  • Kevin. says:

    I agree that you shouldn't put in real/borrowed 'data' as a placeholder, but I am okay with cartooning in what data you need and/or expect to put there.

    Is it "storyboarding" to put down that panel A will be the western blots showing how well the RNAi worked? Or that the mutants are actually mutant?

    When do you start the Punchlist? When you write the grant, after the experiments are in, or somewhere in between?

  • qaz says:

    Storyboarding is the only way I know to communicate. A scientific paper is not a report of an experiment. It is a report of a discovery. The storyboard is how we figure out if we have all the pieces to the discovery, if it makes sense, if we've missed anything.

    There's nothing wrong with saying "if the data come out this way, then we have a complete story, but if it doesn't then we don't". As long as you check yourself. As long as you are OK with the data not coming out that way. As long as you are honest. There is no substitute for honesty. If the data don't come out the way you wanted, then that's interesting. You change your theory not your data. But that doesn't mean you don't have a theory of what you think the data might look like.

    PS. How is storyboarding any different from having a hypothesis?

    It's that Asimov line - "The most exciting phrase in science, the one that heralds new discoveries, is not 'Eureka', but 'That's funny....' " You'll never know that the data don't fit your expectations if you don't have expectations.

  • drugmonkey says:

    I literally cannot believe you people.

  • odyssey says:

    How the hell can you start writing a paper without all the relevant data? You have the data or you don't. There is no "expect".

  • DJMH says:

    I storyboard before I have even half of figure 1. I do it usually with an outline of how I think a paper might shape up, but sometimes with cartoons (hand-drawn). It is also known as "planning your experiments" if that helps.

    Generally, of course, those storyboards are thrown to the wind when experiment 1b shows an unexpected result, and that's just fine. I just make new story boards, or outlines or what have you. The point is to have a plan, not to carry out every experiment from the standpoint of "I wonder what happens if.."

  • neuropop says:

    It's not storyboarding to organize final (collected) data figures into Fig. 1, Fig. 2 etc. before writing the manuscript, is it?

  • Pinko Punko says:

    DM- define your terms very explicitly, or else you just have a train wreck of a discussion.

  • Kevin. says:

    Maybe DM is just in a field where he cannot possibly change his experimental plan of the study once it is started because that's P hacking, double-blinded, or whatever. Thankfully, I can do a new experiment every day based on yesterday's results.

  • Newbie PI says:

    I disagree with DM because storyboarding really helps me to plan experiments and to assign specific tasks to my technician versus grad students versus undergrads. If the experiments don't work or we get unexpected results, we adjust the story.

    On the other hand, my postdoc PI would send me a full list of seven figures with panels A-Z every month or so based on a random idea he had. It was all purely hypothetical and was such a waste of his time (but apparently it was fun for him). I mostly ignored his ideas, but I know at least one of his grad students would follow these outlines like they were the gospel truth and do exactly what DM was talking about -- repeat experiments until getting the result the PI predicted. So yeah, storyboarding can be bad in the wrong hands.

  • Dave says:

    ...re-running experiments until you get it to look the way you want

    I always run an experiment once and call it good.

  • SidVic says:

    'storyboardin'? What the hell.

  • becca says:

    DM- lots of people learned to storyboard as *the way* to draft a manuscript. It's like giving a speech from an outline. The planning is a basic part of the creative intellectual process. You may not use the particular approach, but that doesn't mean it's useless.

    I think the real danger is exactly what Newbie PI suggests- when trainees try to get the data PIs want, it can quickly morph from "doing the experiment the way they had in mind, with the appropriate controls" to "seeking proof for the PI's pet hypothesis". That is a huge problem regardless of whether a storyboarding process is the plan, or a grant application, or an email of "hey can you run this experiment for me next week?". Trainees learn to read their PIs quickly, as a survival mechanism, and it is perilous to assume their desire to tell the PI what she wants to hear isn't the driver of much self-delusion and outright fraud. The problem isn't the methodology people use to plan, the problem is the incentives they have to follow the plan even when it proves unwise.

  • drugmonkey says:

    I know at least one of his grad students would follow these outlines like they were the gospel truth and do exactly what DM was talking about -- repeat experiments until getting the result the PI predicted.

    That is a huge problem regardless of whether a storyboarding process is the plan, or a grant application, or an email of "hey can you run this experiment for me next week?".

    Laying out the panels you "need" with so called placeholders is a far frickin cry from the narrative in the research plan of a grant application or the iterative process of chatting about next directions with the staff who are on a particular project. It sets an expectation of how the experiments are supposed to come out or need to come out so that you can get that paper into the right journal with its nice and neat little "complete story" bullshit narrative.

    It is no fucking coincidence that these supposed "placeholder" panels are so commonly the meat of retraction excuses. It encourages cheating to lay out your storyboarded expectations/demands for how the science has to come out.

    And faking is just the barest tip of the iceberg. This influences practices of running the experiment over and over until it "works" the way it is needed to work. And that sort of thing can never really be detected very easily. And you can draw a straight line toward aspects of the supposed replication crisis.

  • Jonathan Badger says:

    It seems really weird to think about storyboarding a manuscript in some way could influence bias in the data and analysis. What is DM implying? That he writes the paper *before* the data is collected and analyzed? Who does that?

  • Established PI says:

    I don't think my research is remotely related to DM's and I also find the "storyboard" concept foreign and troubling. We never even start writing a paper until we can put together an abstract that describes the general question, what we did to address it, what our findings were and how that changes the way we think about the problem (be it a great leap or incremental progress). Placeholders are for red0s of existing experiments that are rather ugly (splotches, gel smiling, too many time points) but whose result is clear - never for an experiment or control whose result we think we can predict. Perhaps someone can provide a counterexample from their field, but planning out a "story" before most of the experiments have been done seems to be a recipe for confirmation bias.

  • Namesaste_Ish says:

    Storyboarding in my lab:

    Drug X is super interesting because it can stop stupidity but has the unwanted side effect of making folks shitte like crazy. Future non-misogynistic Nobel winner in my lab has solid data that after a few doses of Drug X, our famous new transgenic rats won’t shitte their brains out.
    What's the purpose of this paper? To show if Drug X make our Shitteless Rats lacking gut 5HT receptors shitte their brains out.
    Like everyone important, we have labeled the entire gut in shitless rats ontogenetically Flaming Hot Chocolate Brown Fluorescence.

    Storyboard
    Working Title:
    Authors and people who don't deserve authorship but we are a circle jerk factory so more authors

    Figure 1 Pictures of our cute new shitteless rats and their shitty normal rat buddies*
    Figure 2 D/R curve to induce a one-time shittestorm (this data must be there or we wouldn't have started studying this, so no placeholders) in normal shitty rats
    Figure 3 Amount of shit vs. time
    Figure 3 Weight of animals vs. time
    Figure 3b Pictures of animals +/- shittestorm
    Figure 4 Optogentic analysis of gut pathology in shitteless and shitty rats
    Figure 5 Shitty rats vs. non-shitty rats given a 5HT reuptake inhibitor then Drug X. Shit v time.

    This is my storyboard. Do I know what's happening when they get some reuptake inhibitor for Figure 5? Nope. And the paper will need 4 more figures, but this is a standard blueprint.
    When figure 5 data comes thru and I don't understand it, that's the most interesting part.
    I don't know how it will turn out. Nobel winner-to-be and I look at the data together. We talk about why, because I don't know the fuckken answers, then we work more, we both read more and talk more.
    It aunt a fuckken check list, but some of your paper is standard in every fuckken field, Ted.

    * These are only male rats because we said we'd do both sexes but got our budget cut, so see you later females.

    Jesus, I like this blog better when you write about risotto. Where are the fuckken risotto dishes?

  • duke of neural says:

    This seems like a pointlessly broad rule that will vary by research field and researcher.

    SOME researchers may, in some situations cling to that storyboard idea, possibly to the point of faking necessary data, I suppose.

    For me personally, when I don't plan out figures in advance, I find it easier to convince myself to go off in tangents that ultimately don't have any place in a coherent paper. They're always neat experiments to me, but as DM said, if they don't get published, they're not really worth anything.

    When I have a list of what experiments I'd need to make a publishable story, it's easier to keep myself on track doing relevant experiments that might already be boring to me.

    Perhaps my field just offers me more creative direction. Perhaps other fields, papers are more rigidly formatted, "Figure three is where the patch clamp experiments go, figure four is where the western blot goes." And I am new at this, maybe in a few years it will be obvious to me what experiments I'd need to do next.

  • A Salty Scientist says:

    If your storyboard is a choose-your-own-adenture, fine (i.e. Hypothesis is A, Experiment to test is B, If results are C continue on to D, If results are E continue on to F...). God help you if your storyboard is linear.

  • lurker says:

    I literally cannot believe you people.

    Just trying to keep your blogge honest. Lots of legit scientists do story-boarding. Story-boarding is a legit way to do science. The practice does not automatically lead to dishonesty. It's not like other types of " "-boarding...

  • Dave says:

    It is no fucking coincidence that these supposed "placeholder" panels are so commonly the meat of retraction excuses

    What are you on about here? What evidence do you have that retractions are 'commonly' the result of 'placeholder panels'?

  • jipkin says:

    so what's your stance on after the data are collected but before all possible analyses have been performed with said data?

    I've been finding it quite useful to play around with storyboards and figure layouts as they show where the empty holes are and lead me to do some additional analyses here and there. I don't literally put fake pictures in the empty spots... they're empty for a reason. But the process (again, largely post-data collection) is useful for uncovering some of those spots in the first place.

  • drugmonkey says:

    Dave- try reading more closely.

    jipkin- my issue of the moment involves mostly the fantasy of what it "should" look like prior to data collection. Especially with wrong-data placeholders. Figuring out the story once the data are available is very different.

  • Pinko Punko says:

    I would say that the common strategy that I am familiar with is as follows: lots of experiments in the discovery phase. Then, imagine the narrative for the paper with the existing data. Does this reveal holes in the argument or name obvious where additional approaches might increase how compelling the story is? Once those are identified then those are what you work on to get to finality in the manuscript. None of this precludes the narrative changing in new data force a reexamination. The "outline" or "storyboard" enables a different view of the data in sum as opposed to isolated parts.

  • qaz says:

    I've got a simpler rule. Don't fake your data.

    Or this one: Let the science lead you.

    The problem is that too many scientists were never taught how to write. You need to get to the point where it's OK to throw out your storyboard and write a new one if your data change. This is something any true storyboard team in movies would do. You should see how movies change from the initial storyboard to the final product. Storyboarding is a process of figuring out how your narrative goes.

    Come on, don't you find that while writing a paper, you discover you are missing a figure? Say a new control that you didn't know you needed until you were writing the paper? And you don't have any expectations of what that figure will look like? Really? I have trouble believing even the great DM is that zen.

  • Mikka says:

    Get off your high horse, DM. Don't tell me you don't plan your experiments ahead, to organize your thoughts about the possible directions and account for all the necessary controls. Or even to just set an minimum publishable end point in advance. Is that not "storyboarding", even if you don't do figure mockups?

    If I just piled the data on until I had a story to tell with it it would take me decades to publish, and it would be horribly inefficient.

  • Paul Brookes says:

    In this and the previous post on "place-holder" figures, DM might be referring to this interesting retraction from J. Neurochem...

    http://www.ncbi.nlm.nih.gov/pubmed/26059260

    The excuse essentially boils down to "we used placeholder figures, then an earthquake ate my homework, and we accidentally submitted the paper with the made up data still in there".

  • drugmonkey says:

    That is a very frequent variety of excuse.

  • jmz4 says:

    I do most of my "storyboarding" when I'm putting together posters and presentations. Nothing like having to get up in front of actual people and defend your work to make sure that it has a coherent and sound narrative. Although, I err on the side of presenting preliminary (clearly labeled as such) and unpublished work, which I gather is increasingly unusual. If the preliminary stuff doesn't replicate, it sadly goes in the trash. And yeah, I'll admit to trying to replicate and exciting positive result or crucial stepping stone more often than I really should (all along inventing quasi-legit excuses about expiration dates and buffers). If my PI really likes a result that turns out to be fake, it will die a humiliatingly drawn out death by overwhelming negative replication and additional controls (eventually).

    As far as top down expectations from PIs influencing the science, yeah, it happens a lot IME. PI's should probably avoid storyboarding altogether, and leave it up to the trainees. There's too much pressure to get the data "right' if the PI is the one directing the narrative. Let the PI be the big-picture gal, fitting it with existing work, explaining discrepancies, and figuring out what it all means in the broader context of biology.

    Overall, I dislike the dissembling that most paper narratives enforce on the science, and would like to see an alternative to the traditional paper arise.

  • imager says:

    Storyboarding is one good way to tell the postdoc or student what experiments (s)he need still to run to get the paper done and not to divert at all different moments into other interesting avenues. Storyboarding helps us to keep on track and come uo with the experiments needed to make the story complete - not to make data up we need. If the data are different then the story and the board will look different down the line.

  • Ola says:

    If you're gonna do this shittio, then do it on a board, preferably a dry-erase board. If you use a word document or powerpoint file that will one day become the manuscript itself, it's not story-boarding.

  • […] been reading about millennial science manuscript […]

Leave a Reply