Reading Graphs – An Example of a Decoding Interview
(transcribed by Grace Reyes, Carthage College)
June 8th, 2017
DP: David Pace
KD: Kari Duffy
LC: Leslie Cameron
PR: Peter Riegler
DP: -what it is.
LC: But, I’m not going to be the only one interviewed, aren’t you going to be part of the, yeah-
PR: Yeah, right that is what I am suggesting because I wouldn’t feel comfortable to sit on that side, so to say, because I feel, um, that is important for my teaching as well.
DP: Um, I don’t think I have, well here’s something *clears throat*…
LC: Kari has one too?
KD: Well just run video without, with it at the ceiling. I don’t have a recorder per say, but you can run video without a picture. Oh, there you go.
DP: Well let’s just try that for a couple of seconds.
PR: Well I would suggest that for several reasons, one of them I indicated during lunch, that after fruitful interview you feel energized and “oh I’ve understood that!” and two days later it evaporated. So you can look at that, and the other one is in your step number seven, I suppose, in the decoding cycle that it could come out something that is worth to be shared with other people. And we should have a #lay down#.
DP: So, let’s, if you could record it-
PR: Yeah, so, I would, sorry-
PR: So let’s start the recording, at least the level seems to be fine.
DP: Okay, do you think we’ve got, do you want to test or are you pretty confident about it?
LC: I’m going to close the door.
PR: Well the graphical feedback seems good.
DP: Great, great.
KD: So are we both asking you two questions?
DP: And the central (essential) question we come back to-
PR: I forgot to tell, so your task might be to drag him out, if he gets you know, gets on our, gets on our side so to say.
KD: Okay, okay.
DP: If I go unconscious.
LC: That’s right, we’ll all be on the unconscious side.
2:02: Start of Interview
KD: So both of your bottlenecks are data interpretation?
LC: We have the same, we’re sharing a bottleneck.
KD: You’re having a hard time-
DP: And your last name again?
DP: Cameron, that’s right.
KD: Having students, teaching students how to interpret the data.
PR: Well I have to say, I’ve never taught them!
LC: That’s exactly what I was gonna say, that’s exactly what I was gonna say, right.
DP: Alright, so, why don’t you start with a statement of your, of your notion of what students can’t do. Where, where do you encounter this in a course? Ah, what’s a good example for each of you where students get stuck?
LC: So I’ll, I’ll give an example from when I very first got started with this which was having students do online laboratories, where outside of class, they were ah, doing a cognitive task on a computer that collected data, um, and the online program then provided them with a graphical representation of the data from all the students in the class, and their assignment, um, was to both write about the data and speak in class, present to their peers, an analysis of the the data. Only a graph, not statistical, just “here’s a graph and here’s a representation of the data we collected”. And um, and it seemed to me that students had a hard time describing those data to their peers, to me and to their peers, and also had a difficult time writing about it. So it wasn’t um, it just wasn’t clear what they were, and it seemed to me that they often missed steps, so they, there would be simple things they didn’t do to help orient people to the graph and to the axes and to what they were actually showing. So they they acted sometimes as though it was sort of self-evident and they would blow right through, um, without giving a clear description. So that was, so for me that was the starting point.
DP: Okay, good. What about you, what’s a good example, Peter?
PR: Um, although, being a trained physicist, I don’t, in my classes I don’t, I’m not that much into contact with data but I’m thinking of my Numerics class where I typically ask students to generate data for for given purpose, so generate data by themselves by some computing task, and then there should be a step of interpretation and, and getting at some conclusions, typically at- Yeah the the the intention is to help them build a model how computations are done in a computer and experience that by themselves.
PR: So doing some micro-experimentation.
DP: Okay, and what do you see instead of what you want.
PR: Well what I’m seeing is that students tend to interpret these tasks as it’s their task to generate the data and report the data, period. But not interpreting the data, not trying to make sense of that. Not trying, yeah, to think about that. It’s that like, prior to generating the data or even afterwords what is what I would expect in that situation? Either beforehand or afterwards, is that plausible? Is that something I would expect, is that strange?
PR: And that’s actually what I want to do in these exercises, because at times the computer doesn’t work in an intuitive way and I want them to generate data where they get astonished and basically asking “What’s going on here? Why is that?”.
DP: I think I mentioned to you last year that wonderful case, that on the college board exams in the United States, once there was a question, that um, there was the army was moving a series of soldiers, and there were so many buses and so many troops, and the simple question: “How many buses were needed?” And they, you know, had several options, and a third of the students put down 12.5 buses. Yeah. There’s not a half bus.
DP: But they simply divided and didn’t- so the significance of the data was completely irrelevant, they didn’t consider that as an issue at all. Well this is going to be interesting, it’s something we haven’t done before where we have two people in different fields with somewhat similar things so I think what we’ll be finding both where are the similarities and the differences. So why don’t we start off, when you hear each other’s bottlenecks, where is there overlap and where is there difference, do you think right now at this stage?
LC: And I heard, I actually heard three things in there, one had to do with an expectation of what the data will look like, in other words a hypothesis, a prediction of what-what might happen. There was clarity in describing the data, and then there was interpretation of the data. And I think I was starting with just “Could we get a description of the data?”. A clear description of the data. Um, and before, maybe even before the hypothesis and the interpretation.
LC: But it seems like, as I was discovering in the previous workshop, any one of these problems seems to have many sub-problems.
LC: You know? So I think for me that’s where, I would be, and I’m working, I don’t know about you Peter, but I’m working with undergraduates, maybe second year undergraduates. So, so the first step seemed to just get to that-
LC: -simple description, when you look at a graph can you describe what’s on it without missing steps? Without saying something that I would understand but that your peer might not.
LC: Not using jargon.
DP: So is that okay, we’ll start with that?
PR: That’s fine with me, um, now I would like to ask a question, or maybe two. Um, what is your metrics to clearly define, to clearly describe-decribe, describe data? And the other question that came to my mind is: do you think you can do that in a context-free situation? Like have a histogram here and I don’t know what it is about?
LC: Mhm, right.
PR: That simply says “frequency over x”?
LC: Right I’ve never tried to do that, I’ve only ever tried to do that in the context.
PR: Okay, uh huh.
LC: I think that, that seems like sort of the abstract versus the concrete, so in statistics you know, can you, that sounds to me like a question of “do you understand what an interaction between variables are? Do you understand the abstract if the variables were A and B, or do you need a concrete situation?”
LC: Again I would, I’d have to think about this but I would, I ah, it seems to me that it would be easier to start with the concrete. I never try and describe the abstract.
PR: Hmm, okay.
LC: I never try and work with the abstract.
DP: So let’s, let’s take *clears throat*, imagine that you’re, that our goal is to find out what your practice is and barge in any time you want.
DP: Especially if I seem to not, to be missing something. But we need to get at your practice. So you’re looking at, and both of you are looking at, at data. In, in a graph? Should we figure that’s the typical way to- So you’re looking at a draft-graph, and you’re going to explain it to somebody else. What the, what the significance is. What’s shown on the graph.
DP: What do you do?
LC: And I just say, not necessarily the significance-
DP: Yes, okay.
LC: But just, what’s there?
DP: Mhh, okay.
LC: So let’s start with what’s, what are the axes? What’s, are there labels on the axes that you know, need some descriptions, so understanding. Um, and ours it’s often the y-axis which is our dependent variable, what we’ve measured, right? Is that, it’s often a measure of accuracy or reaction time. So, a reminder to the audience that this is what’s actually on that axis. And then what was manipulated in this experiment? So you need to have some understanding of the methodology, um, to be able to- So how did you generate those data? What was manipulated in order to generate reaction time under different conditions?
KD: Do you think it needs to be even more basic than that? In second year are you making assumptions that they even understand the axes? Or how to set up?
LC: I’m making an assumption that they know it, yup-
KD: And is that a reasonable assumption?
LC: No. Probably not. I mean we spend a lot of time talking about what variables are. And trying to make sure that they understand the difference between a dependent and an independent variable, what was manipulated, what was measured.
KD: And do you have proof of their understanding before you move on to the next step?
LC: It’s an evolution, so you’re not, certainly not everyone is going to get it every every time, but it’s, this is a 14 week class and they’re going to encounter it week after week, and we’ll get students at the end who will still reverse them.
DP: So you’re looking at the graph, and there’s the x, you look at the x, I don’t know is the one you look at first?
LC: Exactly. Probably the y axis, I think maybe the y axis.
DP: Okay, so you start with the y axis. And what goes through your mind as you look at the y axis?
LC: Um, well it’s a measure of, it is what was measured in the experiment, um so I guess I would look at it and, so what is it and how would that have been generated?
DP: Okay. What tells you the answer to those questions, where do you go to find the answer?
LC: The method, what was, right? So it would have to step back to what did the participant actually do in this experiment in order to generate those data?
DP: Okay, alright. Explain to me more about how you do that. Because I’ve never taken a psychology course so I’m a great person to interview.
LC: Ah- So, and I can use examples? Right?
LC: So it’s, it let’s say is a reaction time, it’s reaction time on the y axis, that means: how much time did it take for someone to generate a response, usually a button press, um, after a stimulus was presented um and they had to make a judgement about that stimulus.
DP: Okay. Alright.
LC: That’s a lot of jargon!
DP: Yeah, a lot of words there.
DP: So there’s a story behind the y axis.
DP: In a certain sense. But you have to understand the story-
LC: -That’s right.
DP: -The history of the, the that generated-
DP: -that y axis.
LC: And, and to, and I – mhh – this might be what I am not supposed to be doing in this interview, but I often tell the students that the exciting thing about psychology and why psychology is better than physics –
LC: Is because… My husband’s a physicist, um, is that, is that those variables are complicated. And that whatever you see on your graph is a construction from, it’s an experiment that you designed, right? You thought up in your head, “Let’s manipulate this and measure that.” And then we’re going to try and make an interpretation about human behavior based on that. But it’s, it’s completely dependent on how you decided on as you say, and I like that, the story. Right? The story, the story which is the method.
LC: Which I think in the natural sciences is often um more of a protocol, where for psychologists or maybe it’s something that’s presented-
PR: Yeah, yeah I see your point. More often-
LC: Right? Whereas where we’re having to generate these methods, um, yeah, create this way of, it’s not always obvious what to measure and what to manipulate.
DP: That’s a really interesting thing and I wonder if the students, I mean if I understand you correctly, when you look at the y axis you’re seeing an event.
DP: Um, you’re seeing residue of an event.
DP: And I wonder if the students do that or not? Or they just see a number.
PR: And currently I wonder whether I, as a physicist, would look at the data in the same way, psychological data.
LC: Right, right.
PR: In the same way that you do. I mean as an expert in my field.
LC: Mhm. Right.
PR: So, you’re pointing out something where, where differentiating to the natural sciences, and yes, you might be right that there is more into, different way, into looking at the data than I would do that.
DP: Wouldn’t the way in-
PR: If it’s a protocol thing-
PR: In physics, yeah.
DP: One of the places where students like often get confused is in transitions from abstract to concrete-
DP: Back and forth.
DP: And it seems like you move from something very abstract which is the y axis to something more concrete, which was a series of measurements were made under these circumstances with human beings of a certain population, a whole set of things-
DP: Which you could see in a table and it’s all there. That’s been turned into this, you know, a point on this line.
DP: Um, and that may be problematic for them, they may not think that that’s something very natural, something in your position, but not necessarily natural built into the human psyche, ultimately for everybody.
DP: It took humans a long time to figure that out-
DP: I mean it was what, I mean the 17th century before people were really graphing things, I think. Uh, so all those years before nobody did it.
LC: Mhm, mhm.
DP: So your students are kind of-
DP: Pre-Descartes or something like that.
LC: Mhm, mhm.
DP: Um, so the question is, how do you move back and forth from the events which, which led to a number, and the graph itself?
LC: Mhm. *Pause* Um, I – I think it is, I mean very, I think it doesn’t happen in one step-
LC: I bet it is, I’m thinking about trying to read an article, um, you know, something challenging in my field, right? Something that’s cutting edge that I’m reading for the first time, um, and there’s a lot of back and forth between looking at the graph and looking at the methodology. And going back and forth. Do I, wait, what did they, do I understand because it’s not usually- And it may be more the independent, what was manipulated may be more complicated because the, there are relatively few dependent variables that we deal with. Um, but I think I do a lot of going back and forth, and, and again I use this as a, in contrast with the natural sciences-
LC: Correctly or incorrectly I’ll say um, it’s not obvious what that thing, and we had to do some, make some assumptions in order to um, to decide to use, to measure that variable or to manipulate that variable. And so when I’m trying to understand what somebody else has done, then I- I think there must be a lot of iterative back and forth: what was the methodology? Now can I, can I get to the – and as I say this I think that’s not something that I make transparent to students, probably.
DP: So when you get back behind the graph, itself you’ve got I’m assuming, um, a number which emerges from some mechanical device-
DP: And there’s that thing which gives you a number – 16.
DP: And so you know, 16’s here, so the student has to grasp that-
DP: But they also have to grasp the context in which that, that measurement was made. It’s a much more, as usual we’ll have this, I’m realizing how hard this is when you really start thinking about it.
LC: Yeah, exactly! No, and when you say “16” I think well that’s funny because it would never be 16.
LC: Sixteen is too fast, right?
DP: Yeah, I know, I know but I don’t measure anything-
LC: No, I know, that’s a good point because I look at graph and if I saw, this is one of the problems that students have if they generate their own data and they can generate a completely-ah, I’m sure you deal with this in physics, right? They give you a number and they don’t put the, the units on, this is something that drives my husband crazy.
LC: Right, so they don’t put the units on, but they give you 16, 16 what? Sixteen bananas, 16 milliseconds, there’s no way that somebody had a reaction time of 16 – right? So-
DP: Right, right, or 12.5 buses?
LC: Is this even… Or 12.5 buses. Is this even in the realm of possibility for the reaction time of the eye or the finger or you know the, whatever it is that’s generating that.
DP: So that’s the secondary thing that you’re also doing-
LC: -It is, in fact-
DP: And you’re doing something else which we haven’t fully defined, but you’re also running a program in your brain that tells you whether or not this is a plausible-
DP: -Ah, answer. Or whether something’s been screwed up.
LC: Like the expectation, right?
LC: Not exactly the hypothesis, but is this even in the realm of –
PR: Yeah, I wouldn’t expect whatever your typical reaction times are-
PR: If it’s 16 hours I would say, “well that’s-
KD: Because that-
LC: Something weird, something weird’s happened-
PR: So I don’t have expectations.
DP: Weren’t you the one that, wasn’t it last summer that you had some student that they came up with um, how, how fast something was and the car was driving 24,000 miles an hour-
DP: And now we’re-
PR: Yeah, yeah and maybe I, well I can’t remember that but that’s quite typical.
PR: That’s quite typical, quite, quite typical.
DP: Well that’s something we’ve got to put aside for right now, but it’s out there-
LC: Right, right-
DP: Part of what you do is run a program about the, about the range of an, of answers that would be credible. So you get some of them out of the way quickly.
DP: But, let’s go back, you’ve got a number and you’ve got a machine that produced the number-
DP: But that’s nothing-
DP: That’s the, absolutely useless for your purposes, in itself. So what do you add to that?
LC: Well, just a one number, well, well what I thought you were going to ask, I thought you were asking ah, was ah, that number is always some ah, average, some amalgamation of-
LC: -Of a whole bunch of, right? The students sat and did a whole, they sat for 15 minutes in front of their computers pressing buttons and they all did it and now we’ve got one number.
DP: Yeah, okay.
LC: So that’s also something-
DP: That’s also a piece of it, yes.
LC: Right, right. But, but your question – yeah, sorry-
PR: #unintelligible# the question is, is that clear to students? I, I recently um, I, in, in grading the theses I had the question looking at some data, is that a single event or is that an average?
PR: And the, the figure caption didn’t tell me and the figure didn’t tell me, so I realize now that’s a question now that I immediately ask at that point. So, is it single event or is it, is it average data? Um, you just said that, um, typically for you it’s average data-
PR: I wonder whether this is clear to students.
LC: Right, I wonder too.
DP: Yeah, there’s a big piece. So-
LC: And this is one of the difficulties of going, I realize now, and something that may be helpful, going from that computer, so when I use these online laboratories the computer does all that computation. It takes all of those single numbers, ah, single events, averages them for an individual, averages them across the whole class, and then spits out, you know, a bar graph let’s say of those averages. That is a lot of computation that happens that is completely ah, unavailable to them, right? Hidden from them.
LC: And maybe part of this is working with the, generating, maybe generating graphs is something that would help in the int- in the understanding and being able to describe the data.
DP: Maybe. Maybe. So, part of the, so you’ve got the number there and then you recognize that that number is a composite of the different numbers that had a range. There’s something in that, uh, probably. You never know how deep to go, but it strikes me that there’s something in that that the student could get messed up. Are the students expecting a right answer?
DP: And getting a range of answers? I mean you accept that as perfectly normal, you know? But I can imagine that the student that thought, well if you did the experiment right you’d always get the right answer, you get the same answer.
DP: You know what I mean?
LC: The idea of variability and-
PR: Uncertainty I would add, not something which I have experienced, now consciously time and again that, the idea that there’s uncertainty attached to the data is unheard of.
LC: For your field. For mine that’s what we live in, is that, if I understand you.
PR: I, I live in, unheard of for the students so I-
LC: Oh, I, I see, I see-
PR: So, for them it’s, it’s hard to even think that there is this uncertainty in the data.
LC: Yeah, right, right.
PR: So like, if you measure 7.5 seconds for whatever, so it’s 7.5 seconds and not something within a measurement uncertainty.
LC: Right, right, right.
PR: -and affected by other side effects.
DP: Yeah, so there’s a whole problem there with understanding the nature of data as a statistical pattern rather than a mechanical one.
DP: Um, that’s going to be a big thing for some students I suspect. And this is digression here, which I probably shouldn’t make but I’ve thought about it in terms of the misunderstandings of science and history and a bunch of other things, right now the one word I’d like to destroy is discovery.
DP: Because of the way discovery is used, and ah, I remember watching a show about fossils, and this, and this guy is chipping away these #unintelligible# and he turns back and you see him get the fossil, and then bam! They know, oh! A new species, etc. And I’ve read stuff about, you know, some people think that this is one organism, some people think it’s three, you know, I mean some people just, that you work on it for the longest time before you move from the discovery to, to something that’s a plausible hypothesis that you can defend.
DP: And that concept I think is, the students are probably imagining you just discovered what the reaction time is, period. It’s not like you have to multiply, do multiple testings to get a reasonable answer, that’s a, that’s a different universe than one of discovery where you just find it. There it is.
LC: Uh huh.
DP: And there’s an element of construction, ah, present in the data-
LC: So that, that’s interesting to me because I think the most interesting conversations, if I understand the point you’re making, making the most in conversations we have in class is dealing with the unexpected, right, so they’re, so I think they do have some sense of what’s expected in part because of the computer program actually will actually tell them, so in this particular example that we’re talking about they would have been told what to expect. Um, and then as a, if our class average doesn’t look like the expectation, and in fact, it shows on the graph as a matter of fact, um, like, the results of thousands of people look like *this* and then you ask “Does the class data look like that?” And it’s, you know, readily apparent. Um, so part of the conversations then become, and the thing I think is most exciting about being an experimental psychologist is, if you didn’t get that, right, that um, what was expected, then why didn’t you? What are the explanations, how, you know, what might have happened in this experiment-
LC: What would you need to do differently in order to… Both explain what you got and sort of, get, get to the expected result.
DP: As opposed to a student who might think, “Oh, this must be wrong”?
DP: Because it doesn’t match the expectations.
LC: Exactly. Exactly.
DP: Um, *laughs* #indistinct#, but it, on the other hand, going back to your, it’s never 12.5 buses.
LC: Right, right.
DP: But it might be some kind of buses that, number of buses that would surprise you.
LC: Well, and figuring and understanding because actually now I’m thinking, they actually get their own, they get their own output of their own results, in this particular example, um and so, ah, individual subjects are, oh the data are usually quite messy, right? From an individual you’re not going to get anything-
LC: Um, but inspecting those that look quite variable, um, oh I lost my train. Um. You were, you were saying can you just repeat what you.. No, maybe not?
KD: Messy data?
DP: Yeah, yeah.
LC: Yeah, oh, oh, oh! There, I got it! So, whether, if they get something that’s sort of, again, I guess this is related to something we already said, within the realm of possibility, or did they lose internet connection and they got garbage data?
LC: Right? You know what I mean? There’s like, there’s like, it’s possible you could have done this, it’s a bit unexpected-
LC: But it’s not a mechanical problem, it’s not a…
DP: Okay, so one of the things you do then at a certain point looking at the data, you have two forking paths: one is there’s something faulty in the recording process-
DP: Somewhere along the way-
LC: Sixteen can’t be right.
DP: Yeah. On the other hand, you’ve got the possibility of something unexpected and you, and those are two paths-
LC: Yes, that’s right.
DP: -in your mind, probably not in the student’s mind.
DP: This is good, I think we’re getting somewhere.
LC: Yeah no, it’s-
DP: Um, I’m just imagining another, um, piece that you’re probably doing you tell me, but I’m not-
DP: I’m not an experimental psychologist so, but you must have a notion of population and the relationship of different populations from results. That, that uh, if you ask, you know, four-year-old African Americans and you look at them and their reaction time might not be the same as some other group.
DP: So, there must be some ways in which you are relating what’s being done to the notion of the specific population that you’re investigating.
LC: So the um, for us, so much of experimental psychology um, for better or for worse is done with college sophomores.
DP: Yeah, right.
LC: Aha, these guys are college sophomores so they kind of are, but those are often the critical thinking questions that we would ask, you know. Well, what if? You know, would you see this earlier in development or would you see this in anther cultural context? I tend to want to make the argument that we’re more alike than we are different as I was saying earlier –
DP: But you still must have in your mind the notion that the population-
LC: the Population – exactly, absolutely.
DP: That the population is a variable that you need to be considering.
LC: Yeah, exactly, exactly. Absolutely.
DP: If you over-generalize outside of that population, you’ve left science in some way and done something else, um… So that’s another piece.
DP: That seems to be, I think the students would have trouble fully doing what you want them to do with the graphs if they don’t have something like that.
LC: Mhm. Although I think that they might be more likely than I am to say, “but this is just us.”
DP: Okay. Okay.
LC: Yeah, in fact, I think they might be more, um, I might be, partly because I study perception, which I really fundamentally think that people are more alike than they are different-
LC: I think the more complex the human behavior, the more likely culture matters, um, anyway..
DP: And something that’s really important in this process is that you come up with certain mental operations that aren’t problematic to the students. So, you know, it’s not that every one of these has to be taught, you know maybe that’s something that they have no trouble with at all.
DP: You know, but something else they do.
DP: So, testing some of this stuff is useful to find out where, where the problem comes from. So what else do you do? You’ve got the, you get the number, they get 42, the answer-
LC: Forty-two might be right-
LC: Should be, right? *Laughs*
DP: So that’s the answer-
DP: Everything except nothing. Um, so you’ve got your number, what’s a number we’ll accept? You said the number I picked was 16 but-
LC: Let’s say 500.
DP: Five hundred, okay, so you’ve got 500, you look at it, and, and so now you’re thinking about the process that produced that number.
DP: What questions do you, are there questions that you ask yourself at that point about that process and its’ relationship to the number?
LC: I don’t think so until I’ve looked at the other axis.
LC: Right, because it’s always about that number relative to another number.
LC: That’s just generated under a different condition. So now I’m going to have to go back to the method. So what did, what was manipulated? And look for that on the X-axis.
DP: Okay, anything else with the y before we move on-
LC: I don’t think so. The y is a simple one.
LC: I think. Even though it took us a long time to talk it through. But-
DP: Okay, so now we’re going to the x, and you, so one of the steps that you do is that you would, I take it from what you said, typically read y and then x.
LC: I’d have to think about that, at the moment I seem to think that, that that’s what I do.
PR: I would agree with that. At the moment you said it, I for the split of a second I doubted it, then I realized yes, I do that myself-
DP: That raises an interesting question and I’ve never thought about this, where there may be things that are a little bit better but not, not a, um, game breaker, where there may be paths that are a little simpler that you want students to do but they can get there another way-
DP: Other times if they don’t follow the path you’re doing, they’re out in the wilderness and they never get there. So-
DP: So it’s possible that the x and y axis, it’s just a little better to do the y, but you get there from the x. Or maybe that there’s something essential getting y straight first?
PR: Because that’s the dependent variable.
DP: Yeah, yeah.
PR: That’s the one you’re interested in.
LC: Yeah, yeah.
PR: That’s the reason.
LC: And it varies as a function of the other, not the other way around. Which they do confuse, sometimes, I think.
DP: Okay, so, so doing y first then you think is probably a rule that, that they should be following?
DP: Okay, good.
LC: At the moment.
DP: Great. Okay so now we’ve gotten to x. We’ve made some progress here!
DP: So you look at the x value, you’ve done all the things that we’ve mentioned and you’ve, and all this fairly sophisticated ways of thinking about the number and the event and the statistical meaning of that in terms of variation, in terms of the population-
DP: I’m summarizing to get it straight in my mind, but those are all things you did with the y, right?
LC: Yep, yep.
DP: And each one can be problematic to the students?
DP: X, now we’re to x.
DP: So you’ve got that, the meaning of y in your mind, so what are you gonna do with x?
LC: So, I think I do, then I, then I have to know what happened in the experiment, so what was the method, what was manipulated? Specifically, um, what were those, so let’s just take the simplest case: there are two bars, condition A and condition B, and do I know exactly what happened in condition A and exactly what happened in condition B? Because the question is, is the number on the y axis different in those two conditions?
DP: Do you need… Do you, um, need to ask yourself if there is in fact only one variable being measured on the x axis?
LC: In the case, yeah, yes-
LC: And that is, that is absolutely implicit. I guess the way I would ask them that is, “how many independent variables were manipulated in this experiment?” If they’re just two numbers, it’s only one and two conditions of one.
DP: Unless somebody screwed up the experiment and unless there’s something wrong-
LC: Yeah, but well. Exactly, but then that’s the complicated question about are there confounds? Is there something here that impacted the data other than the manipulation that you made, right? Like-
DP: But that’s not a question you’d be asking yet? Is that right?
LC: No, no-
DP: It’s down the road, but you, you-
DP: It’s important to ask that question.
LC: The first, right, yeah.
DP: Okay, so what are you doing first with the x axis? Let’s talk about that some more.
LC: Again, I think I’m going back to the method, do I understand exactly what happened in the method that gives me, that allows me to understand ah, what that number represents, what the condition was, under which that number was generated.
DP: So you’ve got two points on the graph-
DP: Here, and you want to know what’s different on the x axis at this point, from this point. Is that…?
LC: That, I’m gonna ask, right, are the, um, are the numbers that were generated in those, the average reaction time generated in those two conditions, well what were the conditions?
DP: Yeah, what’s the-
LC: So then I can ask, are those two things different from each other?
DP: So what’s the difference between this point and that point-
DP: On this axis?
DP: Well what, with the, yeah so we’re going back again from the abstract representation of the graph to, to an event.
DP: What were, so how were the events different at, at this point?
DP: And, and maybe I’m being too, too simplistic about it, but I’m trying to be as stupid as I can to imagine-
LC: No, it’s, it’s good!
DP: -All the pieces that are there, it’s not hard to be stupid for me in this area because I don’t deal with graphs so this is a million miles away from my normal practice. Okay, so, what does that mean that you, I mean how do you go about recognizing the difference in the events that produce the different numbers on the, that axis. Is there, is there..
LC: So in the simplest, someone someone has written, it’s written down somewhere, right? It’s written down I mean in, if you’re reading a paper, in the paper, it’s written down in the, in the method of, um, maybe in the computer program it told you what those conditions were, but um… You do have to understand what they, what they mean.
DP: So you have to move from a verbal description to a, to the significance of the number?
LC: Right. Yeah, we’re not even there yet at thinking about the comparison.
LC: Right? It’s first just what is that condition?
DP: So, uh, I’m assuming you have to know where in the source to look for the information that’s gonna be about-
DP: -the x axis.
DP: And um, how do you, I mean, are the conventions of writing articles-
LC: There are. Absolutely.
DP: So clear that you can just say, “always look in this place-
LC: Pretty much.
DP: -for this thing?”
LC: Pretty much, yeah.
DP: But of course the students have to know that convention.
LC: They need to know the APA format, right? Which that, and that actually you’ve just given me a great, a great rationale, they often complain about having to learn that convention of writing. But this is a perfect example of why they should be really glad that we have it.
LC: Because then it gives, then you know where to find the information that they’re looking for.
DP: So that example I used, I think you were there, the days are all running together, so many things that happened today. But the example of the geologist who asked the students the four questions about, um, I can show you her video. I think it’s on here later on, yeah I think it’s on this, but I’m gonna stop now. But, she found out that, in fact, that whole notion, they were unable to find, recognize what it means when something’s in this part of the article versus what it means in that part.
LC: Right, right, right.
DP: So there’s a whole, that’s another piece-
DP: -that you’ve got, yeah.
PR: It might be cultural, I mean, I am currently missing a point in your…
PR: Um, procedure. Um, so we started with the y axis, indep-dependent variable, went on to the x axis, independent variable, in your example it turned out to be categorial?
PR: Could we call it numerical? Something else, and that’s actually the missing point. So I would look at, okay, how is the data organized? Is it categorial, is it numerical, is it on a logarithmic scale?
PR: All these thoughts of things would presumably happen at that time slot? I’m not sure, it might also be that I’m looking at the dependent and the independent and then I do this – okay, how are the axes organized, is it numerical or is it logarithmic?
LC: I get the sense, yeah…
PR: Like, is that of importance for you?
LC: Ah, it, it is in the next step if you need to do a statistical analysis. Well, presumably you would, but I wonder whether your y axis tends to be less discrete, like it has, like you’re measuring ah, over-
PR: Let’s say over order of magnitudes and then I would put it on a logarithmic scale-
LC: Oh, I see, I see, yeah, yeah, right.
PR: Or it would be put on a logarithmic scale and, for me, that would be important information in reading the graph.
LC: And in a lot of these it wouldn’t be, they’re just, you know, two or three conditions.
PR: Okay, so it’s-
LC: Yeah, it is in some-
PR: In, in your field variety is less than maybe-
LC: Maybe, maybe-
PR: Than in my field. Okay.
LC: And I know in some of the work that I do, that does matter-
LC: But in the stuff, most of what our students would be encountering it’s not, they um.
DP: So the numbers are just numbers? They’re just the normal number sequence, there’s-
DP: -no complexity of logarithmic scale or anything like that?
LC: Well they’re not even necessarily numbers, they’re categories, right? So they’re this condition in which this happened. The thing, places where, or, let me, the other place where you might wind up with something that’s more continuous is like time.
LC: But then we’re gonna have discrete measures of, of time.
DP: That’s what-
LC: Right, we can’t generate data that, that’s um-
PR: Your measurements are always discrete, of course-
LC: Well true, but you have more, you’re likely to have more of them-
PR: And if you measure reaction time, you wouldn’t presumably use a logarithmic scale-
LC: Well that is-
PR: Because it’s not our orders of magnitude.
LC: Right, right. Yeah.
PR: Yeah, I just wanted, I had the impression when I would read graphically represented data that looked, well not a lot, but I would put effort into decoding how the data is represented on the axis.
DP: Well there’s also the question, though, of the units of measurement, which are gonna affect how the thing looks. I mean obviously you can play by having -short- small or big units of measurement are gonna make differences in the graph look very, very differently.
DP: So there’s-
LC: That’s a whole ‘nother-
DP: Yeah, so they have to be thinking about, what are the units of measurement over here.
LC: Right, yup.
DP: Is it surprising if the graph goes way up? Or is that a normal, is that not representative, how much difference does, do the points on the x axis make in terms of numbers?
LC: Right. And that’s a very, very tricky thing that’s happening there because, depending on how, this is where statistics come in, right? Because you can make a graph, as you say, you can expand the y axis and make reaction times look very different from each other but it’s only you know, two milliseconds and it-
DP: Right, exactly-
LC: -is not either statistically nor functionally-
DP: -Yeah, right-
LC: – an important difference.
DP: And that’s an important example, are you measuring in milliseconds or seconds or minutes or-
DP: It’s gonna make a different graph, so they have to have some kind of a, of a sense of the units on the graph.
DP: If I understand what’s going on.
DP: Well this is getting more and more complicated, isn’t it?
LC: Yes, I agree.
DP: Curiouser and curiouser. Um, okay, so that’s another thing they have to do, let’s go back. You’re looking at the x axis.
DP: You’re recognizing what in the world produced the differences in, in those numbers that come out on the, or the categories of numbers that show up on the x axis. You’re thinking about what the scale is, what else are you doing on the x axis?
LC: Well, in the case that we’re talking about, which is a simple, right? There was this manipulation and that manipulation, I feel like I’m kind of done. And the next, and then I’m on to the next step of asking… So, so this is precisely, this is precisely the problem. That, what I was finding with students is that they were skipping everything that we’ve just described. Right? They weren’t, they don’t tell you what’s on the axes, and, in fact, if you work with things like Excel, Excel often doesn’t put an axis label on, right? So if Excel doesn’t put an axis on, a label on, then they’re not gonna put a label on. And then they move directly to what I would say, now, after having done all that work, the meat of the matter which is: is there a difference? Which is a statistical question. But they’ve moved entirely beyond that, ah, they’ve got, they’ve got that without doing this other more basic work.
LC: Which as we’ve just described is very complicated work. And then the next step is where they, where they actually really do struggle as well, which is in trying to assess whether or not these things are different and how would you know if they’re actually different? Which is the statistical-
DP: Is it a “forty-two problem”, in the sense that they have a number but because they haven’t done this work the significance of that number really isn’t clear to them? Or, or not?
LC: I don’t even think we’re at signif-, if by significance you mean-
DP: You haven’t gotten there yet?
LC: -like functionally significant, like meaningful?
DP: Yeah, yeah.
LC: Uh, I don’t think they’re quite there yet.
LC: Yeah. But, but if I could get them, but now, so now we’re to, now this has helped me clarify that the problem, I don’t even want them to go yet to the problem of whether or not there’s something important-
LC: -that’s happening. I just want them to describe what is there.
DP: That’s right.
LC: So that then we can do, and if we’re on the same page with that point, and I’m doubting that they’re, that they’re actually there with me, for precisely the reasons that we’ve just talked about because there’s a lot of work that’s going on that they may or may not be doing… Then, I’m frustrated because I feel that they’ve, they’ve missed that, they’ve gone right to the meat and they’re not able to really understand the meat because they didn’t understand the foundation.
PR: Can we start with talking about data interpretation? Currently I would say we are at the step of… Data perception?
LC: Description, yeah, right? Just description.
PR: Yeah, even before that I would say.
LC: Just even for, I see, perception-
LC: I see, what they-
LC: Yeah, right, right, right-
PR: What do I, looking at this graphical representation what do I actually perceive?
PR: I feel like it has not been described by the students.
LC: That’s exactly right, and they speak as though it’s self-evident. And, and by the way, if it’s the, there are cases I think where the students, if they have understood it, which we do the same thing, right? It’s self-evident, it feels self-evident and so, and this is particularly true if they’ve worked on a pro-, like a big project themselves, not something where the data were handed to them in this way, but they actually worked on a research project in the lab… And, again, they do this thing where they start, you know, three-quarters of the way into the explanation as opposed to stepping back.
LC: And at the risk of going in a direction I’m not supposed to go, and I, I’m mindful of the time, I want to make sure you get the chance too, I, I realize just in the previous session that some of the work, I, I’ve often wondered, “where did I learn to do this?” And I do remember, as a graduate student, giving presentations where we would be practicing for a presentation at a conference and we would describe, and you know you only have-
LC: -twelve minutes, right? And so, and in those cases we spent a lot of time, faculty and grad students together, talking about, “okay, well wait, wait, wait, you can’t say that because you didn’t tell them, you know, this piece.” Or, “that wasn’t clear so they’re not gonna, you’re gonna lose them at that, and you”, and there it’s so important because the time is so tight.
DP: Yeah, it’s my, and recorded this the other day, but my, you know, colleague Mia #?# always says, really points that out that what you’re describing is apprenticeship, a personal, face to face apprenticeship in which you’re soaking up the rules, often without being conscious of them.
LC: Yeah, yeah.
DP: When you try to do that in a class, a lecture class of 150 students or something – it doesn’t work.
DP: Because they’re not seeing it, they’re not, you know, that’s probably why we have to do all this, this stuff.
DP: So you’ve got them in some ways to begin to see the graph in some more fundamental way-
DP: And you’ve, you’ve led them past a whole lot of mistakes that they could, they could make. Um, where do we need to go now? What’s, what’s missing here? What´s the next step?
LC: Well for me, for me the the clarity that I have that I’m going to forget in 48 hours-
DP: When you’re gonna have the recording, hopefully-
LC: -is that, um, it’s about the axes. Right? And I often, I have heard myself say, “well wait, wait, wait, what’s on the axes, what’s on the x axis?”
LC: “What’s on the y axis?” Right? But not thinking about, well that’s, and especially if it’s listed there, they can just say, “well it’s reaction time”. But then I haven’t thought about the work together.
DP: Of course that’s what we always do. I often use this as an example because I’m so guilty of it, but having written on history papers, um, you know, interpretation question mark, thesis question mark, and realizing that if they knew what interpretation, what I meant by interpretation, thesis, they wouldn’t have done it.
LC: Right, right, right.
DP: Uh, and it’s the same thing, it’s, look at the x axis, well-
DP: Yeah. Okay, it’s there-
LC: Well it says this and they can read you what’s there.
DP: But they can’t, they can’t read it.
DP: They can, they can see the number but they can’t read the axis because they aren’t doing the things that you, like you just did.
LC: So, it’s, it’s perception or seeing or um, or des-, and it, and then, and then description. And if I got there I think I would be really quite happy.
LC: For the moment, right?
DP: Now is there anything they need to do with the, with the slope of the, of the line or anything like that that’s, that’s essential that you do automatically, do you think? Or is that just getting too much into a much more complicated thing?
LC: Oh, okay, so the thing, the next the next piece of it is to have some, I know, this I feel very explicitly, that I have, I have intuitions about what kinds of differences are actually both meaningful and statistically significant, right? And so I can look at it and if it’s sort of self-evident, right, but they can’t and that has to do, I think with, and we don’t need to follow up on this, but that has to do with what’s the variance? And what is the statistical test actually doing? And, um, and that I’m, for the moment, less worried about.
PR: Currently I feel a little bit uneasy, um, because for me this, this distinction between perception of the data and description of the data seems to matter. Uh, but I am not aware of what I’m doing. I have the, I share with you this analyzing, perceiving part and then I have the feeling that this is all subconscious I believe, or became, became a habit that I would ex-, describe the data to myself. Maybe within the split of a second.
PR: This, this graph relates response time to two categories.
PR: Period. That might be it.
PR: So that’s the, after having, you know, perceived the graph I, how do you feel-
LC: But I think you’re, I think you’re right, and I think what we’ve just talked about is the fact that the students, if it’s labeled, might even say that and you might still have the sense of… But the, but the real understanding of what that is, is missing.
LC: So they might even be able to say the words, but it’s, but do they know what that, what that means? Not what the data, not on an interpretation, but just, again-
PR: True, just describing-
LC: Just describing.
PR: It relates, it doesn’t, still I haven’t said how the relation is.
LC: Exactly, exactly.
PR: It might be unrelated actually what, what the graph is, is telling me.
LC: Right. Yep.
DP: But it’s about, what’s the subject of the graph, is that what you’re getting?
DP: You know, as an example of a graph about relationship of this and that.
DP: Um, I’m thinking too that there’s a video we have, um, an interview, a decoding interview uh with Gregor Novak, uh, one of the founders of, of Just in Time Teaching, and he says um, goes through, he talks about, I mean not in detail like this, but he talks about students reading graphs in physics. And then Joan Middendorf’s the interviewer, “well, so you want students to do all these things you just did”, and he said, “no, no, they have to be able to move past that. They have to be able to automatically look at the graph and get it.” So, so, all of the scaffolding is something you build but it has to go away-
DP: -to really, in his case at least, to do the physics. So, you know, it, because you don’t do the, you don’t actually stop and do all these things-
DP: -you go, boom, boom, boom, boom.
DP: And then you go on to the subject.
DP: So it’s, it’s like ah, piano lessons or something.
DP: Yeah, to go back, you know you have to learn it slow, but then you don’t think about it anymore when you go to the dynamics and all that kind of thing. But it has to be practiced until-
LC: Right, exactly-
DP: -you get, ‘til it disappears like you before you can really function in the discipline very well. So that’s another piece of this-
DP: -is that you want this to disappear eventually.
DP: But only after it’s become part of the reality of the process so much.
PR: Until it became a habit.
PR: I guess for us it’s a habit.
PR: For the students it’s not, yet.
DP: Yeah. Yeah. So, um, let’s look at your data a little bit if you guys are up for a little bit more.
PR: Sure, okay.
DP: I’m enjoying this by the way-
LC: You okay?
DP: -this is really. You are, you are an A-1 subject, by the way.
LC: Oh geez.
DP: I mean you stay with it, you answer questions, you follow up, you think, you know, we’re not hitting blocks. It’s really nice. It’s really nice work.
LC: Thank you.
DP: So, how’s yours similar or different?
56:00: Move to Peter’s Bottleneck
PR: In my physics classes I don’t teach physics anymore, um, I would say this is absolutely similar. We had a somewhat different setting here because you framed in like you were reading a research paper and, and thinking um, how to get along with the data. Um, you were quite often referring to going back to the methodology, and uh, I wasn’t sure whether I would do that, but at a certain point I, I agreed with you, I, it’s making, making connections with what is this all about. I, I maybe would put more of the emphasis on, okay what is the point we want to clarify here? Um. And that, that might be the difference.
LC: Meaning the, meaning the interpretation?
PR: I mean, there’s certainly a difference. You’re measuring reaction times and wondering whether it’s dependent on this or that or, in physics typically we ask is there a relation between speed and mass or something like that.
LC: Right, right.
PR: That’s all. Um, there certainly is a difference. Um, so, in, in the case I’ve brought up it’s less about um, interpreting graphical data. Um, now I have the feeling that it is about, well, I said before, about relating it to the, to the question at hand. Checking whether it is plausible in the first place, do we have that? And then, trying to see how this data is connected to the hypothesis and whether it’s, it provides evidence, whether it provides insights, whether it provides astonishments. Now that I’m talking about that, it might be completely different settings.
LC: It’s the next, that’s the next step, right?
LC: For me, right?
PR: Yeah, yeah, yeah, sure.
LC: That’s absolutely where I wanna be, but-
PR: Uh, so, what’s happening with the students, then-
LC: But can you, can you get to them interpretation before you’ve done this other work?
LC: Right, so, yeah.
LC: And, and do your students, I mean I’m talking about second year undergraduate, I don’t know what the level is that you’re working with.
PR: What I’m currently talking about would be first year master students.
LC: So they may-
PR: Typically I work with undergraduate students, um, so I have to remind myself that in that situation that I described earlier, students would generate their own data on their computer. And, so what would the problem be? I guess one step that they would need to do themselves, I’m realizing the difference to your case, they would need to organize the data. If you have a graphical representation that already is sort of organization of the data. Um, in my case, students would have to do that themselves. Which I have-
PR: Of course. Um-
DP: Well can we start right there? From there?
PR: Yeah. Yeah.
DP: The first question is, is the um, what goes on the x and what goes on the y axis. Is that fairly obvious to them, or is that an issue?
PR: Well in that, in the case I’m currently thinking of that’s not applicable, uh, well-
PR: Well, I would say it’s obvious you wouldn’t represent it graphically in that way or you wouldn’t have a traditional standard of graphical representation, but that, again, might not be clear to the students.
PR: -I realize. Um, so in that situation, it’s um, they get fairly simple numerical computation tasks and they should predict the result by doing the calculation by hand and then doing it on the computer. And the data, the, the task is designed in such a way that the difference will be observable. Um, so I wouldn’t represent that graphically. Um, but-
DP: So what’s the, what’s the problem that you would, what’s an example of a problem you would be working with?
PR: I’m sorry, say your question again.
DP: Well you, you sort of talk about reaction time-
DP: What, what would be a concrete situation you would use?
LC: Your mass and weight and speed-
DP: Yeah, what’s your-
PR: Yeah, I’m currently thinking of my Numerics class and-
PR: -I was trying to describe um, so the problem with floating point computations on a computer is that they are not exact. So there comes an element of uncertainty then and I want the students to realize that.
PR: The task is, um, so here’s some number of numerical computations like 10 minus five times 10 to the minus 15 plus something. Um, and to do that on the computer, so that is another task in between because I need to program that, that thing. But at a certain point come up with data that the computer produces data. And, I feel that’s where the data interpretation, then, should start. There is a machine producing data. Producing numbers. And we already have the issue of how to organize the data, to me it had been unclear how to organize it, and simply saying I wouldn’t organize it in a graphical way, I would organize it in a tabular way. But that might not be clear to the students.
PR: So that’s my expertise, I, I believe that for this sort of problem, that would be the preferred uh, represen-, representation of the data and not be preferred by community, cultural conventions, but-
DP: But typically what, and, and there can be cultural conventions, there can be aesthetics, there also can be what works and what doesn’t work.
DP: Um, so if you imagine someone being at that forking path between the table and the graph-
DP: In a particular situation um, what might suggest one is better than the other?
PR: Mhh… (long pause) No doubting that one is better than the other-
PR: Well it might be actually cultural, um, yeah in order to answer that I have to imagine myself, what would be the outcome if you would represent this data graphically?
PR: So I’m trying to do that in my head. And that basically, now you can argue, now, now all sort of questions start, of course, as soon as I, as I draw a graph: so what is the dependent variable, um what is the independent variable, so the dependent variable of course is the, is the result that the computer generates, but you would have, you want to do a graph so you have to basically doing this thinking, which I’ve just done. So what is my dependent variable? I would have to clarify that. Um, in a table I would not have to do that, I believe. At least not at that point.
DP: You just fill in the squares.
PR: Yeah, yeah, yeah. Just fill in the numbers. I suppose you have done that and I’m just still trying to imagine what the graphical representation would look like. Um. Well then, we would come to the x axis, um, in that example it’s, it’s arith-arithmatic combination of three numbers. So, what do I need to do? Do I need three axes for independent variables? That’s what I would have to clarify. Is there a way to code it into one axis? Would I need to have several diagrams, um, because I can’t- wouldn’t be able to draw this four-dimensional thing, which I would need, uh, in stats, so I guess now these sort of questions, I – I would stop now and say okay that this is way too complicated, I would go to the table.
DP: So it’s some kind of methodological Ockham’s Razor that’s operating here.
PR: Yeah. Yeah, sort of. So I became aware of it if I were actually thinking about the, the, the best way to represent that I, certainly I am, and all this what I made explicit might have happened, might have happened in the split of a second, of course-
LC: Can I, can I, I’m sorry-
LC: Can I ask a question about the bottleneck? Because it seems that this is demonstrating a slightly different bottleneck for you and for me. Or actually-
PR: That could be.
LC: It’s one that I share with you but I, it’s not the one I was talking about right now.
LC: So, for the, for the one we were just talking about, it was: here’s a graphical representation, you are likely, as a psychology student, going to encounter these things and are going to need to be able to understand them and describe them. Um, but at another point in your college life, you will also have to generate them.
LC: And those are, I think two, and so you’re talking about making decisions, which our students have to do too, is this a time when I should have a table, or is this a time when I should have a graph, is it a histogram, is it a whatever, right? – But that is a different, that strikes me as a very different question then.
LC: Or a different bottleneck than-
LC: But then so what is the, for, for, and now I realize I’ve made a leap here, which was I’m gonna, I’m gonna deal with things that are already made.
PR: Yeah, in your case-
LC: They were already-
PR: In your computer program had done this work.
LC: It did. And that’s what-
PR: In my case, students have to do that.
LC: And my, and other cases my students will have to do that.
LC: But this is not, this particular one-
PR: Yeah, so-
LC: -is not one of those. So they, when they see, and back to the questions you were asking, when they see, when they’re reading a journal article they have to figure out what’s behind that graph-
LC: And they won’t have the spreadsheet.
LC: To figure out, right, they won’t have the raw data. And some cases they will. So, ah, so my question for you is: when thinking about the bottleneck, is the bottleneck in, the one that you’re more concerned with, the one of them making decisions about creating a graph or not or is it one about describing data? Does that make-
PR: Um… Let me try to answer that from a different angle, because, nowadays there’s often this “oh my god” situation.
PR: And that is, you know, this, this task has a, have particular goals in mind. That the students should generate, self-generate some insights. And now I realize, oh my god, there is this difficulty in that task and that’s an important skill they need to learn.
PR: But not for that particular task.
PR: So, I, I could easily take that out by saying, “Okay, compute the data and organize it in a table, period.”
PR: Just to, you know-
PR: Get them out of that situation.
LC: Out of that problem.
DP: Which goes to point out, *clears throat*, how far-
PR: So, then in trying to answer your question I realized at that point that it’s, it’s not important, yeah, it is a bottleneck, I agree with you, but I’ve realized that’s misplaced here, ah, I should help my students just to bypass this whole thing. In this task.
DP: One of the things that decoding has done for me that I think has been the most effective is begin to make me really think strategically. And you were just thinking strategically.
LC: Mhm, mhm.
DP: This is not the time to fight this battle-
DP: You know? I should go around, and have them do this-
PR: Yeah, right. This is-
DP: And later on we’ll do that. And I think we tend not to think strategically enough when we think about teaching. The decoding sets it up in terms where you really begin to say, like you have all these things out there, where do I start? What’s the sequence? What’s the reasonable order?
DP: And that’s a whole set of questions that come once you’ve got them on the table. I think that’s really important. Yeah.
DP: More needs to be written on that, I think, in fact.
PR: You know where I really want to get to is the students asking what is this data telling me?
PR: And, in that situation, I haven’t posed any research questions or hypothesis, saying, “Okay, there’s the hypothesis that numerical computations are not exact, use that task to verify or falsify that hypothesis.”
PR: Or think of a way to verify it to support that. So in, in, in that situation I’m not operating in classical hypothesis-tasting scenario, just generate some data and then look at them, and yeah, I actually want them to be astonished by that, so “Wow, that’s weird, I wouldn’t have expected that.”
PR: That’s the task in, in this-
PR: Classroom exercise. And, what I’m experiencing is that most of the time they are not astonished by that.
LC: They’re so bamboozled by the-
PR: That could be!
LC: trying to figure out whether it’s a graph or a table.
PR: So at a certain point you have the number, have the numbers, and for them it seems to the end of the story.
DP: I keep coming back to it, but 42 seems so relevant to the situation.
DP: And it’s like, as if the people that have the computer run for whatever in Hitchhiker’s Guide for three millennia or something and they get the answer 42 and they say “oh, it’s 42!”
LC: Yeah. *Laughs*
DP: And that’s it! So, what is-
PR: Well luckily it’s not for all of them, for some of them-
PR: They, they, they are shocked. And then searching for bugs in their computer program. Yeah.
PR: Yeah, which, in a way is good-
PR: Because they believe it should come out exact but then realize that it’s not the case.
DP: So the question is, for those who don’t do that, they’re not doing something that you’re doing, and that the other students are doing. I mean there’s some reason why, uh, well I mean taking an analogy, that you know, if you put a spot on the head. If you put most animals to sleep-
DP: Put a spot on their head, show ‘em a mirror and won’t react.
DP: But primates will.
DP: So it means that the primates are doing something that other people aren’t. So the people that get surprised by this, like the primate that says, “what, what is this thing”-
DP: Uh, is doing something. So what are you doing that’s creating this exactly, this astonishment?
PR: That’s a good example. I, I generate an expectation.
PR: That’s what I actually guess I want my students to do. I mean this task is designed to get into a conflict and for that you need an expectation.
PR: The, the expectation preferably, from a pedagogical point of view, would be that they expect that the computer computes an exact result. And then they get the result and see, well that doesn’t confirm my expectation. What’s going on here?
DP: Yeah. My guess is anybody in your class, if they were asked that question, if I’m, you know, will I get consistent answers from the computer? You know? Would say, “oh yes, of course.”
DP: So it’s, I don’t think it’s that they don’t have that expectation-
DP: It’s that that expectation somehow doesn’t get, somehow part of the process. It doesn’t get activated.
DP: Does that make sense?
DP: So there’s something you’re doing.
PR: So there’s something that’s missing actually.
DP: Yeah. Um…
PR: Well no, it’s not missing. Sorry, because I asked them to do the, the, the computation by hand-
PR: Before. So they have two things to compare. They have to-
DP: But isn’t it like the students who check the box for 12.5, uh, buses? That they, that there’s no point at which they look at that answer and say, “what does that tell me?” You know? Or what does that signify? It’s just, isn’t it just, I’m imagining a student for which it’s just a number.
DP: And there’s no point at which they process “what does that number mean? What does it tell me about the world? Why is it surprising that this”-
DP: Something is not happening.
PR: In that sense it’s similar. Right? That’s the interpretation of the data, I would say.
PR: In other words I would say in another sense it’s not similar, uh, because it’s simply not plausible that you have 12, well, maybe it’s the same. Um… Yeah, so I agree that’s-
DP: So, *clears throat*, so part of the job is stopping and asking questions-
DP: About what’s going on here.
DP: And some of them are doing it and some of them aren’t.
PR: Is, is, is this what I expect?
PR: Is this plausible? That’s why I just wanted to make the differentiation so that if I have a reaction time of 16 hours for pressing a computer button?
PR: That’s not plausible! Alright? If I have a reaction time of 16 hours for answering an email, well-
PR: It might be-
LC: Oh boy.
PR: That might be plausible.
DP: That’s a wonderful example. That’s really good. Um, yeah, so, but they, there’s two things involved. One is knowing to ask the question, “is this plausible?” And the other point, then, is to have criteria for plausibility.
PR: Yeah. Yeah.
DP: Um. Because you, your examples really make that clear.
DP: That you have one criteria for email and one criteria for when someone hits your, you know hits your knee with a, with a hammer or something, that’s just, you know, different. Different levels of plausibility there. So, the first problem you have is to, and this case, the main problem seems to be that they don’t ask the question at all.
PR: That’s my feeling, yeah.
DP: That anybody even stops to ask that question at all is probably gonna get a, “well oh, that’s weird.”
LC: And this does seem like my, the what my husband was, um, comes up against when he has students give him answers without units.
LC: Right? Where he says, “But you’re not-” so they’re not thinking about, they’re just looking for an answer-
DP: That’s right.
LC: Any answer. Give me an answer, give me a number.
LC: Right? And not thinking through what that actually means. And somehow I feel I must have that same problem, but it doesn’t get expressed in the same-
LC: And it may be that there are fewer units actually. Or it possibly that is-
DP: Well part of my expectation is that the people in physics and math have been trained to do that by the system.
DP: In psychology it’s a-
DP: It’s something new. But-
PR: Yes, I mean I have been painstakingly trained by you know, labeling axes and-
PR: Telling units, um, I would say in a heuristic manner. But never mind. But I guess the important thing is, um, not about “You missed the unit here! You have to add the unit.” The number is meaningless.
DP: But in fact-
PR: And, and, and the number with the unit is also meaningless-
PR: Unless you relate it to something.
DP: So have the students been taught to do that?
DP: But in this math, math uh, quizzes, tests, homework assignments in which there’s no units, there’s no meaning, there’s just manipulated numbers?
PR: So that’s a problem-
DP: So they’ve been trained to do that.
PR: with mathematics, I agree with you.
PR: I mean, mathematics is so abstract that even the meaning is abstract in a way, sometimes.
PR: The meaning of the application.
DP: That’s right, but even beyond that it’s taught in a way that’s just repetition of the model.
DP: Without any kind of, of self-reflection in it. So that’s part of the problem. Is that they, they need to learn, and this is not really part of our reflection but it’s a fascinating question: how do you teach people to be amazed?
DP: And what experiences do you give them in which, which they begin to realize that’s part of the process? Um, and you can imagine all sorts of, of almost like joke answers, you know, absurdities that come out which, what’s absurdity? Are you, you know, playing with it, but, but something about that needs to be done, it sounds like with your students.
LC: I have a colleague who, when she shows students things that she thinks is really amazing, like she wants them, she’ll actually stop and say, “I need to hear the ‘Oh wow!’”
LC: You know? You need to say that!
LC: Whether you feel it or not, I wanna hear, “oh wow”.
DP: Yes. Yeah, that’s right. And, and giving them some, something first that they’re definitely going to say “oh wow”, I mean a perceptual, some kind of perceptual thing that just-
LC: Although that doesn’t always, I do have one that almost always gets an “Oh wow.” You know. A color after-effect that you, you adapt to blobs of color, um, and then you’re shown ah, you’re actually shown first a black and white, it’s a black and white picture of a castle. I’ll bring it, maybe tomorrow I’ll bring my computer tomorrow and show you, it’s a black and white picture of a castle, you adapt to these sort of blobs and then when you see the black and white castle again it looks like it’s in color. And it’s, it’s all color after- so right, so the sky looks blue and it, you know, lasts for a couple of seconds but it really looks like a color photograph. And I always stand away from the computer like, “I’m not doing anything, it’s a black and white picture!” And they’ll see it and you know, they-
DP: There’s the gorilla video which I’ve never actually watched-
DP: You know about that?
LC: Inattentional blindness.
DP: Yeah, there, there’s a video in which people are passing a basketball-
LC: People are playing, yup, so you have people-
PR: Ah, yeah okay.
PR: Yeah, yeah.
DP: And the gorilla walks by.
LC: And a gorilla, a guy in a gorilla suit walks through and 50% of people don’t see it.
DP: Something, in some way they’ve gotta be taught that part of the process is being amazed. And that seems like an odd thing to have to teach.
DP: Yeah, well, anyway.
LC: Yeah, but you know what’s funny is that kids have that naturally.
DP: Oh, absolutely.
LC: It gets beaten out of them. And then when they’re, you know, cool teenagers they can’t be amazed by anything.
DP: That’s right.
LC: And then we’re getting them at the point when they’re, you know-
DP: And that’s, that’s just training. They’ve been trained to be stupid.
LC: They’ve been trained, yeah.
DP: At this point.
DP: Um. But, I think that may be part of it. So you’re going to have to model amazement.
DP: In some fashion. And other students may be helpful there too. You know.
LC: And they do appreciate it in us, right? When we’re amazed and excited about what we’re, yeah-
DP: Think about those rooms, too, where you, ah, you, you know, you enter the room-
LC: The Ames room?
LC: The Ames room, I’m sorry, go ahead.
DP: I don’t know what you call it, but everything’s the wrong, not the size you think it is-
DP: Because lines and all like that, you know.
LC: Yup, that’s the Ames room.
DP: Ames room, interesting. Um, anyways, there’s some amazing things. But after you get them amazed, what do they need to do next? Do they recognize that this is not what I expected it to be? You’ve got data that amazes them, what do they do, what do they need to do next, do you think? Or do we need to move to another problem, we solved this problem.
PR: Um, well I can’t, I could tell you, but then we’re way beyond data interpretation.
PR: Um, so then a little research, probably would start, like why is that, the case?
DP: Okay. So you come up with a hypothesis, possible-
PR: Yeah, yeah.
DP: Generate, generate possible reasons why this thing would happen that was there. Um. I’m getting into the modeling, but I love the modeling part. A little bit here. But I can imagine some, some metaphor where you take them and put them in a situation where, where they experience something amazing, astonishing, and then, and then they go through the process of figuring out what caused that effect to happen in some other part of the world where they could, their world where they could understand that. And then you go back and you say, “Okay now we’re gonna do the same thing but with this. I want you to do this stuff and we’re gonna look at what’s weird about this.”
DP: And what could make this weird thing happen. What is it? But that get into a whole other thing. But what are you assuming must be the case that, that’s not present here. I mean there’s a whole bunch of questions you would have to ask yourself to get to the meaning of that.
PR: Yeah. I mean this, this could continue um, it’s the first class activity and it’s basically set to frame the course or to have this, this #indistinct# at the beginning to say, “Wow my computer doesn’t work as, as I expect it to work.”
PR: And then there’s a whole class of answering questions that-
PR: Might have been generated there or might come up later, so I’m not, currently not concerned with what happens after the data interpretation.
DP: I know people who I’ve worked with, let’s see if I can remember this exactly, people I’ve worked with in computer science before, I know one, one of the problems that they, ah, have expressed is that students, um, they, they don’t wanna de-bug until the thing’s done. They don’t wanna pay attention to mistakes being generated. Until the whole thing is, is finished.
DP: And that that’s not very efficient.
DP: They make-
DP: A whole mess of things and it strikes me that, that this process of considering what you’re doing and what’s astonishing etcetera is probably important in that whole process, that you start doing that from the beginning. What, what fits, what doesn’t fit, what’s going on here? Why is this kind of odd? Until you end up with the whole system crashing and you have to re-write it again, you know, something like that.
PR: Yeah, I guess this is a complicated problem because you have to um, to break apart a big thing into smaller things and, and encapsulate them and have meaningful structures.
PR: And then think about, how could I test them, um the smaller things that you have created.
DP: Yeah, yeah.
PR: I believe this is a difficult-
DP: So are we-
PR: Yeah, I’m fine-
LC: This is fabulous!
PR: Yeah actually I’m not fine!
PR: Yeah, I have to confess that I’m really shocked. I’m angry with myself, um yeah, because it never occurred to me that, you know there’s you know, complexity in there that shouldn’t be.
DP: Well see and that’s the-
LC: I think it’s fabulous.
PR: Yeah, yeah, sure I am happy-
LC: I am so excited. Yes.
PR: I am happy, but on the other side angry with myself.
DP: Yes, now that’s what-
PR: And this is the basic thing, um, don’t mix in too many things.
PR: When you do something in the lab, and the class is my lab.
LC: Right. Right, right.
PR: And it happened to me that I mixed in too many-
LC: Right, right, right.
PR: Too many things.
DP: Yeah, but this is actually my failure as a, as interviewer because I should have set that up more because that’s one of the things that typically happen-
PR: No, I’m fine. I’m fine.
DP: A few people go, I taught all those years and I never told them the most basic thing that they had to know I never taught-
DP: You can see guilt in the, people sometimes, you can see them flush, their faces get red-
DP: And they feel guilty.
PR: Yeah, yeah.
DP: Um, because, oh my god, the students, I gave them that assignment and it, it didn’t make any sense because they didn’t know this step at all. And you know, so part of the process is to say, you know, we’re all experimenting here, we’re all trying to find out things in the past, we’ve just gone in the dark.
DP: So we’re gonna see things which we wish we’d seen a long time ago, but we see it now. And if you share it with somebody else, you’ve made up, made up for your sins.
PR: That’s fine with me. I mean, on a higher level I’m happy.
PR: That it happened to me and that I got to that point and when I do the class next time I would tell them, “Okay, organize it in a table, such that this is out of the way, but #indistinct#, I’m emotionally taken.”
DP: It’s almost like there’s two paths. And one leads to like this nice garden and the other path you have to fight through the brush and there are monsters and you eventually can get there.
DP: And you forgot to tell them to take the right path, you know. But, it’s life.
LC: Very exciting.
DP: And your steps were really clear, I thought. And, and I expect it to be useful to a lot of different people.
DP: Not that they all would see the same thing-
DP: But what you’re doing would, would um illuminate what they’re trying to do.
DP: Even if it’s not the same.
DP: So that is very, very valuable.
LC: Interesting. Well I, well I didn’t know and I’ve been trying to think over the course of, since yesterday, since your presentation about “oh what are those steps?” And I, I can, really on my own I was only able to come up with a couple of things. So this was very helpful for me-
LC: To be able to think about… Especially that going back and forth, that was new, that was, “Oh, I must be doing that, um-“
LC: You know, without knowing that that’s what I’m doing.
DP: There’s certain, certain patterns that appear repeatedly.
DP: And uh, one of them is the asking questions, that students tend not to ask questions and experts do ask questions.
DP: Another is that very often that movement from the abstract to the concrete is problematic, more problematic than experts realize.
DP: And that came up I think in yours.
DP: Uh, sometimes there’s a, an order in which you do things that the students don’t know. They don’t know to follow the order, they go all over the place.
DP: Um, students tend to want to do things too fast.
DP: And that’s, that’s like a big problem in so many ways. So if you can get this to us in some fashion. We need to exchange emails.
PR: Yeah, I think I ah, stop that here and then I-
PR: Transfer it to my computer? Um, that-