This is better

Ben Alpert from Khan Academy responded to my open letter to Sal Khan and provided a link to an improved set of decimal exercises.

They are better.

You end up having to compare two-digit decimals to one-digit decimals.

Screen shot 2013-08-06 at 9.00.35 AM

My major objections now boil down to pedagogy, on which point I understand that I will make no progress with Khan Academy, so I won’t make the effort. I’ll leave that to Frank Noschese.

If you accept that people learn mathematics by doing lots of multiple choice exercises, then all I have left are technical details.

They are these:

  • In the U.S., money is a good enough model to get students through two-digit decimals. It is not uncommon for children to be able to reason about two-digit decimals, but not generalize to three- and four-digit decimals.
  • These are randomly generated, I assume. And the probability of getting two-digit decimal>one-digit decimal seems artificially low. As I ran through a bunch of these, I began to build a model in which I could (1) treat comparisons with same number of decimal places as whole numbers, and (2) claim that the one-digit decimal is larger. So I ran an experiment. Twenty exercises using my model. I got 90%. (See video of a repeat of this experiment; I don’t know that I did 20 this time, but I did a bunch and only got one wrong)
  • Related to this, there is no need to click through the hints. None of the decimals came out equally (i.e. no 0.1 v. 0.10). So when I got a wrong answer, I just chose the other inequality. Pattern matching and process of elimination allowed me to avoid instruction of any kind and to get an A.

See, here’s the thing. Teaching requires a mix of knowledge and assumptions on which to base decisions. When everything is pre-programmed, deeper knowledge is required in order to create meaningful instruction, not more analytic data.

Carnegie-Mellon is working on a deep model for diagnosing student misconceptions with decimals [pdf] (and presumably many other domains). Again there’s the pedagogy thing, but I am impressed with the effort to build a solid theoretical foundation for their work. Here is a sample of a taxonomy of decimal misconceptions they have developed.

Thanks to Frank Noschese for the find.

Without that deep knowledge base, all that’s left are assumptions. Which is fine, as long as the assumptions are not flawed.

7 responses to “This is better

  1. “As I ran through a bunch of these, I began to build a model in which I could (1) treat comparisons with same number of decimal places as whole numbers, and (2) claim that the one-digit decimal is larger. So I ran an experiment. ”

    Nice. It makes me wonder how hard it would be to build software “students” who each answer KA questions following a given misconceived model as a way to test the problem sets (and to aid in diagnosing real students). Presumably they already do some automated testing on their data collection and on the interface. Maybe they could at least be gotten to improve the problem sets by pitching it as an AI problem, since they don’t seem to find pedagogy alone to be very interesting.

  2. Michael Paul Goldenberg

    I remain skeptical that deep knowledge – of content, of pedagogy, or of the crucial pedagogical-content variety – is the goal or credible domain of the Khan Academy and its approach to mathematics teaching.

    While it may be toying with some potentially useful tools, I find it doubtful that any that are come from Mr. Khan himself, based on the ample video of evidence of his teaching style, choices of examples, and inability to imbue what he teaches with the smallest modicum of enthusiasm. He continues to teach in a tone that says, “This is just another set of things to be memorized and/or practiced unto “perfection.” Nothing to actually SEE here, however, so move on to the next set of rote exercises and my next deathly-dull off-the-cuff chalk-talk.”

    While there is more to KA than those dull lectures, the notion that there’s something revolutionary going on at Khan Academy can’t stand without addressing the telling charge that: 1) he’s taking traditional mathematics instruction with all its potential ills and transferring it, badly, to the on-line digital world. The big “advantage” is that students can rerun these talks over and over. It’s certainly conceivable that on the nth viewing, a given student can finally follow the steps and get the same result Sal does, but at this point, it’s pure speculation as to how many viewings that would be and what, exactly, the student has gained, if anything, as a result. But with various cheap recording devices out there, students could conceivably make their own “tapes” from live lectures in school, PLUS have an advantage KA doesn’t provide: the ability to pause the lecture by – wait for it – asking an immediate question of a live instructor.

    Now, of course, a lousy classroom teacher is no better than a lousy recorded teacher except that the former might just stumble into a decent answer to a spontaneous question. Not so Sal Khan or any other talking head. At best (and thus far I see none of this coming from Khan) a recorded teacher could, based on research and past practice, anticipate good questions from students (“good” here meaning both those that help expose misunderstanding and confusion, and those that push the topic towards further depth or connections to applications, other mathematical topics, or actual experience), and build as much of the resulting conversation and discussion into the recorded format itself.

    Could KA do that? Maybe. But I think it would take a much better teacher than Sal Khan is ever likely to be in order to start assaying such a complex task. And it would take a more insightful instructor than Sal to realize that such a structure would be a good idea.

    I know: I’m a hateful, status-quo supporting, jealous, fearful traditionalist trying to defend his income from the innovative and selfless Sal Khan. Except that I’m not a teacher for the most part and anything but a traditionalist or supporter of bad pedagogy. Quite the contrary. I work against the status-quo at every turn. Where I sit, Sal Khan is really part of the worst propensities of status-quo mathematics instruction. He helps propagate more ignorance than insight. And that is one major reason I find him objectionable.

  3. This is awfully pervasive in math. STudents figure out these mildly amazing strategies like yours — “one digit is greater than two digit.”
    Often, they believe that’s the rule they’re supposed to memorize. Then when somebody tells them their rule is wrong… they think the rules are changing . T hey have no idea that they constructed the rule; they think that’s what they were “taught.” I suppose if they were taught in a constructivist classroom that might be true 😉
    I’m thinking of all kinds of fascinating rules in algebra about “moving that to the other side,” or “subtract from both sides” (of whatever seems long enough to take the space).

  4. I write algorithmic questions, so this got me thinking about how I would try to avoid the pitfalls you’re seeing in the KA exercise. It seems you’d want the question to hit a bunch of cases:

    1. Equal number of decimal places
    1a. Same number of non-zero values: .034,.012; .4,.5; .00024,.00062
    1b. Diff number of non-zero values: .034,.003; .00024,.00143
    1c. With trailing zeros: .034,.030

    2. Different number of decimal places
    2a. different number of leading zeros:
    2ai. same number of non-zero values: .03,.1; .0042,.015
    2aii. diff number of non-zero values: .0123,.004; .4,.065
    2aiii. same number except for extra 0: .005,.05
    2b. same number of leading zeros:
    2bi. Shorter number is larger: .08,.0153; .5,.415
    2bii. Longer number is larger: .0623,.034; .316,.22
    2c. truncated are equivalent: .234,.23
    2d. equivalent with trailing zero: .34,.340

    Then repeat that whole mess but adding a whole number in front of the decimal point. Anything that could make it even better?

  5. It’s worth noting that, as I understand the KA mission, learning from KA exercises alone is *not* ideal—the tool supports entirely self-driven learning, but isn’t designed for it. Instead, under the ideal model, the teacher/parent/coach/etc. is able to diagnose a struggling student’s misconceptions as they arise and help resolve those particular mistakes better than any automated process can.

    However, it sounds like even the ideal model fails when the tools misidentify a confused student as a successful student, since the teacher won’t know to step in. That’s a very strong criticism of this particular exercise, and, as KA hires even more content creators, I think it’d be wise to explicitly identify many common misconceptions for each concept and confirm that the exercise properly filters students who manage to learn some warped version of the actual concept. This new decimals exercise arguably skews too strongly in favor of resolving *one* particular misconception, but is definitely a step in the right direction—and, as I understand it, creators of manually curated exercises now have some basic tools to help them identify common errors and provide specific guidance to those users, which isn’t a perfect solution (a human will always be better!) but oughta help resolve some issues without demanding the human’s time and energy, which oughta especially help those without the benefit of a human teacher.

    I’m curious about the focus on the decimals material in particular. What are some other KA materials that could pay more attention to ways students often misunderstand the concept? Especially if it were phrased constructively rather than derogatorily, I’m sure the KA team would love to continue to receive this kind of detailed feedback—or, if the primary message *is* that all of KA is broken, I’m interested in hearing more about that, too. Is KA too broken to fix and should just be scrapped, is the idea of online materials fundamentally broken to begin with, or should we instead work to improve the existing KA resources?

  6. Actually, thanks to Kevin Hall for the find, but that’s okay 🙂 (See comments in your last blog post).

    And the Carnegie Mellon article I linked to also includes a reference list with a couple journal articles describing decimal misconceptions. But if you want a more complete list (and Ben Alpert, this may be of use to you), here it is:

    (Scroll down to “math education – learning of decimals”)

    I’d like to point out that whether you think KA is good now or not, it’s clearly getting better quickly. Whatever problems you have with the site now, I bet you won’t have them in 3 years. A lot of the difficulty they have had comes from issues of scale. Rather than working on their content up to now, they’ve been working on their infrastructure…for example, recoding their site so it can be easily translated between languages without having to be completely reprogrammed. I think it makes sense for them to work on these infrastructure problems first and then address content. But I agree they have over-hyped their product for now.

  7. Glad to find others critical of Khan Academy. I represent an interesting perspective as a youtube content creator myself. The goal of my videos are always to support what I do in the classroom through providing the “rewind” option on the theoretical discussions and then provide examples using my language and techniques.

    It is nearly impossible to recreate the teacher/student interchanges that occur in a well designed class via video. However, I do agree with one of the commenter’s that it is essential for a content creator to have significant classroom experience to be able to anticipate and point to possible misconceptions and mistakes common to students.

    I am glad to have stumbled onto your blog and look forward to reading more.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s