Funded by the US. Department of Education’s Institute of Education Sciences under the Effective Mathematics Education Research Grants proposal program for $1.4 million over 4 years starting in 2003
Neil T. Heffernan of Worcester Polytechnic University
Kenneth R. Koedinger and Brian Junker of Carnegie Mellon University
Steven Ritter, Carnegie Learning Incorporated
The movement towards high stakes testing promises to encourage rigor and accountability in middle school mathematics, but there is a danger that a too-narrow focus on testing will take time and attention away from mathematics instruction. The fundamental dilemma that teachers face in trying to use assessment to guide instruction (i.e., to figure out what is the next best thing for a student to try to learn) is that because assessment takes time away from instruction, how can teachers be sure the time spent assessing will improve instruction enough to justify the cost of lost instructional time. We propose to address this dilemma by building and experimentally evaluating the effectiveness of a web-based "Assistment" system for middle school mathematics. This system will 1) quickly predict a student’s score on a standard-based test, 2) provide feedback to teachers about how they can specifically adapt their instruction to address student knowledge gaps, and 3) unlike other assessments system, provide an opportunity for students to get intelligent tutoring assistance at they same time as reliable assessment data is being collected behind the scenes. This system will result from the collaboration of leading experts in cognitive psychology, psychometrics, and educational technology and will address the formative assessment dilemma through a unique combination of innovative features. First, by providing instructional assistance during assessment, teachers can justify having students spend time every week using the Assistment system. Second, by collecting continuous metrics of student understanding and performance, like how much help the student needs, Assistments provide more information per problem than the discrete right-or-wrong metrics of current paper, and on-line, assessments. Third, by providing scaffolding questions within test items, Assistments assess individual components of knowledge, not just overall performance. At the same time, they provide more focused instruction than feedback given by on-line multiple-choice systems. Thus, instead of getting only a single bit of information per item, we get a couple of bytes of information per item. We argue that such Assistments should particularly address the needs of the minority and low social-economic status students that we proposed to work with as the learn middle school algebra. We also present a series of the experiments to investigate the effectiveness of this intervention in leading towards higher student achievement.
KENNETH R. KOEDINGER
A. Professional Preparation
Ph. D. Cognitive Psychology. Carnegie Mellon University. December, 1990.
M.S. Computer Science. University of Wisconsin, Madison. May, 1986.
B.S. Math and Computer Science. University of Wisconsin, Madison. With distinction. May, 1984.
B. Recent Appointments
Associate Professor with Tenure. Human-Computer Interaction Institute. School of Computer Science, Carnegie Mellon University. 2001 to present.
Senior Research Scientist. Faculty position equivalent to Associate Professor. Human-Computer Interaction Institute. School of Computer Science, Carnegie Mellon University. 1999 to 2001.
Co-Founder, Board of Directors, and Consultant. Carnegie Learning, Inc. 1998 to present.
C. Career Summary
My
multi-disciplinary preparation has been critical to my research goal of
creating educational technologies that dramatically increase student
achievement. Toward this goal, I create
"cognitive models", computer simulations of student thinking and
learning, that are used to guide the design of educational materials, practices
and technologies. These cognitive
models provide the basis for an approach to educational technology called
"Cognitive Tutors" in which we create rich problem solving
environments for students to work in and provide just-in-time learning
assistance much like a good human tutor does.
I have developed Cognitive Tutors for mathematics and science and have
tested them in the laboratory and the classroom. In a whole-year classroom study with our Algebra Cognitive Tutor,
I have shown that students in our experimental classrooms outperformed students
in control classes by 50-100% on targeted real world problem solving skills and
by 10-25% on standardized tests. My
research has contributed new principles and techniques for the design of
educational software and has produced basic cognitive science research results
on the nature of human thinking and learning. I have
authored 47 peer-reviewed publications, 2 textbooks, 6 book chapters,
and 41 other papers and have been a Project
Investigator on 15 major grants.
I currently codirect the Pittsburgh Advanced Cognitive Tutor Center and
am managing teams of cognitive scientists, programmers, and teachers to create
integrated learning solutions that include text materials, teacher training and
Cognitive Tutors. I am a co-founder of
Carnegie Learning Inc., a company marketing these technology-enhanced learning
solutions to schools and colleges across the country.
D. Pertinent Publications
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4 (2), 167-207.
Aleven,
V.A.W.M.M., & Koedinger, K. R. (2002). An effective metacognitive strategy:
Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2).
Corbett, A. T., Koedinger, K. R., & Hadley, W. H. (2001). Cognitive Tutors: From the research classroom to all classrooms. In Goodman, P. S. (Ed.) Technology Enhanced Learning: Opportunities for Change. Mahwah, NJ: Lawrence Erlbaum Associates.
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
Koedinger,
K. R., Suthers, D. D., & Forbus, K. D. (1999). Component-based construction of a science learning space. International
Journal of Artificial Intelligence in Education, 10.
Ritter, S. & Koedinger, K. R. (1996). An architecture for plug-in tutoring agents. In Journal of Artificial Intelligence in Education, 7 (3/4), 315-347. Charlottesville, VA: Association for the Advancement of Computing in Education.
Neil T. Heffernan
A. Professional Preparation
Ph. D. Computer Science. Carnegie Mellon University. 2001
M.S. Computer Science. Carnegie Mellon University. 1997
B.A. Computer Science and History. Amherst College. Summa cum laude. 1993.
B. Recent Appointments
Assistant Professor. Worcester Polytechnic Institute. Department of Computer Science. July, 2002 to present.
Post-Doctoral Researcher. Human-Computer Interaction Institute. School of Computer Science, Carnegie Mellon University. 2001to 2002.
C. Career Summary
Dr. Neil Heffernan graduated summa cum laude from
Amherst College in History and Computer Science. Neil taught mathematics and science to eighth grade students as
part of Teach for America, a program
that selectively recruits top candidates to teach in inner-city schools, Neil
attended Carnegie Mellon University's Computer Science department to do
research in creating educational software that leads to higher student
achievement. For his dissertation, Neil
built the first intelligent tutoring
system that incorporated a model of tutorial dialog. This system has been shown to lead to higher
student learning, by getting students to think more deeply about problems. It is based upon detailed studies of student
learning as well as studies of experienced human teachers. The system (free at www.AlgerbaTutor.org) is
the most widely used web-based intelligent tutoring system. It has been used by thousands of students
and teachers and has been awarded many educational awards. Carnegie Mellon has applied for a patent for
this unique web-based tutor. As a
post-doc, Neil managed a team of four programmers and PhDs to create authoring
tools to make it easier to build intelligent tutoring systems. The first version of these tools was
successful used in the “CIRCLE 2002
Summer Schools for Building Intelligent Tutoring Systems” where researchers
from around the world came to learn how to build intelligent tutoring
systems. Neil is a Spencer Foundation /
National Academy of Education Postdoctoral Research Fellow. Neil is now an assistant professor at
Worcester Polytechnic Institute, where one of his projects is organizing
"The Learning Open" (www.LearningOpen.org), an interdisciplinary
collaboration between educational software researchers and classroom teachers
to study what are the benefits of different instructional approaches.
D. Pertinent Publications
Heffernan, N. T. (accepted 2003) Web-Based Evaluations Showing both Cognitive and Motivational Benefits of the Ms. Lindquist Tutor 11th International Conference Artificial Intelligence in Education
Heffernan, N. T., & Koedinger, K. R. (2002) An Intelligent Tutoring System Incorporating a Model of an Experienced Human Tutor. In the Proceedings of the Sixth International Conference on Intelligent Tutoring System 2002. Biarritz, France.
Heffernan, N. T., & Koedinger, K. R. (2001) Results from a Web-Based Tutor for Writing Algebra Expressions for Word-Problems. Sciences et Techniques Educatives. A French journal named "Educational Sciences and Technology."
Heffernan, N. T (2001) Intelligent Tutoring Systems are Forgotten the Tutor: Adding a Cognitive Model of Human Tutors. Dissertation. Computer Science Department, School of Computer Science, Carnegie Mellon University. Technical Report CMU-CS-01-127.
Heffernan, N. T., & Koedinger, K. R. (2000) Intelligent Tutoring Systems are Missing the Tutor: Building a More Strategic Dialog-Based Tutor. AAAI Fall Symposium on Building Dialogue Systems for Tutorial Applications.
Heffernan, N. T. & Koedinger, K. R. (1998). A developmental model for algebra symbolization: The results of a difficulty factors assessment. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society, (pp. 484-489). Hillsdale, NJ: Erlbaum
Heffernan, N. T. & Koedinger, K.R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension. In Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum. [The Marr Prize winner for best student paper.]
brian W. Junker
A. Professional Preparation
Ph.D., Statistics, University of Illinois, 1988.
M.S., Mathematics, University of Illinois, 1986.
B.A., Mathematics, University of Minnesota, cum laude,1980.
B. Recent Appointments
Professor of Statistics, Carnegie Mellon University, January
2001—present (Associate, 1996—2001; Assistant, 1991—1996).
Visiting Research Scientist, Learning Research and Development Center, University of Pittsburgh, September 1999—July 2000.
Postdoctoral Fellow in Psychiatric Statistics, Carnegie Mellon University, 1990—1991.
Visiting Assistant Professor of Statistics, University of Illinois, 1988—1990.
C. Career Summary
BRIAN W. JUNKER
is Professor of Statistics, Carnegie Mellon University. Dr. Junker received his
Ph.D. in statistics from the University of Illinois in 1988. He came to
Carnegie Mellon University in 1990, where he began as a postdoctoral fellow in
the Program in Psychiatric Statistics, jointly operated by the Department of
Statistics at Carnegie Mellon and Western Psychiatric Institutes and Clinic,
University of Pittsburgh. During the
1999-2000 academic year, he as a Visiting Research Scientist at the Learning
Research and Development Center, University of Pittsburgh. His research
interests include the statistical foundations of latent variable models for
measurement, as well as applications of latent variable modeling in the design
and analysis of standardized tests, small-scale experiments in psychology and
psychiatry, and large scale educational surveys such as the National Assessment
of Educational Progress. Dr. Junker is a Fellow of the Institute of
Mathematical Statistics, a member of the Board of Trustees and the Editorial
Council of the Psychometric Society a associate editor and editor-elect of Psychometrika. He also served on the National Research Council (NRC) Committee
on Embedding Common Test Items in State and District Assessments, and served as
a consultant to the NRC Committee on the Foundations of Educational and
Psychological Assessment. He is
currently a member of the Design and Analysis Committee for the National
Assessment of Educational Progress.
D. Pertinent Publications
Baxter, G. and Junker, B. W. (2001). Designing Developmental Assessments: A Case Study in Proportional Reasoning. Paper presented at the Annual Meeting of the National Council of Measurement in Education, April 2001, Seattle, WA. See also http://www.stat.cmu.edu/ ~brian/rpm/.
Junker, B. W. (2001). On the interplay between nonparametric and parametric IRT, with some thoughts about the future. In Boomsma, A., Van Duijn, M. A. J. and Snijders, T. A. B. (Eds.) (2001). Essays on item response theory, pp. 247—276. New York: Springer-Verlag.
Junker, B. W. and Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258—272.
Junker, B. W., Koedinger, K. R. and Trottini, M. (July 2000). Finding improvements in student models for intelligent tutoring systems via variable selection for a linear logistic test model. Presented at the Annual North American Meeting of the Psychometric Society, Vancouver, BC, Canada.
Junker, B. W. and Sijtsma, K. (2000). Latent and manifest monotonicity in item response models. Applied Psychological Measurement, 24, 65—81.
Junker, B. W. (1999). Some statistical models and computational methods that may be useful for cognitively-relevant assessment. Prepared for the Committee on the Foundations of Assessment, National Research Council. (See http://www.stat.cmu.edu/~brian/nrc/cfa/.) Source material for Chapt. 4 of Pellegrino, J., Chudowsky, N., and Glaser, R. (eds.) (2001). Knowing what students know: the science and design of educational assessment. National Academy Press, Washington, DC.
Steven B. Ritter
A. Professional Preparation
Ph.D., Psychology, Carnegie Mellon University, 1992.
Sc.B., Cognitive Science, Brown University, 1985.
B. Recent Appointments
Senior Cognitive Scientist, Carnegie Learning, 1998—present
Postdoctoral Associate, Carnegie Mellon University, 1993—1998.
C. Career Summary
STEVEN
B. RITTER is Senior Cognitive Scientist at Carnegie Learning. Dr. Ritter
received his Ph.D. in psychology from the Carnegie Mellon University in 1992,
working with Brian MacWhinney. Starting in 1993, he began postdoctoral research
with John Anderson at Carnegie Mellon University, where he was instrumental in
developing and evaluating the intelligent tutoring systems that became the
basis for Carnegie Learning’s products. In 1998, he was one of the co-founders
of Carnegie Learning. He is the author of numerous papers on the design,
architecture and evaluation of Intelligent Tutoring Systems and served as
chairman of the IEEE Learning Technology Standards Committee working group on
tool/agent communication. Dr. Ritter guided the development of the Problem
Situation Authoring Tool (pSAT), which is an intelligent authoring environment
for encoding word problems in the Cognitive Tutor for Algebra. He has also been
responsible for some of the earliest and most complete web-based intelligent
tutoring systems. Dr. Ritter is currently directing Carnegie Learning’s
Teaching Practices study, which is a project aimed at using video protocol
analysis to understand how teachers are implementing the Cognitive Tutor
software and curriculum in different environments. The study is especially
focusing on ways that teaching pedagogy changes in teachers’ first, second and
third years implementing it.
D. Pertinent Publications
Ritter (1997). PAT Online: A Model-tracing tutor on the World-wide Web. In P. Brusilovsky, K. Nakabayashi & S. Ritter (Eds.), Proceedings of the Workshop on Intelligent Educational Systems on the World Wide Web, Kobe Japan.
Ritter, S. and Anderson, J. R. (1995). Calculation and strategy in the equation solving tutor. In J.D. Moore & J.F. Lehman (Eds.),Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society. (pp. 413-418). Hillsdale, NJ: Erlbaum.
Ritter, S., Anderson, J., Cytrynowicz, M., and Medvedeva, O. (1998) Authoring Content in the PAT Algebra Tutor. Journal of Interactive Media in Education, 98 (9) [www-jime.open.ac.uk/98/9]
Ritter, S. and Blessing, S. B. (1998). Authoring tools for component-based learning environments. Journal of the Learning Sciences, 7(1), 107-131.
Ritter, S. and Blessing, S. B. (1996). A programming-by-demonstration tool for retargeting instructional systems. In D.C. Edelson and E.A. Domeshek (Eds.), Proceedings of the Second International Conference on the Learning Sciences, (pp. 292-299). Charlottesville, VA: Association for the Advancement of Computing in Education.
Ritter, S. and Koedinger, K. R. (1995). Towards lightweight tutoring agents. In Proceedings of the Seventh World Conference on Artificial Intelligence in Education (pp. 91-98). Charlottesville, VA: Association for the Advancement of Computing in Education.