Evaluation at CTL
Innovations in teaching and learning call for innovations in evaluation. CTL's interdisciplinary evaluation teams have a reputation for applying cutting-edge approaches to studying both early-stage and mature reform initiatives in schools and community settings, providing clients data that clients can use to improve their program designs and analyze implementation and impact. CTL's researchers are experienced in conducting mixed-method evaluation studies and in using diverse study designs, including quasi-experimental and experimental outcome studies. CTL's approach is unique among providers of evaluation services in that:
We work in partnership with clients to develop a shared understanding of their initiative that drives an evaluation design and analyses that are customized to its government, commercial, and foundation clients’ needs.
We focus on learning, drawing from learning sciences research and from CTL's own design projects to investigate how students and teachers learn with technology and to help clients improve their initiative's potential to support learning.
We use a principled approach to assessment design in our evaluation studies, assuring that measures of program impact are valid measures of the important knowledge and skills an initiative is designed to improve.
We study real classrooms in their systemic context, paying attention to participants' perspectives on implementation and analyzing the social, institutional, and political factors that make technology integration possible and those that make it challenging.
We are advancing the field of evaluation in educational technology by developing tools other researchers and evaluators can use, developing new methods in evaluation, and publishing our findings.
People
Current Projects
- Catalyzing Network Expertise
- College Success Foundation (CSF) evaluations
- Contingent Pedagogies for Conceptual Teaching and Learning
- Cyber-Enabled Design Research to Enhance Teachers' Critical Thinking Using a Major Video Collection on Children's Mathematical Reasoning
- Evaluation of Kids’ Survey Network
- Evaluation of the Microsoft Innovative Schools Program
- Evaluation of the Ready to Learn Initiative
- InnovaTE3 Girls Innovating with Technology as Entrepreneurial Environmental Engineers
- Intel® Getting Started Course Program Evaluation
- Intel® Teach Essential Online Course Global Program Evaluation
- Intel® Learn Program Evaluation
- Knowledge Management for Program Improvement: Evaluation of the San Diego County Office of Education's Knowledge Management Systems
- Measuring Learning: A guidebook for gathering and interpreting evidence
- National Technology Activities Task Order
- One Economy's AccessAll Project
- Scenario-Based Learning
- Study of Data-Driven Decision Making
- SunBay Digital Mathematics
- Teacher Learning of Technology-Enhanced Formative Assessment (TEFA-TL)
- The Universally Designed Science Notebook: An Intervention to Support Science Learning for Students with Disabilities
- Thinking With Data
- Universal Design of Inquiry-based Middle and High School Science Curricula
- Wireless Internet Learning Devices (WILD)
Selected Publications
-
Dieterle, E. (in preparation). Science video games. In A. Hirumi (Ed.), Games in science education. Eugene, OR: International Society for Technology in Education.
Read more
-
Hilberg, S., Joshi, A., & House, A. (Forthcoming, 2009). "Washington State Achievers Program: Influence of the Scholarship Program on Low-Income College students' Achievement and Aspirations." Journal of College Student
Retention: Research, Theory & Practice.
Read more
-
Yarnall, L., Vahey, P. and Swan, K. (in preparation). Designing Geospatial Exploration Activities to Build Hydrology Understanding in Middle School Students. To appear in Teaching Science and Investigating Environmental Issues with Geospatial Technology: Designing Effective Professional Development for Teachers.
Read more
-
Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing Data-Informed Decision Making in Schools--Teacher Access, Supports and Use. Report prepared for U.S. Department of Education, Office of Planning, Evaluation and Policy Development. Prepared by SRI International, Menlo Park, CA.
Read more
-
Means, B., Toyama,Y., Murphy. R., Bakia, M., and Jones, K. (2009). Evaluation of Evidence-based Practices in Online Learning: A Meta-analysis and Review of Online-learning Studies. Washington, D.C.: U.S. Department of Education.
Read more
-
Penuel, W. R., Fishman, B. J., Gallagher, L. P., Korbak, C., & Lopez-Prado, B. (2009). Is alignment enough? Investigating the effects of state policies and professional development on science curriculum implementation. Science Education. 93(4), pp. 656-677.
Read more
-
Penuel, W. R., McWilliams, H., McAuliffe, C., Benbow, A., Mably, C., & Hayden, M. M. (2009). Teaching for Understanding in Earth Science:
Comparing Impacts on Planning and Instruction
in Three Professional Development Designs for Middle
School Science Teachers. Journal of Science Teacher Education. 20:415–436.
Read more
-
Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuel, W. R., Nussbaum, M., Claro, S. (2009, October). Scaffolding group explanation and feedback with handheld technology: impact on students’ mathematics learning. Educational Technology Research and Development, doi: 10.1007/s11423-009-9142-9.
Read more
-
Swan, K., Vahey, P. Kratcoski, A., van ‘t Hooft, M., Rafanan, K., and Stanford, T. (2009). Challenges to Cross-Disciplinary Curricula: Data Literacy and Divergent Disciplinary Perspectives. Presented at the Annual Conference of the American Educational Research Association, April 2009, San Diego, CA
Read more
-
Vahey, P., Lara-Meloy, T., Knudsen, J. (2009). Meeting the needs of diverse student populations: Findings from the Scaling Up SimCalc project. In Swars, S. L., Stinson, D. W., & Lemons-Smith, S. (Eds.). Proceedings of the 31st annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. Atlanta, GA: Georgia State University. pp. 416-424.
Read more
-
Crawford, V. M., Schlager, M., Penuel, W. R., & Toyama, Y. (2008). Supporting the art of teaching in a data-rich, high performance learning environment. In E. B. Mandinach & M. Honey (Eds.), Linking data and learning (pp. 109-129). New York: Teachers College Press.
Read more
-
Denner, J., Kareem, Z., Koch, M., & Peterson, K. (2008 March,). Girls and information technology: Innovative approaches to narrowing the gender gap. The Build IT portion of the presentation at AERA 2008. New York City, NY.
Read more
-
Dieterle, E., & Clarke, J. (2008). Multi-user virtual environments for teaching and learning. In M. Pagani (Ed.), Encyclopedia of multimedia technology and networking (Second ed.). Hershey, PA: Idea Group, Inc.
Read more
-
Gallager, L., Means, B., & Padilla, C. (2008). Teachers' Use of Student Data Systems to Improve Instruction: 2005 to 2007. Report prepared for U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. Prepared by SRI International, Menlo Park, CA.
Read more
-
House, A., Tirumala-Bhanot, R., & Michalchik, V. S. (2008). Intel® Teach Essentials Online: Global Pilot Evaluation Summary. Menlo Park, CA: SRI International.
Read more
-
House, A., Tirumala-Bhanot, R., Lundh, P., & Michalchik, V. S. (2008). Intel® Teach Essentials Online: U.S. Formative Evaluation. Menlo Park, CA: SRI International.
Read more
-
Koch, M. "Build IT Poster." (2008 February). ITEST Poster Session. Presented at NSF's ITEST PI Meeting. Washington, DC.
Read more
-
Koch, M. (2008 February). Build IT data summary. Presented at NSF’s ITEST PI Meeting, Washington, DC.
Read more
-
Koch, M., & Gorges, T. (2008 February). Scaling Build IT. Presented at NSF’s ITEST PI Meeting, Washington, DC.
Read more
-
Means, B. (2008). Levers for Improving Mathematics Learning in High-Poverty High Schools. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., & Gallagher, L. P. (2008). Comparing three approaches to preparing teachers to teach for deep understanding in Earth science: Short-term impacts on teachers and teaching practice. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., McWilliams, H., McAuliffe, C., Benbow, A., Mably, C., & Hayden, M. M. (2008). Preparing teachers to teach for deep understanding: A curriculum-based approach. The Earth Scientist XXVII (2), 21-24.
Read more
-
Roschelle, J., Tatar, D., Shechtman, N., & Knudsen, J. (2008). The role of scaling up research in designing for and evaluating robustness. Educational Studies in Mathematics. 68(2).
Read more
-
Shear, L. & Gorges, T. (2008, March). Knowledge management for Program Improvement: Developing and leveraging capacity at the San Diego County Office of Education. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY.
Read more
-
Shear, L., Means, B., Mitchell, K., House, A., Gorges, T., Joshi, A., Smerdon, B., & Shkolnik, J. (2008) "Contrasting Paths to Small School Reform: Results of a 5-year Evaluation of the Bill & Melinda Gates Foundation’s National High Schools Initiative." Teachers College Record, 110(9), p. 1986-2039.
Read more
-
Vahey, P., Knudsen, K. (2008). Can a Technology-enhanced Curriculum Improve Student Learning of Important Mathematics? Presented at the National Council of Supervisors of Mathematics. April 2008, Salt Lake City, UT.
Read more
-
Vahey, P., Lara-Maloy, T. (2008). Using a Technology-Enhanced Curriculum to Improve the Learning of Important Mathematics for English Language Learners. Invited presentation to the 2008 CREATE conference, October 2008, Minneapolis, MN.
Read more
-
Vahey, P., Lara-Meloy, T. (2008). Supporting All Students in Learning Advanced Math: Findings From a Technology-enhanced Curriculum Study. Presented at the 40th Annual National Council of Supervisors of Mathematics Conference. April 2008, San Jose, CA
Read more
-
Bryan, M., Nemes, O., & Koch, M. (2007 February). Girls in Build IT. Presented in Reaching and Teaching the New Learner at NSF’s ITEST PI Meeting. Washington, DC.
Read more
-
Courneya, J., & Koch, M. (2007 February). Sharing best practices in youth development. Presented at NSF’s ITEST PI Meeting. Washington, DC.
Read more
-
Crawford, V. M. (2007, June). Creating a powerful learning environment with networked mobile learning devices. Educational Technology Magazine: The Magazine for Managers of Change in Education, 47(3), 47-50
Read more
-
Dieterle, E., Dede, C., & Schrier, K. (2007). "Neomillennial" learning styles propagated by wireless handheld devices. In M. Lytras & A. Naeve (Eds.), Ubiquitous and pervasive knowledge and learning management: Semantics, social networking and new media to their full potential (pp. 35–66). Hershey, PA: Idea Group, Inc.
Read more
-
Gallager, L., Means, B., & Padilla, C. (2007). Teachers' Use of Student Data Systems to Improve Instruction. Report prepared for U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. Prepared by SRI International, Menlo Park, CA.
Read more
-
House, A., Tirumala-Bhanot, R., Michalchik, V. S., & Lundh, P. (2007). Intel® Teach Essentials Online: Interim Report for U.S. Formative Evaluation. Menlo Park, CA: SRI International.
Read more
-
Joshi, A., Hilberg, S., & House, A. (2007). Washington State Achievers Program: Influence of the scholarship program on low-income college students’ achievement and aspirations. Paper presented at the American Educational Research Association, 2007 Annual Meeting, Chicago, IL.
Read more
-
Koch, M. & Penuel, W.R. (2007 April). Designing for design learning. In CHI 2007 Conference Proceedings. Workshop on Converging on a Science of Design through the Synthesis of Design Methodologies. Reading, MA: ACM Press.
Read more
-
Koch, M. et al. (August, 2007). Build IT: Girls Developing Information Technology Fluency through Design. Annual Report Year 2. Menlo Park, CA: SRI International.
Read more
-
Koch, M., & Nemes, O. (2007 October). Introduction to Build IT workshop. Presented at Girls Incorporated Region III Conference 2007. Providence, Rhode Island.
Read more
-
Michalchik, V., Sussex, W., & Gorges, T. (2007). Intel® Learn Program evaluation findings. Menlo Park, CA: SRI International.
Read more
-
Mitchell, K., Bakia, M., Chen, E., & Yang, E. (2007). Local Strategies and Practices for Using Technology in K-12 Education. Menlo Park, CA: SRI International.
Read more
-
Pasnik, S., Strother, S., Schindel, J., Penuel, W. R., & Llorente, C. (2007). Report to the Ready To Learn initiative: Review of research on media and young children's literacy. New York, NY and Menlo Park, CA: Education Development Center and SRI International.
Read more
-
Penuel, W. R., Korbak, C., Sussex, W., & Frank, K. A. (2007). Catalyzing Network Expertise: Year 1 Report. Menlo Park, CA: SRI International.
Read more
-
Penuel, W.R., & Riel, M. (2007). The "new" science of networks and the challenge of school change. Phi Delta Kappan. 88(8). pp. 611-615.
Read more
-
Penuel, W.R., Boscardin, C.K., Masyn, K., Crawford, V., (2007). Teaching with student response systems in elementary and secondary education settings: A survey study. Educational Technology Research and Development, 55, 315-346.
Read more
-
Remold, J. (2007). Educational Technologies and Social Inequality in Brazilian Public Schools. Rio de Janeiro; The Edelstein Center Editions. (also printed in Portuguese with the title Desigualdade Social e as novas tecnologias no sistema escolar)
Read more
-
SRI International. (2007). "I think it makes me not give up on dreams. What students say about HERO." Interim Report for College Success Foundation. Menlo Park, CA: SRI International.
Read more
-
Yarnall, L., Toyama, Y., Gong, B., Ayers, C., & Ostrander, J. (2007). Adapting scenario-based curriculum materials to community college technical courses. Community College Journal of Research and Practice, 31(7), 583-601.
Read more
-
Dieterle, E., & Dede, C. (2006). Building university faculty and student capacity to use wireless handheld devices for learning. In M. van ‘t Hooft & K. Swan (Eds.), Ubiquitous computing in education: Invisible technology, visible impact (pp. 303–328). Mahwah, NJ: Lawrence Erlbaum Associates.
Read more
-
Hilberg, S., Joshi, A., & Means, B. (2006). Washington State Achievers Scholarship Program Case Study: Final Report. Presented by invitation to The Bill and Melinda Gates Foundation, Research Advisory Council, Seattle, WA.
Read more
-
House, A., Shear, L. F., Cole, S., Wolman, M., Joshi, A., & Uekawa, K. (2006). Findings from the Evaluation of the Bill and Melinda Gates Foundation's National School District and Network Grants Program: The Challenge of Sustaining High-School Reform. Paper presented at the American Educational Research Association, 2006 Annual Meeting, San Francisco, CA.
Read more
-
Koch, M. (2006) Build IT curriculum. Featured in Consumers Guide to After School Science Resources at http://www.sedl.org/cgi-bin/mysql/afterschool/science.cgi?location=search&show_resource_id=45
Read more
-
Koch, M. (April, 2006). Build IT: Girls Building Information Technology Fluency Through Design In M. Koch, J. Denner, D. Emery, & R. London (presenters) BayCHI-Kids Event: Developing girls' technology fluency. Menlo Park, CA: SRI International
Read more
-
Koch, M. (October, 2006) Infusing IT/STEM skill standards in informal ITEST learning ITEST Webcast
Read more
-
Koch, M. et al. (August, 2006). Build IT: Girls Developing Information Technology Fluency through Design. Annual Report Year 1. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R. (2006). Report on professional development plans and materials for the TIDES Project. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., Bienkowski, M., Gallagher, L., Korbak, C., Sussex, W., Yamaguchi, R., Fishman, B. J. (2006). GLOBE Year 10 evaluation: Into the next generation. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., Gallagher, L., & Kreikmeier, P. (2006). TIDES: Report on technical quality of 7th grade student assessment. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., Gallagher, L., & Kreikemeier, P. (2006). Report on technical quality of 6th and 8th grade student assessments for the TIDES Project. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., Kreikmeier, P, & Venezky, D. (2006). Mapping standards to big ideas: A task for assessing teachers’ comprehension of curricular purposes in Earth science. Menlo Park, CA: SRI International.
Read more
-
SRI International. (2006). Review of Evaluation Findings for the Intel® Learn Program. Menlo Park, CA: SRI International.
Read more
-
Chen, E., Heritage, H. M., & Lee, J. J. (2005). Identifying and Monitoring Students' Learning Needs with Technology. The Journal of Education for Students Placed At Risk, 10(3), 309-332.
Read more
-
Chen, E., Howie, J., Leon, S., & O'Neil, (2005). The Effects of Feedback on Performance on a Computer Negotiation Task. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Canada.
Read more
-
Heritage, H.M., & Chen, E. (2005). Why data skills matter in school improvement. Phi Delta Kappan, 86(9), 707-710.
Read more
-
Heritage, M., Lee, J., Chen, E., & La Torre, D. (2005). Upgrading American's use of Information to Improve Student Performance. CSE Report 661.
Read more
-
Mansori, S., Breeden, L., Dreier A., Llorente, C., & Weagle V. (2005). The U.S. Department of Education community technology centers program: Analysis of Grant –funded Projects 1999-2002. Newton, MA: EDC.
Read more
-
Penuel, W.R., Fishman, B.J. GLOBE Teacher Survey on Professional Development (2005). Menlo Park, CA: SRI International.
Read more
-
Chen, E., Heritage, H.M., Baker, E., & Lee, J.J. (2004). Evaluating the Web-Based Quality School Portfolio/Research Report. QSP Year Two Report submitted to the U.S. Department of Education.
Read more
-
Means, B., Murphy, R., Javitz, H., Haertel, G., & Toyama, Y. (2004). Design Considerations for Evaluating the Effectiveness of Technology-Related Teacher Professional Development. Menlo Park, CA: SRI International.
Read more
-
Penuel, W. R., & Means, B. (2004). Implementation variation and fidelity in an inquiry science program: An analysis of GLOBE data reporting patterns. Journal of Research in Science Teaching, 41(3), 294-315.
Read more
-
American Institutes for Research and SRI International. (2003). High time for high school reform: Early findings from the evaluation of the National School District and Network Grants Program. Washington, DC: Authors.
Read more
-
Chen, E., Aten, H., Nambiar, V., & Howie, J. (2003). The Impact of the KidZLit Program on Elementary School Students. Paper presented at the annual meeting of the American Educational Research Association, Chicago, Illinois.
Read more
-
Chen, E., Burnam, B., Aten, H., Howie, J., & Nambiar, V. (2003). Predicting Teachers' Computer Use in the Classroom. CSE technical report.
Read more
-
Haertel, G. D., & Means, B. (2003). Evaluating educational technology: Effective research designs for improving learning. New York: Teachers College Press.
Read more
-
Hoadley, C., Foley, B., Agarwal, R., Chorost, M., Gaffney, and T., Lewis, S. et al. (2003). Collaborative learning in immersive audio engineering: Evaluation report for Year II. University Park , Pennsylvania: Penn State University.
Read more
-
Shear, L., & Smerdon, B. (2003, April). Mapping the terrain: Year 1 of the evaluation of the Bill & Melinda Gates Foundation's National School District and Network Grants Program. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Read more
-
Smerdon, B., Garet, M., O'Day, J., Miller, L., Means, B., & Mitchell, K. (2003, April). Charting a course: Evaluation design of the National School District and Network Grants Program. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Read more
-
Yarnall, L., Penuel, W. R., Ravitz, J., Murray, G., Means, B., & Broom, M. (2003). Portable assessment authoring: Using handheld technology to assess collaborative inquiry. Education, Communication, Information, 3(1), 7-55.
Read more
-
Chen, E., Burnam, B., Aten, H., Howie, J. & Nambiar, V. (2002). Implementation of a Problem Solving Software in High School Science Classrooms. Paper presented at the annual meeting of the American Educational Research Association. New Orleans, Louisianna.
Read more
-
Crawford, V. M., & Toyama, Y. (2002). WorldWatcher Looking At The Environment Curriculum: Final External Evaluation Report. Menlo Park, CA: SRI International.
Read more
-
Hoadley, C., Gaffney, T., Lewis, S., and Foley, B. (2001). Collaborative learning in immersive audio engineering: Evaluation report for year I. Menlo Park , California: SRI International.
Read more
-
Shear, L., & Penuel, W. R. (2002). Putting the ‘learning’ in adventure learning: Design principles for technology-supported classroom inquiry. Journal of Curriculum Supervision, 17, 315-335.
Read more
-
Slotta, J. D., Clark, D., & Cheng, B. H. (2002). Integrating Palm Technology into WISE Inquiry Curriculum: Two School District Partnerships. In Proceedings of CSCL ‘02 Computer-Supported Collaborative Learning. Mahwah, NJ: Lawrence Erlbaum.
Read more
-
Vahey, P. & Crawford, V. (2002). Palm Education Pioneers Program Final Evaluation Report. Menlo Park, CA: SRI International.
Read more
-
Means, B., & Haertel, G. (2001). Technology supports for assessing science inquiry. Paper commissioned by the National Academies, Board on Testing and Assessment. To appear in Proceedings of the Workshop on Technology and Assessment.
Read more
-
Means, B., Penuel, W. R., & Padilla, C. (2001). The connected school: Technology and learning in high school. San Francisco: Jossey-Bass.
Read more
-
Murphy, R., Penuel, W., Means, B., Korbak, C., Whaley, A. (2001). E-DESK: A Review of Recent Evidence on the Effectiveness of Discrete Educational Software. Menlo Park, CA: SRI International.
Read more
-
Coleman, E. B., & Penuel, W. R. (2000). Web-based student assessment for program evaluation. Journal of Science Education and Technology, 9(4), 327-342.
Read more
More...
Related Keywords

