�����Ƶ

In the late 2000s, Association of American Universities (AAU) staff recognized that its member institutions were vulnerable to criticisms concerning the quality of undergraduate STEM (science, technology, engineering, and mathematics) teaching, learning, and retention, such as those raised in the 1998 Boyer Commission Report on educating undergraduates in the research university.1 At the same time, the rise of MOOCs (massive open online courses), growing calls for higher education accountability at the state and federal level, and mounting pressures to justify the cost and value of an undergraduate degree at a research university were topics of growing discussion among the AAU membership. Following reports such as Rising Above the Gathering Storm by the National �����Ƶ of Sciences, a significant degree of national attention was also placed on the need to improve STEM education to ensure that an adequate pipeline of domestic STEM talent was being produced to generate the ideas, products, and industries that would drive future U.S. global competitiveness.2 It was also well documented at the time that over 40 percent of students who entered research universities intending to major in a STEM field did not complete their degrees or ended up earning degrees in non-STEM disciplines.3

During this same time period, research on teaching and learning had also led to the development of instructional methods that were more engaging and effective at helping students learn. The positive impact of these improved teaching methods had been extensively documented in STEM fields and was summarized in reports issued in 2010 by both the National �����Ƶ of Sciences and the President’s Council of Advisors on Science and Technology.4 A comprehensive meta-analysis of 225 studies revealed that undergraduate students in classes with traditional lectures are 1.5 times more likely to fail than students in classes that use active learning methods.5 Evidence was also emerging (and now is firmly established) demonstrating that learning gains from using these teaching approaches in highly structured classrooms are particularly good for students from disadvantaged and diverse backgrounds.6 Active learning also has been shown to confer disproportionate benefits to female students in male-dominated fields.7

Reflecting on undergraduate STEM education from 2009 to 2012, the AAU found that most university efforts to support student success in STEM fields occurred outside the classroom, and that a majority of STEM faculty members remained inattentive to scholarship on effective pedagogy. Student-centered, evidence-based teaching practices were not yet the norm in most undergraduate STEM education courses, and the desired magnitude of change in STEM pedagogy had not materialized.8

Most scholarship on STEM educational reform has focused on individual faculty members and the students in their classrooms. This literature often centers on microlevel assessments of the classroom, which are crucial to assessing the effect of pedagogy on student learning and informing the broad audience of instructors about what works. Much less evident is research about the larger institutional and external environments, including the costs and political challenges in scaling up reforms.9 Concern about more macrolevel environments requires a change in assessment from looking solely for benefits and learning outcomes at the course or program level to a more nuanced consideration of factors that facilitate, impede, or influence widespread transformation in undergraduate STEM education.

Education scholar Ann Austin has recommended that sustainable STEM reform requires engaging institutional leaders such as department chairs, deans, and presidents in rethinking institutional structures and culture.10 Relying on her well-documented systems approach to change, Austin has also suggested that external stakeholders, such as disciplinary societies, government agencies, and employers, are crucial to long-lasting change.11 Transforming undergraduate STEM education requires multiple facilitators or “levers” pushing for change that can counterbalance the forces that sustain ineffective instructional practices and that address the systemic obstacles that work against educational innovation and reform.12

In 2011, the AAU launched the Undergraduate STEM Education Initiative, which was designed to assist AAU institutions in widely implementing effective teaching practices in STEM education and supporting student learning and persistence in STEM. This ambitious project has sought to increase the importance and value of effective undergraduate STEM teaching in the nation’s leading research universities and continues to promote the implementation of a systemic view of educational reform within academia.13

Since its launch, the initiative has made significant progress in advancing these goals. At the institutional level, although many of the interventions are still in progress, initial data and analyses point toward their positive impact. Of the eight initial AAU STEM Initiative project sites, all have reported some improvement in student learning outcomes. The magnitude and significance have varied according to the different stages of the reform process across the institutions and departments. Several campuses have experienced dramatic reductions in achievement gaps, especially for women, underrepresented minorities, and first-generation students. Reports of decreased DFW (D grades, F grades, and withdrawals from a course) rates are common, as is increased student persistence and success in subsequent courses as measured by grade performance. AAU project sites also found improved performance on exams designed and sponsored by disciplinary societies to assess knowledge of core disciplinary concepts (that is, concept inventories). Some campuses also have tracked the effects of instructional interventions on more general psychological factors, such as self-efficacy, metacognition, and student attitudes toward science.14

However, evidence alone is not enough to change faculty behavior. As AAU principal investigator James Fairweather has explained, “research evidence of instructional effectiveness is a necessary but not sufficient condition” for faculty to change their teaching practices. Fairweather has suggested that the assumption that “the instructional role can be addressed independently from other aspects of the faculty position, particularly research, and from the larger institutional context” is misguided.15 Given the size and scale of higher education, changing individual faculty members or even isolated departments will have minimal impact. To achieve long-lasting and broadly disseminated educational reforms, efforts must go well beyond this microlevel focus.

As such, AAU member campuses are implementing specific strategies to catalyze change, partner with academic departments, and support faculty members to improve the effectiveness of undergraduate education. Moreover, two cross-cutting resources–data and funding–are being committed to advancing these strategies. Before describing these approaches, it is important to discuss the academic department and its role in facilitating long-term sustained change. The department is the location where these strategies obtain buy-in and commitment, as well as connect with faculty members in the university.

In 2017, the American �����Ƶ of Arts and Sciences’ Commission on the Future of Undergraduate Education published a report that examined the current state of American undergraduate education, projected the nation’s short-term and long-term educational needs, and offered recommendations for strengthening all aspects of undergraduate education.16 One of the primary recommendations of the Commission is for institutions to make a systemic commitment to the improvement of undergraduate teaching. According to the Commission, strengthening college teaching will require institutional collaboration with academic departments. In a supplemental report, the Commission recommended that institutions provide sustained support for department chairs to enable them to become more knowledgeable about the research base on effective college teaching and help them create teaching improvements in their home departments.17

From its inception over six years ago, the AAU’s Undergraduate STEM Education Initiative has recognized that academic departments are the primary loci for cultural change and that academic units and colleges are central to improving the quality of undergraduate education. Institutions rely on individual academic departments to coordinate and manage the academic process.18 Departments determine course offerings, curricula, and teaching assignments; appoint and promote teaching and administrative staff; and manage essential services for faculty members and students. Moreover, faculty members typically identify more strongly with their departments than with their university as a whole because their identities are most closely tied to their academic disciplines and because academic work is primarily carried out in separate department-based worlds.19 Thus, the department is the primary unit in which faculty members see themselves as having the greatest influence, and the space in which they can create desired change.20

The department chair plays a significant leadership role at a university. In addition to leading their departments, chairs also situate their departments within institutional context and priorities: they lie at a pivotal junction between the administration and the faculty, maintaining the department as well as meeting the needs of the institution.21 The chair is a linchpin that connects institutional priorities and faculty work by translating messages from senior institutional leaders, and interpreting questions, issues, and concerns expressed by faculty members.22 Department chairs have meaningful, ongoing interactions with faculty members, students, and other department chairs. They advocate within the university for the interests of those engaged in their particular fields.23 Department chairs can help create cultures in their units where teaching excellence is valued and rewarded.24 Overall, the work of department chairs has an immediate and lasting impact: their actions affect the daily experience of faculty members, staff, and students. Research has shown that department chairs are responsible for 80 percent of administrative decisions on campuses.25

Consistent with the importance of department chairs in reforming undergraduate STEM education, the AAU convened teams of department chairs from member campuses in 2015 and 2018. During these workshops, the AAU discussed the evidence of improved learning gains and STEM-major retention in classes using engaged and structured teaching methods. The chairs then discussed topics such as creating inclusive and welcoming classroom environments, using data to inform and assess curricular innovations, introducing practices to evaluate and reward teaching effectiveness, and developing productive partnerships between academic departments and centers for teaching and learning. By engaging STEM department chairs in these critical teaching and learning issues, the AAU has worked to increase the magnitude and speed of change in the quality and effectiveness of undergraduate STEM education at research universities.

The AAU Undergraduate STEM Education Initiative has found that departments taking collective responsibility for improving the effectiveness of their foundational courses are the ones most likely to emphasize evidence-based active-learning strategies. Collective responsibility is related to developing a uniform departmental vision of educational improvement among faculty members and implementing strategies necessary to support a cycle of continuous improvement.

The AAU has observed six key drivers for the continuous improvement of undergraduate STEM teaching and learning.

Embedding discipline-based education and pedagogical expertise in departments to assist in educational improvement. To improve introductory foundational STEM courses, many AAU institutions are investing in faculty members who have subject matter expertise, a deep understanding of effective pedagogy, and experience in using evidence-based teaching practices. Crucial to the effective use of these personnel is finding ways to incorporate them in departmental decision-making about teaching and curricula.

These education-based faculty appointments vary widely across institutions. Some are discipline-based education researchers hired in tenure-track faculty lines. Others are faculty members in lecture positions aligned with a promotion track, which provides some level of employment security. Some are postdocs who provide expert pedagogical guidance to faculty members. Appointments vary by title, tenure-track status, teaching load, research expectations, performance expectations, and promotional level.26 The role and responsibility to advance institutional, college-wide, or department-based educational improvement efforts by faculty members in these positions is a function of hiring expectations and the acceptance by departments and institutions of their contributions to improving undergraduate education.

Embedding faculty with disciplinary and educational expertise in departments can assist in department-wide educational improvement, including the design and teaching of foundational courses. When linked with colleagues across departments, these individuals can also assist in achieving broader, institution-wide, systemic STEM teaching reforms. Although many of these faculty members exclusively teach (especially introductory courses), when given the opportunity, they are quite effective as change agents, leading refinements in course curriculum and assessments, helping tenure-track faculty teach these introductory courses more effectively, conducting educational research and assessments, and linking their academic department to other university teaching and learning units. These broader departmental reforms are more difficult to achieve if these newer types of faculty members are not given time to engage in nonteaching activities. Acceptance and support from departmental leadership and tenure-track faculty members of individuals with instructional expertise are essential to make maximum use of their expertise to promote long-lasting reforms in teaching and learning.

A number of AAU institutions are testing an adaptation of an expert-guided course-transformation process. For example, TRESTLE (Transforming Education, Stimulating Teaching and Learning Excellence) is a multi-institution, National Science Foundation–funded project that studies and implements a model for improving STEM education at public research universities. The embedded pedagogical experts lead their department colleagues through department planning and course transformation using a backward mapping design process consistent with improvement science. This process is complemented by a curriculum-mapping step to promote a sense of shared ownership of courses and curricula and to generate a common vision. In addition, TRESTLE is building intellectual communities around evidence-based educational improvement, within and across departments and institutions. It is also collecting and making visible evidence of the impact of reforms on teaching and learning.

Creating inclusive and welcoming classroom environments. Unstructured learning environments can lead to unfairness, feelings of exclusion, and collisions of students’ cultural backgrounds with the learning environment. In a structured learning environment, the instructor designs classroom interactions with the intention of maximizing student learning.27 Adding structure to learning environments can mitigate unfairness, promote feelings of inclusion, and foster student success.28 At some institutions, faculty members are participating in mentee-mentor coteaching teams to implement inclusive, evidence-based teaching methods designed to close achievement gaps in foundational science courses. Inclusive teaching has two main components: putting more structure into a course by giving clear instructions so that all students know what to do before, during, and after class; and facilitating class discussion so that everyone can participate.

Another effective strategy to create inclusive learning environments is to partner with undergraduate students. Undergraduate learning assistants (ULA) are undergraduate students who have done well in the class previously. They help facilitate learning activities during instructional time. Frequently, ULAs support instructors making extensive use of innovative, evidence-based pedagogies in their courses. Depending on the university, recognition for ULAs range from course credit to financial stipends. In addition to the support ULAs offer instructors, studies indicate that learning-assistant programs have several benefits, such as improved learning outcomes and knowledge retention for students who take courses with ULAs compared with students who take parallel courses without ULAs; reduced DFW rates in courses that have ULAs; and, after being a ULA, students have equivalent knowledge to graduate students in the field.29 Undergraduates can also be employed to facilitate peer-led team learning (PLTL). PLTL groups typically consist of six to eight undergraduate students who work together to solve problems and are facilitated by a peer leader. Peer leaders are undergraduate students who have previously taken and performed well in the course. PLTL is designed to help students become conscious of the problem-solving process. It also helps students develop important collaboration skills, including how to approach problems effectively as a group, how to communicate well, and how to exchange and critique ideas in a collaborative environment. Peer leader training is an important component of the program. Peer leaders often are enrolled in courses to learn how to be mentors for their groups; they form a collaborative group of their own to help one another address common PLTL challenges.30

Employing collaborative active learning techniques in the classroom is an important strategy for achieving student engagement and for enhancing learning. Numerous studies provide significant evidence that engagement is critical to student success. In collaborative and flexible learning spaces, faculty members are using innovative teaching and learning strategies that promote higher-order thinking skills that lead to better understanding and improved ability to transfer knowledge to other applications. These rooms are often an important catalyst for faculty members to redesign courses and are cited by students as providing more inclusive learning environments.31

Implementing practices to value, evaluate, and reward teaching effectiveness. College and university efforts to improve undergraduate teaching and learning require the recognition of faculty who use teaching practices shown to support student learning. Despite decades of scholarship to develop rich, multisource systems for evaluating teaching, these methods have not been broadly implemented into or recognized within faculty reward systems.32 Many departments, colleges, and institutions are now developing innovative efforts to support the implementation of higher-quality approaches to teaching evaluation.

Evidence shows that stated policies about teaching alone do not strongly influence faculty behavior, much less encourage academic culture to more highly value teaching. A richer, more complete assessment of teaching quality and effectiveness for tenure, promotion, and merit is necessary for systemic improvement of undergraduate education.33

Several institutions have adopted strategies to create an environment in which the continuous improvement of teaching is valued, assessed, and rewarded at various stages of a faculty member’s career, and is aligned across the department, college, and university levels. The AAU has developed a matrix to map the landscape of efforts working to improve policy and practices related to the evaluation of faculty work.34

Staff at some centers for teaching and learning are developing frameworks and rubrics to provide a more comprehensive view of faculty teaching. These tools are often designed to structure departmental evaluation of faculty members’ teaching with defined expectations and dimensions of effective teaching practice. At other institutions, in partnership with centers for teaching and learning, departments are using a variety of tools (such as the Classroom Observation Protocol for Undergraduate STEM and the Decibel Analysis for Research in Teaching) to help conduct more effective observations of faculty teaching.35 In some instances, a radical revision of teaching observations is underway. As an example, some faculty members observe classes taught by others with evidence-based instruction. Instead of evaluating that instructor’s performance, they write a self-reflection on their own teaching to include in annual reviews. Faculty senates are also leading efforts to reconsider the institutional process for the evaluation of teaching.

Developing productive partnerships between academic departments and units dedicated to educational effectiveness. Across the AAU, a variety of institutional structures exist to support faculty members in improving the quality and effectiveness of teaching and learning. The AAU has recognized that when academic departments develop productive working partnerships with units dedicated to educational effectiveness, it results in change at scale.

This reflects a core principle that the ultimate responsibility for teaching quality lies with the department, especially the department chair. This effect occurs through three main mechanisms: determining the curriculum (typically developed by a faculty committee and enforced by the chair), making teaching assignments, and evaluating faculty teaching. Many institutions have recognized the interdependence of support units and departments in improving teaching and learning. They are elevating and reorganizing the traditional teaching center into a full division or more closely aligning it with university leadership, oftentimes an associate provost responsible for teaching innovation or excellence with a direct reporting line to the provost. By expanding and more centrally locating these teaching responsibilities at higher levels within the university, the institution can make its expectations for teaching more explicit to academic units. More centralized leadership provides the necessary scaffolding for individual faculty members who wish to incorporate evidence-based teaching approaches into their course or department-level projects that promote student learning, create inclusive classrooms, and retain highly qualified students. Individual faculty members are also provided assistance to design and conduct assessments to evaluate curricular innovations as well as determine the impact of pedagogical changes on student learning. In this new light, centers for teaching and learning can bridge instructional teams (faculty, graduate students, undergraduates, and postdocs) and experts in assessment, technology, pedagogy, and student support. Increasingly, these support units provide department chairs with a suite of necessary information to generate appropriate conversations and reflection on teaching and teaching quality.

Finally, these more visible and institution-wide units are better positioned to compete for extramural grant funds to facilitate course transformation, teaching development efforts, and cultural change across the institution around teaching. In some instances, more visible centers for teaching and learning have helped departments to submit proposals and receive grant funding from the institution to encourage and facilitate high-impact learning practices, technology-enhanced learning, and a culture of educational excellence at the department and college levels.

Using data to inform and assess curricular innovations. Research universities can facilitate STEM education improvement by supporting the development and use of institution-wide data and analytical tools on student instruction and learning outcomes. It is critical that data collected by the institution are compiled and shared with departments in ways that help them and their faculty members to enhance continually the quality of their STEM instruction. Central to the successful use of data analytics is to distinguish between the types of data useful for individual faculty members designing and assessing their courses and the types of data that can be used to inform departmental decision-making. For example, information about incoming student backgrounds, demographics, and past performance (such as SAT scores); pre- and post-tests to assess student understanding of core concepts; data from various course observation protocols; and data provided by student evaluations and assessments can be helpful to individual faculty members. Data regarding student performance in subsequent courses, DFW rates over multiple semesters, and data that enable comparisons across various sections of a class can be useful to the department. The ease and efficiency of the use of data are also important factors in broad acceptance of teaching-related metrics. Last and most important, data must be seen as part of the policy- and decision-making process. Among the more important lessons learned on the use of data in educational reform are that actionable and supported strategies based on data analytics must be developed within academic units.

Several AAU institutions are developing analytical tools to examine student demographics, student preparation, student performance, student choice, curricular complexity, instructional resources, and student learning. The aim is to foster a cycle of progress in which faculty members and administrators move from awareness and understanding to a continuous cycle of action and reflection.

Creating new business models. Systemic improvement of undergraduate STEM education at research universities should not be done from one grant to the next. Although this is an acceptable approach for supporting research, it is not appropriate for the institutional instructional mission. Symbolically and practically, establishing and maintaining lasting business models and organizational structures that support STEM educational reform are required elements for eventual institutionalization. Systemic changes in undergraduate STEM education require long-term administrative financial support. Institutionalization of reform efforts will frequently require funds for personnel, infrastructure, and space. In the past, institutions have not fully taken advantage of the fundraising potential that exists around efforts to improve the quality of their STEM teaching. However, more and more institutions are finding that donors are inspired by these new teaching practices and learning environments and will provide funds to support these efforts.

Likewise, new funds are needed to support embedding high-level, teaching-oriented faculty with deep disciplinary expertise within departments. Universities have for many years sought funds to endow research chairs within departments. These chairs often have been funded by industry or industrial leaders. Endowed chairs can provide more space and recognition for faculty wanting to devote time to helping their departments improve the quality of their teaching. Institutions and departments would be wise to capitalize on this growing interest by endowing education-oriented chairs within their departments and providing these faculty with the resources needed to enhance and improve teaching in their departments. At the same time, industry would be wise to seek to support such endowed chairs to help ensure a well-trained STEM workforce in disciplines critical to their continued success.

While the AAU is working to help universities advance these critical catalysts necessary for systemic change in undergraduate STEM education and to leverage the influence of peer institutions, challenges remain. 

First, institutions and departments need to find ways to better value the contributions of individuals (such as teaching professionals and teaching faculty) working to achieve the university’s educational mission. The AAU has observed at research universities a significant challenge in recognizing the academic unit as a team of faculty members all making contributions to undergraduate education. In addition, the value of activities to improve undergraduate education, particularly the more invisible elements of teaching (such as course or curriculum redesign and assessment), is weighted differently across and within institutions. And as faculty members work to demonstrate effectiveness in research, teaching, and service as part of the promotion and tenure process, it is often unclear where to discuss this work. The AAU has found differing opinions by deans and department chairs within universities on this topic. Some consider efforts such as collaborating with faculty colleagues on a curriculum design as a service role or as part of committee work. Others consider this task a core element of teaching. This ambiguity can make it difficult to reward faculty for making key contributions to the full range of departmental educational objectives.

For faculty members hired to provide pedagogical, discipline-based expertise with long-term contracts and the opportunity for professional advancement, there is considerable debate about teaching loads, research expectations, how contributions to improving courses or mentoring faculty members in evidence-based pedagogy are counted in annual review, as well as policies about their rights to participate in department governance and service committees. The AAU has observed that departments are relying on these faculty members to make significant educational improvements to foundational introductory courses but have not figured out how to provide these faculty members voice in departmental governance or how to give faculty members credit for their teaching and educational leadership contributions. This growing tension must be addressed.

Second, for sustainable undergraduate STEM education reform, departments will need to create environments to support the interactions necessary to build trust and respect among the whole team of faculty members and address some of the critical barriers to undergraduate education improvement. A recent study by higher education scholar Adrianna Kezar examined the role of the AAU in scaling improvements in undergraduate STEM education.36 The study found that through in-person convenings of faculty members and campus leaders, the AAU has facilitated a community of change leaders by creating an environment in which they can share challenges, learn from peers in similar institutional contexts, and provide multiple dimensions of support to one other. Moreover, the AAU has found that the in-person component of networking is important. Even in our technological age, physical proximity matters for collaboration: productive collaborations are driven by face-to-face interactions in shared spaces.37 Networks are central to facilitating and scaling change since they provide the emotional support and sense of community necessary for participants to feel that they can safely take risks and experiment together. 

Third, there is the challenge of expanding beyond STEM. Effective teaching and learning and creating inclusive and welcoming classroom environments are critical not just in STEM but for all disciplines. The AAU is pleased that member institutions are expanding their efforts to include the full range of disciplines represented on their campuses. The AAU has found such efforts in the social and behavioral science courses that enroll a large number of students as well as in foundational or general education curriculums. In these spaces and in the humanities, there is growing recognition that these courses are also important for student learning and for departmental budgets; they account for a significant amount of credit hours and tuition every term. Such courses are also very difficult to teach, enrolling students from a wide variety of backgrounds, interests, and goals, as well as endeavoring to prepare them for subsequent study across a range of fields.

Finally, institutions must commit to a cycle of continuous improvement. At the national policy level, we have begun to see a more coordinated effort to improve undergraduate education across relevant organizations and actors.38 We have observed a shift away from isolated directives within individual disciplines and nationally funded efforts that do not require long-lasting reforms within academic institutions. Today, many funders are designing solicitations with expectations for projects to build and sustain institutional change.39 At the institutional level, universities are designing institutional structures and committing to leadership roles necessary to support the diverse, complex pathways students take to earn degrees, as well as ensure effective teaching for the growing diversity of learners.40 Universities are also engaging in a reflective practice of assessing institutional improvement efforts in teaching and learning and then adjusting practice at multiple levels of the university. Ultimately, effective undergraduate education will require a sustained institutional commitment to a continuous cycle of improvement. The AAU will continue to work to promote the use of evidence-based teaching practices and drive systemic change to improve the quality of undergraduate education at research universities.

Endnotes

• 1Boyer Commission on Educating Undergraduates in the Research University, (Stony Brook, N.Y.: Boyer Commission on Educating Undergraduates in the Research University, 1998).
• 2The National Academies of Sciences, Engineering, and Medicine, Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future (Washington, D.C.: The National Academies Press, 2007).
• 3Elaine Seymour, Talking about Leaving: Why Undergraduates Leave the Sciences (Boulder, Colo.: Westview Press, 2000).
• 4Susan R. Singer, Natalie R. Nielsen, and Heidi A. Schweingruber, eds., (Washington, D.C.: The National Academies Press, 2012); and Executive Office of the President, President’s Council of Advisors on Science and Technology, Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics (Washington, D.C.: The President’s Council of Advisors on Science and Technology, 2012).
• 5Scott Freeman, Sarah L. Eddy, Miles McDonough, et al., “,” Proceedings of the National �����Ƶ of Sciences 111 (23) (2014): 8410–8415.
• 6Sarah L. Eddy and Kelly A. Hogan, “” CBE–Life Sciences Education 13 (3) (2014): 453–468; David C. Haak, Janneke Hille Ris Lambers, Emile Pitre, and Scott Freeman, “,” Science 332 (6034) (2011): 1213–1216; Kyle F. Trenshaw, David M. Targan, and James M. Valles, “,” Proceedings from ASEE NE ’16: The American Society for Engineering Education Northeast Section 2016 Conference (Kingston, R.I.: American Society for Engineering Education Northeast Section, 2016); and Michael T. Crim-mins and Brooke Midkiff, “,” Journal of Chemical Education 94 (4) (2017): 429–438.
• 7Mercedes Lorenzo, Catherine H. Crouch, and Eric Mazur, “,” American Journal of Physics 74 (2) (2006): 118–122.
• 8Winston A. Anderson, Utpal Banerjee, Catherine L. Drennan, et al., “,” Science 331 (6014) (2011): 152–153; Singer et al., Discipline-Based Education Research; and Marilyne Stains, Jordan Harschman, Megan K. Barker, et al., “,” Science 359 (6383) (2018): 1468–1470.
• 9Charles Henderson and Melissa H. Dancy, “,” white paper prepared for the Characterizing the Impact and Diffusion of Engineering Education Innovations Forum, February 7–8, 2011 (Washington, D.C.: National �����Ƶ of Engineering, 2011).
• 10Ann E. Austin, “,” white paper commissioned by the Coalition for Reform of Undergraduate STEM Education (Washington, D.C.: Association of American Colleges and Universities, 2014).
• 11Ann E. Austin, “” (Washington, D.C.: National Academies National Research Council Board on Science Education, 2011).
• 12Anderson et al., “Changing the Culture of Science Education at Research Intensive Universities”; and Andrea L. Beach, Charles Henderson, and Noah Finkelstein, “,” Change: The Magazine of Higher Learning 44 (6) (2012): 52–59.
• 13Nancy Kober, “,” Reaching Students: What Research Says about Effective Instruction in Undergraduate Science and Engineering (Washington, D.C.: The National Academies Press, 2015); and Association of American Universities, Progress toward Achieving Systemic Change: A Five-Year Status Report on the AAU Undergraduate STEM Initiative (Washington, D.C.: Association of American Universities, 2017). The AAU’s report documents the results of the AAU STEM Initiative to date. The AAU is committed to expanding the initial effort indefinitely by integrating continued support for undergraduate STEM education improvement into its ongoing portfolio of work.
• 14Melanie M. Cooper, “” Journal of Chemical Education 92 (8) (2015): 1273–1279; and Michael J. Obsniuk, Paul W. Irving, and Marcos D. Caballero, “,” paper presented at the Physics Education Research Conference, July 29–30, 2015, College Park, Maryland.
• 15James Fairweather, “” (Washington, D.C.: National Academies National Research Council Board on Science Education, 2008).
• 16Commission on the Future of Undergraduate Education, The Future of Undergraduate Education, The Future of America (Cambridge, Mass.: American �����Ƶ of Arts and Sciences, 2017).
• 17Aaron M. Pallas, Anna Neumann, and Corbin M. Campbell, Policies and Practices to Support Undergraduate Teaching Improvement (Cambridge, Mass.: American �����Ƶ of Arts and Sciences, 2017).
• 18Richard Edwards, “The Academic Department: How Does It Fit into the University Reform Agenda?” Change: The Magazine of Higher Learning 31 (5) (1999): 17.
• 19Adrianna Kezar, Sean Gehrke, and Susan L. Elrod, “Implicit Theories of Change as a Barrier to Change on College Campuses: An Examination of STEM Reform,” The Review of Higher Education 38 (4) (2015): 479–506.
• 20John Tagg, “Why Does the Faculty Resist Change?” Change: The Magazine of Higher Learning 44 (1) (2012):6–15.; and Rebecca L. Matz, Cori L. Fata-Hartley, Lynmarie A. Posey, et al., “,” Science Advances 4 (10) (2018).
• 21Alan T. Seagren, John W. Creswell, and Daniel W. Wheeler, (Washington, D.C.: The George Washington University School of Education and Human Development, 1993).
• 22Austin, “Promoting Evidence-Based Change in Undergraduate Science Education.”
• 23Jeffrey L. Buller, The Essential Department Chair: A Comprehensive Desk Reference, 2nd ed. (San Francisco: John Wiley & Sons, 2011).
• 24Fairweather, “Linking Evidence and Promising Practices in Science, Technology, Engineering, and Mathematics (STEM) Undergraduate Education.”
• 25James B. Carroll and Mimi Wolverton, “Who Becomes a Chair?” New Directions for Higher Education 126 (3) (2004): 3–10.
• 26Carl Wieman, Improving How Universities Teach Science: Lessons from the Science Education Initiative (Cambridge, Mass.: Harvard University Press, 2017); and Seth D. Bush, Michael T. Stevens, Kimberly D. Tanner, and Kathy S. Williams, “,” Science Advances 5 (6) (2019).
• 27Kimberly D. Tanner, “,” CBE–Life Sciences Educa-tion 12 (3) (2013): 322–331.
• 28Viji Sathy and Kelly A. Hogan, “,” The Chronicle of Higher Education, July 22, 2019.
• 29Valerie Otero, “,” American Journal of Physics 78 (1218) (2010).
• 30Regina F. Frey, Angela Fink, Michael J. Cahill, et al., “,” Journal of Chemical Education 95 (12) (2018): 2103–2113.
• 31The University of Arizona, “.”
• 32Daniel J. Bernstein, “,” Change: The Magazine of Higher Learning 40 (2) (2010): 48–51; Daniel Bernstein and Mary Taylor Huber, “What Is Good Teaching? Raising the Bar through Scholarship Assessed,” presentation at the International Society for the Scholarship of Teaching and Learning 3rd Annual Conference, Washington, D.C., November 9, 2006; Charles E. Glassick, Mary Taylor Huber, and Gene I. Maeroff, Scholarship Assessed: Evaluation of the Professoriate. Special Report (San Francisco: Jossey-Bass, 1997); Pat Hutchings, Making Teaching Community Property: A Menu for Peer Collaboration and Peer Review (Sterling, Va.: Stylus, 1996); and Pat Hutchings, Mary Taylor Huber, and Anthony Ciccone, The Scholarship of Teaching and Learning Reconsidered: Institutional Integration and Impact (San Francisco: Jossey-Bass, 2011).
• 33James S. Fairweather, “The Ultimate Faculty Evaluation: Promotion and Tenure Decisions,” New Directions for Institutional Research 114 (2002): 97–108; and Mary Taylor Huber, “,” New Directions for Institutional Research 114 (2002): 73–84.
• 34Association of American Universities, “” (Washington, D.C.: Association of American Universities, 2018).
• 35Michelle K. Smith, Francis H. Jones, Sarah L. Gilbert, and Carl E. Wieman, “The Classroom Observation Protocol for Undergraduate STEM (COPUS): A New Instrument to Characterize University STEM Classroom Practices,” CBE–Life Sciences Education 12 (4) (2013): 618–627; and Melinda T. Owens, Shannon B. Seidel, Mike Wong, et al., “,” Proceedings of the National �����Ƶ of Sciences 114 (12) (2017): 3085.
• 36Adrianna Kezar, Scaling Improvements in STEM Learning Environments: The Strategic Role of a National Organization (Washington, D.C.: Association of American Universities, 2018).
• 37Matthew Claudel, Emmanuel Massaro, Paolo Santi, et al., “An Exploration of Collaborative Scientific Production at MIT through Spatial Organization and Institutional Affiliation,” PLOS One 12 (6) (2017).
• 38American Association for the Advancement of Science, (Washington, D.C.: American Association for the Advancement of Science, 2011); National Science Foundation, , workshop report (Arlington, Va.: National Science Foundation and American Society for Engineering Education, 2013); National Research Council, (Washington, D.C.: The National Academies Press, 2013); and Executive Office of the President, National Science and Technology Council, Committee on STEM Education, (Washington, D.C.: Executive Office of the President, 2013).
• 39Catherine L. Fry, ed., (Washington, D.C.: Association of American Colleges and Universities and the Coalition for Reform of Undergraduate STEM Education, 2014); and Emily R. Miller and Tara King, eds., (Washington, D.C.: Association of American Universities, 2019).
• 40National Academies of Sciences, Engineering, and Medicine, (Washington, D.C.: The National Academies Press, 2016).