Political Science Learning Outcomes Final Report

PRE-CONFERENCE WORK

Prior to our workshop, PI Golich sought to gather relevant articles, books, and sample learning outcomes specific to the discipline of political science. Despite many hours of searching--herself, by student assistants at Fullerton and San Marcos, and by staff at the CSU Chancellor's Office--few such materials were found; two articles and five learning outcomes samples–three discipline-specific and two general education social science–were assembled for distribution at the workshop. Due to the paucity of discipline-specific materials, some general resources were also included. Subsequent searches yielded several other helpful articles, websites, sample learning outcomes, and general resources.

Golich also sought to contact political science department chairs to solicit interest in and participants for the workshop. This experience provided the first inkling that communication among CSU political scientists was infrequent at best and nearly non-existent at worst. The Chancellor’s Office did not have the information; no disciplinary council–similar to those in English, history, sociology, or foreign languages–was in place; and the simple effort to identify, much less contact, disciplinary chairs was amazingly enormous.

 

DAY 1 — Morning

The session was facilitated by Candy Young–Professor of Political Science at Truman State and well known for her work in defining and assessing learning outcomes in political science. Lessons learned from Truman State’s 15-year effort to change the university culture to one which embraces assessment were helpful in our efforts to begin the process of defining learning outcomes for political science. Briefly, she offered several key observations about the process of transforming the campus and insights about its effect on the campus.

 

QUALITY

Professor Young noted that "quality" is typically defined by inputs–who attends, who teaches, what resources are available, e.g., library books)–and by reputation. In reality, although inputs are important, outcomes are also critical and might even tell a more important story. So what are the outcomes? An agreed upon set of skills, knowledge, and attitudes which defines

• What students know and are able to do when they leave campus with a college degree, and, more narrowly,
• what students know and are able to do with a political science degree.

CAMPUS CULTURE

Campus culture is a keystone to success in moving to an environment where the focus in on outcomes rather than (or in addition to) inputs. It is critical that administrators begin by demonstrating interest in graduating highly qualified, well educated students. This can begin to redirect the conversation to one focused on value added.

Also critical is a broad definition of who the "customer" of our education is: Such a definition should include at least the following:

1. Students who seek an education,
2. Society-at-large and, for public institutions such as Truman State or the CSU, tax payers who pay the largest portion of the bill, and
3. Faculty colleagues, who will "receive" students in class "y" and need them to have learned some set of skills, content, and attitudes in the previous courses to succeed.

One unanticipated, but welcome, result of this shift in attitudes is an approach to advising which suggests that, if students are weak in a certain area–e.g., math or writing–they should be advised to take more classes in order to gain competencies rather than fewer in order to avoid difficult learning experiences.

The goal is to create a campus culture that embraces a willingness to question itself; to ask

1. how are we doing?
2. where do we want to be?
3. how will we get there?
4. how will we know when we have arrived?

 

LEARNING OUTCOMES DEFINED

To help us move to a place where we could begin to define learning outcomes for the political science graduates at our multiple campuses, Professor Young clarified levels and types of learning outcomes and specified some of the decisions made by the Political Science Department at Truman State.

Levels

University-Program Curriculum-Course-Student

Categories

1. KNOWLEDGE (the content of the curriculum)

1. General Education (creates the knowledge base upon which disciplinary knowledge is built)
2. Major (specifies the knowledge each graduate should share AND refines/reinforces the skills and attitudes developed in GE; within the major there is both

i.Core knowledge (courses required of ALL students) — a fairly substantial number of units providing a foundational breadth of the discipline

(1) When building this Core, need to think carefully about a) efficient use of faculty resources b) skill matrix c) course sequencing rationale d) what kind of common experience students should have

(2) At Truman State this Core =

a) American (1^st year)

b) International Relations (1^st year)

c) Methodology (2^nd year)

d) Comparative Politics (3^rd year)

e) Public Policy (3^rd year)

f) Political Thought (4^th year)

g) Senior Seminar (4^th year)

ii. Electives (a knowledge base which builds upon the Core)

(1) When building this Electives Set, need to think about

a) flexibility

b) "customized" degree options

c) increased expertise in particular areas

2. SKILLS (the skills competencies graduates should share)

1. General Education (facilitates learning and performance success)

1. writing
2. speaking
3. computing
4. listening
5. reading
6. collaborating
7. analyzing
8. quantifying

2. Major

1. argumentation
2. debate
3. scientific method
4. statistical packages
5. quantitative methods
6. qualitative methods
7. government documents
8. logical reasoning
9. research design

3. ATTITUDES

1. citizenship
2. ethical awareness
3. multicultural sensitivity
4. confidence
5. satisfaction with the university and with the political science curriculum

 

Assessment

Some discussion of assessment ensued with the following thoughts offered:

1. Outcomes can be assessed using

a. quantitative measures

b. qualitative measures

c. self-evaluative measures (self-confidence surveys; exit interviews)

d. "objective" measures (usually standardized testing)

i. Educational Testing Service — Major Field Assessment Test (MFAT)

ii. CT — General Education pre- and post-test

2. Whatever approaches are used, it is important to employ multiple methods of assessment to get an accurate picture of what is going on in a curriculum.

3. Other assessment elements

a. Retention Tenure Promotion — all faculty members must use assessment to demonstrate how they have improved their teaching and learning

b. Keeping in mind that campus culture is important

1. don’t blame
2. find and focus on what is going well
3. collect data in the aggregate, not by individual classroom
4. stay focused on assessing the curriculum

c. Build on what is already being done; start with thinking about what data we already HAVE? (e.g., course syllabi — course objectives are articulated)

Implementation Strategies

1. Some thoughts and tips from Truman State’s experiences can inform the CSU process:

a. Prepare an assessment plan that allows for some comparison with

comparable institutions.

b. Prepare faculty to be able to read and evaluate portfolios; pay $500/week;

think in terms of 60 faculty reading in 3 groups of 20.

c. Since portfolios are read by teams of faculty from across the institution —

intergenerational and interdisciplinary teams — use this as an opportunity to increase collegiality across the university.

d. Prepare a standard protocol for interviews:

1. what are you here?
2. why did you pick political science?
3. what kept you here?
4. what discouraged you?

2. Campus-wide

a. Ask about the challenge in the courses at each level

b. Lower division, upper division, general education, major

3. Sample Skills Matrix TS adopted =

American Government Þ argumentation

International Relations Þ debate/or simulation

Methodology Þ quantitative research project

Comparative Politics Þ qualitative research project

Public Policy Þ government documents

Political Thought Þ research

proposition papers

4. Capstone

a. teaches the classics

b. is team taught

c. for 3 hours credit

d. in a seminar format

e. utilizes the following strategies for assessing student

i. Knowledge of the major

• papers tied to weekly reading assignments
• class discussion
• comprehensive exam
• research design

ii. Skills

• speaking — extemporaneous
• class discussion
• Q/A session following presentation on research design — as discussant and presenter
• exit interview

Benefits of this Approach

Professor Young summarized her presentation by indicating several benefits which derive from this approach to curriculum development, evaluation, and revision:

• The campus community — students, faculty, staff, administrators, employers, external community — has a conscious plan with which to work.
• The plan provides a comprehensive view of the curriculum.
• The plan communicates expectations to everyone involved, including high standards of performance and appropriate responsibility (e.g., student learning requires work!).
• The plan facilitates a more efficient deployment of resources — human, physical plant, financial capital.
• The plan creates synergies among otherwise disparate elements of a college education.

 

DAY 1 — Afternoon

In the afternoon, we turned to thinking about the learning outcomes we would want on our campuses. We had four groups; the table below depicts their early thoughts.

 

Initial Thoughts on Learning Outcomes

in CSU Political Science Curricula

Group I

• Excellent written and verbal communications skills
• Senior writing project or portfolio assessed by at least two faculty
• Required oral presentations
• Quantitative skills
• Required methods course, quantitative methods used in UD courses
• Critical thinking (e.g., information competence)
• Curriculum
• Core knowledge (department-specific)
• Curriculum

 

Group 2

• Discussed problems of implementing this kind of program
• Writing skills
• Need collegial "buy-in"…to know what others are doing…so we can know what we can expect from our students
• Relationship to administration — committed to creating culture of evidence in non-punitive arena

Group 3 — CSUSM

• Discussed campus strategies regarding how to get started
• Used the Diamond Skills Matrix (p. 52) as a start and blended items from

Dr. Young’s list to get at

• Knowledge
• Skills
• Attitudes
• Decided
• To use senior seminar as a starting point for collecting data
• To link to curriculum more consciously to Lower Division GE and major preparation
  • In the case of CSUSM, this means close work with North County Higher Education Association (NCHEA) — a consortium of our two primary feeder community colleges and us
• Think of locally developed exams…combine with random sample of students taking national exam
• Developed a grant proposal for a local professional development opportunity to help move forward on this agenda

Group 4

• Skills
• Student experience questionnaires
• Skill matrices
• Attitudes cluster — views of the major
• National subject exams
• Portfolio procedures
  • Design
  • Assessment
  • Administrative
• Analytical
• Foundation
  • Reading
  • Hypothesis testing
  • Critical thinking
• Normative
• Ethical reasoning
• Cultural sensitivity/competence
• Empirical
• Research design
• Quantitative
• Qualitative
• Information Competency
• Communication
• Argumentation and debate
• Data visualization/presentation
• Writing
• Speech/listening
• Civic effectiveness?
• Knowledge
• Power and authority
• Historical foundations

as they are played out in each of the sub-fields:

• American government
• Comparative government
• International Relations
• Political theory
• Policy/administration
• Methods

 

DAY 2

ASSESSMENT

On the second day our discussion turned to assessment of learning outcomes. We agreed on several important points. First, we concluded that formative assessment — aimed at continuous improvement — would generate far better results than evaluative (or summative) assessment — targeting accountability. Formative assessment allows faculty to seek open and honest feedback regarding the quality of instruction, advising, placement, and other elements of their program. Summative assessment would create incentives for departments to avoid genuine evaluations and inhibit their ability to improve. Since we are committed to delivering the highest quality education possible — given the fiscal and workload constraints under which we labor and the state mandate we have to accept the top one-third of high school graduates as well as transfers from community college graduates — we are open and willing to make changes in our programs if and when necessary. Therefore, we want to collect meaningful information which can help to ensure that we can consistently provide outstanding curricula and instruction.

Second, we concluded that the best assessment can only take place in a trusting, safe, and supportive environment. This requires the best efforts of administrators and faculty alike; we each bear some responsibility for building and maintaining a culture where real evidence is allowed to inform real change. Departments and programs must feel safe to acknowledge the need for change to initiate meaningful reform efforts. Therefore, assessment should remain separate from accountability. To this end, ideas were suggested:

1. Assessment is for program improvement purposes: The sole purpose of assessment should be to improve the quality of programs; it should not provide a rationale for punitive measures, including resource allocation. If programs are penalized as a result of faculty assessment efforts, this will destroy any incentive to do a fair assessment. Assessment data should not move above the departmental level and certainly none should be sent to the central administration.
2. Assessment should not single out individuals: Assessment, like learning outcomes, should be a collective responsibility. Just as no one individual can teach a student everything s/he needs to learn in college, no one person should be identified in learning outcomes assessment.
3. Assessment should be kept separate from enrollment pressures: Assessment should not emphasize quantity at the cost of quality. Assessment should be used to improve quality and the results should not be thwarted by external enrollment pressures.
4. Adequate resources are needed to do assessment right: Good assessment is not cheap, and there is no sense in doing poor assessment. Adequate time and money will be needed for it to be done right. Indeed, our primary consultant — Candy Young, Professor of Political Science from Truman State University — noted that this campus, which is so highly regarded in the arena of learning outcomes and assessment, took at least five years operating under nearly ideal conditions of strong leadership to begin to change campus culture significantly. We would be wise to move slowly in our efforts to change campus and system cultures in California as well.
5. Assessment required the use of multiple measures, including both quantitative and qualitative instruments: Quantitative assessment involves review of enrollment, advising, graduation, and job placement efforts; standardized testing numbers; faculty accomplishments; and so forth. Qualitative assessment measures include focus groups; interviews with graduating students; capstone courses; portfolios; a common essay; and contact with alumni 10 years after they graduate. Interviews with students and alumni might include questions about best and worst educational experiences and how the program helped them.

Our discussion elicited several other suggestions, including

1. Begin with a pre-evaluation study: Ideally, CSU political science departments will conduct some pre-evaluation studies to determine

1. what data is needed and what is available
2. what data is assessable and by what kind(s) of measures, and
3. what support agencies are available campus- and system-wide to assist in the assessment processes (e.g., institutional studies offices.)

1. Follow with a pilot — experimental — project: Before launching a full-fledged evaluation process, a pilot project should be implemented in an effort to work out potential pitfalls and glitches in the plan.
2. Ensure efficient data collection: Whenever possible,

1. existing data should be used, and
2. every effort should be made to avoid duplicating data collection efforts.

1. Develop a pre- and post-test for students: to begin to assess value-added information.
2. Keep evaluation efforts campus-specific: Although some aspects of political science learning outcomes will be common among CSU campuses, as noted in the Chancellor’s Office RFP

1. Learning outcomes goals and their assessment should be campus-specific, given the nature of campus missions, faculty expertise, and student interest.
2. In addition, campus resources — personnel, financial capital, and data banks — require evaluation to be unique to each campus.

1. Define a role for external evaluation: Again, ideally,

1. Assessment should provide an external point of view, and
2. external reader of student papers might be hired.

1. Keep classroom instruction at the core: The demands of assessment should not be so great that it takes significant resources from classroom instruction.

POLITICAL SCIENCE COUNCIL

Another discussion on our second day concerned the need for a system-wide political science council. We distributed a questionnaire to solicit feedback on this topic: Of 30 workshop participants, 15 said that such a council was desirable; two said is was not necessary; the remainder did not respond. Ideally, a political science council would include two members from each campus — preferably a tenured and a non-tenured faculty member. Participants strongly recommended that council membership not be limited to department chairs. Following the practice of the Faculty Development Council, a political science council could meet in conjunction with the Western Political Science Association meetings, when they are held in California (typically, every other year.)

The list of topics such a council could address is strong testimony to the need for one!

1. Learning Outcomes: What do we want our students to know and be able to do when they graduate with a political science major and how should this affect the content of the programs;
2. Assessment: How will it be used; how data will be interpreted and who will do it; the role of faculty; and system-wide criteria; and overall program evaluation
3. Program Quality: How to use assessment to improve program quality, including program development, professional development, workload, promoting professional activities? Discuss ways to improve the RTP process.
4. Future of the Discipline: How might we shape or respond to future trends in the discipline? To what extent will it become oriented to potential practitioners (e.g., vocational)? How will current enrollment trends affect the discipline and how might we work to stop the decrease and to initiate new interest in the discipline?
5. Resources and Future Needs: How can we share resources where appropriate and avoid detrimental "turf battles"? What kinds of consortia grants might be available? How might we integrate technology into our teaching, learning, and research responsibilities? Initiate and sustain important CSU system discussions, such as distance learning, technology-mediated instruction, and competency-based instruction? Coordinate relations with community colleges, graduate schools, and job placement efforts.

 

ENROLLMENT IN POLITICAL SCIENCE

A final discussion concerned the national decrease in political science enrollment. We strategized about how enrollments in political science might be improved. Our brainstorming session revealed a number of suggestions:

1. Outreach: develop liaison with

1. Community colleges
2. High schools
3. Community
4. Graduate schools
5. Job placement entities
6. Lobby local government concerning job requirements and job placement
7. Explain more effectively the value of a political science degree

1. Course Delivery and Offerings:

1. Consider offering courses throughout a wider range of hours including weekends and Friday evenings
2. Offer courses in the community
3. Consider other alternative methods of curriculum delivery
4. Develop more political science critical thinking courses for GE
5. Adjust to meet the needs of the population, especially groups left out, such as minorities
6. Expand intern opportunities and experiences, including service learning possibilities

1. Curriculum Design:

1. Recruit more pro-actively in introductory courses
2. Connect successful alumni with current students, especially in introductory courses
3. Consider some vocational education — for potential practitioners at multiple levels
4. Emphasize job placement
5. Offer a course about how to make it in the profession, including resume preparation
6. Emphasize public administration or political science minors for students in business schools
7. Develop professional tracks and connections with high enrollment schools like business schools
8. Bring in relevant local employers — politicians, lawyers, non-profit managers, and other career people open to political science hires

 

PRE-CONFERENCE WORK

Lessons Learned

We did not invite enough participants from each campus. Ideally, at least four faculty could have participated from each campus so that the conversation started here would have launched a "critical mass" cohort capable of leading and continuing the conversation on individual campuses.

We should have sent out materials before the workshop. Even through experience suggests that such materials are not really read ahead of time–maybe at breakfast the day of the workshop–some folks clearly would have taken advantage of this opportunity and benefited from doing so. Our conversation might have been even better with more informed voices at the table…we were mostly new to the process and idea of identifying and assessing learning outcomes.

We should have clarified logistics roles and responsibilities more clearly. Although everyone involved in the workshop worked very hard, some miscommunications might have been avoided with clearer lines of responsibility.

Above all, we agreed that defining and assessing learning outcomes should definitely be led by faculty and stay within the purview of faculty to maintain a formative approach to the process. We will need to work hard and remain vigilant to keep the process from shifting to an evaluative approach; we want to keep learning outcomes and assessment non-punitive for all stakeholders. For faculty to be in control of defining criteria, they must step forward to manage the process.

Continuing Work

Given the ideas regarding the need for a disciplinary council to help us continue the work begun at this workshop, PI Golich has

• Developed a current list of political science department chairs and their contact points,
• Collected catalogue copy from each campus with a political science program and created a table of unit requirements--core and elective–for the major,
• Revised and updated the list of participants and the learning outcomes workshop, and
• Updated and re-instituted the CSU political science listserv for department chairs (and we may want to consider another or changing this to be for Council members).

in addition, Professor Buck has continued his work to analyze CSU political science data and to develop a table that would help in our efforts to understand and respond to the changes with appropriate actions related to our curriculum and enhancement.