11-28-2011

This paper explores the relationship between student major and industry of employment and its application to higher education accountability. Data provided by statewide longitudinal data systems (SLDS) have enabled state educational agencies and colleges to follow students into the workforce. While most studies have focused on wage outcomes, this study shows how to use SLDS data to understand the correlation between major and industry. The transition into the workforce is an important outcome since it is an assessment of a college’s ability to develop specific, targeted sectors of the economy. We use SLDS data from Iowa to follow community college alumni from 2002 through 2008.

Click here for a printable booklet version of IR Apps 32.

6-10-2011

Critics of testing for admission purposes cite the moderate correlations of admissions test scores with success in college. In response, this study applies formulas from classical measurement theory to observed correlations to correct for restricted variances in predictor and success variables. Estimates of the correlations in the population of high school graduates are derived from two of the several formulas in the literature. This article describes limitations and encourages additional investigation into the use of the formulas for estimating correlations in unrestricted populations.

Click here for a printable booklet version of IR Apps 31.

3-2-2011

Assessment drives significant change in higher education. Colleges and universities are being asked to publish expected learning outcomes for each of their programs; provide evidence that the expected learning outcomes are realized by students; and demonstrate how such data collection and analyses lead to continuous improvement of student learning, the curriculum, and the university. This is a significant challenge, especially for large schools with many students.

This paper presents a working yet evolving model of an institutional assessment and analysis plan; obstacles and challenges associated with implementing the plan; innovative strategies—some technological—employed to address the obstacles and challenges; a college-wide application of a systems approach to assessment; and lessons learned and future strategies. Institutional, college-wide, discipline-specific, and individual faculty perspectives to a systems approach of bridging assessment, learning outcomes, and accreditation are presented. The importance of student involvement throughout the process is highlighted.

Click here for a printable booklet version of IR Apps 30.

12-15-2010

Artificial Neural Network (ANN) and Support Vector Machine (SVM) approaches have been on the cutting edge of science and technology for pattern recognition and data classification. In the ANN model, classification accuracy can be achieved by using the feed-forward of inputs, back-propagation of errors, and the adjustment of connection weights. In conjunction with the computational shortcut of kernel functions, the SVM classifier maps input data from the input space into the high-dimensional feature space, and seeks an optimal hyperplane to separate data from different classes. Both ANN and SVM machine learning algorithms can be used to establish nonlinear relationships between variables and rank the importance of variables, thereby, contributing to the effectiveness of medical curriculum assessment. The purpose of this investigation is to shed light on how to construct the most suitable ANN and SVM curriculum assessment models based on student perceptions.

Click here for a printable booklet version of IR Apps 29.

9-24-2010

Quality measurement is essential in every form of research, including institutional research and assessment. This paper addresses the erroneous assumptions institutional researchers often make with regard to survey research and provides an alternative method to producing more valid and reliable measures. Rasch measurement models are discussed and a demonstration is provided, thus highlighting the utility of the Rasch models in higher education research and practice.

Click here for a printable booklet version of IR Apps 28.

7-29-2010

Geographic analysis has been adopted by businesses, especially the retail sector, since the early 1990s (Thrall, 2002). Institutional research can receive the same benefits businesses have by adopting geographic analysis and technology. The commonalities between businesses and higher education institutions include the existence of trade areas, the provision of services to clients (students), and the identification of clients geographically by their addresses. Among the valuable information that institutions of higher education can create using business geography are psychographic profiles of student populations, commuting patterns, the underlying demographics of the institution’s trade area, and the ability to plan for new facilities to meet the needs of the market. Understanding these geographic characteristics can assist in identifying institutional objectives and planning how to best implement these objectives.

Click here for a printable booklet version of IR Apps 27.

6-1-2010

In the university setting, the issue of faculty morale typically has been linked to a variety of perceived inequities, including faculty salary. New approaches for analyzing two different, but related, types of inequity are proposed. One approach addresses whether salary compression, often perceived by faculty to exist, actually does exist; the other approach addresses whether end-of-term annual salary increases, often perceived by faculty not to reflect a cost of living component, actually do. What sets these two approaches apart from the others suggested in the literature are that they lack a high level of mathematical complexity, yet they still have the ability to control for confounding sources of variation, they are easily carried out even by someone with minimal statistical expertise, and results from them are easily understood by a broad audience. The two new approaches are applied to real data from a private research university in the Northeast and results from these analyses are discussed.

Click here for a printable booklet version of IR Apps 26.

1-28-2010

Students are demanding more convenient and less time-consuming forums in which to be engaged in all areas of their education, including sharing their ideas about their educational experiences. Web surveys are more prevalent as a technologically advanced research medium being used throughout higher education. As such, this methodology is proving to be an effective means of allowing students to provide their input. One arena that has yet to be fully explored is whether or not the use of incentives will encourage a higher response rate among students participating in web-based student survey panels.

Click here for a printable booklet version of IR Apps 25.

11-10-2009

This study uses a regression model to determine if a significant difference exists between the actual budget allocation that an academic department received and the model’s predicted budget allocation for that same department. Budget data from a Southeastern Master’s/Comprehensive state university were used as the dependent variable, and the budget for each department consists of money used for salaries (personnel) and money used for equipment, travel, and other expenditures (non-personnel). Independent variables included in the model were the number of professors, credit-hour production, number of degrees conferred, and a market ratio variable.

Click here for a printable booklet version of IR Apps 24.

9-1-2009

Using a Markov chain approach in which the transition matrix contains the overall transitions in grades from high school to university, we compute the expected changes in undergraduate performance between the beginning and the end of a student’s degree. The data are compiled from graduates at the University of Barcelona (Spain) for the period 1996–2003. The empirical results show small changes in the ergodic solution if transition probability of improved grades is increased, at least for the period considered. However, educational policies aimed at increasing transitions from lower to upper states seem to be more effective at lower levels. Likewise, the transitions from high school grades to university grades differ according to the kind of high school that students attended (public, private religious, or private non-religious).

7-1-2009

This paper demonstrates the application of system dynamics modeling in university decision-making. In particular, we develop a system dynamics model to predict the potential impact of a change in tenure policy (elimination of the tenure cap) on the proportion of tenured faculty at Bryant University. Since this result can have important implications for both faculty and administration, the potential impact of such a change must be carefully assessed in advance. We demonstrate that system dynamics methodology is useful for this purpose. It allows decision-makers to simulate the dynamic of faculty structure through time to predict the potential impact of the policy change. In this case, the model predicted with a high degree of certainty that the proportion of tenured faculty over the 10-year prediction period would be within tolerable limits after the tenure cap was removed. This reassured both faculty and administration, and the policy change was implemented. Realized results over the three years since the change in tenure policy were consistent with model predictions, underscoring the potential usefulness of the methodology in academic decision-making and planning.

6-1-2009

The goal of the study was to develop a practical application to help a large doctoral/research-extensive public university promote student-athlete success by identifying at-risk student-athletes. A longitudinal model using survival analysis was used to identify factors that impact a student-athlete’s ability to persist and graduate and his or her duration of enrollment.

4-1-2009

High school course-taking plays a critical role in shaping K-12 and post-secondary educational outcomes. Previous educational research often quantifies coursework in a qualitative manner. This research introduces institutional researchers to the use of a profile analysis method with an application of Profile Analysis via Multidimensional Scaling (PAMS) for identifying high school coursework patterns in a quantitative fashion. Four prototypical student coursework profiles were identified: (a) rigorous; (b) Advanced Placement vs. honors; (c) non-foreign language vs. foreign language; and (d) English and social science/history vs. mathematics and science. Subgroup analyses provide evidence that male and female students differed in their patterns of coursework; females completed relatively more English and foreign language and males completed relatively more mathematics and science. Furthermore, students from lower income families adhered to the rigorous and foreign language profiles less than students from higher income families.

3-23-2009

The Student Right-to-Know Act requires colleges to provide institution-specific information on graduation rates for students initially enrolling full-time in the fall term. Not all students enroll in that fashion, especially at two-year institutions. We use data on degree-seeking students from the 1996/2001 Beginning Post-Secondary Survey to identify students for whom statistics are and are not reportable under the Act and to track their progress. Results indicate the published progress rates are substantially higher than the progress rates for the non-reportable populations, whether students enter a two-year or a four-year institution. While progress rates for the two samples are significantly correlated within four-year institutions, they are not within two-year institutions. For those beginning at two-year institutions, the progress rates reported under the Student Right-to-Know Act are indicative of neither their absolute nor their relative (cross-institution) probability of success. Policy makers and prospective students will not make efficient decisions without better information.

2-8-2009

Data mining provides both systematic and systemic ways to detect patterns of student engagement among students at hundreds of institutions. Using traditional statistical techniques alone, the task would be significantly difficult—if not impossible—considering the size and complexity in both data and analytical approaches necessary for this task. This study presents a step-by-step review on how the data mining technique is utilized to develop an institutional typology based on student behavioral data. The result provides a fresh angle to understand similarities and differences among four-year undergraduate colleges and universities, shifting away from previous institutional typologies, such as those based on institutional mission, resources, or reputation. The institutional engagement typology is derived through student behavioral data, and therefore, is advantageous in that it retains one of the most important components in understanding higher education—student behaviors. This data mining-based study broke new conceptual and methodological ground, and its resulting institutional learning engagement typology offers new perspectives on peer institution comparison, congruence between students and their institutions, as well as policy development regarding educational quality.

1-15-2009

This study is an investigation of the derivation of scores that predict whether or not prospective first-time freshmen will apply or will enroll and whether or not first-time freshman enrollees will graduate using data from the ACT (American College Testing) assessment. Using a regression methodology, four basic scores are derived to be independent of academic ability, which is indicated by a fifth score. Using cross-validation populations, each of the scores is shown to predict the desired behavioral criterion quite well, and each should serve its intended purpose. The paper discusses potential uses of the scores and examines the inclusion or exclusion of no-response items (where the individual did not give a response), the optimal number of data items to include in an enrollment management score, and other characteristics of the scores.

11-15-2008

This study is an investigation of the derivation of scores that predict whether or not prospective first-time freshmen will apply or will enroll and whether or not first-time freshman enrollees will graduate using data from the ACT (American College Testing) assessment. Using a regression methodology, four basic scores are derived to be independent of academic ability, which is indicated by a fifth score. Using cross-validation populations, each of the scores is shown to predict the desired behavioral criterion quite well, and each should serve its intended purpose. The paper discusses potential uses of the scores and examines the inclusion or exclusion of no-response items (where the individual did not give a response), the optimal number of data items to include in an enrollment management score, and other characteristics of the scores.

1-18-2008

Enrollment forecasting is the central component of effective budget and program planning. The integrated enrollment forecast model is developed to achieve a better understanding of the variables affecting student enrollment and ultimately perform accurate forecasts. The transfer function model of the autoregressive integrated moving average (ARIMA) methodology and linear regression model are major forecasting techniques. The structural approach embedded in the models allows the researcher to construct candidate models, eliminate inappropriate ones, and retain the most suitable model. In addition, the expert system for the ARIMA model is a supplementary tool used to verify the resulting models in terms of model structure and forecasting accuracy.

The enrollment series of interest is the 1962 – 2004 student enrollment for Oklahoma State University (OSU). Fifteen independent variables are used in an attempt to increase explanatory power. These variables include demographics (Oklahoma high school graduates and competitor college enrollment from the University of Oklahoma), state funding, economic indicators, (e.g., state unemployment rate and gross national product), and one-year lagged mographics and economic indicators.

The best ARIMA and linear regression models yield remarkably high R-squared values and exceptionally small mean absolute percentage errors (MAPEs), respectively. Moreover, they contain two identical demographics: Oklahoma high school graduates and one-year lagged OSU enrollment. Hence, the first-order autoregressive models appropriately depict the longitudinal and aggregated OSU enrollment series. An additional linear regression model shows that one-year lagged Oklahoma high school graduates and three economic indicators significantly contribute to OSU enrollment. This integrated enrollment forecast model has demonstrated its model validity and accuracy. Hence, it could be replicated for comparable universities elsewhere.

10-19-2007

One of the most valued approaches to identify course prerequisites relies on statistical techniques, typically requiring institutional-research support. Yet these techniques are often inadequate for prerequisite-identification purposes. A discussion of the reasons for the inappropriateness of these techniques is presented. A review of common practices used to identify prerequisites is also presented, and some basic notions related to the concept of prerequisites are discussed. A tool conceived by one of the authors’ institutional-research offices for the purpose of identifying prerequisites is presented and demonstrated.

07-31-07

To provide quality education, a university needs to make available a well-equipped computing center. However, such centers are expensive, and their provision is a problem for administrators when budgets are tight. Hence, it is important that money be invested in services that will enhance user satisfaction the most. This study explores the relationship between service quality and user satisfaction in a university computing center. Two hundred and seventy-four successful questionnaires were collected from faculty members, staff, and students of Soochow University in Taiwan. Analysis of the data revealed that network infrastructure, consultancy and maintenance, and system quality are particularly important, and should be considered core services of a computing center. Other services, such as network function, classroom management, and administrative procedure are considered supporting services.

02-26-07

In applying institutional research to higher education challenges, our profession frequently looks at our responsibilities and roles. It seems that we have had this preoccupation since AIR became a professional association in the 1960s. Peterson (1999) identifies 12 major professional self-analyses extending from 1960 forward in a succession of what he terms an “endless debate over the nature and role of institutional research.” Delaney (2001) reviews the findings of eight additional studies since 1976 that describe the challenges and opportunities facing institutional researchers.

11-01-06

This study examined the transfer trends for, and significant differences between, transfer students who graduated from community college career-oriented programs and students who graduated from liberal arts programs and looked at possible factors which affect these trends and differences. The results indicated students who made better grades in the Community College made better grades in the Senior College. Also, for transfers who graduated from career-oriented programs, older students and students who had been placed in remedial courses earned higher senior college GPA than their younger counterparts and students who did not attend remedial courses. For transfers who graduated from liberal arts programs, males and Hispanic students did not do as well as other students.

07-19-06

This exploratory data mining project used distance-based clustering algorithms to study three indicators of student behavioral data collectively called AB-Index, and established a typology of six types of learners for a suburban community college. The study is based on the notion that student behavioral data are a good basis for new ways of doing research studies rather than using non-behavioral data, such as gender or race and intended educational goals. The discoveries from this data mining endeavor are meaningful for understanding and measuring students' behaviors. The study encapsulated and discussed several fresh and novel topics and analytical approaches. The study uncovered previously unknown differences in both output (FTES) and outcomes (GPA, Persistence) across the learner types which may greatly enhance a college's ability to monitor the changes and to make appropriate adjustment to enrollment and teaching strategies. The study noted the lack of predictive power of traditional indicators, such as race or gender, across learner types within the typology. The study also employed several less often used data visualization techniques, such as drop-line charts and the Web graph.

05-03-06

The effects of various college preparation programs, class ranking, and student background characteristics on college retention were studied. The data were obtained from the National Education Longitudinal Study (NELS):1988-2000 and NELS:88/2000 Postsecondary Education Transcript Study. The sample contained 4,445 first-time freshmen students who enrolled in four-year institutions between 1992 and 1994. Using survival analysis techniques, the focal point of the study was to examine longitudinal impact of high school programs on college retention. Participation in ACT/SAT preparation courses reduced the likelihood of departure by 42% or 55% in the second or third year in college, while receiving assistance with the financial aid application increased the odds of departure by 89% in the second year.

02-14-06

There was a time when virtually all students pursuing the master’s degree were required to submit to and successfully complete comprehensive examinations, most of which featured both written and oral elements. In this more traditional period, the acquisition of the master’s degree was in large part a qualifying ritual for admission to a Ph.D. program which, in turn, assumed a subsequent professional life in the academy. With the gradual shift in the character of the degree from the purely academic to the at least quasi-professional, alternative approaches to evaluating student performance are emerging as exit requirements for the master’s degree. This paper reports the results of a survey, featuring both quantitative and qualitative components, designed to investigate the status of the traditional comprehensive examination, as well as the nature and distribution of alternative assessment methods.

11-03-05

Using a typological schema derived from freshman survey data and other empirical measures, this study examines the link between students' traits upon entry to college and their college academic performance and skill development in various areas as measured at the exit point. The findings indicate the typological schema is predictive of student outcomes in terms of self-reported gains and future plans, validating the definitions of student types to a significant extent. Also, the findings help institutional leaders reflect upon questions of alignment between institutional mission focus on the one hand and student interests and aptitudes on the other: How well do various aspects of our programs meet the distinctive needs of diverse students? Whom do we serve well, and whom less well, and in what ways?

08-24-05

The selection of faculty in academic institutions is an important process - one that has long-lasting effects on an institution's ability to fulfill its mission. Faculty influence the quality of the education delivered, the effectiveness of the programs and activities offered, and the financial efficiency of the delivery processes. Failed searches waste time and incur needless expense. Inadequate searches - those that result in candidates who are poorly qualified or lack organizational fit can have profound negative impact on these three key strategic elements. Hiring the wrong person may lead to dysfunctional departments, dissatisfied students, and, eventually, repeat efforts. Applying a sound process, one that structures the search, identifies and relates the selection criteria, allows for qualitative and subjective assessments, and encourages full participation of search committee members, can enhance the desired outcome, i.e., identification of best candidates that will contribute to the quality, effectiveness, and efficiency of higher education.

06-30-05

Logistic and Cox regression methods are practical tools used to model the relationships between certain student learning outcomes and their relevant explanatory variables. The logistic regression model fits an S-shaped curve into a binary outcome with data points of zero and one. The Cox regression model allows investigators to study the duration and timeline of the critical events, which are also a binary and dichotomous measure. This paper introduces logistic and Cox regression models by illustrating examples, implementing step-by-step SPSS procedures, and further comparing the similarities and differences of the model characteristics. Logistic regression analysis was conducted to investigate the effects of the explanatory variables such as pre-admission variables, college cumulative GPAs, and curriculum tracks on student licensure examination. Moreover, logistic regression analysis was employed to quantify the effect (odds or odds ratio) of specific explanatory variables on the binary outcome holding other variables constant. With regards to Cox regression analysis, the outcome variable of interest was the timing of experiencing academic difficulty--dismissal, withdrawal, and leave of absence. The Cox regression model was used to detect when students were most likely to experience academic difficulty beyond their matriculation. The model also allowed the investigators to measure the effect (relative hazard or hazard ratio) of specific risk factors on the academic difficulty after adjusting for other factors. Identifying the occurrence of critical events along with the explanatory variables, college administrators and faculty could implement intervention strategies to ensure student success.

04-06-05

In 2003, the Association for Institutional Research (AIR) initiated the Best Visual Presentation (BVP) award to acknowledge the contributions made through new ways of professional communication, in addition to those made through more traditional scholarly formats. Fueled in part by advanced technologies, as well as by changing notions of organizational decision-making processes and individual learning, visual presentations are increasingly important in enhancing our understanding of issues relevant to the higher education enterprise. The ability to develop effective visual presentations is an important addition to the narrative and quantitative techniques more traditionally employed in scholarly and professional settings. Visual presentations are important for communicating with various audiences. The award recognizes expertise in this area, and is expected to help elevate professional norms surrounding this important institutional research function.

01-21-05

Service-learning courses focus on both service experience and academic learning. Academic content is covered in both the classroom and the service experience, and the service experiences are reflected upon and processed in the classroom. Based on educational values, potential outcomes can be classified as development of personal competence, interpersonal relationship, and perception of community service as a responsibility of charity or perception of community service as a responsibility of social justice. The Student Service-Learning Course Survey (SSLCS) is designed to measure these four outcome domains. It draws attention to the dichotomy between the two kinds of citizenship and operationalizes the concepts by developing questions to measure the differences. The present study explored the factorial validity of SSLCS and the factorial invariance across gender groups using confirmatory factor analysis. The results of our study indicate that the four factors of SSLCS are validly measured and the partial factorial invariance across gender groups lends support for comparison between female and male groups.

09-08-04

This manuscript summarizes an institutional research study carried out at Bowling Green State University (BGSU) concerning factors affecting time to bachelor's degree attainment. Tuition sensitivity and concern about efficient use of institutional resources point to the need for decreasing students' time-to-degree. This study enlarges upon an earlier one; it investigated the effects of state- and institutionally-sponsored policies that were designed to decrease time-to-degree, and also some additional factors such as student participation in learning communities and first year programs. Time-to-degree decreased in four years since the previous study. Participation in a tuition discount program, total student credit hours earned, average credit hour load per semester, and student credit hours transferred were among the strongest predictors of time-to-degree. The study highlights the use of descriptive and bivariate statistical techniques, as well as important considerations in the use of applied multiple regression.

05-26-04

The ordinal regression method was used to model the relationship between the ordinal outcome variable, e.g., different levels of student satisfaction regarding the overall college experience, and the explanatory variables concerning demographics and student learning environment in a predominantly minority health sciences center. The outcome variable for student satisfaction was measured on an ordered, categorical, and four-point Likert scale--'very dissatisfied', 'dissatisfied', 'satisfied', and 'very satisfied'. Explanatory variables included two demographics, e.g., gender and ethnic groups, and 42 questionnaire items related to the satisfaction of faculty involvement, curriculum contents, support services, facilities, and leisure activities at the college. The major decisions involved in the model building for ordinal regression were deciding which explanatory variables should be included in the model and choosing the link function (e.g., logit link or complementary link) that demonstrated the model appropriateness. In addition, the model fitting statistics, the accuracy of the classification results, and the validity of the model assumption, e.g., parallel lines, were essentially assessed for selecting the best model. The research findings indicated that explanatory variables such as faculty competence and student-faculty relations were significantly associated with the satisfaction of the overall college experience. This discovery suggests that faculty members have played a major role in creating a pleasant environment to facilitate student satisfaction. In addition, the curriculum content regarding health promotion and disease prevention was significantly associated with the satisfaction of the overall college experience. It may also provide strong evidence that a specific component of the medical curriculum addressed student needs and contributed to the fulfillment of the medical college goal, e.g., delivery of primary care through health promotion and disease prevention.