Evaluating the Potential Incorporation of R into Research Methods ...

I was recently completing some professional development activities that required me to write a report on a self-chosen topic related to diversity in student backgrounds. I chose to use the opportunity to reflect on the potential for using R to teach psychology students research methods. I thought I'd share the report in case it interests anyone.

Abstract

Research methods is fundamental to psychology education at university. Recently, open source software called R has become a compelling alternative to the traditionally used proprietary software called SPSS for teaching research methods. However, despite many strong equity and pedagogical arguments for the use of R, there are also many risks associated with its use. This report reviews the literature on the role of technology in research methods university education. It then reviews literature on the diversity of psychology students in terms of motivations, mathematical backgrounds, and career goals. These reviews are then integrated with a pedagogical assessment of the pros and cons of SPSS and R. Finally, recommendations are made regarding how R could be best implemented in psychology research methods teaching.

Introduction

Training in research methods is a fundamental component of university education in psychology. However, for many reasons subjects in research methods are challenging to teach. Students have diverse mathematical, statistical, and computational backgrounds; students often lack motivation as they struggle to see the relevance of statistics. These issues are compounded by undergraduate majors in psychology that typically have several compulsory research methods subjects. Given the competition for entry into fourth year and post-graduate programs, such research methods subjects can be threatening to struggling students.

As with many other universities, research methods in psychology at Deakin University has largely been taught using software called SPSS. This software is typically taught as a menu driven program that is used to analyse data enabling standard data manipulation, analyses, and plotting. While SPSS is relatively user-friendly for standard analyses, there are several problems with teaching students how to use it. In particular, it is very expensive; thus, students can not be assumed to have access to it either from home for doing assignments or in future jobs. In addition, while SPSS makes it easy to perform standard analyses, it is very difficult to alter what SPSS does to perform novel analyses. Thus, for many reasons some lecturers are seeking alternative statistical software for teaching research methods.

While there are many programs for performing statistical analysis, one particularly promising program, known simply as "R", has emerged as a viable alternative to SPSS. R is open source so it is free for students and staff. Thus, students can use R from home when completing assignments, and can use it in any future job. It has a vast array of statistical functionality. Despite these benefits, it does present several challenges to incorporation into psychology. Analyses are typically performed using scripts. It is often less clear how to run certain analyses. The program often assumes a mental model of a statistician rather than an applied researcher.

Thus, the current report had the following aims. The first aim was to evaluate the pros and cons of using R to teaching psychology students research methods. The second aim was to evaluate how best R could be incorporated. In order to achieve these aims, the report is structured into several parts. First, the general literature of software in statistics education is reviewed. A particular focus is placed on diversity in student backgrounds in applied fields. Second, the backgrounds and career goals of psychology students are presented with reference to the literature and practical experience. Third, the pros and cons of using R versus SPSS is presented. Finally, ideas about how best to incorporate R into statistics education are reviewed.

Statistics education and the role of software

There is a substantial literature on statistics education and the role of statistical software in statistics education. Tiskovskaya and Lancaster (2012) provide one review of the challenges in statistics education. Their review is structured around teaching and learning, statistical literacy, and statistics as a profession. Of particular relevance to teaching statistics in psychology they outline several problems and provide relevant references to the statistical literature. With references taken from their paper, these issues include: inability to apply mathematics to real world problems (e.g., Garfield, 1995); mathematics and statistics anxiety and motivation issues in students (e.g., Gal & Ginsburg, 1994); inherent difficulty in students understanding probability and statistics (e.g., Garfield & Ben-Zvi, 2008); problems with background mathematical and statistical knowledge (e.g., Batanero et al 1994); the need to develop statistical literacy which translates into everyday life (e.g., Gal, 2002); and the need to develop assessment tools to evaluate statistical literacy. Tiskovskaya and Lancaster (2012) also reviewed potential statistics teaching reforms. They note that there is a need to provide contextualised practice, foster statistical literacy, and create an active learning environment.

Of particular relevance to the current review of statistical software, Tiskovskaya and Lancaster (2012) discuss the role of technology in statistics education. The importance of technology has increased as computers have become more powerful. This has enabled students to run powerful statistical programs on their computer. Some teachers have used this power to focus instruction on interpretation of statistical results rather than computational mechanics. Chance et al (2007) further note the value of using interactive applets to explore statistical concepts and taking advantage of internet resources in teaching.

Chance et al's (2007) review also summarises several useful suggestions for incorporating technology in statistics education. Moore (1997) notes the importance of balancing using technology as a tool with remembering that the aim is to teach statistics and not the tool per se. Chance et al (2007) notes particularly valuable uses of technology include analysing real datasets, exploring data graphically, and performing simulations. Chance et al (2007) also review statistical software packages for statistics education noting both the advantages and disadvantages of menu-driven applications such as SPSS.

Chance et al (2007) offer several recommendations for incorporating technology into statistical education. First, they highlight the importance of getting students practicing not just performing analyses, but also focusing on interpretation. Second, they recommend that tasks be carefully structured around exploration so that students see the bigger picture and do not get overwhelmed with software implementation issues. Third, collaborative exercises can force students to justify to their fellow students their reasoning. Fourth, they encourage the use of cycles of prediction and testing, which technology can facilitate (e.g., proposing a hypothesis for a simulation and then testing it).

Chance et al (2007) summarise the GAISE report by Franklin and Garfield (2006) on issues to consider when choosing software to teach statistics. These include (a) "ease of data entry, ability to import data in multiple formats, (b) Interactive capabilities, (c) Dynamic linking between data, graphical, and numerical analyses, (d) Ease of use for particular audiences, and (f) Availability to students, portability" (p.19). Franklin and Garfield (2006) also discuss a range of other implementation issues, such as the amount of time to allocate to software exploration, how much the software will be used in the course, and how accessible the software will be outside class. Garfield (1995) suggest that computers should be used to encourage students to explore data using analysis and visualisation tools. Running simulations and exploring resulting properties is also particularly useful. Thus, overall these general considerations regarding statistics education can inform the choice of statistical software. However, the above review also highlights that choice of software is only a small part of the overall unit design process.

Psychology students and the role of statistics

Pathways of psychological studies in Australian universities typically involve completing a three year undergraduate major in psychology, then a fourth year, followed by post-graduate professional or research degrees at masters or doctoral level. As a result of student interest, specialisation, and competition for places, there is a reduction over year levels. From my experience both at Melbourne University and Deakin University, a ball park estimate of the student numbers as a percentage of first year load, would be 40% at second year, 35% at third year, 10% at fourth year, and 3% at postgraduate level. This is from one to two thousand students at first year. Of course these are just rough estimates, but the point is to highlight that there are huge numbers of students getting a basic undergraduate education in psychology; in contrast, the few that go on to fourth year have both a high skill level in psychology also different needs regarding research methods.

Psychology students are taught using the scientist-practitioner model. A big part of science in psychology is research methods and statistics. Students typically complete two or three research methods subjects at undergraduate level, another unit in fourth year, and potentially further units at postgraduate level. The diverse nature of psychology student backgrounds, motivations, and career outcomes can make research methods a difficult subject to design and teach. Psychology undergraduate students also have diverse career goals and outcomes. Many go on to some form of further study. Those that exit at the end of third year have diverse employment outcomes. For example, Borden and Rajecki describe one US sample finding that income was lower than many other majors and that roles included administrative support (17.6%), social worker (12.6%), counsellor (7.6%) along with a diverse range of other jobs. Of those that go on, some will continue with research, but others will go into some form of applied practice.

In terms of research methods in psychology, there are a diverse range of goals. First, research methods is meant to help all students learn to reason about the scientific literature in psychology. Second, for students who continue with psychology research methods should give students the skills to be able to complete a quantitative fourth year and postgraduate thesis. For a subset of students, quantitative skills is part of their marketable skillset that they can take into future employment. Furthermore, for a small group of students who go on to do their PhD and then join academia, research methods skills are fundamental to the continuation of good research and the vitality of the discipline.

In addition to diverse aims are the diverse student backgrounds in psychology. In particular, there are typically no mathematics pre-requisites. By casual observation many students seem motivated to find work in the helping professions, and particularly as clinical psychologists. Many studies have discussed the challenges of teaching statistics to psychology students. For example LaLonde and Gardner (1993) proposed and tested a model of statistics achievement that combined mathematical aptitude and effort with anxiety and motivation as predictors.

Thus, in combination this diversity in background and student goals introduce several challenges when teaching research methods. For some students the main goal is to introduce a moderate degree of statistical literacy. For others, it is essential that they are at least able to analyse their thesis data in a basic way. A final group of advanced students needs skills that will allow them to model their data in a sophisticated way to contribute to the research literature. Thus, there is a tension between presenting ideas in an accessible way for all students versus tailoring the material for advanced students so they can truly excel.

This tension exists in many different aspects of research methods curriculum. Research methods can be taught with varying degrees of mathematical rigour and abstraction. Teaching can emphasise interpreting output or it can emphasise computational processes. It can also vary in the prominence of software versus ideas. In particular the correct choice of statistical software can substantially interact with these issues of balancing rigour with accessibility. In particular, tools like SPSS are more limited than R, but such limits can make standard analyses easier.

Aiken et al (2008) reviewed doctoral education in statistics and found that most surveyed programs were using SAS or SPSS primarily. They described a case study in curricular innovation in terms of novel topics emerging followed by initiatives from substantive researchers. Textbooks and software that make techniques accessible to psychology graduates also facilitate the teaching process. In some respects, as R has become more accessible through usability innovation and as the needs of data analysts have become more advanced, the argument for R has become more compelling.

Whether to use R in psychology research methods

Pros and cons of R

The above review thus provides a background for understanding both statistics education in general and the diversity in the background and goals of psychological students. The following analysis compares and contrasts R and SPSS as software for teaching research methods in psychology. This initial comparison focuses on price, features, usability and other considerations.

In terms of price, an initial benefit of R is that it is free. It is developed under the GNU open-source licence. It is free to the university and free to students. In contrast a student licence to SPSS for a year is around $200; A professional licence is around $2,000; and SPSS charges expensive licencing fees to the university. R would make it easier to get students to complete analyses from home. Requiring students to purchase SPSS creates equity issues and may even encourage some students to engage in software piracy. If as Devlin et al (2008) suggest that essential textbooks create economic hardship, even more expensive statistical software would compound this problem.

In terms of features, SPSS and R both run on Windows, OSX, and Linux. They both support most standard analyses that students may wish to run. However, R has a larger array of contributed packages. SPSS has several features including a data entry tool, a menu-driven GUI, and an output management system for tables and plots that R does not have. R makes it a lot easier to customise analyses, perform reproducible research, and simulations.

In terms of flexibility SPSS and R both have options for performing flexible analyses. However, R makes it a lot easier to gradually introduce customisation by building on standard analyses. It is also flexible in how it can be used because of the open source licence. R is particularly suited to advanced students who can benefit from the easier pathway it provides for growing statistical sophistication.

In terms of usability R and SPSS are quite different. R assumes greater knowledge about statistics. SPSS has an interface that is more familiar to standard Windows-based programs. R is a programming language with a less consistent mental model to standard Windows programs. R has a steeper initial learning curve, but shallower intermediate curve. R encourages students to gradually develop statistical skills. In particular R has several quirks which create difficulties for the novices (e.g., learning details of syntax, escaping spaces in file paths, treating strings as factors versus character variables, etc.). There are also many things that are easy in SPSS that are difficult in R. Some examples include: variable labels and modifying meta data, editing loaded data, browsing loaded data, producing tables of output, viewing and browsing statistical output, generating all the possible bits of output for an analysis, importing data, standard analyses that SPSS already does, and interactive plotting.

R and SPSS can also be compared in terms of existing resources. There are many online resources for both R and SPSS. Psychology-specific R resources exist but are less plentiful than for SPSS. Furthermore, existing psychology supervisors, research methods staff, and tutors are probably more familiar with SPSS which may cause issues when transitioning teaching to R. That said, many supervisors either train their students directly in the software that they want their students to use or they let the student handle details of implementation.

Mental Models

When choosing between SPSS and R it is worth considering the mental models required to use SPSS and R. These mental models both guide what needs to be trained and also may suggest the gap that needs to be closed between students' initial mental models and that which is required by the software.

The SPSS mental model is centred around a dataset. The typical workflow is as follows: (a) import or create data; (b) define meta data; (c) menus guide analysis choice; (d) dialog boxes guide choices within analyses; (e) large amounts of output are produced; (f) instructional material facilitates interpretation of output; (g) output can be copy and pasted into Word or another program for a final report. Custom statistical functions or taking SPSS output and using it as input to subsequent functions is not encouraged for regular users. Thus, overall the system guides the user in the analysis.

In contrast, R requires that the user guides the software. Thus, the R workflow is as follows: (a) Setup raw data in another program; (b) import data where often the user will have multiple datasets, meta datasets, and other data objects (e.g., vectors, tables of output); (c) transform data as required using a range of commands; (d) perform analyses, where command identification may involve a Google search or looking up a book, and understanding arguments in a command can be facilitated by internal documentation and online tools; (e) because the resulting output is minimal, the user often has to ask for specific output using additional commands; (f) much of what is standard in SPSS requires a custom command in R, but also much of which does not exist in SPSS can be readily created by an intermediate user; it is much easier to extract out particular statistical results and use that as input for subsequent functions; (g) while output can be incorporated into Word or Excel, users are encouraged to engage in various workflows that emphasise reproducible research.

Summary

Thus, overall SPSS is well suited to a menu-driven standardised analysis workflow which meets the needs of many psychology students. R is particularly suited to statisticians that need to perform a diverse range of analyses and are more comfortable with computer programming and statistics in general. R requires greater statistical knowledge and it encourages students to have a plan for their analyses. R also requires students to learn more about computing including programming, the command-line, file formats, and advanced file management. The emphasis on commands creates a greater demand on declarative memory which in turn makes R more suited to students who will perform statistical analysis more regularly. However, the flexibility and nature of R means that it can be used in many more contexts than SPSS such as demonstrating statistical ideas through simulation.

Overall, there are clearly pros and cons of both SPSS and R. R is particularly suited to more advanced students. Occasional users may be more productive initially with SPSS. That said, the many students who never go on with any data analysis work, may learn as much or more by using R. It also remains an empirical question to see how different psychology students might handle R. Thus, the remainder of this report focuses on what implementation of how R could be implemented most effectively.

How to use R in psychology research methods

When considering implementation of R in psychology, it is useful to look at existing textbooks and course implementations. When considering textbooks, it is important to note that psychology tends to use a particular subset of statistical analyses. It also often has analysis goals that differ from other fields. For example, there is a greater emphasis on theoretical meaning, effect sizes, complex experimental designs, test reliability, and causal interpretation. While there are many textbooks that teach statistics using R, only recently have books emerged that are specifically designed to teach R to psychology students. The two main books are Andy Field's "Discovering Statistics Using R" and Dan Navarro's "Learning Statistics with R". An alternative model is to take a more generic R textbook or online resource and combine it with a more traditional psychology textbook such as David Howell's "Statistical Methods for Psychology". In particular, there are many user friendly online resources for learning R such as http://www.statmethods.net/ or Venables, Smith and the R Core Team's "An Introduction to R". Whatever textbook option is chosen an important part of learning R involves learning how to get help. Thus, training should include learning how to navigate online learning resources and internet question and answer sites that are very effective in the case of R (e.g., stackoverflow.com).

Dan Navarro (2013) has written a textbook that teaches statistics to psychology students using R. Navarro (2013) presents several argument for using R instead of a different commercial statistics package. These include: (1) the benefits of the software being free and not locking yourself into expensive proprietary software; (2) that R is highly extensible and has many cutting edge statistical techniques; and (3) that R is a programming language and learning to program is a good thing. He also observes that while R has its problems and challenges, overall it provides the best current available option. Thus, overall, his approach is to inspire the student to see the bigger picture about why they are learning R. Navarro then spends two chapters introducing the R programming language. Starting with simple calculations, many basic concepts of variables, assignment, extracting data, and functions are introduced. Then, standard statistical techniques such as ANOVA and regression are presented with R implementations.

Overall, both these textbooks provide insight into how R could be implemented. Teaching with R provides some opportunity to teach statistics in a slightly deeper way. However, various recipes can be provided to perform standard analyses. Teaching R also requires taking a little extra time to teach the language. The menu-driven interface to R called R-Commander also provides a way of introducing R in a more accessible way. The infrastructure provided by R also provides the opportunity to introduce many important topics such as bootstrapping, simulation, power analysis, and customised formulas. Weekly analysis homework not easily possible with SPSS could consolidate R specific skills.

An additional issue of implementation relates to when R should be introduced. Fourth year provides one such opportunity where the students that remain at this level tend to be more capable and have some initial experience in statistics. Fourth year research methods is a very important subject. It is often designed to prepare students to analyse multivariate data. It is also designed to prepare students to be able to analyse data on their own including preliminary analyses, data cleaning, and transformations. R supports all the standard multivariate techniques that are currently taught at fourth year level. These include PCA, factor analysis, logistic regression, DFA, multiple regression, multilevel modelling, CFA, and SEM. R also makes it easier to explore more advanced methods such as bootstrapping and simulations.

Conclusion

Ultimately, it is an empirical question as to whether using R would provide a more effective tools for research methods education in psychology. It may be useful to explore the idea with some low-stakes optional post-graduate training modules in R. Such programs may give a sense of the kinds of practical issues that arise with students when learning to use R. If R is to be rolled out to all of fourth year psychology, this would be a high risk exercise. It would be important to evaluate the student learning outcomes in a broad way. In particular, it would be important to see any effect on analysis performance in fourth year theses.

References

• Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America. The American Psychologist, 63(1), 32-50.
• Batanero, C., Godino, J., Green, D., and Holmes, P. (1994). Errors and Difficulties in Understanding Introductory Statistical Concepts. International Journal of Mathematical Education in Science and Technology, 25 (4), 527?547.
• Borden, V. M., & Rajecki, D. W. (2000). First-year employment outcomes of psychology baccalaureates: Relatedness, preparedness, and prospects.Teaching of Psychology, 27(3), 164-168.
• Chance, B., Ben-Zvi, D., Garfield, J., and Medina, E. (2007). The Role of Technology in Improving Student Learning of Statistics. Technology Innovations in Statistics Education, 1(1). url: http://www.escholarship.org/uc/item/8sd2t4rr
• Devlin, M., James, R., & Grigg, G. (2008). Studying and working: A national study of student finances and student engagement. Tertiary Education and Management, 14(2), 111-122.
• Franklin, C. & Garfield, J. (2006). The GAISE (Guidelines for Assessment and Instruction in Statistics Education) project: Developing statistics education guidelines for pre K-12 and college courses. In G. Burrill (Ed.), 2006 NCTM Yearbook: Thinking and reasoning with data and chance. Reston, VA: National Council of Teachers of Mathematics.
• Gal, I. (2002). Adults' Statistical Literacy: Meanings, Components, Responsibilities. With Discussion. International Statistical Review, 70(1), 1-51.
• Gal, I. & Ginsburg, L. (1994). The Role of Beliefs and Attitudes in Learning Statistics: Towards an Assessment Framework. Journal of Statistics Education, 2(2). url: http://www.amstat.org/publications/jse/v2n2/gal.html
• Garfield, J. (1995). How Students Learn Statistics. International Statistical Review, 63(1), 25-34.
• Garfield, J. and Ben-Zvi, D. (2008). Developing Students' Statistical Reasoning: Connecting Research and Teaching Practice, Springer.
• Lalonde, R. N., & Gardner, R. C. (1993). Statistics as a second language? A model for predicting performance in psychology students. Canadian Journal of Behavioural Science, 25, 108-125.
• Moore, D.S. (1997). New pedagogy and new content: the case of statistics. International Statistical Review, 635, 123-165.
• Navarro, D. (2013). Learning statistics with R: A tutorial for psychology students and other beginners (Version 0.3) url: http://ua.edu.au/ccs/teaching/lsr
• Tishkovskaya, S., & Lancaster, G. (2012). Statistical education in the 21st century: a review of challenges, teaching innovations and strategies for reform. Journal of Statistics Education, 20(2), 1-55.
• Wiberg, M. (2009). Teaching statistics in integration with psychology. Journal of Statistics Education, 17(1), 1-16.

Source: http://jeromyanglim.blogspot.com/2013/07/evaluating-potential-incorporation-of-r.html

Susan Rice the Who jon bon jovi jon bon jovi Kliff Kingsbury Amish Mafia Dave Grohl