The Importance of Peer Feedback in Online Education

Ertmer, P. A., Richardson, J. C., Belland, B., Camin, D., Connolly, P., Coulthard, G., et al. (2007). Using peer feedback to enhance the quality of student online postings: An exploratory study. Journal of Computer-Mediated Communication, 12(2), 412-433.

The importance of feedback to students in education is a known factor, yet the research of how feedback impacts the online learner is something Ertmer and colleagues (2007) find little studied. In their article, Using Peer Feedback to Enhance the Quality of Student Online Postings: An Exploratory Study, Ertmer et. al (2007) sought to investigate the impact peer feedback has on both the overall quality of student posts and on their perceptions of the value of both giving and receiving feedback as part of an online class.

To frame their need for a study, Ertmer et. al (2007) presented a literature review. Feedback, they note, serves to help the learner evaluate their knowledge and can assist them in altering their viewpoints when presented new information. To do this, good feedback, according to the authors,  should help to “clarify what a good performance is” for the learner, assist the learner in developing their ability for “self-reflection and assessment,” help the learner gain “high quality information” about their learning, focus faculty-student interactions towards learning, increase motivation and self-esteem for the learner, help the learner “close the gap” between current performance and their final performance goals, and assist the educator in accessing information towards improving the quality of teaching (Ertmer et. al. 2007, 413-414). When it comes to online education, Ertmer and colleagues (2007) emphasized how instructor feedback can act “as a catalyst for student learning” and argue that, to be most effective, studies showed that online feedback should be timely, specific and consistent and can extend from formative to summative formats (p. 414). However, good feedback can overload the time and effort abilities of  the faculty member. To offset the increased workload good online feedback requires, Ertner and colleagues (2007) proposed investigating utilizing peer feedback as part of their instructional design. The advantages to this, they argued, is that it could increase the timeliness of feedback while helping the student to be part of the community of learners.  Studies summarized by Ertmer et. al (2007) indicate that the giving and receiving of feedback helps to increase collaborative meaning construction, increases the overall quality of discussions, and can give the learner greater “understanding and appreciation for their peers’ experiences and perspectives” (Ertmer et. al., 2007,p. 415). This, Ertmer et. al. (2007) can increase student motivation and satisfaction with a course and can increase learner autonomy. However, the authors also outlined that the literature indicates several drawbacks in using peer feedback. These included increasing student anxiety about participating in peer feedback, inexperience of students in providing quality peer feedback, and general negative perceptions about the overall quality of peer feedback.

Based on this information,  Ertmer et. al (2007) developed three research questions.These questions were (p. 416):

  1. What is the impact of peer feedback on the quality of students’ postings in an online environment? Can the quality of discourse/earning be maintained and/or increased through the use of peer feedback?
  2. What are students’ perceptions of the value of receiving peer feedback? how do these perceptions compare to the perceived value of receiving instructor feedback?
  3. What are students’ perceptions of the value of giving peer feedback?

To answer these which they utilized a case study of peer-feedback in a graduate level course. Within this course which ran a single semester, 15 students were required to post to weekly discussion questions and comment on the post of one classmate. For the first 6 weeks, only the course instructors provided feedback to each student. This feedback on their posts consisted of a numerical score relative to what level of Bloom’s taxonomy they demonstrated on their post, and overall comments regarding the quality of the posts made. After week six, students then were asked to provide feedback on two peers postings for each discussion using the same system as demonstrated by the faculty member earlier in the class. This feedback was moderated and collected by the faculty member, anonymized, and given to the posting student usually within two weeks of the original post deadline. The authors gathered qualitative and quantitative data from interviews with participants, scored ratings of the students’ weekly discussions, and the students’ pre- and post surveys.  The pre-surveys occurred at the end of the faculty feedback period and the post-survey following the end of the peer feedback perio. It is also worth noting that the authors opted not to use the students’ peer feedback scoring for their analysis as they felt there were inconsistency in scoring and instead two of the project researchers scored all the student posts and these were used to assess the first research question.

In evaluating the impact of peer feedback on the quality of student postings. Ertmer et. al (2007) compared the average scores on postings during the instructor feedback period (M=1.31) to that during the peer feedback period (M=1.33).  The data showed no significance difference in the quality of postings . Quality did not improve or decrease when compared between the forms suggesting that “peer feedback may be effective in maintaining quality…once a particular level of quality has been reached” (Ertmer et. al , 2007, p 421-422). These results were then compared to qualitative data from interviews conducted during the peer feedback period. Student interview responses, the authors argue, showed that students felt they used the peer feedback to improve their writing. In their discussion of their paper, the authors present that the lack of change in quality may be attributed to several factors including the variability in posting quality between required and optional posts (all of which were included in the tabulations), use of a limited post scoring system (two levels only), and discussion questions which were not designed intentionally to elicit higher-level analyses.

In evaluating the perception of the value of receiving peer versus instructor feedback as well the value of giving feedback, came from data of the pre- and post surveys as well as the interview responses. Overall the authors noted that students surveys showed a significant increase in the perceived importance of feedback from the start of the course to the end. But that overall feedback from the instructor was perceived as being more important than peer feedback at both the beginning and end of the course. Interview responses indicated to the authors that this may stem from student perceptions of the overall quality peer feedback and the belief that bias may be present in peer feedback. however despite this, students within the study did feel as if peer feedback was valued. This is reflected also in the fact that students valued giving and receiving feedback as part of their learning process. In discussing the greater perceived value for instructor feedback,  the authors noted that this was what was seen in other studies but that factors such as the up to two week delay in getting peer feedback (due to instructor intermediate processing) may have contributed to students perceiving this as being less timely and therefore of less value. In addition, concern about impact on grades (since peer feedback was incorporated into peer grades for the boards) may have also increased the anxiety associated with providing and receiving peer feedback.

In reflecting on the limitations of their study, the authors acknowledge the issues presented by a small study of relatively short duration and consider that future work should include more training for students on both the benefit of peer feedback and how to effectively rate peers feedback.  From this the authors conclude that feedback in general is of value in online courses and that, based on interview, students valued and learned from peer feedback even if their perceived of instructor feedback as being more important. They wrapped up their paper, by providing some general recommendations faculty could consider when trying to implement peer feedback in their online courses.

In examining this work, the use of both qualitative and quantitative data to reflect on three specific research questions was well situated.  The fact that this is an exploratory study suggests the authors are seeking to improve upon the general development of assessing feedback. Given this, there were several aspects of the study which were problematic which I think could be better addressed in future studies. In considering the impact of feedback, the fact that feedback was delayed in reaching the recipient (for up to two weeks) makes one question how much was the actual scoring of posts a measurement of the impact of the feedback the student received from peers (since these were delayed) versus the general progress of the student in developing their own self-regulating abilities after having received faculty feedback for the first five weeks of the term. Secondly, in reflecting on the discussion question design, the author’s noted that issues of construction may have limited how much students were prompted to approach the higher levels of Bloom’s taxonomy in their responses since many of the questions “were not particularly conducive to high-level responses” Ertmer et. al , 2007, p 426).  I would have liked to have seen a breakdown data between the differing discussions to see if perhaps there is a pattern that is being lost due to some underperforming questions. In addition, when reflecting on use of feedback and the scoring system, the authors assumed familiarity with using Bloom’s taxonomy  given their study population but, given that several indicated problems in using the system to score posts, there may be underlying population variations which could perhaps be impacting their data. Overall I would be interested in seeing if a expanded version of this study addressing the limitations noted by the authors and above could be applied to a larger population of undergraduates to see if these patterns hold true to lower level students.

 

New Literacies: Risks, Rewards, and Responsibilities

“To be literate tomorrow will be defined by even newer technologies that have yet to appear and even newer discourses and social practices that will be created to meet future needs. Thus, when we speak of new literacies we mean that literacy is not just new today; it becomes new every day of our lives” (Leu, 2012, p. 78)

New literacies are the “ways in which meaning-making practices are evolving under contemporary conditions that include, but are in no way limited to, technological changes associated with the rise and proliferation of digital electronics” (Knobel and Lankshear, 2014, p. 97). It involves examining how, through the use of digital technology, the learner of today can come to identify, understand, interpret, create and communicate knowledge in novel and often unconventional ways.  While the incorporation of new literacies allows the educator to meet students where they are at, to engage and enliven learning through the relevancy and interest of the learner, restructure the power dynamics of learning, and to extend learning beyond the classroom, the approach of the educator towards engaging with new literacies is often a daunting undertaking.  In her article, Hagood (2012) highlights the processes by which teachers were introduced to and implemented new literacies into their classrooms. Working with a group of 9 middle school teachers during bi-monthly meetings over the course of a year, the author (2012) provided them with a three phase process towards introducing new literacies. These phases included an introduction phase to learn about new literacies, an exploration phase of the skills and tools necessary for new literacies, and a design and implement phase. The output was an inquiry-based project incorporating new literacies the educators could use in their classes. Using the participants’ reflections on this process, Hagood (2012) outlined their takeaways towards the implementing new literacies so as to lessen push-back, increase interest for participation and overall increase teacher satisfaction with incorporating new literacies. These included starting small and learning to implement new literacies through pre-existing assignments,  test trying new literacies to facilitate learning when traditional avenues fail, and expecting to fail and retry as part of the process for developing their educator skills with new literacies. Hagood (2012) noted that while many of the participants recognized the fact that students were well ahead in their connectedness to digital technology, that this was not the motivator for their implementation of new literacies. Rather it was the fact that many of the participants felt invigorated by what they saw their students capable of producing, the increased engagement of their students, by their own personal growth, and by their renewed enjoyment of teaching through new literacies. In addition, the educators felt that they developed a collaborative network which not only pushed them to stay on task but also made them feel more invested in sharing what they had learned thereby reiterating the connectedness to context and people that comes with new literacy.

While this article lacks in any quantifiable data with regards to how implementing digital literacy impacted student and teach motivation and student success within these classes, the incorporation of the teacher’s voices in reflecting on what resulted carries great weight in thinking about how this introduction of new literacies must be transformed into workable practices for the educator. This was a single small group in a single school from a single training year and Hagood (2012) presents no follow-up or check-in to see how these teachers are fairing in their use of new literacies in the following years. Have they expanded their incorporation of new literacies beyond the one inquiry-based project and how did they do this? Or perhaps they limited themselves to the one project, change projects, or abandoned new literacies altogether? What obstacles came about over time which impacted how they developed their skills and their overall implementation of new literacies? These are questions this article doesn’t address but are of interest when thinking about how to aid educators in exploring and adopting new literacies. What did their students think of these new literacies

In thinking about research, the above questions bear greater examination.  It would be interesting to expand upon this towards examining the best processes for implementing new literacies by examining outcomes such as motivation, efficacy, self-directedness, and overall success for both student and teacher.

Hagood, M. C. (2012) Risks, Rewards, and Responsibilities of Using New Literacies in
Middle Grades. Voices from the Middle, Volume 19 Number 4, May 2012

Leu, D. J., & Forzani, E. (2012). New literacies in a Web 2.0, 3.0, 4.0, …∞ world. Research in the Schools, 19(1), 75-81

Knobel, M., & Lankshear, C. (2014). Studying new literacies. Journal of Adolescent & Adult Literacy, 57(9), 1-5

 

 

 

Digital Games, Design and Learning: A Meta-Analysis

Clark, D. B, Tanner-Smith, E.E, and Killingsworth, S.S. (2016) Digital Games, Design and Learning: A Systematic Review and Meta-Analysis. Review of Educational Research 86(1):  79-122.

Within this article, Clark, Tanner-Smith and Killingsworth (2016) offer a refined and expanded evaluation of research on digital games and learning.  To ground their study, the authors summarize three prior meta-analyses of digital games. It is from these three studies and their findings that the authors develop a set of two core hypotheses about how digital games impact learning  that were tested in their meta-analysis. These two core hypotheses were further examined for that the authors term as moderator conditions and from this the authors developed sub-theories for each core theory to also test in their meta-analysis. Utilizing databases spanning “Engineering, Computer Science, Medicine, Natural Sciences, and Social Sciences” the authors sought research published between 2000 and 2012 to identify studies which examined digital games in K-16 settings, which addressed “cognitive, intrapersonal and interpersonal learning outcomes”(p. 82) and had studies which either had comparisons of digital games versus non-game conditions or utilized a value-added approach (something the prior meta-analyses ignored) to compare standard and enhanced versions of the same game. In addition they required a set of criteria for these studies to meet which included specifics on game design, participant parameters, and pre and post testing data which could be used to assess change in outcomes. Overall, they identified 69 studies which met the parameters outlined in their research procedures. From this population they discerned the following signficant patterns:

  1. In studies of game versus non-game conditions in media comparisons, students in digital game conditions demonstrated signficantly better outcomes overall relative to students in the non-game comparisons conditions (p. 94). This was significant for both cognitive and interpersonal outcomes (p.95). The number of studies with interpersonal outcomes was too small for statistical significance.
  2.  In studies of standard game and enhanced game versions through value-added comparisons, students in enhanced games showed “significant positive outcomes” relative to standard versions (p. 98). While overall there were too few studies with specific features for cross comparisons, the one feature of enhanced scaffolding (personalized, adaptive play)was present in enough studies and showed a significant overall effect (p. 99).
  3. Overall in examining game conditions, games which allowed the learner multiple play sessions performed better than those of single game play when compared against non-game conditions. Game duration (time played) seemed to have no impact on overall impact. (p. 99) These results did not vary even when considerations of the visual aspects of the game were measured.
  4. Despite what was seen in previous meta-analyses, there was no difference in outcomes for games paired with additional non-game instruction versus those without the additional non-game instruction. (p. 99)
  5. There was significant differences with player configurations within games. Overall, single player games had the most signficant impact on learning outcomes relative to group game structure and these outcomes were higher in single player games with no formal collaboration or competition. (p. 100). However games with collaborative team competition had signficantly larger effects on learning outcomes when compare to single competitive player games.
  6. Games with greater engagement of the player with actions within the game had greater impact than those with only a small variety of actions of the screen which did not change much over the course of play.
  7. Overall the visual and narrative perspective qualities of the games both simple and more complex game designs showed effectiveness in learning outcomes but overall schematic (schematic, symbolic or text-based) games were more effective than cartoon or realistic games

In reflecting on their findings, the authors recognized some limitations present based upon both their search parameters and their methodological breakdowns for analysis and encourage further examination of studies which fell outside of their range (for example simulation games) and greater examination of the subtleties of the individual studies included within their analysis before any larger generalizations can be made as to the specifics of best practices for game design.

Perhaps the most interesting aspect of this study is not the outcomes it presents for future study (even though these are great food for thought about intentional game design for educational purposes) but the proposition it makes that educational technology researchers should “shift emphasis from proof-of-concept studies (“can games support learning?”) and media comparison analyzes (“are games better or worse than other media for learning?”) to cognitive-consequences and value-added studies exploring how theoretically driven design decisions can influence situated learning outcomes for the board diversity of learners within and beyond our classrooms” (p. 116).

 

 

Online learning as online participation

Hrastinski, S. (2009). A theory of online learning as online participation. Computers & Education, 52(1), 78–82

In this article, Hrastinski (2009) presents the argument that online participation is a critical and often undervalued aspect of online learning and that models which relegate it to solely a social aspect for learning are ignoring its larger contributions to how students connect to materials and each other in the online environment.  In support of his ideas, Hrastinski (2009) offers an overview of literature on online participation which highlights that online learning is “best accomplished when learners participate and collaborate” (p.  79) and this translates into better learning outcomes when measured by “perceived learning, grades, tests and quality of performances and assignments” (p. 79).  In order to evaluate online participation, Hrastinski (2009) presents a conceptualization of online participation as more than just counting how often a student participates in a conversation but rather reflects on the online participation as “a process of learning by taking part and maintaining relations with others. It is a complex process comprising doing, communicating, thinking, feeling and belonging which occurs both online and offline” (p. 80). Hrastinski (2009) in reflecting on the work of others, offers up a view that participation creates community which in turn supports collaboration and construction of knowledge-building communities which foster learning between each other and the group at large. This learning through participation requires physical tools for structuring this participation and the psychological tools to help the learner engage with the materials.  This suggests examining aspects of motivation to learn within the structure of designing materials directed towards participation. He presents this means we should be looking at participation through more than just counting how much someone talks or writes but developing activities which require engagement with others in variety of learning modes.

While the importance of participation being seen as a critical component of online learning and the idea of reflecting on ways in which students may reflect online participation through more than just discussion boards is a good thing to see. Hrastinski (2009) offers little in terms of concrete examples to demonstrate how he sees this theory of online participation playing out through these different learning modes. While he may not have included examples as a way of preventing a formulaic approach to considering online participation, the inclusion of either examples or greater descriptions with how he sees faculty being able to construct both the physical and psychological tools of online participation would have been helpful for those less familiar with these to visualize the increasing ways they can apporach structuring online engagement.

As I have a deep interest in examining ways in which community and culture are structured through online classes and the impacts this has on learning, I found this article both intersting and encouraging for research avenues. In particular the rethinking he proposes on how we see online participation being constructed is encouraging and I would like to see the ways in which faculty and students may seem this idea of “what is participation” similarly or differently and the connection these perceptions have on how they both approach online larning and how they evaluate online learning.

 

 

Unpacking TPACK…

Gómez, M. (2015). When Circles Collide: Unpacking TPACK Instruction in an Eighth-Grade Social Studies Classroom. Computers in the Schools32(3/4), 278–299.

Coming into teaching from a graduate program in anthropology where the concern was not on how to teach but on how to research, the idea of evaluating the knowledge needed to effectively teach much less teach with technology is novel to this author.  Thus while the overall importance of Mishra’s and Koehler’s (2006) work on Technological Pedagogical Content Knowledge (TPCK) towards understanding the practice of teaching with technology is evident to this author, the actual process of implementation within the actual class design was difficult to visualize. To clarify the steps to how Mishra’s and Koehler’s model is applied and is implemented within course design, Gomez’s (2015)  illustrated applying TPACK to a case study of a single 8th grade teacher and two social studies classroom. Using data collected through classroom observations, formal and interviews, and the analysis of artifacts produced, Gomez used a constant comparative approach to organize the data along themes which related to the
intersections of TPACK: technology knowledge (TK), content knowledge (CK), pedagogical knowledge (PK), technology content knowledge (TCK), technology pedagogical knowledge (TPK), pedagogical content knowledge (PCK), and technological pedagogical content knowledge (TPCK) and examined when and how these intersected within the framework of the class. Interestingly , when interviewing the teacher of the class, he offered up that he was designing his class not with TPACK in mind but rather as a way to reach his desired goal – to teach students to think historically – and that technology is only a tool that helps him to engage them in doing this by helping him to shape the lesson in a way that meets this goal.

Overall this is only a single case study so aspects of design towards implementation are bound to vary by teacher, school and students. The act of selecting this class and teacher was not random, rather the teacher was recommended to the researcher as someone who uses technology regularly in the classroom. In addition, the school utilized was a K-12 private school withone-to-one technology and thus it this scenario presents one where there is a great degree of technological access and affordances which may not be available to all teachers and schools. Gomez recognizes these limitations and approapriately makes no generalizations from these oberservations and interviews which should be broadly applied.

Despite this, this articles is offering one example of how in TPACK might be implemented in course design. Based on what Gomez (2015) observed, he does acknledge that this case example does breaks down the idea that the components of TPACK must be intersecting concurrently. Rather he notes “TPACK no longer becomes the intersection of these three types of knowledge, but rather it becomes the layered combination of these three
types of knowledge” (p. 295). In addition, Gomez (2015) highlights how teachers may approach TPACK very differently in implementation as the teacher of the 8th grade classes studied indicated that “teaching effectively with technology (TPACK) begins with an understanding of what he wants his students to learn” (p. 296). Therefore he frames TPACK within a framework of what he wants students to know.  Gomez presents that this may be a common way that teachers may implement TPACK and therefore “understanding the role students play in making decisions about using technology in instruction” should be considered more within the TPACK design (p. 296).

Mishra, P. and Koehler M.J. (2006) Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record, 2006, Vol.108(6), p.1017-1054

Promoting Student Engagement in Videos Through Quizzing

Cummins, S. Beresford, A.R. and Rice. A (2016) Investigating Engagement with In-Video Quiz Questions in a Programming Course. IEEE Transactions on Learning Technologies 9(1): 57-66

The use of videos to supplement or replace lectures that were previously done face-to-face is a standard to many online courses. However these videos often encourage passivity on the part of the learner. Other than watching and taking notes, there may be little to challenge to the video-watching learner to transform the information into retained knowledge, to self-assess whether or not they understand the content, and to demonstrate their ability to utilize what they have learned towards novel situations. Since engagement with videos is often the first step towards learning, Cummins, Beresford, and Rice (2016) tested whether or not student can become actively engaged in video materials through the use of in-video quizzes. They had two research questions: a) “how do students engage with quiz questions embedded within video content” and b) “what impact do in-video quiz questions have on student behavior” (p. 60).

Utilizing an Interactive Lecture Video Platform (ILVP) they developed and open sourced, the researchers were able to collect real-time student interactions with 18 different videos developed as part of a flipped classroom for programmers. Within each video, multiple choice and text answer based questions were embedded and were automatically graded by the system. Videoplay was automatically stopped at each question and students were require to answer. Correct answers automatically resumed playback while students had the option of retrying incorrect ones or moving ahead. Correct responses were discussed immediately after each quiz question when payback resumed. The style of questions were on the level of Remember, Understand, Apply, and Analyse within Bloom’s revised taxonomy . In addition to the interaction data, the researchers also administered anonymous questionnaires to collect student thoughts on technology and on behaviors they observed and also evaluated student engagement based on question complexity. Degree of student engagement was measured by on the number of students answering the quiz questions relative the number of students accessing the video.

According to the Cummins et. al. (2016), that students were likely to engage with the video through the quiz but that question style, question difficulty, and the overall number of questions in a video impacted the likelihood of engagement. In addition, student behaviors were variable in how often and in what ways this engagement took place. Some students viewed videos in their entirety while others skipped through them to areas they felt were relevant. Others employed a combination of these techniques. The authors suggest that, based both on the observed interactions and on questionnaire responses, four patterns of motivating are present during student engagement with the video – completionism (complete everything because it exists), challenge-seeking (only engage in those questions they felt challenged by), feedback (verify understanding of material), and revision (review of materials repeatedly). Interestingly, the researchers noted that student recollection of their engagement differed in some cases with actual recorded behavior but, the authors suggest this may actually show that students are not answering the question in the context of the quiz but are doing so within other contexts not recorded by the system. Given the evidence in student selectivity in responding to questions based on motivations, the author’s suggest a diverse approach to question design within videos will offer something for all learners.

While this study makes no attempt to assess the actual impact on performance and retention of the learners (due to the type of class and the assessment designs within it relative to the program), it does show that overall in-video quizzes may offer an effective way to promote student engagement with video based materials. It is unfortunate the authors did not consider an assessment structure within this research design so as to collect some assessment of learning. However given that the platform they utilized it available to anyone (https://github.com/ucam-cl-dtg/ILVP-prolog) and that other systems of integrated video quizzing are available  (i.e. Techsmith Relay) which, when combined with key-strokes and eye movement recording technology, could capture similar information does open up the ability to further test how in-video quizzing impacts student performance and retention.

In terms of further research, one could visual a series of studies using a similar processes which could examine in-video quizzing to greater depth not only for data on how it specifically impacts engagement, learning and retention but also how these may be impacted based on variables such as video purpose, length, context and the knowledge level of the questions.  As Schwartz and Hartmann (2007) noted design variations with regards to video genres may depend on learning outcomes so assessing if this engagement only exists for lecture based transitions or may transfer to other genre is intriguing. As the Cummins et. al (2016) explain, students “engaged less with the Understand questions in favour of other questions” (p.  62) which would suggest that students were actively selecting what they engaged with based on what they felt were most useful to them. Thus further investigation of how to design more engaging and learner centered questions would be useful towards knowledge retention. In addition, since the videos were sessions to replace lectures and ranged in length from 5 minutes and 59 seconds to 29 minutes and 6 seconds understanding how length impacts engagement would help to understand if there is a point at which student motivation and thus learning waivers. While the authors do address some specifics as to where drop-offs in engagement occurred relative to specific questions, they do not offer a breakdown as to engagement versus the relative length of the video and overall admit that the number of questions varied between videos (three had no questions at all) and that there was no connection between number of questions and the video length. Knowing more about the connections between in-video quizzing and student learning as well as the variables which impact this process could help to better assess the overall impact of in-video quizzing  and allow us to optimize in-video quizzes to promote student engagement, performance and retention.

Schwartz, D. L., & Hartman, K. (2007). It is not television anymore: Designing digital video for learning and assessment. In Goldman, R., Pea, R., Barron, B., & Derry, S.J. (Eds.), Video research in learning science. pp 349-366 Mahwah, NJ: Lawrance Erlbaum Associates.

Video Podcasts and Education

Kay, R. H. (2012). Exploring the use of video podcasts in education: A comprehensive review of the literature. Computers in Human Behavior, 28, 820-831

While the use of podcasts in education is growing, the literature to support their effectiveness in learning is far from concluded. Kay (2012) offers an overview of the literature on the use of podcasts in education a) to understand the ways in which podcasts have been used,  b) to identify the overall benefits and challenges to using video podcasts, and c) to outline areas of research design which could enhance evaluations of their effectiveness in learning. Utilizing keywords, such as ‘podcasts, vodcasts, video podcasts, video streaming, webcasts, and online videos” (p. 822), Kay searched for articles published in peer-reviewed journals. Through this she identified 53 studies published between 2009 and 2011 to analyze. Since the vast number of these were of focused on specific fields of undergraduates, Kay presents this as a review of  “the attitudes, behaviors and learning outcomes of undergraduate students studying science, technology, arts and health” (p. 823) Within this context, Kay (2012) shows there is a lot of diversity in how podcasts are used and how they are structured and tied into learning. She notes that podcasts generally fall into four categories (lecture-based, enhanced, supplementary and worked examples), can be variable in length and segmentation, designed for differing pedagogical approaches (passive viewing, problem solving and applied production) and have differing levels of focus (from narrow to specific skills to broader to higher cognitive concepts).  Because of the variability in research design, purpose and analysis methods, Kay (2012) approached this not from a meta-analysis perspective but from a broad comparison perspective with regards to the benefits from and challenges presented in using video podcasts.

In comparing the benefits and challenges, Kay (2012) presents that while there are great benefits shown in most studies, some studies are less conclusive. In examining the benefits, Kay finds that students in these studies are coming into podcasts primarily in evenings and weekends, primarily on home computers and not mobile devices (but this will vary by the type of video),  are utilizing different styles of viewing and that access is tied to a desire to improve knowledge (often ahead of an exam or class). This suggests that students are engaged in the flexibility and freedom afforded them through podcasts to learn anywhere and in ways that are conducive to their learning patterns. Overall student attitudes with regards to podcasts are positive in many of the studies. However, some showed a student preference for lectures over podcasts which limited the desire of the student to access them. Many studies commonly noted that students felt podcasts gave them a sense of control over their learning,  motivated them to learn through relevancy and attention, and helped them improve their understanding and performance. In considering performance, some of the studies showed improvement over traditional approaches with regards to tests scores while others showed no improvement. In additional while some studies showed that educators and students believed there were specific skills such as team building, technology usage and teaching skills the processes as to how these occur were not shared. In addition, some studies indicate technical problems with podcasts and lack of awareness can made podcasts inaccessible to some students and that several studies showed that students who regularly accessed podcasts attended class less often.

In reflecting on this diverse outcomes, Kay presents that the conflict evident in understanding the benefits and challenges is connected to research design. Kay (2012) argues that issues of podcast description, sample selection and description and data collection need to be addressed  “in order to establish the reliability and validity of results, compare and contrast results from different studies, and address some of the more difficult questions such as under what conditions and with whom are video podcasts most effective” (p. 826).  She argues that understanding more about the variation in length, structure and purpose of podcasts can better help to differentiate and better compare study data. Furthermore, Kay asks for more diverse populations (K-12) and better demographic population descriptions within studies so as to remove limits on ability to compare any findings among different contexts. Finally, she presents that an overall lack of examination of quantitative data and overall low quality descriptions of qualitative data techniques undermine the data being collected. “It is difficult to have confidence in the results reported, if the measures used are not reliable and valid or the process of qualitative data analysis and evaluation is not well articulated.” (p. 827) From these three issues, Kay recommends an overall greater depth to the design, descriptions, and data collection of research is needed in video podcasting research.

While literature review offers a general overview of the patterns the author witnessed in the studies collected, there are questions about data collection process as the author is unclear as to a) why three prior literature reviews were included as part of an analysis and b) as to whether the patterns she discusses are only from those papers which had undergraduate populations (as is intimated by her statement on this – as noted in italics above) or is it of all samples she collected. The author also used articles published in peer-reviewed journals and included no conference papers. It is unclear what difference in data would have resulted from including these other sources.

Overall the most critical information she provides from this study is the fact that there is no unifying research design that underlies the studies on video podcasts and this results in a diverse set of studies without complete consensus on the effective use of podcasts in education and overall little applicability on how to effectively implement video podcasts. The importance of research design in creating a comparative body of data cannot be understated and is something which should be considered in all good educational technology research. Unfortunately, while Kay denotes the issues present in how various studies are coding and how data is collected and analyzed in the studies she examined, she does not address the underlying research design issues much when thinking about areas of further research.  While this is not to lessen the issues she does bring up for future research, the need for better research design is evident and given little specifics by Kay.  One would have liked a more specific vision from her on this issue since greater consideration towards the underlying issues of research design with regards to describing and categorizing video podcasts, sampling strategies and developing methods of both qualitative and quantitative analysis are needed.

 

Intentional Design for On Screen Reading

Walsh, G. (2016) Screen and Paper Reading Research – A Literature Review. Australian Academic & Research Libraries, Vol.47(3), p.160-173

As more students move into on-line courses and as more faculty consider incorporating open educational resources (O.E.R) into their courses, the impact of screen reading and learning material design on reading comprehension and overall learning is of essential consideration.  Walsh (2016), desiring to help academic librarians gain knowledge on issues of online reading, examines the current research (last 6 years) with regards to reading comprehension and the screen versus paper debate. Overall Walsh found no consistency in research design among the studies she examined, making cross-comparisons difficult. However, she concludes that “most studies find little differences between the print and screen reading for comprehension” (p 169). But, she notes, most were not focused on scholarly readings and those that did “concluded that participants gain better understanding of the content when reading from paper” (p 169).

Overall, this article offers a synthesis of recent scholarly literature (2010-2016) located in information management databases. While the scope of the study does not specify the exact search parameters used nor if search parameters were used to eliminate any studies from consideration, it does offer a brief overall glance at some of the literature that exists on this subject from an information management perspective. If the author had opened up this research to examine databases within learning, education and educational technology, additional research may have been found. However despite this limited search parameter, the information within this article, when synthesized together, highlights several aspects to screen reading which should be considered within educational technology.

In her article, Walsh (2016) notes that when considering reading and comprehension, neuroscience research suggests that deep reading is necessary for “furthering comprehension, deductive reasoning, critical thought and insight” (p 162) but that there is variation in the areas of the brain which are stimulated by print reading and versus those stimulated by screen reading. This variation may indicate that there may be some impingement upon the screen reader’s “ability to reflect, absorb and recall information as effectively as in formation in the paper form” (p 162) and may encourage more shallow or skim reading. While not specifically addressed by Walsh but when considered further, this information suggests that educators which rely on-screen based reading to help students gain material knowledge for their course may need to develop activities which work to promote deeper reading in students. This is not something students learn early on due to the predominance of paper assigned materials in early education. At the same time, this may not be a skill that can be developed with something as simple as giving them a set of questions to answer after having read. Kuiper et. al (2005) offered that, when examining how students searched the Internet, how the teacher structured the task impacted how the student approached the content. In the case of screen reading, well-structured tasks (to borrow from Kuiper et. al) may support only a seek-and-find strategy and not necessarily support the ability of the student to creatively and critically come to comprehend and synthesize the materials.

Walsh’s review also offers information which shows that the content’s format, intention and its length can impact how much the student may learn from screen reading. Walsh (2016) notes that even though students read off of screens for entertainment, when it comes to academic documents, students prefer to print off a document rather than reading it on the screen. This preference is related to not only the “high level of concentration and text comprehension” necessary but that academic reading also required the reader to interact with the document through annotating, highlighting and bookmarking passages for reference (p 163).  Walsh’s research suggests that students do not perceive themselves as being able to accomplish as much with screen reading of academic documents as print reading.  This perception is critical since even though many students within the studies indicated interest in screen reading, they doubted their own ability to be competent with it. This perception of competence could potentially undermine student interest in engaging with the reading fully. Thus, while Walsh does not specify this within the article, it does recommend that an educator who utilizes screen based academic reading as part of their course may need to offer more guidance to the readers with regards to both how they may engage with the reading (through digital annotation, tagging and bookmarking) and more encouragement for students to build self-confidence in their abilities.  In addition, Walsh (2016) highlights research showing there is very little difference in outcomes of performance between screen readers and print readers for shorter content but that for longer, more complex materials, learning and information retrieval can be impacted when reading from a screen. Furthermore text which were less data and fact based, which were less visual, and required more cognitive reasoning were easier to read in paper format than on-screen. These two points would suggest that a simple transformation of printed text to a digital format for screen reading – a common practice among educators and journals alike, may not be sufficient for materials to be comprehended as easily as the text version. Rather that utilizing technology to optimize the reading experience through visuals, textual divisions, and structured hypertext may benefit the comprehension of more complex longer materials.

Finally Walsh presents research which outlines how the platform characteristics with regards to design, user interaction and navigation can impact comprehension. The research Walsh presents suggest that platform structures not only create technical frustrations but may limit the level of engagement the student can have with the reading or increase the level of distractions they can experience. Not all readings are equally optimized for learning for all students in all platforms. Therefore this could recommend to the educator that careful consideration of platform tools (navigate, annotate, explore), overall student familiarity with a platform and its usability, and the ability of the educator and student to turn off and on hypertext/pop-ups should be considered when selecting for digital materials.

These points, taken together, suggest that educators need to have a more thoughtful, approach to the incorporation of digital reading materials in their courses and that students may be better served by educators approaching onscreen reading with more intentional design than is currently in use.

Additional References

Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in K–12 education: Strategies for supporting students in searching and processing information. Review of Educational Research, 75, 285–328

 

Designing Effective Qualitative Research

Hoepfl, M. C. (1997) Choosing qualitative research: A primer for technology education researchers. Journal of Technology Education, 9, 47–63

According to Hoepfl (1997), research in technology education has largely relied on quantitative research, possibly due to its own limitations in knowledge and skill on qualitative research design. Desiring to increase the implementation of qualitatively designed research, Hoepfl offers a “primer” on the purpose, processes and practice of qualitative research. Presenting qualitative research as expanding knowledge beyond what quantitative can achieve, Hoepfl (1997) sees it as having three critical purposes. First it can help understand issues about which little is known. Secondly it can offer new insight on what we already know. Thirdly, qualitative research can more easily convey the depth of data beyond what quantitative can. In addition, since qualitative data is often presented in ways which are similar to how people experience their world, he offers that it finds greater resonance with the reader. With regard to the processes of qualitative research, Hoepfl (1997) denotes that due to its nature, qualitative research design requires different consideration  as the “particular design of a qualitative study depends on the purpose of the inquiry, what information will be most useful, and what information will have the most credibility” (p.50). This leads to a flexibility – not finality – of research strategy before data collection and a de-emphasis on the confidence of data being a result solely of random sampling strategies and numbers. This flexibility in design strategy means a great deal of thought must be made on how to best situate data collection with recognition that actions in field may require adjustments of design as some questions fail or if new patterns emerge. In terms of strategies, the author offers up purposeful sampling options and discusses how maximum variation sampling may lead to both depth of description and sensitivity for emergent pattern recognition. He also outlines some of the various forms of data available in qualitative research and the stages of data analysis. In doing this, Hoepfl (1997) recognizes that qualitative data is much more difficult to collect and analyze than quantitative data and that often the research may require numerous cyclical movements through the various stages of collection and analysis. Importantly he addresses the practices of the researcher and reviewer in considering authority and trustworthiness in qualitative research by examining issues of credibility, transferability, dependability and confirmability.

In examining Hoepfl’s work, he offers a quality start to understanding the strengths and struggles of qualitative research. He correctly argues that the ability for qualitative research to have increasing acceptance within technology education rests on the ability of the researcher to address the questions of authority and trustworthiness which are more easily (albeit possibly erroneous) accepted in quantitative research. However  there were other aspects which are inherent in qualitative research which he gives almost no treatment to at all. These include consideration of  how relationships become built and defined between subjects and researcher and the impacts these can have on subject behavior. Hoepfl (1997) makes mention of these relationships and the risk of altering participant behavior denoting that “the researcher must be aware of, and work to minimize.” (p.  53) but  he offers no process for either recognizing when this occurs within the data nor how to actually go about minimizing this.  When it comes to the ethics of human subject interaction, Hoepfl (1997)  denotes that “the researcher must consider the legal and ethical responsibilities associated with naturalistic observation” (p. 53) but earlier offered that limiting the knowledge of the researcher’s identity and purpose or even hiding them may be appropriate. This is a problematic statement given informed consent guidelines and outlines a key aspect of information missing in this primer – that of how to consider human subject research ethics within qualitative research design. Since Hoepfl is offering a general guide to qualitative research and since the existence of IRB’s and the primacy guidelines of informed consent were established in 1974 by the National Research Act,  one would have expected at least some consideration of those guidelines, a mentioning of informed consent, or at least a discussion of how to handle the sensitive data that may come with qualitative data collection.

In reflecting on the applicability of Hoepfl’s work to my research interests, the emphasis on what qualitative research can bring to the educational technology table is enlightening as I did not recognize how much of a new approach this was to education as it was something of a staple to my anthropological education. Of particular interest was Hoepfl discussion of maximum variation sampling. He cites Patton in saying

“The maximum variation sampling strategy turns that apparent weakness into a strength by applying the following logic: Any common patterns that emerge from great variation are of particular interest and value in capturing the core experiences and central, shared aspects or impacts of a program” (Hoepfl, 1997 p.52)

This statement and his discussion of trustworthiness connected to a recent article I read on generalizing in educational research written by Ercikan and Roth’s (2014). In particular, the authors discuss the reliance on quantitative research for its supposed ability to be generalized but then break down this assumption to argue that qualitative data actually has more applicability since, if properly designed, can create essentialist generalizations. These are:

“the result of a systematic interrogation of “the particular case by constituting it as a ‘particular instance of the possible’… in order to extract general or invariant properties….In this approach, every case is taken as expressing the underlying law or laws; the approach intends to identify invariants in phenomena that, on the surface, look like they have little or nothing in common”(p. 10).

Thus by looking at “central, shared aspects” denoted by Hoepfl through maximum variation sampling and discerning the essential aspects which underlie the patterns, qualitative research could “identify the work and processes that produce phenomena.” Once this is established, the testability of the generalization is done by examining it to any other case study. If issues of population heterogeneity are also considered within the design of the qualitative data collection, the authors then argue that the ability to generalize from data is potentially greater with qualitative research.

Additional References

Ercikan, K. and Roth W-M (2014) Limits of Generalizing in Education Research: Why Criteria for Research Generalization Should Include Population Heterogeneity and Uses of Knowledge Claims. Teachers College Record Volume 116 (5): 1-28

A Consequence of Design – Considering Social Inequality in Educational Technology Research

Tawfik, A. A., Reeves, T., & Stich, A. (2016). Intended and Unintended Consequences of Educational Technology on Social Inequality. Techtrends: Linking Research & Practice To Improve Learning60(6), 598-605

Technology has often been considered a potential route for addressing inequalities of access and quality within education.  However, Tawfik et. al. (2016) consider such a perspective to be premature. In examining the educational system, the authors argue that the significant inequalities present among populations, based on aspects of socio-economic status, location, race and ethnicity, have not been addressed much in educational technology research and, in cases where it has been considered, differences in outcomes are exhibited by these populations. Denoting that student’s racial, ethnic and socioeconomic background have influence on educational attainment and achievement, Tawfik et. al (2016) examine how this inequality, with regards to technology, is present in some form at all levels of education – from early education (through access to media and apps associated with learning) to construction of college applications, to in-class and online learning, and through lifelong education.  Their review of the literature offers that not only is inequality evidenced in issues of access, interpretation and application of technology by students but also in the ability for teachers to access technology and the professional development related to learning technologies. The authors come to the conclusion that, while there is evidence of success of educational technologies to address gaps, there is also evidence that they can exacerbate them,  inadvertently increasing educational inequality.  Tawfik and colleagues (2016) argue that greater consideration of the consequences of educational technology on societal inequalities needs to be considered as part of the design, development and implementation of educational technology research

In reflection, Tawfik et. al (2016) offer a broad examination of the intended and unintended consequences of educational technology and offer food-for-thought on what good educational technology research needs to consider. While not a complete review of all literature as it relates to technology and educational inequality, the authors supplement their points of issue with cited examples to support their conclusions.  They see a failing within research design and appropriately make a reasoned argument that greater reflection on issues of social inequality – as it related to educational technology among both students and educators – needs to be considered. Perhaps intended to spur the conversation and not so much to guide it, their recommendations for specific ways to implement this within research design are not forthcoming; making one to wonder just how they see this induction of greater consideration of social inequality into educational research being implemented.

In consideration of research design, the argument can been made that, for research to have application and impact policy, understanding the populational structures under which an assessment is done and to which it can be applied, is critical to the generalizability of the results. If educational technology has both the ability to lessen and widen the gaps in educational achievement in ways often not predicted in research design, consideration of aspects of socio-economic status, race and ethnicity should be examined.  This is especially true if one wants to move into strategizing  implementation since proper determination of situational generalizability is necessary. Moreover, given that educational technology can influence the potential outcomes of groups with regards to attainment and achievement, a reflection on its role in both decreasing and increasing educational inequality is essential towards a critical understanding of what we do in this field.