摘要 :
We report the results of a study comparing two concurrent think-aloud approaches for usability testing: the classic think-aloud (CTA) and an interactive think-aloud (ITA). The think-alouds were compared in respect of task performa...
展开
We report the results of a study comparing two concurrent think-aloud approaches for usability testing: the classic think-aloud (CTA) and an interactive think-aloud (ITA). The think-alouds were compared in respect of task performance and usability problem data. We also analyse the utility of the interventions used within the ITA in eliciting useful participant utterances. The most useful interventions were those focused on seeking explanations and opinions; these generated more utterances about user difficulties. Requests for clarifications, particularly about actions, resulted in fewer useful utterances: participants responded with simple procedural descriptions. In comparing the CTA and ITA, we found no differences in the number of successfully completed tasks, but the ITA did elongate the test session. The ITA led to the detection of more usability problems overall, and a greater number of causal explanations. However, the ITA produced more low-severity problems than the CTA.
收起
摘要 :
Teachers need ways to efficiently assess students' cognitive understanding. One promising approach involves easily adapted and administered item types that yield quantitative scores that can be interpreted in terms of whether or n...
展开
Teachers need ways to efficiently assess students' cognitive understanding. One promising approach involves easily adapted and administered item types that yield quantitative scores that can be interpreted in terms of whether or not students likely possess key understandings. This study illustrates an approach to analyzing response process validity evidence from item types for assessing two important aspects of proportional reasoning. Data include results from an interview protocol used with 33 middle school students to compare their responses to prototypical item types to their conceptions of composed unit and multiplicative comparison. The findings provide validity evidence in support of the score interpretations for the item types but also detail important item specifications and caveats. Discussion includes recommendations for extending the research for examining response process validity evidence in support of claims related to cognitive interpretations of scores for other key mathematical conceptions.
收起
摘要 :
Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgments related to the quality of performance are typically operationalized ...
展开
Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgments related to the quality of performance are typically operationalized as ratings against success criteria specified within a rubric. The extent to which the rubric is understood, interpreted, and applied by assessors is critical to support valid score interpretations and their subsequent use. Therefore, the purpose of this study was to examine evidence to support a scoring inference related to assessor ratings on a clinically oriented performance-based examination. Think-aloud data showed that rubric dimensions generally informed assessors' ratings, but specific performance descriptors were rarely invoked. These findings support revisions to the rubric (e.g., less subjective, rating-scale language) and highlight tensions and implications of using rubrics for student evaluation and making decisions in a learning context.
收起
摘要 :
This paper describes a comparative study of three usability test approaches: concurrent think-aloud protocols, retrospective think-aloud protocols, and constructive interaction. These three methods were compared by means of an eva...
展开
This paper describes a comparative study of three usability test approaches: concurrent think-aloud protocols, retrospective think-aloud protocols, and constructive interaction. These three methods were compared by means of an evaluation of an online library catalogue, which involved four points of comparison: number and type of usability problems detected; relevance of the problems detected; overall task performance; and participant experiences. The results of the study showed that there were only few significant differences between the usability test approaches, mainly with respect to manner of problem detecting, task performance and participant experience. For the most part, the usability methods proved very much comparable, revealing similar numbers and types of problems that were equally relevant. Taking some practical aspects into account, a case can be made for preferring the concurrent think-aloud protocols over the other two methods.
收起
摘要 :
Usability is an important step in the software and product design cycle. There are a number of methodologies such as talk aloud protocol, and cognitive walkthrough that can be employed in usability evaluations. However, many of th...
展开
Usability is an important step in the software and product design cycle. There are a number of methodologies such as talk aloud protocol, and cognitive walkthrough that can be employed in usability evaluations. However, many of these methods are not designed to include users with disabilities. Legislation and good design practice should provide incentives for researchers in this field to consider more inclusive methodologies. We carried out two studies to explore the viability of collecting gestural protocols from sign language users who are deaf using the think aloud protocol (TAP) method. Results of our studies support the viability of gestural TAP as a usability evaluation method and provide additional evidence that the cognitive systems used to produce successful verbal protocols in people who are hearing, seem to work similarly in people who speak with gestures. The challenges for adapting the TAP method for gestural language relate to how the data was collected and not to the data or its analysis. (c) 2005 Elsevier Ltd. All rights reserved.
收起
摘要 :
In this study we analysed what computational thinking skills pupils used when programming a small robot to solve a given problem, a simple programming task in which pupils had to select codes and put them in the correct place. Com...
展开
In this study we analysed what computational thinking skills pupils used when programming a small robot to solve a given problem, a simple programming task in which pupils had to select codes and put them in the correct place. Computational thinking is the process of (re)formulating a problem so that it becomes possible to use computer technology to solve it. Five computational thinking subskills were needed to solve the given problem: problem decomposition, abstraction, data ordering, testing and debugging. Forty-two pairs of pupils from grades 2 through 6 (ages 6-12) worked on the programming task. To capture their thoughts, they were asked to think aloud. The pairs were videotaped while working on the programming task. A rubric was developed to score the presence and quality of the use of the five subskills. Data analysis showed that just under half of the pairs succeeded in solving the task, mainly the pairs from the higher grades. For three of the five subskills (problem decomposition, abstraction, testing), the greatest improvement in approach occurred between placing code 1 and code 2. Successful pairs worked in a structured way.
收起
摘要 :
The think-aloud method is a widely used method for evaluating the usability of websites and software. However, it can also be used with cartographic products, an area which has been neglected up to now. It is a method in which tes...
展开
The think-aloud method is a widely used method for evaluating the usability of websites and software. However, it can also be used with cartographic products, an area which has been neglected up to now. It is a method in which test participants verbalise all their thought processes aloud. The participants are given a test scenario containing tasks to be completed. The method aims to reveal the participants' subjective attitudes toward a product in order to evaluate its usability. The present paper describes the use of the think-aloud method to evaluate the usability of a cartographic work—the regional atlas of the Moravian-Silesian Region. The study includes (I) a complete review of the method, based on the studies conducted; (II) testing tools for working with recorded data; (III) designing an experiment for evaluating the usability of the atlas; and (IV) the resulting qualitative and quantitative evaluation of the atlas based on the obtained results. During the study, three approaches were proposed to process and analyse the audio recordings. The first option was to separate the audio recordings into individual annotations and analyse them. The second option was to convert the recordings to text and perform a linguistic analysis. The third supplementary option was to use all the material produced and to analyse it subjectively and retrospectively, from the researcher's perspective. All three options were used in the final assessment of the atlas. Based on the participants' statements, any shortcomings in the studied atlas were identified for each topic (e.g., non-dominant maps or exceedingly complex infographics), and recommendations for their elimination were proposed.
收起
摘要 :
Many user-centred studies of electronic information resources include a think-aloud element - where users are asked to verbalise their thoughts, interface actions and sometimes their feelings whilst using these resources to help t...
展开
Many user-centred studies of electronic information resources include a think-aloud element - where users are asked to verbalise their thoughts, interface actions and sometimes their feelings whilst using these resources to help them complete one or more information tasks. These studies are usually conducted with the purpose of identifying usability issues related to the resource(s) used or understanding aspects of users' information behaviour. However, few of these studies present detailed accounts of how their think-aloud data was collected and analysed or provide detailed reflection on methodological decisions made. In this article, we discuss and reflect on the methodology used when planning and conducting a think-aloud study of lawyers' interactive information behaviour. Our discussion is framed by Blandford et al.'s PRET A Rapporter ('ready to report') framework - a framework that can be used to plan, conduct and describe user-centred studies of electronic information resource use from an information work perspective.
收起
摘要 :
Quantitative researchers traditionally use cognitive pretesting methods such as think-alouds to determine whether participants are interpreting close-ended survey items as intended. I applied this method to evaluate the score vali...
展开
Quantitative researchers traditionally use cognitive pretesting methods such as think-alouds to determine whether participants are interpreting close-ended survey items as intended. I applied this method to evaluate the score validity of quantitative survey instruments with a middle school and high school sample of students. Through these applications, the cognitive pretesting method also revealed to be a useful tool for cueing participants’ reflection on their experiences related to the phenomenon of interest. Qualitative researchers traditionally use more open-ended interviews guided by general “how” and “why” questions, which might not serve as sufficient prompts to elicit participants’ reflection on their experiences, especially younger participants. To date, there are no illustrative examples focused on how having participants think-aloud as they complete a quantitative survey can inform qualitative-focused or mixed research questions. Therefore, the purpose of this article was to illustrate how the cognitive pretesting method, where participants think-aloud as they complete a survey, can be used as an additional technique to gain insight into their experiences of a phenomenon. The phenomenon in this illustration was defined as the participants’ experiences with the science content that they were learning in school. Limitations and methodological challenges and recommendations for overcoming these barriers are also discussed.
收起
摘要 :
Initiatives to integrate engineering design in the elementary science classroom have become increasingly evident in both national reform documents and classroom practice. Missing from these efforts is a purposeful attempt to captu...
展开
Initiatives to integrate engineering design in the elementary science classroom have become increasingly evident in both national reform documents and classroom practice. Missing from these efforts is a purposeful attempt to capture students' designerly thinking and dialogues as they engage in the process. The purpose of this study was to investigate how elementary school students approach and engage in engineering design using concurrent think-aloud protocols. Data from seven concurrent think-aloud protocols among triads of elementary students across seven classrooms were analyzed to identify how students conceptualize design. Researchers employed a transfer problem and think-aloud protocol analysis to assess students' transfer of learning from classroom science based engineering design-based experiences. Results indicate that elementary student triad design teams were able to define a design problem, identify constraints and criteria, and generate multiple design ideas to solve the problem. Protocol timelines were generated using NVivo software to capture sequence of the triads' coded cognitive strategies crucial in understanding which triads used a systematic approach to solving the problem from triads that randomly brainstormed ideas. If design is to become a pedagogical approach to teaching science or other STEM-related subjects, attention must be given to how students learn design and function within design. Concurrent think-aloud protocol provides a promising means of assessment of such efforts.
收起
原文获取