-
Notifications
You must be signed in to change notification settings - Fork 1
UIST 2012 Reviews
Dear Pourang Irani,
Your UIST 2012 submission
432 - CrashAlert: Enhancing Peripheral Alertness for Eyes-Busy Interaction while Walking
is still under review consideration and you are now allowed to submit a rebuttal in PCS. The deadline for submitting a rebuttal is Friday, June 1st. The reviews are now available online at https://precisionconference.com/~uist12a/ and are also enclosed below.
For more information about the UIST review process, please see: http://www.acm.org/uist/uist2012/AuthorGuide.html
Rebuttal Information: Authors of papers that have not been rejected at this stage will have a week to submit a rebuttal if they feel that the reviewers have made substantive errors, or to answer specific questions posed by the reviewers. The rebuttal period will be between May 25 and June 1, 2012. The rebuttal is confined to 4000 characters in length, and it must be self-contained. For instance, URLs to additional material are not allowed. The rebuttal period is for addressing factual errors in the reviews, not for getting revised text or new results into the review process. Any such novel material should be disregarded by the reviewers.
Hrvoje Benko and Celine Latulipe ([email protected]) UIST 2012 Program co-Chairs
------------------------ Submission 432, Review 4 ------------------------
Title: CrashAlert: Enhancing Peripheral Alertness for Eyes-Busy Interaction while Walking
Reviewer: primary Overall rating: 3 (scale is 1..5; 5 is best) Reviewer expertise: 4 (scale is 1..4; 4 = "Expert")
Overall Rating
3 (Borderline: could go either way.)
Expertise
4 (Expert)
Summary and contribution
The authors introduce CrashAlert, a mobile system intended to identify obstacles in a user's path while they are walking and interacting with a mobile device. The authors describe the design of CrashAlert and describe the results of a study in which CrashAlert was used by participants on a walking course.
The Review (prior to rebuttal)
CrashAlert is an interesting system that addresses a little explored problem. The use of the depth camera to detect obstacles has potential. However, there are several limitations of the work as currently presented.
First, it's hard to get a sense of the overall system from the current paper. The figures are very small - both the screen shots and the image of the hardware. There is no clear picture in the paper of the user wearing the entire device, and the user interface screenshots are likewise hard to see. The video helps, but the figures could be improved in the paper.
Second, it's difficult to know whether the design of the CrashAlert visualization is any good relative to other possible visualizations. The authors mention other possible implementations, including a vibrotactile display and a radar view, but do not test them. It would be especially helpful to know how much information is really needed: do users need to know where obstacles are, or simply that an obstacle is nearby? A broader exploration of the types of possible alerts would be extremely beneficial.
Third, the study as presented offers a null result - the visualizations don't significantly affect performance using the application. It is good to know that adding the visualizations doesn't harm performance, but the paper could be improved by more carefully exploring how use of the interface affected the participants and their use of the system. This issue might also be addressed by adding additional levels of task difficulty to help expose performance differences.
Overall, this paper explores an interesting area. However, the current paper presents a limited contribution due to the above issues.
Other comments
- I would appreciate more information about obstacles in the study. How many obstacles did participants encounter beyond the "actor"?
The Meta Review (Primary)
Overall, opinions about the study were mixed. The primary concerns seem to be about the limited scope of the design (and lack of exploration of other design options), and the limited degree of study analysis, so these issues should be considered in the rebuttal.
R1 is generally pleased with the work, but recommends the addition of some related work, additional screenshots of the system, and a more robust description of how well the system detects various types of obstacles.
R2 notes that CrashAlert's performance is promising, but questions whether the task was complex enough to be affected by the situational demands. R2 suggests using a different study task, but it's possible that there may be insights from the current study that shed light on whether the task was too simple to show CrashAlert's effects.
R3 agrees that the paper is well written and that the system is interesting, but questions the novelty of the technical contribution, and suggests that additional design features (such as tactile feedback and a 360 degree display) could increase the novelty of this system. R3 also requests that additional information about the study be included.
Issues to address in the rebuttal:
- Were other designs evaluated in any way? If so, it would be helpful to know about them.
- How well does the system detect different types of obstacles (e.g. wires, objects off the ground)
- If possible, include additional information about the study participants (e.g. average walking speed), and additional statistical analysis.
Final Decision
------------------------ Submission 432, Review 1 ------------------------
Title: CrashAlert: Enhancing Peripheral Alertness for Eyes-Busy Interaction while Walking
Reviewer: external Overall rating: 4 (scale is 1..5; 5 is best) Reviewer expertise: 3 (scale is 1..4; 3 = "Knowledgeable")
Overall Rating
4 (Probably accept: I would argue for accepting this paper.)
Expertise
3 (Knowledgeable)
Summary and contribution
This paper describes a mobile solution to aiding user navigation when the eyes are occupied. The distance and location of obstacles on the user’s path are provided through the visualization of objects. A usability study is reported where findings have suggested that visualizations play a role in detecting obstacles when in motion. The main contributions from the paper include the design of the mobile interface itself (transforming images from depth camera into visually salient information) and an evaluation study examining the usability of the application within an indoor environment.
The Review (entered before the rebuttal period, and uneditable thereafter)
The paper is well structured and well written, covering a wide range of studies relating to walking user interfaces, and describing the challenges of obstacles which individuals may incur when the eyes are busy. I suggest adding a reference to the implementation section (page 4) relating to the use of depth perception to support navigation for blind users (Zollner et al., 2011: NAVI – A Proof-of-Concept of a Mobile Navigational Aid for Visually Impaired Based on the Microsoft Kinect) and describe in more depth the importance of evaluating mobile technologies under more realistic scenarios (Kjeldskov and Stage, 2004: New techniques for usability evaluation of mobile systems. International Journal of Human Computer Studies 60, (2004), 317–335). I suggest also providing a screen shot, clarifying for the reader, what feedback the participants saw when playing the Whack-A-Mole game when an obstacle was detected.
I also suggest describing limitations of the system. One issue that some blind users face include hanging obstacles (e.g. wires) which are not easily detected using a long cane, as the cane is making contact with the ground. As the camera itself is positioned at a tilt towards the ground, these obstacles may still be encountered. Is there a way in which the system could be designed to support this? In the future work section, I also suggest describing the ways in which you could test the interface under more realistic conditions (e.g. outdoor environments) to determine if the system meets users’ needs.
Additional review comments (entered after the start of rebuttal period)
------------------------ Submission 432, Review 2 ------------------------
Title: CrashAlert: Enhancing Peripheral Alertness for Eyes-Busy Interaction while Walking
Reviewer: external Overall rating: 3 (scale is 1..5; 5 is best) Reviewer expertise: 4 (scale is 1..4; 4 = "Expert")
Overall Rating
3 (Borderline: could go either way.)
Expertise
4 (Expert)
Summary and contribution
This paper presents a system to help the mobile device user avoid collision to obstacles or other people while walking. The system, CrashAlert, uses a depth-sensing camera to sense whether any obstacle is in front of the user. The paper explains different visualizations of the view of the camera to inform the user of the existence of any obstacle. The user study revealed that the participants could dodge or slow down with fewer heads-ups when playing a game on a mobile device while walking.
The Review (entered before the rebuttal period, and uneditable thereafter)
I think that the system is interesting. My concern is that the user may still have to devote much visual attention to the visualization provided by CrashAlert. Thus, the system may help the user focus on the mobile device, but not necessarily on her tasks. The task used in the user study may not be appropriate to highlight the benefits of CrashAlert.
First, even with CrashAlert, it seems that the user still has to pay much attention to the environment. The overall visual attention to the environment might not be greatly different between using CrashAlert and not. The user study revealed that the number of heads-ups decreased in the condition of using CrashAlert. This is good news as the results indicate that the participants relied on the CrashAlert visualizations to avoid collisions. However, the game performance was not improved. This may be effects by the game design or other factors. But it could explain that CrashAlert did not liberate the user from paying attention to the environment much because the participants had to pay the visualizations to avoid collisions. It would be great if the paper can examine more directly how the visual demand to the environment was reduced with CrashAlert.
Related to the previous note, it would have been useful if the study included a non-walking condition. The performance in the stationary condition would tell us how well the participants could play the game when they did not have any distractor. And by comparing the results in each condition with this reference condition, the user study could have shown how much CrashAlert could contribute to the user’s focus on the game rather than the environment.
A more complex task might highlight the benefits of CrashAlert better. As Oulasvirta et al. showed in their CHI 2005 paper, the user’s attention to the environment can break up the focus on her task into a few second chunks. Thus, if a task in the user study requires the user to focus longer than a few seconds, participants might have significantly lower performance in the condition without CrashAlert than the ones with it. Unfortunately, the whack-a-mole game seems to be too simple (probably requires attention only for a couple of seconds). This might be one reason why the user study did not reveal any improved performance in conditions with CrashAlert.
Antti Oulasvirta, Sakari Tamminen, Virpi Roto, and Jaana Kuorelahti. 2005. Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '05). ACM, New York, NY, USA, 919-928.
There is an iPhone App called Type n Walk (http://www.type-n-walk.com/). The idea is somewhat similar to this work, but it uses a built-in camera and raw image. I wonder how CrashAlert could be better than it.  
Additional review comments (entered after the start of rebuttal period)
------------------------ Submission 432, Review 3 ------------------------
Title: CrashAlert: Enhancing Peripheral Alertness for Eyes-Busy Interaction while Walking
Reviewer: external Overall rating: 2 (scale is 1..5; 5 is best) Reviewer expertise: 3 (scale is 1..4; 3 = "Knowledgeable")
Overall Rating
2 (Probably reject: I would argue for rejecting this paper.)
Expertise
3 (Knowledgeable)
Summary and contribution
This paper presents a system that provides obstacle detection for eyes-busy interaction with mobile devices. CrashAlert makes use of the Kinect and ambient visualisations for mobile devices. The user study results are somewhat promising.
The Review (entered before the rebuttal period, and uneditable thereafter)
Overall this is a reasonably well written paper and addresses an important issue in mobile interaction. However, there are several issues that prevent me from rating this paper highly:
Technological Novelty According to the UIST guidelines, technological insights should therefore take priority over user studies in the review. Unfortunately, in this case, the novelty of the CrashAlert system is somewhat limited. Although the use of depth cameras in mobile situations is interesting and technically challenging, there are several systems that make use of depth cameras for a similar purpose. There is a great deal of existing research in the area of sensory substitution systems for the visually impaired. For example, the NAVI system by Zöllner et al. is a mobile navigation aid that uses the Kinect and vibrotactile feedback to help visually impaired people navigate through indoor environments. Furthermore, Mann et al. built a navigation system for visually impaired people that incorporated the Kinect and tactile feedback in a helmet.
Feedback The authors argue that an ambient visual display is the most appropriate method for CrashAlert. Tactile feedback was not used because it is ‘limited in providing context and detail information’. However, Brown et al. have already shown that multiple dimensions of information can be encoded in tactile stimuli. Furthermore, the comments from the experiment participants indicate that feedback alerts would be sufficient and it may not be necessary to provide the ambient visualisations. This reduces the contribution of the paper as a large portion of the paper focuses on the different visualisations. This also suggests that tactile feedback may be more suitable than first assumed.
The contribution of this research could be greatly increased by providing alerts about hazards in a 360 degree range around the user (for example, a car approaching from behind the user is just as dangerous as one approaching from in front of the user!) and also by investigating different modalities of feedback or including feedback about the direction in which the object is moving.
Experimental Methodology The authors state that there were 4 conditions in the experiment but there were only 8 participants. For the experiment to be fully counterbalanced there should have been 24 participants. In terms of the results, it would have been interesting to include a larger amount of information. For example, what was the average walking speed of the participants with and without CrashAlert? The results could also be strengthened by including some statistical analysis of the different maneuvers taken with each visualisation technique.
Additional review comments (entered after the start of rebuttal period)