Chi2014 reviews

CHI 2014 Papers and Notes

Reviews of submission #1006: "tPad: Rich Interaction with Transparent-Display Mobiles"

------------------------ Submission 1006, Review 5 ------------------------

Reviewer: primary

Your Assessment of this Paper's Contribution to HCI

This work provides a general overview of the design space of interactive transparent displays and extends this design space with two (possibly new) interaction styles: flipping (switching back-and-forth between the device's front side and the back side) and surface capture (optically capturing and augmenting surfaces directly below the display). The two interaction styles are prototyped and experimentally tested.

Overall Rating

2.5 . . . Between possibly reject and neutral

Expertise

3 (Knowledgeable)

The Review

While several HCI papers have been already devoted to transparent displays, this submission reviews previous work and proposes a few seemingly original and useful ideas. Therefore it may be a significant contribution to HCI.

A major issue with this paper is the lack of a clear focus. I see at least three possible contributions:

1. A taxonomy of interaction techniques for transparent displays
2. An exploration of flipping interactions with transparent displays
3. An exploration of surface capture interactions with transparent displays

The current submission provides a mix of 1, 2, and 3, but does not develop any of them in enough depth.

About 1): the taxonomy covered in section "TRANSPARENT MOBILE INTERACTIONS" is interesting and potentially useful, but not developed enough: what are the dimensions of this taxonomy? What is the rationale for this classification? For the taxonomy to be useful, a table or figure needs to be provided where all previous work is laid out. Only then can we see how previous work relate and if the taxonomy is complete. Similarly, the two proposed interactions should appear as filling holes in this taxonomy. Ideally, the taxonomy should be presented as a design space with meaningful and independent dimensions. Eliciting ideas from end users is not enough: a taxonomy or design space should be the product of extensive analytical reasoning by the authors themselves.

With respect to the potential contributions 2) and 3), many interesting ideas are proposed (I find the surface capture idea very promising, and especially the "Extraction" scenario -- I envision this being used for, e.g., opening PDF references in a scientific paper print-out) but the contributions are not highlighted clearly enough. Do the contributions lie in the novelty of the ideas proposed? If yes, then they need to be clearly contrasted with previous techniques. The initial literature review seems appropriate, but it is separate from the description of the interaction techniques, which make it very difficult to judge how novel the proposed techniques are.

Does the contribution lie in the hardware solutions? I doubt, since the hardware implementations are only rough prototypes and the interesting (but challenging) problem of how to actually support surface capture is not addressed (and the possible technical solutions discussed p.9 are not overly realistic).

Does the contribution lie in the evaluations? It could, but the evaluations are actually fairly limited: the ecological validity of the application switching tasks is not discussed, and the solution does not seem to scale well to more than 2 applications (the paper claims "this difference is maintained as the number of multitasking applications increases" but this is not what Fig 10 shows). The image capture experiment yields interesting results, but does little to explain why surface capture is faster than camera capture. In particular, I expect this to be very sensitive to the experiment setup, including the initial position and orientation of the devices. This is not explained. Overall, the evaluations could bring interesting insights, but the experiment designs are not discussed in enough depth, both in terms of choice rationale and limitations. The experimental data and findings also deserve more discussion, both in terms of causes and implications.

To summarize, this in an article with possibly interesting contributions, but it is trying to do to much. None of the potential contributions is developed enough to clearly warrant a publication to CHI. The authors should focus on either introducing a design space, or on exploring and validating one of the two interaction styles proposed. A few ideas that are only very briefly discussed (like the paragraph on "Extraction") seem to deserve a full paper.

The Meta-Review

All reviewers agree this paper makes interesting contributions, especially since transparent devices are becoming more available and may hit the market soon (R2,R4). In that sense, this is currently an important design space to explore and the paper may inspire future work.

Overall reviewers appreciated the initial literature survey, although the specific techniques proposed are not contrasted well enough with similar or alternative solutions (see below).

The report on the participatory design session is not especially useful, and reviewers suggest to either expand it by providing more details (R1), or remove it altogether (R1,R4). I personally think this part could be removed.

All reviewers find the interaction techniques proposed reasonable and potentially useful. Reviewers also point out that none of the techniques is entirely new. Similar techniques have been proposed on other form factors, and sometimes on transparent displays. Re-implementing existing techniques on transparent displays does provide some value, and is "somewhat, but not too interesting" (R2).

A serious shortcoming of the paper is the lack of a proper comparison with alternative approaches. For example, for augmented reading tasks, the pros and cons of surface capture should be clearly contrasted with other approaches such as Anoto (R1), regular tablet computers and mobile AR (R1,R4) or scanners (R2). The same is true for dual display input, which is not contrasted clearly enough with other techniques (R1).

Similarly, the two evaluation studies are reasonably well-designed, but they provide no explanation for the differences in performance (R1,R4). For example, it is hard to understand why surface capture outperformed camera capture, and as a result, the study "falls short of providing true insight that could inform the design of other techniques" (R1). According to R4, the "authors should have questioned their results" and provided insights that could inform designers of such applications. The absence of such discussions is arguably a serious shortcoming of the paper, and makes it much less useful for the CHI audience than it could have been. A deeper discussion is needed especially since the findings in themselves are not that surprising (R3). R1 also asks for more details about the different conditions. Without such details, the results are hard to interpret.

Overall, reviewers are split but I see slightly more negative than positive. While R3 is very positive and thinks this could be a "valuable addition to the CHI conference", none of the other reviewers seems to be willing to push hard for this paper to be accepted. R1: "I am not quite convinced there is enough contribution there"; R2: "I this lean slightly towards an accept."; "somewhat, but not too interesting."; R4: "Right now the paper is not strong enough to convince me to argue for accepting this paper."

I think the paper's problem is that it is trying to say too much. In the end, none of the possible contributions is covered with enough depth (also see AC review above). As R4 states, "overall this paper has a bit of everything.". It might be possible to fix the paper: R1 and R4 suggest parts that can be trimmed down or removed, and R1 provides detailed recommendations for reorganizing the paper. However, this would involve some heavy rewriting, probably too much for a single review cycle. At the moment, I would rather recommend the authors to develop either their design space or one of their techniques and make it into a stronger, more focused paper*.

(*) During the discussion phase, R1 commented that the submission does not have to be split into several papers and suggested that if the authors "improved the write-up and improved the discussion overall, a paper with a synthesis of contributions like this could still be very appropriate at a more technology-focused venue like UIST or ITS."

------------------------ Submission 1006, Review 1 ------------------------

Reviewer: external

Your Assessment of this Paper's Contribution to HCI

The paper presents an exploration of transparent tablets / readers with dual-sided touch input as well as orientation sensing capability. The paper does a quick exploration of a number of techniques, from annotation and tracing, to image capture, to interaction with underlying registered content (e.g. features of a known document). The paper then has a small experiment which demonstrates some advantages for task switching by flipping the device over and quick image capture on contact.

This is a potentially interesting topic and I think the paper has some ideas to contribute, but it also is not totally clear to me after reading exactly what is and is not new here, as compared to previous related papers.

Overall Rating

3.0 - Neutral: Overall I would not argue for accepting or rejecting this paper.

Expertise

4 (Expert )

The Review

The paper emphasized how the Elicitation of Interaction Techniques employed a "user centered design approach" multiple times, but was actually kind of vague on what exactly this entailed, and the details of the methodology and findings in this section of the paper were left largely to the imagination. This is kind of one of those paper sections where you either have to cut it, or add more detail for it to be truly useful and insightful. As it is, I just have to trust your summary of the findings rather than having rich enough detail to see if there are other ideas and insights that would strike me from having done the study. Right now it is in between and therefore not that useful, even though in principle such an inquiry could have been good fodder for the paper. That there weren't clearly or markedly new ideas that arose from this didn't help too much -- many of the basic ideas proposed felt familiar from previous related works.

Several of the proposed interaction scenarios revolve around active reading type of tasks, and here it would have been nice to get a better understanding or rationale around why the authors expect transparent displays to have significant advantages over other approaches -- including the Anoto-pen style of interactive paper (which has the disadvantage of producing marks on the original, and the lack of output unless coupled with a projector), or especially, camera-mediated AR-style output through a mobile display. On this latter approach a key missing reference in my view is the PACER work of Liao et al, which uses the camera on a (non-transparent) mobile to do some of the same kinds of things. So when it gets into the details like this, I was not totally convinced that the transparent approach was really different or better, even thought it does seem like it should have some unique advantages or different interaction properties that could be leveraged.

Liao, C., Q. Liu, et al. (2010). PACER: Fine-Grained Interactive Paper via Camera-Touch Hybrid Gestures on a Cell Phone. CHI'10.

The Overlay interactions made sense, but at the same time it was hard to see how these were necessarily better than sketching on top of content on a regular stylus-enabled tablet computer. The backdrop of the physical object / document is interesting, but also limiting -- the user can't do a pinch-to-zoom gesture, for example, to overlay strokes at a finer level of detail than the 1:1 size of the original.

I found the dual display input interactions the most interesting, but still there are related ideas there from LucidTouch and back-of-device interactions that have been explored in several other papers already. I also thought the DualFlip technique (for delimiting motion-gesture interactions on mobile phones), while not exactly what you are after here, would have been a good reference on the basic idea of and recognition of flipping the device over as a mode switch (which is essentially what the app-switching technique amounts to -- though I will give you that app switching seems like a good idea and a good application of that gesture). I also liked the tap & flip idea, which seems fairly new (though again, probably the Tilt+Touch-in-Motion paper should be cited as a related technique). The Chen paper on dual-display techniques is another key one on the idea of flipping over as a motion-gesture and information navigation technique.

Ruiz, J. and Y. Li (2011). DoubleFlip: A Motion Gesture Delimiter for Mobile Interaction. CHI 2011.

Hinckley, K. and H. Song (2011). Sensor Synaesthesia: Touch in Motion, and Motion in Touch. CHI 2011.

Chen, N., F. Guimbretiere, et al. (2008). Navigation Techniques for Dual-Display E-Book Readers. CHI'08.

It would also be nice to see some of the Anoto-paper-based active reading type of interactions referenced here and discussed a little more either in the related work or in contrast to the Overlay, Extraction, Annotation, and/or Area Trigger techniques. I am not sure the most relevant paper(s) there, but for example some of the work by MacKay and colleagues or Guimbretiere and colleagues would be a good starting point. (Most of the following are references to older, and more general, work but might not be the best for comparing to your specific techniques).

MacKay, W., G. Velay, et al. (1993). "Augmenting Reality: Adding Computational Dimensions to Paper." Communications of the ACM 36(7): 96-97.

Guimbretiere, F. (2003). Paper Augmented Digital Documents. UIST 2003.

Liao, C., F. Guimbretiere, et al. (2008). "Papiercraft: A gesture-based command system for interactive paper." ACM TOCHI 14(4).

The Surface capture interactions, while potentially interesting, were not clear to me in the sense that I couldnt' quite wrap my head around when and why these would be different and preferable to just using the rear-facing camera on a normal mobile or mini-tablet. I think what you are trying to get at is that putting the tPad down on the surface, and having this be sensed as a different state or mode where immediate capture is possible in a more efficient manner, but this is never fully and clearly explained so that I can understand how and why exactly this really is more efficient. This seems to be borne out by the small user study later in the paper, but again there even though an absolute difference in means was found, it was never explained where that performance difference came from -- why one technique outperformed the other. Therefore I was left feeling like there might be something there but I couldn't quite put my finger on what the difference was.

Some parts of the paper could probably be trimmed down without loss of too much of importance. For example, I didn’t think the first paragraph of the Software section added anything of note (and the figure could go with that) and, while some brief mention of the capture-based registration might be in order, it seems a standard enough approach that this whole section (and the evaluation of it) could largely be omitted, because I don’t see anything particularly new or interesting here. This would buy at least half a page to spend more discussion and motivation around the user studies, which I think would be helpful.

The studies were interesting and seemed to show some utility in the techniques under certain conditions. Even though the “Home” and “MultiTask Bar” conditions are kind of standard on mobiles, these actually differ across different devices so you should spell out exactly how they look and act in your study. And actually, you should include some footage of the study tasks in your video rather than just stills; from the descriptions it was not completely clear to me all the steps the users had to go through for the interactions in each condition. Which leads to my next concern/question: there is no rationale or discussion at all as to why the observed performance differences occurred. It was not clear to me why the Tap’N Flip technique should be faster for app switching than the other techniques, for example, and likewise for the capture task I couldn’t understand why the transparent device should have been twice as fast. Clearly some steps are being simplified or a better match to the user’s mental model of the task is being achieved, but in the discussion of results this is not broken down or spelled out as to why the authors think these differences occurred. This circles back to the hypotheses, which I think are weak hypotheses in the sense that they only predict a difference in means between conditions – without being motivated by any specific hypothesis or theory or motivation as to why we should expect to observe differences. So, although the study seems to suggest that the techniques can be faster in the right circumstance, it falls short of providing true insight that could inform the design of other techniques or of related techniques that improve on what is proposed in this paper.

A minor pet peeve here is that I would very much prefer the authors to avoid all the acronyms for the conditions when discussing the results (F, TF, MB and so forth) because it makes it much, much harder for the reader to follow the discussion. Just call ‘em by the names you gave them in the first place. It places a needless barrier to communicating your results clearly.

I think the discussion could have left out the color mixing issue. You already raised this issue in related work, and as far as I can tell you don’t actually have any new technique implemented here yet. If you feel it is important you might raise it briefly as a final reminder of future work that is needed. For that matter, the Advanced Transparent Display Capabilities section is also really a future work issue. Then you return to the current system with a brief discussion of Technical Limitations (which were honest and good points to raise). I would suggest a minor reorganization of your Discussion to first raise and amplify the key questions and answers raised by your technical explorations and study, and they saving the future-looking subsections till the end (or perhaps a separate FUTURE WORK section right before, or blended into, the conclusion).

Another thing that troubles me a little bit is that so many of the paper's references are to various web sites and design concepts. I realize some of these are necessary, but by my count 12/42 references are either to web sites or patents. And this comes at the same time that some relevant published papers are not included. Just to pick on one in particular, though I respect Mike Elgan as a technology columnist, it seemed somewhat out of place to characterize him as an "expert" in an academic paper and then use his web article to make your point.

Overall, I want to like this paper and I want to believe there are interesting roles for transparent displays, but I left the paper feeling like the contribution was a little lighter than I would have liked – particularly if I try to really understand what new roles transparent devices could play beyond things like LucidTouch and beyond the paper-augmentation capabilities explore by the Liao papers. Although the latter are not transparent devices, they do raise the questions of what new capabilities the transparent displays offer here. The best answer seems to be the capture and tap’N’flip techniques, but these are not completely new either. So I think the contribution is modest for a 10-pp paper, and the related work needs to be bolstered from the literature rather than referencing so many concept designs from the web.

There may be enough here for the paper to squeak in, particularly if the authors can improve the related work and also spell out more clearly WHY the techniques seemed to offer performance improvements over the other techniques in the study in a convincing manner. I suggest the rebuttal should focus on that. At present I think this is a borderline paper.

I find myself waffling between a 3.0 and a 3.5 on this. I could see giving it the higher score but for now I would like to see the author's response and comments in the rebuttal for what might go into an improved draft of the paper. At this moment I am not quite convinced there is enough contribution there, even though I like the idea of exploring this design space / form factor of devices.

------------------------ Submission 1006, Review 2 ------------------------

Reviewer: external

Your Assessment of this Paper's Contribution to HCI

The authors demonstrate a set of interaction techniques based on a transparent mobile device, as well as a quantitative study.

Overall Rating

3.0 - Neutral: Overall I would not argue for accepting or rejecting this paper.

Expertise

3 (Knowledgeable)

The Review

I am on the fence on this one. Even though the presented techniques have more or less all been shown on other form factors such as AR or ebook readers, the paper does deliver value by showing that these can be done on transparent mobile devices--and it is the availably of such devices that makes the work relevant. I this lean slightly towards an accept.

This submission is a classic (and thus somewhat formulaic) type of CHI submission that takes a reasonably novel piece of hardware and points out a collection of interaction techniques that one could implement with it. The potential value in this type of paper is that it can quickly bring the community up to speed and inspire broader use of the involved technology as we have seen in the past for example on steerable projectors (Xiang Cao) or on interaction with a proximity sensor table (Medusa, Grossman).

Ultimately the usefulness and impact of this type of paper depends on how novel and inspired the techniques are (and in rare occasions whether they reveal a larger design concept that allows readers to push beyond what is in the paper). Along this metric, I would rate the interactions in this submission as "somewhat, but not too interesting."

#1: AR-style annotations: The interaction could also be done--and have been done--by devices with a scanner, i.e., scan first, then annotate. Might it be more interesting to explore interactions with animated contents, which would then merit the continued presence of the object? Rotoscoping a video, maybe, by placing the device over a video source?

#2: Dual-sided. The authors use flipping merely as a mode switch. This has been proposed by many papers in the past. I would have preferred to also see true dual-sided use, i.e., use of both sides at the same time, as in Ishii's clearboard.

#3: Scanning: While authors mock this one up with an overhead camera, I remember a clever design that really implemented this using a beam splitter. I think it was shown at CHI, potentially as an extended abstract… If I could only remember, but maybe the authors can locate it, which could help that tell readers how this functionality couuld to be implemented in the future without the boom. Also, it was not quite clear to me where the added value of snapping under the screen was compared to other devices featuring a scanner.

#4: Contact AR: Not too different from the related work in AR etc., but still appealing.

In summary, I feel that none of the techniques are "novel novel", but they have certainly never been shown on this type of device. Consequently, the author's designs remind us of some of the things what we already knew how to do in AR and/or mobile readers (such as the work from Guimbretiere's group), but it is good to know that they work on a transparent device. And in the end, it is the availability of such devices that gives the presented work impact.

Given the large number of proposed techniques--some more novel than others--I would enjoy seeing a quick statement in the abstract (and the rebuttal) on the 1-2 techniques the authors think of as most innovative and what sets them apart from the related work.

------------------------ Submission 1006, Review 3 ------------------------

Reviewer: external

Your Assessment of this Paper's Contribution to HCI

The paper describes the prototype implementation of two transparent-display mobile devices and a comparative evaluation of two application scenarios. The contribution is the description of a number of new interaction techniques for transparent-display mobile devices.

Overall Rating

4.0 - Possibly Accept: I would argue for accepting this paper

Expertise

2 (Passing Knowledge)

The Review

This is a really interesting, well written paper that provides a number of contributions to HCI, and I am sure that it would be an interesting and valuable addition to the CHI conference.

The introduction is very well written and the paper does a great job in situating the work by providing a thorough discussion of the related work in this area. Many of the references are online articles, but I don't have a problem with this, as I think this just reflects the innovative nature of the study.

The comparative evaluation is well designed and carried out, considering current best practice solutions and comparing them to the proposed new interaction. I thought that the hypotheses maybe weren't that surprising, but nonetheless still think the paper could be accepted since it provides an original approach in an area that hasn't seen much HCI research so far.

The statistical analysis looks thoroughly done, but as I'm not an expert in statistical analysis, I will leave the evaluation of this part to other reviewers.

Only as a minor comment, I thought it looked slightly strange that there was a third level heading "Discussion" followed by a first level heading "Discussion".

------------------------ Submission 1006, Review 4 ------------------------

Reviewer: external

Your Assessment of this Paper's Contribution to HCI

The paper presents a set of interaction techniques with tables that feature transparent displays. The techniques were obtained through a user-centered design process using a mockup device. Techniques were later evaluated on actual prototypes with good performance and increased user preference.

Overall Rating

2.0 - Possibly Reject: The submission is weak and probably shouldn't be accepted, but there is some chance it should get in.

Expertise

3 (Knowledgeable)

The Review

The paper starts with a very good motivation to point out what makes transparent displays different and useful. The topic is interesting and timely and there needs to be more research to understand the challenges of this kind of devices that will hit the market very soon. The approach the authors take is very reasonable. However, I found the actual results a little underwhelming and the authors should have continued where they stopped and questioned their results. As is I'm not sure if this submission will make the bar for CHI this time.

The paper overall did not point out why transparent tables are advantageous to a standard iPad. Apart from the interaction technique Querying'', all techniques could be implemented on an iPad by involving a camera for the overlay interactions'', a button on the screen for the dual display and input interactions'' (which is arguably easier to use and involves less effort than flipping a tablet) and a camera again for the surface capture interactions'' and ``model-based interactions''. What sets the transparent display apart? Even the querying can be done on existing devices (e.g., using a ruler application on iOS) albeit not by overlaying onto content. But is this really the best you can do with it?

It seems the user-centered design process produced all the obvious ideas people just thought off on the spot. To me they appeared somewhat uninspired and like a laundry list of things you could do on first thought. There was no compelling element in here, though the conclusions claim rich interaction techniques''. I don't think the proposed techniques are rich, nor are they much different from existing techniques. The Surface Capture Interactions'' are similar to traditional AR interaction techniques though the paper initially stated the purpose is to depart from such techniques (which I really would have liked to see). There is a lot of related work in interaction with paper, both digitally as well as physically.

The implementation section takes up a lot of space, but discusses things that are well-understood and explained in the related work already. The authors had better used that space to get to the bottom of transparent displays and why their use might be actually beneficial to existing devices.

The study is very detailed, but why does it only evaluate a subset of the proposed techniques? The discussion seems somewhat devoid of an actual and actionable take-home message. If I were an interaction designer for transparent displays, how would this affect my work?

Overall this paper has a bit of everything. If this work was never meant to set apart transparent displays from existing tables and instead just present a list of things you could do with them the initial motivation should be adjusted. Right now the paper is not strong enough to convince me to argue for accepting this paper.