Specifying Gestures by Example (Paper Report)

Bibliography:

Rubine, D. (1991). Specifying gestures by example (Vol. 25, No. 4, pp. 329-337). ACM.

Link:

http://srl.tamu.edu/srlng_media/content/objects/object-1236962325-cefe7476d664dc727f969660eac672cc/bde-GI-FinalVersion.pdf

Summary:

Previously recognition of gestures was hand-coded.

GDP a sketch program that recognizes gestures. 

GRANDMA (Gesture Recognition Automated in Novel Direct Manipulation Architecture). Design application by assigning gestures to view classes and specifying what the gestures do. 

Recognizes gestures by first building a classifier from examples. Gestures are single strokes. A feature vector is created for each stroke.

Classification. Each class of gesture is specified by the weights it gives for each feature. Each feature vector is created from the measurement of a set of 13 features. Features 12 and 13 have a dynamic component. They measure speed and duration.

Training for the classifier is done with a linear discriminator. First they get a sample estimate of the mean feature vector for each class. Using the mean feature vector for each class they calculate the sample estimate of the covariance matrix for each class and then create the sample estimate of the common covariance matrix using all the sample estimate of the covariance matrices for all classes.
Finally they use the inverse of the sample estimate of the common covariance matrix to calculate the weights for the class evaluators.

In the cases of outliers and ambiguous classification, the probability of correct classification and the standard deviation of the input gesture to its classification is used to determine rejection.

Rejection occasionally occurs on gestures that would be acceptable so it should be turned off if the application supports robust undo.

GSCORE.

Comments:

Seems like a good approach for recognizing gestures. Gestures could be tailored for each user.
Mouse for gestures is outdated. Multi-touch gestures(at least direct manipulation) the norm.

Research Ideas:

Can you use 2D gesture recognition techniques in 3D?

Who Dotted that 'i'? (Paper Report)

Bibliography:

Eoff, B. D., & Hammond, T. (2009, May). Who dotted that'i'?: context free user differentiation through pressure and tilt pen data. In Proceedings of Graphics Interface 2009 (pp. 149-156). Canadian Information Processing Society.

Link:

http://srl.tamu.edu/srlng_media/content/objects/object-1236962325-cefe7476d664dc727f969660eac672cc/bde-GI-FinalVersion.pdf

Summary:

This paper tries to solve the problem of how to distinguish the stroke of one user from a set of other users using pen pressure and tilt data.

Comments:

Research Ideas:

Mechanix: A Sketch-Based Tutoring and Grading System for Free-Body Diagrams (Paper Report)

Bibliography:

Valentine, S.; Vides, F.; Lucchese, G.; Turner, D.; Kim, H.; Li, W.; Linsey, J.; and Hammond, T. "Mechanix: A Sketch-Based Tutoring and Grading System for Free-Body Diagrams." AI Magazine. Winter 2012. 55-66. Print.

Link:

http://www.aaai.org/ojs/index.php/aimagazine/article/view/2437/2347

Summary:

Mechanix is a system designed to aide in the instruction and grading of statics problems of mechanical and civil engineering students. The system leverages sketching to teach statics by "actively engaging" students in the learning process. In one mode of the program students are asked to sketch out trusses and are given constant feedback in their drawings. Such immediate feedback on hand drawn sketches is impossible in large class sizes. This is where sketch recognition allows real-time feed back and automated instruction.

Other similar systems were either too general, did not incorporate the action of sketching the drawings, or were too strict in their drawing order. Mecanix utilizes a powerful low-level recognizer, PaleoSketch, to recognize high-level complex shapes.

The interface consists of a problem statement, standard edit tools, a step-by-step checklist, notepad area, drawing pane, feedback area, and equation pane. Students can check their answers at anytime by clicking a submit button and feedback is instantly provided to guide students.The interface is similar for instructors except they provide critical information and constraints about the truss and non-truss drawings. The problems are saved to the server where they can be retrieved by students. The interfaces also provides information on students submissions.

Students can draw, move, label, color, edit parts, or totally delete shapes. Visual feedback about completed shapes signal mistakes to students.

Geometric recognition of shapes is handled by a bottom-up approach by recognizing low-level shapes through PaleoSketch, and then recognizing groupings of shapes as high-level complex shapes. Trusses are described as two or more convex polygons connected by shared edges. Shared edges are found by removing edges from a connectivity graph and doing a breadth-first search for alternate paths between the two points that made up that edge. Checking answers includes comparing students drawing to instructors drawings and comparison of forces. Feedback is given in the case of found errors.

Problems are also given in non-truss free-body diagrams which are just closed shapes. The comparison between student and instructor drawing then employ three similarity metrics: Hausdorff distance, a modified Hausdorff distance, and the Tanimoto coefficient. The first two used measured the closest distances between points of the two drawings and the last used a ratio of overlapping points. These were then averaged and used as metric to determine similarity.

Also creative response problems where the answer was "open-ended" was also supported. In this case the validity of answers was handled by an artificial intelligence which creates a linear system of equations from the student drawn truss. These are then compared to a list of instructor given constraints and feedback is given.

The system is distributed on servers and load-balancing is used to mitigate stoppages when students submit answers.

Students who used Mechanix scored 40 percent higher on homework assignments.

Comments:

I believe this system is excellent. The advantages of having feedback and sketching the problems by hand had undeniable positive consequences. Some of the tests similarity were foreign to me. I liked how the authors leveraged an existing system, PaleoSketch, to solve a more complex recognition problem.

Ideas for Research:

One Idea for research would be to create a pre-viz system where you could sketch out storyboards for a film or short and have the system solve for camera placement, lighting, and actor placement.

K-Sketch: A “Kinetic” Sketch Pad for Novice Animators (Paper Report)

Bibliography:

Richard C. Davis, Brien Colwell, and James A. Landay. 2008. K-sketch: a 'kinetic' sketch pad for novice animators. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 413-422. DOI=10.1145/1357054.1357122 

Link:

http://dl.acm.org/citation.cfm?id=1357122

Summary:

This paper describes a tool that allows 'non-animators' to intuitively create animations. The main focus of this tool is to make animation accessible to the novice. Ease of use and design that followed the users intuition were an emphasis. Teachers hoping to use animation in storytelling as well as engineers desiring to sketch-up a prototype of a concept, such as the motion of a robots treads over obstacles, comprise a few of the 'non-animators' this tool aims to assist. Both 'animators' and 'non-animators' alike were surveyed to discover the minimal set of operations that needed to be implemented to complete the most number of relevant tasks.

The system allows users to sketch objects on the virtual canvas and translate, rotate, and scale that object through time through an onscreen widget that appears over the object once selected.. To transform an object the user must first select it and, while holding the alt-key, click and either move, rotate, or scale the object in time. And as long the mouse button is held down, time will advance and the object will follow the action created by the mouse. Other features include erasing the object at a certain point in time, creating new objects in time, copying the motion of one object to the other, adding relative motion to the objects frame of reference, and allowing the object to follow in the direction of the sketched path.

The system was tested by asking the users to complete a set of tasks with the K-Sketch system and with the more technical Power Point animation system. Users reported that the K-Sketch system was more natural and easier to use, and they were more willing to animate and show their animations in front of an audience as opposed to with Power Point.

Comments:

I wasn't sure how the system worked until I saw the demonstration videos at http://www.k-sketch.org/. I think the system works great and is exactly what they were going for. The interface looks intuitive and easy to use. It was discussed in class and I also agree that it would be nice to add physic based animation to the tool set.

Ideas for Research:

This paper as well as the discussions in class gives me an idea extending the animation system to include physical simulations like having rigid bodies collide and break and attaching objects together with spring like forces.

ICanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces (Paper Report)

Bibliography:

Daniel Dixon, Manoj Prasad, and Tracy Hammond. 2010. iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 897-906. DOI=10.1145/1753326.1753459

Link: 

http://dl.acm.org/citation.cfm?id=1753459

Summary:

In this paper, the author presents a system to assist in the act of drawing the human face, a very difficult skill to master. The user is given a photo portrait and is asked to follow a set of steps to complete a drawing of the given face. At each step the system seeks to assist the user in placing the salient features that make up the face. The features to be drawn at each step are focused by dimming the rest of the face. User feedback is given in the form of text feedback that points out errors such as an eye being too small as well as visual feedback that highlights where contours should go or which strokes appear misplaced. A template of the desired face is made by first analyzing the photo portrait with facial recognition algorithms to produce a set of features that define a face. A contour drawing of the points is used to judge the correctness of the strokes the user makes as well as determining the effectiveness of the system by comparing the users processed drawing, thorough the facial recognition system, to the desired drawing.

Comments:

I feel that the intention of the system is in the right direction but some of the methods are more mechanical than I would prefer. There are other aspects of drawing that I feel would be helpful. One thing I would have liked to see addressed is the fear of starting the drawing in the first place. Staring at a blank page is frightening to novice and master alike. Somehow facilitating the first and subsequent marks, I feel, would aid in the process of drawing the face. Also I would like to incorporate defocus attention on the drawing action and more on the 'seeing/drawing' connect.

Ideas for Research:

I think that approach to making marks for facial features would be to turn it into a game where the goal is to slice object down the middle. The slices would then 'draw' out the face. This abstracts the drawing act enough that the users own perceptions of how a face looks doesn't get in the way.