21InteractionTechnique--Gesture Recognition and Its Algorithms

1 .1 © 2014 - Brad Myers Brad Myers 05-899A/05-499A: Interaction Techniques Spring, 2014 Lecture 21: Past to Future: Gesture Recognition and Its Algorithms

2 .Announcements Pick up HW 4, quiz 5, & project proposals For projects: Very important not to make stuff up or exaggerate All statements carefully worded and accurate Citations and evidence Ideas for how to assign presentation slots? No “late” presentations! Late penalty for written report 10 points per day! © 2014 - Brad Myers 2

3 .What is a “Gesture” In HCI, an input to a computer where the path of the input is important to its recognition, not just the end points Regular drag-and-drop just cares about where starts and finishes, so does not count as a “gesture” A recognizer is needed to interpret the path – so it may be interpreted incorrectly Can be done with a mouse, a stylus or finger on touchscreen , or hands in the air in front of a camera Can be one or multiple fingers On Smartphones, call “tap” a “gesture” to distinguish between tap, long-press, flick, drag, etc. Depends on properties of the action or of the other actions Location, but also speed, timing, etc. © 2014 - Brad Myers 3

4 .Advantages of Gesture Recognition Very fast to enter Single gesture can give both parameters and command E.g., cross out gesture tells both what to do and to what Large space of potential gestures Can be “natural” Can fit in easily with event-based programming Assuming gestures simply invoke a command © 2014 - Brad Myers 4

5 .Disadvantages © 2014 - Brad Myers 5 No affordance – user has to know they can be done No in-place information on what they look like Can be hard to remember which gesture does what operation (especially if lots) System may recognize them incorrectly Often cannot be entered correctly by users with disabilities, etc. Can be un natural if designed poorly Hard to provide feedback of what is happening, especially if continuous Implementation challenge: creating a good recognizer Designer must decide if rotation and size invariant

6 .Gestures -> Character Recognition See lecture 12 about text entry using hand-printing and hand-writing Rand tablet (1964) PARC Tab QuickWriting (1989) Go PenPoint (1991) Apple Newton (1993) Palm Graffiti (1996) Windows TabletPC (2002) EdgeWrite (2003) © 2014 - Brad Myers 6

7 .Also Already Covered: Gestures in 3D Gestures for 3D manipulation See lecture 16 Mainly pose and path of fingers with datagloves Also elaborate gestures in Teddy See lecture 17 Sometimes gestures with joysticks May depend on path and timing Wii controller gestures Kinect Body poses © 2014 - Brad Myers 7

8 .Also Already Covered: Gestures in 3D Gestures for 3D manipulation See lecture 16 Mainly pose and path of fingers with datagloves Also elaborate gestures in Teddy See lecture 17 Sometimes gestures with joysticks May depend on path and timing Wii controller gestures Kinect Body poses © 2014 - Brad Myers 7

9 .Trainable Gesture Recognizer Applicon (circa 1970).  An interactive trainable computer aided circuit design system using hand-drawn shapes to enter data and commands .  Applicon .  16 mm film.  Video (2:25 min excerpt) From Bill Buxton Lincoln Labs page . See the Wikipedia entry © 2014 - Brad Myers 9

10 .Early Gesture Recognition Buxton, W., Sniderman , R., Reeves, W., Patel, S. & Baecker, R. ( 1979 ) . The Evolution of the SSSP Score Editing Tools . Computer Music Journal 3(4), 14-25. [ PDF ]  [ video ] Can draw gestures for the desired notes to enter music Start location determines pitch © 2014 - Brad Myers 10

11 .Early Gesture Recognition Buxton, W., Sniderman , R., Reeves, W., Patel, S. & Baecker, R. ( 1979 ) . The Evolution of the SSSP Score Editing Tools . Computer Music Journal 3(4), 14-25. [ PDF ]  [ video ] Can draw gestures for the desired notes to enter music Start location determines pitch © 2014 - Brad Myers 10

12 .Go Corp’s “ PenPoint ” OS Founded 1987, released in 1991 Many gestures for editing, navigation, etc. Flick to scroll and turn pages , circle, insert space, cross-out, insert word, get help, … Press and hold to start moving or selecting Special-purpose recognizer for the built-in gestures http://research.microsoft.com/en-us/um/people/bibuxton/buxtoncollection/a/pdf/ Go%20PenPoint%20Getting%20Started.pdf © 2014 - Brad Myers 12

13 .Go Corp’s “ PenPoint ” OS Founded 1987, released in 1991 Many gestures for editing, navigation, etc. Flick to scroll and turn pages , circle, insert space, cross-out, insert word, get help, … Press and hold to start moving or selecting Special-purpose recognizer for the built-in gestures http://research.microsoft.com/en-us/um/people/bibuxton/buxtoncollection/a/pdf/ Go%20PenPoint%20Getting%20Started.pdf © 2014 - Brad Myers 12

14 .Rubine’s Gesture Innovations “Eager recognition” – can recognize a gesture while mouse button is still down as soon as it is unambiguous Either wait for mouse pause, or immediately when unambiguous Allows user to continue with direct manipulation E.g., “L” gesture for rectangle, continue to drag for size “C” gesture for copy, “curlicue” for rotate and scale Multi-finger gestures also supported Two finger drag and resize Video , up through 6:00, 7:00-end © 2014 - Brad Myers 14

15 .Rubine : Gesture recognition algorithm © 2014 - Brad Myers 15 Trained with a small number of examples (e.g., 15) Since done by a person, won’t be identical Examples should vary in whatever ways they will for the user E.g., different sizes? Different orientations? Automatically looks for features of all gestures, that differentiates them Uses a Machine Learning algorithm Statistical Single-Stroke Gesture Recognition Computes matrix inversions, discriminant values, and Mahalanobis distances Experimentally picked a set of 13 features that seemed to work well E.g , “cosine and the sine of the initial angle of the gesture, the length and the angle of the bounding box diagonal, …” Implemented in a system called GRANDMA Video , 6:00 through 7:00

16 .Rubine : Gesture recognition algorithm © 2014 - Brad Myers 15 Trained with a small number of examples (e.g., 15) Since done by a person, won’t be identical Examples should vary in whatever ways they will for the user E.g., different sizes? Different orientations? Automatically looks for features of all gestures, that differentiates them Uses a Machine Learning algorithm Statistical Single-Stroke Gesture Recognition Computes matrix inversions, discriminant values, and Mahalanobis distances Experimentally picked a set of 13 features that seemed to work well E.g , “cosine and the sine of the initial angle of the gesture, the length and the angle of the bounding box diagonal, …” Implemented in a system called GRANDMA Video , 6:00 through 7:00

17 .Improving the Gestures Allan Christian Long Jr., Quill: a gesture design tool for pen-based user interfaces, PhD thesis, UC Berkeley, 2001, (307 pages), pdf How to know if the gestures are too similar? Chris Long took the Rubine recognizer and analyzes if gestures are too “confusable” “Quill” tool Similarity in recognition space not necessarily the same as in human perceptual visual space © 2014 - Brad Myers 17

18 .Improving the Gestures Allan Christian Long Jr., Quill: a gesture design tool for pen-based user interfaces, PhD thesis, UC Berkeley, 2001, (307 pages), pdf How to know if the gestures are too similar? Chris Long took the Rubine recognizer and analyzes if gestures are too “confusable” “Quill” tool Similarity in recognition space not necessarily the same as in human perceptual visual space © 2014 - Brad Myers 17

19 .Improving the Gestures Allan Christian Long Jr., Quill: a gesture design tool for pen-based user interfaces, PhD thesis, UC Berkeley, 2001, (307 pages), pdf How to know if the gestures are too similar? Chris Long took the Rubine recognizer and analyzes if gestures are too “confusable” “Quill” tool Similarity in recognition space not necessarily the same as in human perceptual visual space © 2014 - Brad Myers 17

20 .iPhone Gestures Quick flick down / up / left / right New behaviors in various apps (no affordance) Left and right in Messages, Safari Up and down in home screens Swipe down from top Swipe up from bottom Press and hold Two finger zoom Also in photo Two finger zoom and rotate Google maps Three finger tap – accessibility Shake left-right = undo (sometimes) … © 2014 - Brad Myers 20

21 .Google Glass Gestures https://support.google.com/glass/answer/3064184?hl=en Small touch pad on right side & Motion sensor Activate Glass:  Tap the touchpad to turn the display on Swipe forward and back: affect content being shown Select an item:  Tap Tilt head up / down : display on / off © 2014 - Brad Myers 21

22 .Android Gesture Builder All Smartphones have libraries to support programming apps with gestures Often provided to the code by “events” like “mouse-down”  “swipe-left” Android provides nice tool to define gestures by example Thanks to Pushkar Joglekar, Sam Gruber, Samantha Chiu http://android-coding.blogspot.com/2011/09/gestures- builder-create-your-gestures.html http://android-developers.blogspot.com/2009/10/gestures- on-android-16.html © 2014 - Brad Myers 22

23 .Research: Elaborate Gesture / Picture Recognition Alvarado, Christine and Davis, Randall ( 2001 ).  Resolving ambiguities to create a natural sketch based interface . Proceedings of IJCAI-2001, August 2001.  PDF Recognizes shapes & relationships between shapes Attachment points, anchors, etc. Can then run as a physical simulation YouTube video (4:43) © 2014 - Brad Myers 23

24 .Research: Elaborate Gesture / Picture Recognition Alvarado, Christine and Davis, Randall ( 2001 ).  Resolving ambiguities to create a natural sketch based interface . Proceedings of IJCAI-2001, August 2001.  PDF Recognizes shapes & relationships between shapes Attachment points, anchors, etc. Can then run as a physical simulation YouTube video (4:43) © 2014 - Brad Myers 23

25 .Funny Tyson R. Henry, Scott E. Hudson, Andrey K. Yeatts , Brad A. Myers, and Steven Feiner. "A Nose Gesture Interface Device: Extending Virtual Realities,"  ACM Symposium on User Interface Software and Technology , Hilton Head, SC, Nov. 11-13, 1991 . pp. 65-68. ACM DL or  local copy and slides . © 2014 - Brad Myers 25

26 .Serious ISR and HCII PRESENT a SOCIETAL COMPUTING SEMINAR: Towards Science of Gesture-Based Authentication: Security and Memorability   Janne Lindqvist   Thursday , April 17, 2104, 10:30 a.m., 3305 Newell Simon Hall   We study the security and memorability of free-form multitouch gestures for mobile authentication. Towards this end, we collected a dataset with a generate-test-retest paradigm where participants (N=63) generated free-form gestures, repeated them, and were later retested for memory. Half of the participants decided to generate one-finger gestures, and the other half generated multi-finger gestures. Although there has been recent work on template-based gestures, there are yet no metrics to analyze security of either template or free-form gestures. For example, entropy-based metrics used for text-based passwords are not suitable for capturing the security and memorability of free-form gestures. Hence, we modify a recently proposed metric for analyzing information capacity of continuous full-body movements for this purpose. Our metric computed estimated mutual information in repeated sets of gestures. Surprisingly, one-finger gestures had higher average mutual information. Gestures with many hard angles and turns had the highest mutual information. The best-remembered gestures included signatures and simple angular shapes. We also implemented a multitouch recognizer to evaluate the practicality of free-form gestures in a real authentication system and how they perform against shoulder surfing attacks. Our work shows that free-form gestures present a robust method for mobile authentication. © 2014 - Brad Myers 26

27 .Serious ISR and HCII PRESENT a SOCIETAL COMPUTING SEMINAR: Towards Science of Gesture-Based Authentication: Security and Memorability   Janne Lindqvist   Thursday , April 17, 2104, 10:30 a.m., 3305 Newell Simon Hall   We study the security and memorability of free-form multitouch gestures for mobile authentication. Towards this end, we collected a dataset with a generate-test-retest paradigm where participants (N=63) generated free-form gestures, repeated them, and were later retested for memory. Half of the participants decided to generate one-finger gestures, and the other half generated multi-finger gestures. Although there has been recent work on template-based gestures, there are yet no metrics to analyze security of either template or free-form gestures. For example, entropy-based metrics used for text-based passwords are not suitable for capturing the security and memorability of free-form gestures. Hence, we modify a recently proposed metric for analyzing information capacity of continuous full-body movements for this purpose. Our metric computed estimated mutual information in repeated sets of gestures. Surprisingly, one-finger gestures had higher average mutual information. Gestures with many hard angles and turns had the highest mutual information. The best-remembered gestures included signatures and simple angular shapes. We also implemented a multitouch recognizer to evaluate the practicality of free-form gestures in a real authentication system and how they perform against shoulder surfing attacks. Our work shows that free-form gestures present a robust method for mobile authentication. © 2014 - Brad Myers 26