Gestural Query Specification

Gestural Query Specification

1.Gestural Query Specification Saar Kuzi Arnab Nandi Lilong Jiang Michael Mandel

2.Gestural Interfaces Are widely used, given the proliferation of touch-based devices such as smartphones Gesture Recognition Actions Inference Gesture Gesture Type + Metadata User Interface Application User

3.Gestural Databases G estures performed by the user are mapped to queries Mapping to queries might be challenging: Depends on the query language, the schema, and the data Solution: multi modal interface Fluidity? What if the user is not sure what query to use?

4.Motivating Example Task: find the titles of all the albums created by the artist “Black Sabbath”.

5.Challenges Interaction is continuous Different from the query→result interaction The user expects to receive constant feedback Direct manipulation interface Performing actions on the data by interacting with it The system is expected to guide the user through the process

6.Query Specification Query Task: find the titles of all the albums created by the artist “Black Sabbath” ” Query C ontext D rag the Artist relation into the workspace Select “Black Sabbath” Bring the Artist and Album relations close to each other Select ArtistId as the join attribute Feedback The most likely result Query Intent #1 Query Intent #2 Query Intent #3 Query Intent #4 Query Intent Transition Sequence of Gestures

7.G estural Query L anguage Principles Data Manipulation Closure & Composition Feedback Expressivity




11.Design Considerations Interactive Join Number of gestures M X N MIN(N,M) M X N 1

12.Design Considerations (Cont.) Interactive Join Readable and pragmatic layout

13.System Architecture

14.Gesture Recognition as Classification Gesture Proximity Compatability Query (q) + = The goodness of query q Predicting the most likely query based on the gesture features and the DB state DB

15.Feature Design

16.Classification Example User Gestures Features Before the user begins the gesture “Touching” features receive -∞ for all possible queries User touches the headers of the tables “2 touching points” feature is on Queries that do not include 2 tables are ruled out ”Close” feature is - ∞ for all queries User brings the tables close to each other ”Close” feature is turned on “Stacked” feature is - ∞ We remain only with Join queries Use the “Join” feature to identify the most compatible attributes

17.User Studies Experimental Setup: 30 users Even distribution of proficiency Randomized order of tasks (Learning effects) Limited time (Fatigue effects) Baselines: Console-based SQL (SQLite) Visual Query Builder

18.Completion Time The total time it takes, on average, to complete a task in a given interface

19.User Survey

20.Discoverability How intuitive the interactions of the interface are?

21.Anticipation What is the ability of three different classifiers to anticipate user’s intent?

22.Summary Introducing the gestural query specification problem Proposing a novel gestural querying framework, and system Exhaustive user studies Open Challenges : Complex queries Big Data