Ask Me Again But Don’t Annoy Me: Evaluating Re-authentication Strategies for Smartphones Lalit Agarwal, Hassan Khan, and Urs Hengartner, University of Waterloo https://www.usenix.org/conference/soups2016/technical-sessions/presentation/agarwal This paper is included in the Proceedings of the Twelfth Symposium on Usable Privacy and Security (SOUPS 2016). June 22–24, 2016 • Denver, CO, USA ISBN 978-1-931971-31-7 Open access to the Proceedings of the Twelfth Symposium on Usable Privacy and Security (SOUPS 2016) is sponsored by USENIX. Ask me again but don’t annoy me: Evaluating re-authentication strategies for smartphones Lalit Agarwal, Hassan Khan and Urs Hengartner CheritonSchoolofComputerScience UniversityofWaterloo Waterloo,ONCanada {lagarwal, h37khan, urs.hengartner}@uwaterloo.ca ABSTRACT usersareauthenticatingusingtext-basedpasswords[29]. To Re-authenticating users may be necessary for smartphone mitigate these usability issues, researchers have proposed authenticationschemesthatleverageuserbehaviour,device severaltechniquesthatreducetheauthenticationburdenby context, or task sensitivity. However, due to the unpre- leveraging user behaviour [21, 32, 37], device context [16, dictablenatureofre-authentication,usersmaygetannoyed 24, 25] or the sensitivity of launched apps [17]. whentheyhavetousethedefault,non-transparentauthen- While these schemes reduce the authentication burden on tication prompt for re-authentication. Weaddressthis con- theuser,theymayrequiremid-taskre-authentication. Sch- cern by proposing several re-authentication configurations emes that leverage user behaviour need re-authentication with varying levels of screen transparency and an optional in case of a behaviour mismatch against the current phone timedelaybeforedisplayingtheauthenticationprompt. We user. Similarly, device context-based schemes may need to conductuserstudieswith30participantstoevaluatetheus- establish a user’s identity in case a contextual source (e.g., abilityandsecurityperceptionsoftheseconfigurations. We ambient noise) changes. Taking the sensitivity of launched find that participants respond positively to our proposed apps into account for authentication may also require mid- changes and utilize the time delay while they are antici- task re-authentication. For instance, some users have indi- pating to get an authentication prompt to complete their cated that for a messenger app only opening old messages current task. Though our findings indicate no differences should trigger re-authentication [17]. in terms of task performance against these configurations, we find that the participants’ preferences for the configu- Preliminaryevaluationsshowthatusersliketheconvenience rations are context-based. They generally prefer the re- offered by these schemes [4, 16, 17, 19, 24]; however, a authentication configuration with a non-transparent back- fieldstudyofbehaviour-basedauthenticationshowsthatre- groundforsensitiveapplications,suchasbankingandphoto authentications are a potential issue [19]. More specifically, apps, while their preferences are inclined towards conve- the evaluated scheme used a (simulated) behaviour-based nient, usable configurations for medium and low sensitive authenticationschemethatfocusedontheuser’stouchinput appsorwhiletheyareusingtheirdevicesathome. Wecon- behaviour. Whenever re-authentication was required, the cludewithsuggestionstoimprovethedesignofourproposed user’s current task was interrupted and a re-authentication configurationsaswellasadiscussionofguidelinesforfuture prompt with dark background, similar to the standard An- implementations of re-authentication schemes. droid authentication prompt, appeared immediately. Non- surprisingly the unpredictability of a re-authentication and 1. INTRODUCTION thecontextswitchduetothetaskinterruptionwereannoy- The increased usage of smartphones to access personal and ing to some users. corporatedatarequiresauthenticationatmultiplelevels. A Whilere-authenticationisunavoidabletoprecludemisuseof device-levelauthenticationscheme,suchasaPINorfinger- adeviceoranapp,theunpredictabilityofre-authentication printrecognition,isrequiredtoprotectaccesstothedevice can be reduced by delaying the transition between the cur- while text-based passwords may be required to further es- rent task and the re-authentication prompt through a fade- tablish identity for social networking, banking or enterprise ineffect. Duringthefade-in,theuserisallowedtocontinue apps. Existing studies have shown that the short and fre- interacting with their current task on the device. In addi- quentnatureofsmartphonesessionscreatesusabilityissues tion to the fade-in effect, the re-authentication prompt can for device-level authentication schemes [17] whereas con- beconfiguredtohavevaryinglevelsoftransparencytopro- strained keyboards on smartphones are a bottleneck when vide a visual of the user’s current task in the background. The fade-in effect should reduce the unpredictability of the re-authentication and a visual of the current task of the user should reduce the context switch overhead due to re- authentications. Together these controls have the potential toprovideincreasedusabilityatthecostofreducedsecurity. Copyrightisheldbytheauthor/owner. Permissiontomakedigitalorhard In this paper, we evaluate different configurations of ex- copiesofallorpartofthisworkforpersonalorclassroomuseisgranted withoutfee. plicitauthenticationschemes(suchasPINsorpattern-locks) SymposiumonUsablePrivacyandSecurity(SOUPS)2016,June22–24, whenusedforre-authentication. Ourfocusisonthefade-in 2016,Denver,Colorado. USENIX Association 2016 Symposium on Usable Privacy and Security 221 effectandthetransparencyofthere-authenticationprompt. ing their explicit input. Various IA schemes have been pro- We choose behaviour-based authentication as a target use posedthatauthenticateusersthroughtheirtouchinputbe- case to evaluate the different configurations; however, our haviour[13,21,37],keystrokebehaviour[8,10,14],gaitbe- findings can be generalized to other authentication propos- haviour [12, 27] or device usage behaviour [32, 33]. Several als that require re-authentications. In addition to the re- IAproposalshavebeenshowntoprovideover95%accuracy authenticationconfigurationusedinthepreviouswork[19], [13, 21, 37] and researchers have proposed to use them as a weselectthreeconfigurationsofexplicitauthenticationsch- primaryauthenticationmechanismforuserswhodonotlock emes for re-authentication: (i) The authentication prompt their device or as a secondary authentication mechanism to appearsimmediately(nofade-in)andthebackgroundofthe compliment the existing primary authentication schemes. authentication prompt is transparent to provide a visual of TherearescenarioswhenanIAschemeisunsureaboutthe theuser’scurrenttaskinthebackground; (ii)theauthenti- identity of the user. This uncertainty may be caused by cation prompt appears immediately and the background of an adversary using the device or it could be the result of theauthenticationpromptgraduallytransitionsfromtrans- a false reject. False rejects occur when legitimate users are parenttoopaqueforimprovedsecurity;and(iii)theauthen- misclassified as adversaries. When an IA scheme is unsure tication prompt appears after a four second fade-in delay about the identity of the user, it uses an explicit authenti- andthebackgroundoftheauthenticationpromptgradually cationmechanismtore-authenticatetheuser. Furthermore, transitions from transparent to opaque. if an IA scheme relies on the input behaviour of the user, We perform lab experiments using synthetic tasks to evalu- the false rejects can occur mid-task and re-authentication atethesecurityperception,easeofuse,obstructivenessand requires interrupting the current task of the user [19]. annoyance of PIN and pattern-lock-based re-authentication Context-aware authentication: Several schemes have basedonthedefaultconfigurationfromtheearlierstudy[19] been proposed that leverage device context to reduce au- (asabaseline)andthemodifiedconfigurations. Inaddition thentication overhead [16, 24, 25, 28]. These schemes rely to these qualitative usability metrics, we collect quantita- on a variety of contextual sources, including location, prox- tivedataonthetaskefficiencyandthetaskerrorratefora imitytoWiFiandBluetoothdevices,andambientlightand multifaceted evaluation of these configurations. Finally, we noise. An evaluation of CASA [16] shows that it can re- conduct interviews to gather participants’ perceptions on duceexplicitauthenticationsby68%andalabstudyofthe thesensitivityofdifferentkindsofappsandofparticipants’ scheme proposed by Riva et al. [28] indicates that it can preferred configuration of the re-authentication prompt for reduce the number of explicit authentications by 42%. different apps and different environments. Context-aware schemes can be deployed to sense and assist Our study was completed by 30 participants. Though our in authentication only when users begin their interaction findingsindicatenodifferencesfortheuserperformance(in withthedevice. However,toprecludeattacksfrominformed terms of task efficiency, task error rate, and context switch attackers (such as friends and coworkers), a continuous au- overhead) against these configurations, participants found thentication scenario is more suitable. For instance, a con- all three modified configurations to be less annoying and tinuousproximitysensingschemewillnotallowaninformed less obstructive as compared to the default configuration. maliciouscoworkertounlockthedeviceattheworkplaceand The modified configurations were also at least as easy to thenmovetoasecludedplacetoaccesspersonaldataonthe useasthedefaultconfiguration. Asexpected,theperceived device. Since such scenarios may arise with the legitimate security level of the modified configurations was quite low user of the device (e.g., the device owner moves out of the when comparedto the default configuration. Whilethe low proximityrangewhileusingthedevice,oranambientnoise perceived level of protection was a bottleneck in the adop- sensor may switch off), the device owner may be subjected tionofthemodifiedconfigurationsinhigh-riskenvironments to mid-task re-authentication. and for sensitive content, a significant number of partici- pantspreferredtheproposedconfigurationsoverthedefault App-specific authentication: Hayashi et al. [17] show configurationforlesssensitivecontentandforlow-riskenvi- that all-or-nothing access to smartphones does not align ronments. Wealsocommunicatesuggestionsbythepartici- with user preferences. They find that while the majority pantsonhowtoimprovethedesignofourproposedconfigu- of the users prefer to be authenticated for select apps only, rationsandwediscussguidelinesforfutureimplementations for a subset of apps the users want some functionality to of re-authentication schemes. be available always and some functionality to be available after authentication. For instance, browsing existing en- 2. MOTIVATION tries(suchascontacts)inanappshouldalwaysbeavailable Implicit factors have been proposed to reduce authentica- while modifying or deleting entries should require authen- tion overhead on the web [2], personal computers [22] and tication. Similarly, looking at recent messages should not smartphones[17,25,32]. Ourfocusisonsmartphones. The require authentication while browsing old messages should implicit factors for authentication on smartphones leverage require user authentication. These scenarios require mid- behavioural biometrics [32], device context [16, 24, 25] or task re-authentication of the user. thesensitivityoflaunchedapps[17]. Wenextdescribeeach of these three implicit factors and their potential need to 2.2 NeedforBetterRe-authenticationSchemes re-authenticate a smartphone user. User studies on IA show that users find IA to be more con- 2.1 Re-authenticationScenarios venient and easier to use than traditional authentication Implicit authentication (IA): IA uses behavioural bio- schemes [4, 19]. Evaluations of the context-aware schemes metrics to conveniently authenticate users without requir- show that the reduced authentication overhead is found to 222 2016 Symposium on Usable Privacy and Security USENIX Association be useful and the users indicated that they would use the enteranincorrectcharactertoauthenticatewhenthephone evaluated scheme if it was available on their devices [16, vibrates [6]. 24]. A similar positive experience was reported for an app These approaches are promising; however, they may intro- sensitivity based authentication scheme [17]. duce confounding factors as they have not been adopted While users agree that these schemes are useful and are in- widely. The missing experience of the participants with terested in adopting them, most of these evaluations have thesenewconfigurationdesignmayaffecttheirusabilityper- notinvestigatedtheeffectofre-authenticationswiththeex- ceptions. Since several usability issues can be traced to the ceptionofKhanetal. intheirusabilitystudyoftouchinput- unpredictabilityandcontext-switcheffectsofre-authenticat- based IA [19]. Khan et al. find that for 35% of the partic- ion[19],weperformexperimentstoinvestigatewhetherthe ipants, re-authentications due to false rejects were a source unwantedeffectsstemmingfromunpredictabilityandcontext- ofannoyance. Theparticipantsfoundthere-authentications switches can be minimized for widely deployed authenti- to be frustrating due to their unpredictable nature and the cation mechanisms. Therefore, the main objective of this accompanying context-switch due to authentication inter- study is to investigate whether widely deployed authenti- rupts. Thecontextswitchwasalsoresponsibleforreducing cation schemes can be modified to make them more usable the overall task completion time of the participants. forre-authenticationscenarioswithoutsignificantlycompro- mising on security. Since unavoidable re-authentications are a potential issue in the adoption of IA, we investigate whether the unpre- dictable nature and the context-switch due to authentica- 3.2 ConfigurationParameters tion interrupts can be reduced by modifying how a user is We introduce two configuration parameters for existing au- re-authenticated. We assume that our concepts can miti- thentication prompts: time delay and screen transparency gate these usability issues and thus reduce barriers to the and define the possible values for each of the parameter. adoption of novel authentication schemes that require re- The time delay represents the time it takes between the authentication. transition from the current task of the user to the appear- ance of the re-authentication prompt. This variable sup- 3. STUDYDESIGN&OBJECTIVES ports two possible values: immediate lock (Imm-Lock) and Inthissection,wefirstoutlinedifferentapproachesthatcan gradual lock (Grad-Lock). In the Imm-Lock case, the re- be used for re-authentication. We then provide the ratio- authentication prompt appears immediately (without any nale for our selection of a slightly modified version of the delay)whereasfortheGrad-Lockcase,there-authentication existing authentication prompts through two configuration prompt appears after a predefined interval with a fade-in parameters: time delay and screen transparency. Finally, effect. During this fade-in, the user can continue to inter- we outline the security and usability trade-offs introduced act with the current task. The two possible values provide by these parameters, our constructions of re-authentication different usability and security trade-offs: the secure Imm- prompts with different configurations of these parameters Lock bars the user from interacting with the current task, and the usability expectations from our constructions. while the less secure Grad-Lock is not abrupt and provides the user with an opportunity to interact with the current 3.1 Re-authenticationApproaches task during the fade-in effect thereby potentially allowing Several re-authentication schemes are possible. During the the user to reduce the effect of interruption. For example, design phase, we considered the following: the user can finish reading a sentence. Split-screenconfiguration: Inthisconfiguration,theau- Forourexperiments,wechoseafoursecondtimedelay. Our thenticationpromptandthecurrentusertaskequallyshare selectionwasbasedontheresultsfrompreviousstudiesand the screen space (screenshots are provided in Appendix B). our experiments with both shorter and longer delays. Fer- This enables the user to authenticate within a timeout pe- reira et al.’s [11] study on understanding micro-usage pat- riod with their task in sight. However, it is difficult to en- terns for various smartphone apps revealed that 40% of the sure that the authentication prompt is displayed at a loca- applicationusagelastslessthan15secondsandissufficient tionthattheuserisfocusingon. Incasetheauthentication for a user to read or reply to a message. In a study con- prompt appears in the location where the user is focusing ducted by Yan et al. [38], they find that 50% of the smart- on, it results in the aforementioned usability issues. Never- phone interactions last fewer than 30 seconds. With such theless,thisapproachisworthexploringoncegazetracking briefperiodsofinteractions,itisthereforenecessarytolock solutions for smartphones have matured [23, 26]. the device quickly to prevent any misuse. For the grace pe- riod, we considered and tested delays between two to seven Alternate authentication mechanisms: Alternate au- seconds. During our empirical tests with four participants, thenticationmechanismshavebeenproposedtocountersho- we found that the four seconds delay period allowed the ulder-surfingattacks,whichreducethesizeoftheauthenti- participants to prepare for re-authentication prompts. The cation prompt [20] or allow the user to enter the PIN using shorter delay values did not provide the users with enough simpleupanddowngestures[35]. Similartothesplit-screen time to prepare for the re-authentication prompt, whereas configuration, a challenge for these approaches is the iden- the longer delay values made the users anxious in anticipa- tification of the most suitable placement of the authentica- tion of the re-authentication prompt. tion prompt for re-authentication. Another option is to use mechanisms that provide security using obscurity. For in- Thescreen transparency variableaffectsthevisibilityofthe stance, De Luca et al. [7] have proposed a mechanism that currenttaskbyconfiguringthebackgroundofthere-authen- allows users to enter the secret discretely through the back ticationprompttobeinstantaneouslydark(Imm-Dark,see of the device. In another proposal, the user is expected to Figure 1a), gradually fade from transparent to dark (Grad- USENIX Association 2016 Symposium on Usable Privacy and Security 223 Dark,seeFigure1b)orremaintransparent(Imm-Trans,see turns into dark. However, this configuration also al- Figure 1c and 1d). Similar to the time delay variable, the lows the user to continue interacting with the current three possible states of screen transparency provide vary- task for a grace-period of four seconds before the re- ing degrees of security and usability. The Imm-Dark state authentication prompt appears. During the grace pe- is the most secure one because it hides sensitive data dis- riod, the brightness of the current task is reduced to playedinthecurrenttask;however,thecontext-switchover- indicate the forthcoming re-authentication prompt to headshouldbethemostinthiscasesincetheuser’staskis the user. After the re-authentication prompt appears, not visible anymore. The Imm-Trans state covers the other the users can no longer interact with their task. extreme where sensitive data displayed in the current task remains visible behind the re-authentication prompt; how- 3.4 StudyAims ever, the context-switch overhead should be the least since Weexpectthefollowingpropertiesfromourre-authentication the user’s task remains visible while the user is interacting prompt configurations: with the re-authentication prompt. The Grad-Dark state Imm-Dark-Imm-Lockisthemostobstructivetherefore providesagraceperiodduringwhichtheusercanauthenti- • it should be the most annoying. Furthermore, since it catetoresumethetaskathand;however,iftheuserfailsto providesnovisualcluesonthecurrenttaskoftheuser, do so in a configurable amount of time, the background of task efficiency should be reduced. the re-authentication prompt becomes dark thereby hiding the user’s current task. Imm-Trans-Imm-Lock also immediately locks out the • userbutitspresentationofthere-authenticationprompt 3.3 Re-AuthenticationPromptConfigurations is less intrusive and it provides visual clues on the current task of the user. Therefore, it should be less The four configurations of re-authentication prompts that annoying and more task efficient as compared to the weconstructusingthedifferentmeaningfulcombinationsof Imm-Dark-Imm-Lock configuration. the two configuration parameters are as follows: Grad-Dark-Imm-Lock has similar properties as Imm- • Trans-Imm-Lockbutitprovidesadditionalsecurityby 1. Immediate Dark, Immediate Lock (Imm-Dark- making the current task of the user invisible after a Imm-Lock): Weevaluatethedefaultlockschemeon predefined time interval. Therefore, it should score most Android smartphones to establish a baseline for similar to Imm-Trans-Imm-Lock in terms of usability when it is used for re-authentication. In this con- with a relatively better security perception. figuration the re-authentication prompt appears im- mediately with a dark background, which completely Grad-Dark-Grad-Lockenablestheusertointeractwith • hides the content of the current task, and the user the current task for a grace period and this may in- can no longer interact with the current task. The re- crease the task efficiency of the users. However, the authentication prompt asks the user to enter a PIN user may not take advantage of the grace period and or pattern-lock and the user is able to access the cur- instead wait for the re-authentication prompt to ap- rent task again only after correctly answering the re- pear, which may increase the anxiety and annoyance authentication prompt. This configuration was also of the user. used in the earlier work by Khan et al. [19], as dis- In the rest of this paper, we evaluate whether the four re- cussed in 2.2. authenticationpromptconfigurationsprovidetheaforemen- § 2. ImmediateTransparent,ImmediateLock(Imm- tioned usability properties. Trans-Imm-Lock): There-authenticationpromptap- 4. STUDYDESIGN pears immediately in this configuration and the user Inthissectionweoutlineourdesignofauserstudytoeval- can no longer interact with the current task. How- uate the four re-authentication prompt configurations. To ever, the background of the re-authentication prompt measure the properties of each configuration, we perform a remainstransparent,whichallowsuserstoobservethe lab-basedevaluationwhereparticipantsareinvitedtoexpe- contents of their task. rienceeachconfigurationbyperformingpredefinedsynthetic 3. Gradual Dark, Immediate Lock (Grad-Dark- tasks. After the users experience these configurations, they Imm-Lock): Inthisconfiguration,there-authenticat- are asked to rate and provide qualitative feedback in terms ion prompt appears immediately and the user can no of usability, security perception and their willingness to use longerinteractwiththecurrenttask. Furthermore,the these configurations. In addition to the user feedback, we backgroundofthere-authenticationpromptisinitially measure the task efficiency, context switch overhead, and transparent and the contents of the current task are task error rate against each configuration. Our evaluation visible. Then,thebackgroundofthere-authentication andfeedbacksetuparedesignedtoelicittheefficacyofthese prompt gradually fades into a dark screen and hides configurations for re-authentication in different scenarios. the contents of the current task from the user. If OurstudywasreviewedandreceivedapprovalfromtheIRB the user manages to authenticate before the screen ofouruniversity. Wenowprovidedetailsofourstudydesign has darkened completely, this configuration keeps the in terms of experimental setup and our methodology. user’s current task visible in the background. 4. GradualDark,GradualLock(Grad-Dark-Grad- 4.1 Apparatus Lock): In terms of task visibility, this configuration Whileseveralusecasesexistforre-authentication(see 2.1), § is similar to the Grad-Dark-Imm-Lock configuration wechooseIAastherepresentativeusecaseinthisworkbe- described above. That is, the background of the re- causeitwaseasiertoexplainandconductthantheotherre- authenticationpromptisinitiallytransparentandthen authenticationcasesoutlinedinthepaper. OurchoiceofIA 224 2016 Symposium on Usable Privacy and Security USENIX Association (a) Imm-Dark (b) Grad-Dark (c) Imm-Trans (d) Imm-Trans Figure 1: The proposed configurations with varying values for screen transparency. Figures (a), (b) and (c) show the three possiblevalueswhenapattern-lockbasedre-authenticationpromptisused. Figure(d)showsasamplevalueforaPIN-based re-authenticationprompt. FortheGrad-Darkconfiguration,thebackgroundofthere-authenticationpromptgraduallyturns from transparent into dark. isalsomotivatedbythepriorworkofKhanetal.[19]inthe teraction of the users with the app can be measured, which IAdomainthathighlightstheissueswithre-authentications enables us to compute several metrics in terms of context- incaseoffalserejects. Toensurethateachparticipantexpe- switchoverheadanderrorsmadebytheusers. Fortheemail riencesacertainnumberoffalserejects,weuseasimulated activity,sincetheemailscontainsensitivematerial,theusers IA scheme, as was also done by Khan et al. In particular, performing the email activity should consider the security our scheme simulates IA schemes based on a user’s touch implications of a re-authentication prompt configuration in input or keystroke behaviour. addition to its usability aspects. For our experiments, we select two widely used authentica- TheseactivitieswerebundledintwoseparateAndroidapps, tionmechanismsonAndroid: a4-digitPINandtheAndroid which allowed users to perform tasks. We define a task as pattern-lock(withthesameconstraintsonpossiblepatterns completingthetextentryortheemailactivityalongwitha asinAndroid). Theuserinterfaceofbothschemeswassim- mid-task re-authentication of the user using either the PIN ilar to the Android lock screens (see Figure 1). or the pattern-lock in one of the four configurations. For thetextentrytask,theuserswereinterruptedatpredefined Thefourre-authenticationpromptconfigurationsintroduced intervals, which were triggered based on the key presses by in 3.3 are evaluated using two synthetic activities — a § theusers. Thenumberofkeypressesrequiredtotriggerre- text entry activity and an email activity (screenshots are authenticationchanged acrossdifferenttextentryactivities provided in Appendix A). We choose these activities since for each user but it stayed constant across users for those they represent common smartphone activities (i.e., reading tasksforresultstobecomparable. Similartothetextentry andcomposingemailsandtextmessagesorinteractingwith task, the users were interrupted with a re-authentication social media apps). prompt after a predefined number of swipes for the email task. Theappswereinstrumentedtogatherthetimestamps Text entry activity: This activity displays a 12- • ofevents,includinginputeventsbytheuserandthedisplay digit number to the participants. It also contains a and dismissal events of the re-authentication prompts. The textboxandtheusersareaskedtoenterthedisplayed appsalsocollectedtheerrorsmadebytheusersforthetext numberinthetextboxusingthenumerickeyboardof entry activity and during the re-authentication. We also the device. logged the user interactions, including the keystrokes and Email activity: Intheemailactivity,usersareasked screen touch events, during the grace period for the Grad- • toreadanemailinanemailapp. Theuserinterfacefor Dark-Grad-Lock configuration. The data collected by the theemailappdeveloped forthisactivitylookssimilar apps was instrumental in computing the task completion to the Android Gmail app. Once a participant has rate,contextswitchoverheadandtheerrorrateagainsteach read the email, they are asked to answer a multiple re-authentication prompt configuration. choice question related to the email on a laptop. The emails composed for this activity contained sensitive 4.2 EvaluationMethodology data,whichemphasizedtheneedtoprotecttheemails We evaluate the four re-authentication prompt configura- from adversaries (see Figure 10b for an example). tions using the text entry and email tasks. Each scheme was evaluated in a round that consisted of four text en- The design of the text entry activity ensures that the in- try tasks and two email tasks. Each user was subjected to USENIX Association 2016 Symposium on Usable Privacy and Security 225 five rounds and in each round a different re-authentication leagues starts using their device when it is left unat- prompt configuration was evaluated. For the first round, tended. For this scenario, the apps on the device may the participants performed the tasks without any authenti- be used for a limited time by someone known by the cation,whichallowedustoestablishabaseline. Thepartic- smartphone owner. ipants were allowed to take a break between each task and Home Scenario: In this scenario, we asked the par- eachround. Theorderofthefourre-authenticationprompt • ticipants to consider that their spouse accesses their configurations was randomly chosen for the participants. device while it is left unattended or when they are The participants shortlisted for this study were invited for asleep. Thenumberofadversariesislimitedinthissce- an hour long lab-based study. The participants were first nario as compared to the others and the users may or asked to fill a demographic survey, which asked about their maynotwanttoprotecttheirdatafromtheirspouse. age, gender, and current occupation. They were then asked Aresearcherpresentedthescenariostotheparticipantsand to fill a security preferences survey. In terms of security was available during the interview to answer any questions preferences, we asked the participants about their device participants may have. Participants were given sufficient locking habits, their preferred authentication scheme, and timetoconsiderthepresentedscenarios. Foreachscenario, the adversaries that they wanted protection against. These theparticipantsweretoldthatthere-authenticationprompt pre-study surveys are provided in Appendix D. After the wouldgetactivatedincasethesystemnoticesanysuspicious pre-study surveys, the participants were introduced to IA, activity. Wealsoremindedthemoffalserejectsandthefact thepossibilityoffalserejectsinIA,thetasksandappsused that they may be subjected to re-authentication while they duringthestudy,andthedifferentre-authenticationprompt are using the device. In order to inquire about the security configurations. Theparticipantswerealsotoldthatfalsere- perceptionofanevaluatedre-authenticationpromptconfig- jects were simulated for the purpose of this study. We gave uration, the participants were told that for the purpose of participantstheoptiontoselecttheirpreferredlockscheme these scenarios, they should consider that only IA is pro- (PIN or pattern-lock) and a corresponding secret for the tectingtheirdevice. Sincedifferentusersmayhavedifferent study. We did not assign participants a specific scheme to security preferences for each configuration and each usage avoid any bias due to their inexperience with it. This de- scenario, we initially asked the users to establish the sen- sign decision prohibited us to counterbalance the authen- sitive nature of the apps and usage scenarios. Then the tication methods. The authentication times varied across participants were asked to provide feedback in terms of se- participants. Tocaterforthis,wereportwithin-subjectrel- curity perception, usability and preferred re-authentication ativedifferencesinsteadofabsolutevalues. Theparticipants promptconfigurationforeachofthefourappsundereachof experienced the different configurations in multiple rounds. thethreedeviceusagescenarios. Thefeedbackquestionnaire Afterthecompletionofeachround,theywereaskedtorate is provided in Appendix E. theusabilityandperceivedsecurityoftheconfigurationthat they experienced and to give an overall ranking in terms of Finally,attheendofthestudy,weconductedashortsemi- their preferences by taking both the usability and the secu- structured interview (provided in Appendix F) to gain in- rity of the evaluated configuration in account. Participants sight into participants’ overall impression of the configura- were also asked to indicate their preferences for the eval- tions that they evaluated. uated configurations under different device usage scenarios and were subjected to a semi-structured interview to gain 5. RESULTS furtherinsightintotheirfeedback. Aresearcherwaspresent The data collected through the user studies and the inter- to respond to any questions the participants had. views were recorded and analyzed. The audio responses of the participants were transcribed by one of the researchers. 4.3 UserFeedback We report both the quantitative and the qualitative results The evaluated schemes trade off security for usability and fromthestudyinthissection. Forstatisticalsignificance,we since different users have different security preferences for usedpairedt-testswhencomparingcontinuousdataforthe differentappsanddifferentscenarios,weseekfeedbackfrom within-subjects condition such as the inter-stroke rate for theusersagainstfourappsforthreedifferentscenarios. Pre- each user between grace and non-grace periods. We used vious studies have shown that users prefer a strict security one-way ANOVA when comparing continuous data for the setting for financial and email apps, which contain highly within-subjects condition for the four authentication con- sensitive data, whereas they prefer a relatively relaxed se- figurations (e.g., context-switch overhead). We used chi- curity setting for contacts and other utility apps [17]. We squaredtestswhencomparingparticipants’responsestocat- sought feedback from the users for four apps: a banking egorical Likert-type questions. app,anemailapp,aphotosapp,andacontactsapp. These apps are commonly used and contain varying levels of sen- 5.1 StudyParticipants sitive data of the smartphone user. The participants were We advertised the study through our university-wide mail- asked to consider the following device usage scenarios with ing list and through the graduate student research portal the aforementioned apps available on the device. of our university. The study was advertised with the title BusScenario: Theparticipantshadtoconsiderasit- “Evaluating authentication schemes for smartphones” and • uation where they are traveling on a bus and they ac- werecruitedonlythoseuserswhohadpriorexperiencewith cidentally leave their smartphone behind. A stranger using smartphones. Participants received $10 for their par- picks up their device and starts using it. ticipation for an hour of study. Office Scenario: This scenario asks the participants We recruited 30 participants for the study (see Table 1 for • toconsideraworkenvironmentwhereoneoftheircol- theirdemographics). Alltheparticipantswerestudentsfrom 226 2016 Symposium on Usable Privacy and Security USENIX Association N=30 Gender 60% Females 40% Males Age 33% Under 20 years 57% 21-25 years 7% 26-30 years 3% 31-35 years Lock 26 (87%) Yes device? 4 (13%) No 13/26 Pattern-lock Authentication 5/26 PIN (4 digits) Figure 2: Task completion overhead time for the text entry scheme 6/26 Fingerprint activityrelativetotheBASE ROUND(errorbarsrepresent 2/26 Password 95% confidence interval). Protecting 25/26 Strangers from? 16/26 Friends 14/26 Room-mate pletion overhead is the additional time taken to complete a 14/26 Coworker textentrytaskascomparedtotheBASE ROUNDinwhich 3/26 Spouse, own children a user is not interrupted to re-authenticate. For the task completionoverhead,weonlytakeintoaccountthetexten- Table1: Demographicinformationandthedevicelockusage tryactivitysincetheemailsusedfortheemailactivitywere pattern of the participants. ofadifferentnatureandlengthduringeachround. Ourgoal is to find if there are any re-authentication prompt config- urations that assist the users in completing their text entry our university. The majority of our participants (87%) re- tasks faster. ported that they locked their device. The security prefer- Wefoundthatonaverageuserstook3-4secondslongerwhen ences of participants who locked their devices are provided they had to re-authenticate during a text entry task (see inTable1. Weaskedthefourparticipantswhodidnotlock Figure 2). A one-way between subjects ANOVA was con- their devices for their reason to do so: two indicated that ducted to compare the effect of the four configurations on they had nothing to protect, two wanted their emergency thetaskcompletionoverhead,whichindicatednosignificant contacts to be available and one considered authentication differences across the four configurations (F(3,116)=2.31, to be inconvenient (multiple answers were possible). p=0.08). 5.2 QuantitativeResults Discussion: OurexpectationthattheImm-Dark-Imm-Lock Outof30participants,18participantschosetouseapattern- configurationislessefficientascomparedtothemodifiedre- lockduringthestudy,whiletheremainingparticipantschose authenticationpromptconfigurationsturnsouttobeincor- to use a PIN. Participants were subjected to five rounds in rect. Though, we did not find any significant differences in total. During the first round, participants were not inter- theperformanceoftheconfigurations,theparticipantsmen- rupted for re-authentication. This round was used to es- tionedduringthestudythattheyfeltthattheirperformance tablish a baseline and we use the term BASE ROUND to wasaffectedduringtheImm-Dark-Imm-Lockconfiguration: refer to it. For the remaining rounds, participants tested one of the four configurations in each round. The order of ”Itkindoffreaksmeoutbecauseitistoosudden, the configurations was random during the four rounds. it slows down whatever I was doing.”(P4) During each round, participants completed four text entry tasks and two email tasks. They re-authenticated once for every email and text entry task during all rounds except 5.2.2 Effectoncontextswitchoverhead BASE ROUND. The high rate of re-authentication is not Context switch overhead for the text entry task is defined representative of a real-world scenario; however, our moti- as the time taken by the users to resume their text entry vation was to get participants acquainted with the configu- task once they have re-authenticated. The context switch rationsandtocollectsufficientdatatoevaluatethemetrics overheadisrepresentedbythetimeintervalbetweenthedis- used in this section. During the study each participant re- missalofthere-authenticationpromptandthefirstkeypress authenticatedthemselves16timesduringthetextentryac- onthetextentrytaskoncethere-authenticationprompthas tivity (four times per configuration) and eight times during disappeared. It was not possible to compute this metric for the email activity (twice per configuration). In total, 120 the email task because after re-authenticating a user would re-authentication events, 120 text entry tasks and 60 email complete reading the email text visible on the screen be- tasks were logged per configuration by our apps. fore interacting with the device. Our expectation was that a visual of the user task in the background would reduce 5.2.1 Effectontaskcompletionoverhead thecontextswitchoverhead. Toconfirmthis,weconducted The task completion time is the time taken by the users a one-way between subjects ANOVA to compare the effect to complete a text entry or an email task. It also includes of the four configurations on the context switch overhead. the time taken by the users to re-authenticate themselves However,theresultsindicatenosignificantdifferencesacross while evaluating one of the configurations. The task com- the four configurations (F(3,116)=1.15, p=0.33). USENIX Association 2016 Symposium on Usable Privacy and Security 227 Discussion: Whilenostatisticallysignificantdifferenceswere observed,duringtheinterviews,mostusersfoundtheImm- Dark-Imm-Lock configuration to be abrupt and reported thatitwasdifficulttoresumetheirtaskafterre-authentication: ”Ilostmyplace[context]onwhatIwasdoingbe- fore[thelockappeared],soitismyleastfavourite. It would be too frustrating for me for everyday use, so I would rather take the risk.”(P9) ”You can’t prepare for what’s going to come. It takes more time to pick up after unlock”(P10) Figure3: Inter-keyintervalforthetextentryactivity(error bars represent 95% confidence interval). The top bar rep- 5.2.3 Effectofgraceperiod resents the inter-key interval for the Grad-Dark-Grad-Lock We allowed a grace period of four seconds for the Grad- configuration during the grace period. Dark-Grad-Lockconfiguration. Duringthegraceperiodthe participants could continue working on their task for four seconds before getting locked out. We observe that all par- ticipants took advantage of this grace period by continuing their work during the text entry activity. The average task completiontimefortheGrad-Dark-Grad-Lockconfiguration was 13 seconds and we found that on average users entered 38% of the text during the four second grace period with someusersenteringupto60%ofthetotaltextinthegrace period. A similar trend was observed for the email task where 23% of the swipe events occurred during this period (averagetimetocompletetheemailtaskfortheGrad-Dark- Figure 4: User perceptions of the security of the four re- Grad-Lock configuration was 41 seconds). authentication prompt configurations. We find that the inter-key intervals (time interval between two consecutive key presses) of the users reduced signifi- 5.3 QualitativeFeedback cantly for the Grad-Dark-Grad-Lock configuration during For the apps evaluated in this work, 100%, 73%, 60% and the grace period. The average inter-key interval of users 30%oftheparticipantsconsideredthebanking,email,photo reduced by almost 60% during the grace period when com- andcontactsapptobesensitive,respectively. Theresponses pared to the average inter-key interval during the task (see to the pre-study question regarding the adversaries that Figure 3). A paired t-test was conducted to compare the the participants (who used protection) wanted protection inter-key interval between the grace and non-grace period againstindicatethatdifferentscenariosrequiredifferentlev- for the same text entry activity for each user. The results elsofprotection. Almostalluserswantedprotectionagainst show that inter-key intervals are significantly different be- strangers, which corresponds to the bus scenario. Corre- tween the grace and non-grace period (t(29) = 2.1, p = sponding to the office scenario, 54% of participants wanted 0.04). protectionagainstco-workers. Ontheotherhandonly11% Discussion: Our results indicate that participants took ad- ofparticipantsconsideredthattheyneededprotectionagainst vantage of the grace period by attempting to quickly com- family members, which corresponds to the home scenario. plete the text entry activity. They typed faster than their Wenowpresentthefindingsfromthefeedbackofthepartic- normal speeds during the grace period. ipantsregardingtheusabilityandsecurityperceptionsofthe configurations for each app in the different usage scenarios. 5.2.4 Effectontaskerrorrate Incasetheinputoftheusersmismatchedthedisplayedtext 5.3.1 Securityperceptions for the text entry task, we counted it as an error (with at Figure 4 shows the security perceptions of the participants most one error per task). Our results indicate that users for each re-authentication configuration. Significantly more made errors in 77 out of 600 text entry tasks. However, a (57% more) participants thought that the Imm-Dark-Imm- one-way between subjects ANOVA for the task error rate Lock configuration wasmore secure than theother configu- acrossthefourconfigurationsandBASE ROUNDindicates ration (χ2(3) = 151,p < 0.001). Imm-Dark-Imm-Lock im- nosignificantdifferences(F(4,145)=1.51,p=0.2). Similarly, mediately hides the content on the screen to prevent the whileparticipantsmadeerrorsin43outof240emailtasks, leakage of any sensitive information. Some participants in- the differences were not significant across the different con- dicated that they would take advantage of this increased figurations (F(4,28)=0.28, p=0.84). security at the cost of usability for some apps: Discussion: The task error rate among the configurations werecomparable. Thoughtheinter-keyintervaloftheusers ”If I am sending an important email, I do not duringthegraceperiodreducedsignificantly,itdidnotaffect want anybody else to look at it even for a sec- the task error rate compared to the other authentication ond. It is annoying but it would be the most configurations. beneficial.”(P13) 228 2016 Symposium on Usable Privacy and Security USENIX Association Figure5: Userpreferenceoftheconfigurationsforthebank- Figure6: Userpreferencesfortheconfigurationsforthepho- ing app in different scenarios. 1 represents the most pre- tos app in different scenarios. Only users who consider the ferred configuration while 5 represents the least preferred photos app as sensitive are included (N=18). 1 represents configuration(errorbarsrepresent95%confidenceinterval). themostpreferredconfigurationwhile5representstheleast preferredconfiguration(errorbarsrepresent95%confidence interval). This was followed by the Grad-Dark-Imm-Lock configura- tion,whichwasconsideredtobesecureby33%ofthepartic- ipants. Wefoundthatonly13%and7%oftheparticipants the participants who considered the photos app to be sen- consideredtheImm-Trans-Imm-LockandGrad-Dark-Grad- sitive preferred the Imm-Dark-Imm-Lock configuration for Lock configurations to be secure. As expected, the visible thebusscenario(Figure6). Fortheofficescenario,thepar- taskinthebackgroundisperceivednegativelybymostusers ticipants who were very concerned about protecting their in terms of security. The Grad-Dark-Grad-Lock configura- photos preferred configurations that obscured or gradually tionprovidesaccesstothe devicefora shortperiodof time obscuredtheapp,preventingitfrombeingaccessedbytheir and participants felt that their content was vulnerable dur- co-workers: ingthisperiod. Wenowexplorewhethertheconfigurations thatwereperceivedtobelesssecurewereconsideredappro- priate for some usage scenarios. “I won’t care about my photos with respect to a strangerbutinofficewhereitsmoreprofessional “Ilikedtheideathathowthelockappearsatthe environmentwiththepeopleIknow,Iwouldin- start [during Grad-Dark-Imm-Lock], so if it is crease the security of the scheme.”(P12) someone else, they can’t enter any text message “Ihavealotofphotosthatareverypersonaland and they can’t send anything compared to the I don’t want them [strangers] to see any part of last scheme [Grad-Dark-Grad-Lock] where they them.”(P6) can do anything if they are fast enough”(P4) “Imighthavealreadysharedalotofphotoswith mypartner,soIwouldpreferacomfortablelock TheImm-Dark-Imm-Lockconfigurationwasperceivedmost scheme.”(P6) secure and all participants indicated that they would only considerusingthisconfigurationfortheirbankingappona bus and at the office (see Figure 5). On the other hand, for For the contacts app, the participants were willing to use the home scenario, users had different preferences. 40% of configurations that provided device access for a period be- theusersindicatedthattheywouldstillonlyconsiderusing forelockingthemout. Theywanteditsobecausethiswould theImm-Dark-Imm-Lockconfigurationforthebankingapp allow a stranger to contact them in case they lost their de- at home while 23% of the users indicated that they would vice. The participants were less concerned about securing preferusingtheGrad-Dark-Imm-Lockconfigurationinstead. their contacts at home or office because they felt that they Some of the user comments shed more light on the user shared contacts with individuals at these locations. preferences for the banking app: “If someone picked up my phone and they are “Bankingwouldbeverysensitive,soIwantitto looking at my contacts, they could try to re- get dark as quickly as possible.”(P9) turn it to me through someone in my contacts, “Even with my partner, I won’t feel completely soIwouldchoosesomethingexcepttheonethat securewithmybankingappopenedonmyphone turns dark immediately.”(P7) thatiswhyIwouldpreferimmediatedark.”(P4) “For contacts, now there is an issue of privacy because these are people which they [office col- The feedback from the users was inconclusive for the email leagues]mightalsoknow,soitisimportantthat app and there is no one configuration that users signifi- Iprotecttheirinformationbutatthesametime cantly prefer over the other for the different usage scenar- I don’t want it to be very inconvenient for me ios. On the other hand, for the photos app, the majority of when I look at the contacts.”(P2) USENIX Association 2016 Symposium on Usable Privacy and Security 229