Table Of Content

CrowdTone: Crowd-powered tone feedback and improvement system for emails RajanVaish Andre´sMonroy-Herna´ndez StanfordUniversity MicrosoftResearch Stanford,CA,USA Redmond,WA,USA [email protected] [email protected] ABSTRACT give people access to pools of professional writers on- In this paper, we present CrowdTone, a system designed to demand. However, these services can be costly and time- help people set the appropriate tone in their email commu- consuming (e.g. finding the right people, setting a fair con- 7 nication. CrowdTone utilizes the context and content of an tract,spendingtimeinbackandforthcommunication,etc.) 1 0 emailmessagetoidentifyandsettheappropriatetonethrough Previoussystemsresearchhaslookedintomoreefficientwrit- 2 aconsensus-buildingprocessexecutedbycrowdworkers.We ing solutions that rely on crowds. Work in this space has evaluated CrowdTone with 22 participants, who provided a n proven that, with proper scaffolding, crowds can do expert- totalof29emailsthattheyhadreceivedinthepast, andran a qualityworksuchasshorteningtext,correctinggrammar,and J themthroughCrowdTone.Participantsandprofessionalwrit- finding and formatting citations [4, 16, 18, 24]. Despite all ersassessedthequalityofimprovementsfindingasubstantial 7 thiswork,nosystemsresearchhasfocusedontone,andmore increaseinthepercentageofemailsdeemed“appropriate”or ] “veryappropriate”—from25%tomorethan90%byrecipi- specificallyemails’tone. C ents,andfrom45%to90%byprofessionalwriters.Addition- In order to address the need for tone improvement in email H ally,therecipients’feedbackindicatedthatmorethan90%of communication, we created CrowdTone, a crowd-powered theCrowdToneprocessedemailsshowedimprovement. tone-improvement system. CrowdTone first receives an . s email’smaincontent,alongwithbasicinformationaboutthe c [ sender, the receiver, and open-ended context elicited by the AuthorKeywords user. Then, CrowdTone outputs an improved version of the 1 Crowdsourcing,AmazonMechanicalTurk,Tone email, with a tone appropriate to the context and receiver. v improvement,Emailwriting CrowdTone uses crowd workers from Amazon Mechanical 3 Turk to take emails through a step-by-step tone-scaffolding 9 process that identifies the original tone, improves it through 7 1 INTRODUCTION consensusworkflow,andoutputsthebestresult. 0 Settingtherighttone—attitudeexpressedthroughwords— WeevaluatedCrowdTonewith22participantsrecruitedfrom . inwrittencommunicationsoftendetermineswhetherareader 1 our own organization. We asked participants to provide willinterpretthemessageasthewriterintended[13]. 0 emails they had received that they had perceived as prob- 7 Despite changes in communication practices, email is still lematicwithregardstotone. Forexample,someparticipants 1 one of the most popular forms of communication in profes- shared emails they received from students being rude or in- v: sional settings. Every day, over 200 billion emails are writ- appropriate. Participants were asked to remove the name of i ten worldwide [29]. Despite its popularity, email is prone thesenderandotheridentifiableinformationtomaintaintheir X tomisunderstandingsjustlikeothertypesofwrittencommu- anonymity. Based on the participants’ feedback, more than r nication [33, 15, 35, 20, 11, 5]. For example, researchers 90%oftheemailswereimprovedintone,whilethepercent- a havefoundthatemailsendersoftenhavetheerroneousbelief ageofemailmessagesdeemed“appropriate”or“veryappro- thattheirrecipientswillidentifytheintendedemotionintheir priate” rose from about 25% to more than 90%. When sur- messages [15, 20, 11], whiletherealityisthatclosetohalf veyed,75%oftheseparticipants“agreed”or“totallyagreed” (44%)ofemailrecipientsfailtoidentifytheintendedtoneof when asked whether the CrowdTone emails were of high anemailmessage. Furthermore, thesameresearchersfound quality — matching their writing expectations. Besides, the that when the tone of an email is unclear, recipients tend to tone-scaffolding process was reported as easy and effective interprettheemailbasedontheirstereotypesandexistingas- bythecrowd-workers. sumptionsofthewriter. The core contribution of this work is the design, imple- Thereareafewtoolsthatautomaticallydetecttone[32, 25] mentation, and evaluation of a crowd-powered process that and sentiment [12, 1] in writing, but in order to actually set self-identifies — without user instructions — and improves theappropriatetone,peoplehireprofessionalcopywritersand the email tone with basic context and information provided. proofreadersfortheirimportantcommunications. Though outside the scope of this paper, we believe that our approachandfindingscanbeappliedtodifferentdomainsand Only few people and organizations can afford their own ex- mediaoranyformofwrittencommunications. pert writers, so platforms like Upwork.com and Wordy.com 1 RELATEDWORK five North American writers from Upwork, an online free- The need to communicate a message over email in the in- lancing platform. We selected only writers with a rating of tended way is non-trivial [15, 20, 11, 5]. In fact, several at least 4.5 / 5 that had completed more than 100 hours of online services, such as Wordzen [34] and Crystal [14], are writing-relatedworkonUpwork. Wegavethesewritersfive tryingtoaddressthissameneed. Wordzenfocusesongram- emailseach,andaskedtoimprovethetoneandtoformulate mar improvements, while Crystal makes useful suggestions and document a step-by-step process that begins with tone based on the recipient’s personality, however none of these identificationandendswithtoneimprovement. systemsfocusesonimprovingthetoneperse. Only two of the five writers could formulate a process. The The importance of tone [13], along with the need to under- otherthreefoundthistaskextremelydifficulttoexecute. One standsentiment,hasattractedtheattentionofcompanieslike ofthesethreedescribedtheproblemthisway: Google [12], IBM [32, 1] and Microsoft [2]. These com- “Thereverseprocessisacomplicatedone. Therewritingof panies have public APIs (Application Programming Inter- theemailswasnotasdifficultasbreakingeachbeatdownas faces) for analyzing sentiment. Although those APIs and yourequested. Kindofnewterritoryforanyone” othermachine-basedanalysistools[30]areextremelyuseful forparsinglargedatasets,theyarenotgenerallydesignedfor By translating our learning from this exercise and other re- end-users, and are not specifically designed to help improve sources [28], we explored a few pilot approaches to design- emailtone. ingaworkflowthatcanproducehighqualityoutputusingba- siccontext. Ourapproachutilizedthestep-by-stepworkflow Crowdsourcing is another domain being explored for writ- thatUpworkprofessionalsformulatedbasedontheirexperi- ing assistance. Crowd-powered projects like Soylent [4], ence and expertise. The rest of this section will provide a Legion [22] and Chorus [23] have encouraged researchers detaileddescriptionoftheprocess—frominputthroughthe to harness the crowd for complex tasks — and have even tonescaffoldingphasetooutput. been implemented on different interfaces, including smart- watches[27]. ProjectslikeTurkomatic[21],Ensemble[16], Input Crowdforge[18],andothers[31]havesucceededinconvert- CrowdTonesupportsGUIandRESTbasedinput,andaccepts ing macro tasks into micro chunks that makes it easier for a set of mandatory and optional information to process the crowdworkerstoaccomplishexpert-leveltaskssuchaswrit- email. Primarily intended for a sender of an email, Crowd- ingarticlesandstories. Toneacceptsthefollowinginformation: To accomplish those efforts, it is important for crowd- 1. MandatoryInformation: providesminimalcontext powered systems to focus on task sequencing [6], scaffold- ing,andcrowdcoordination. Crowdcoordinationtechniques (a) Sender: relationship(e.g. intern,student) suchasiterativeandparallelcontributions [10,9,8,7]pro- (b) Recipient: relationship(e.g. adviser,professor) ducevaluableresultsthathavebeenusedtobuildprojectslike (c) Emailsubject CrowdCrit [24]andStoria [17]. CrowdCritusesscaffolding and crowd coordination to provide expert-level critiques to (d) Emailcontent designers. (e) Open-ended context short-description to provide in- formationon,orthestorybehindtheemail IndevelopingCrowdTone,weusedexistingtechniquesfrom the literature, and developed our own sequencing and scaf- 2. OptionalInformation: providesmaximumcontext foldingapproachestoidentifyandimproveemailtone. (a) Gender: senderandrecipient THECROWDTONESYSTEM (b) Nativelanguage: senderandrecipient As shown in Figure 1, CrowdTone inputs email information (c) Hierarchy relationship (e.g. professionally senior, and context, and produces an improved email via a tone- samelevel,orjunior),ifapplicable scaffoldingandconsensusworkflowprocess. (d) Relationshiptype: friendsandfamily,acquaintances, strangers(coldemails) Input: Email subject and Crowd coordination If the sender decides to input mandatory information only, cinofnotremnat, tSioenn, dCeo rn atenxdt Receiver Tpornoece sscsa ffolding Conse nsus Output: Impro ved email tthoepyroavreidperoovpitdioinngalmininfoimrmaaltcioonn,tetxhte.yHwoiwllebveerp,riofvtihdeiyngdemcaidxe- imumcontextthatCrowdTonecansupport. Figure1. CrowdTonesystemoverview: Inputsemailinformationand context,tonescaffoldingandconsensushelpscoordinatecrowdworkers fromMTurktoproduceimprovedemail,improvedemailisproducedas Crowdcoordination—phase1:Thetonescaffoldingpro- anoutput cess,fromtoneidentificationtoimprovement Identifyingandimprovingthetoneofawrittentextisacriti- calandexpert/professional-leveltask. Inthissection,wede- Designingtheworkflowprocess scribeourdesignprocessandthesystemthatproducesthree The design of the process behind CrowdTone was informed improvedversionsoftheoriginalemail,bythreecrowdwork- byaformativestudywithprofessionalwriters. Werecruited ers.CrowdTonedoesnotaccepttone-relatedinstructionasan 2 input. Theprocessfocusesinsteadonenablingcrowdwork- (c) Step 3: Use the list and suggestions from Step 2 as erstoidentifyandimprovethetoneofanemail. self-instruction to revise and improve the email di- rectly. Identify current tone (d) Step4: IteratetheoutputfromStep3tofinetunethe email’stone. Herethecrowdworkersareencouraged Right No Identiftyo nthee right tomakefurtherimprovementsthroughdirectediting. tone? Identify things to Yes improve Overall, each email undergoes and is reviewed by three crowd-workersforconfidentconsensus. Emailsdeemedini- Identify things to Fix the tone, edit improve directly tially to have correct tone goes through two crowd-worker Improve the tone, Polish the tone once 2nd phase of consensus Polish the tone once steps,whileone’swithincorrecttonegothroughfourcrowd- edit directly more, continue editing more, continue editing workersteps. Eventually,producingthreeimprovedversions Figure2. Toneidentificationtoimprovementprocess: Crowdworkers peroriginalemail. gothroughthisprocesstoeliminatetheneedfortone-relatedinstruc- tionsfromrequester This process helps improve the tone without requiring tone- relatedinstructionsfromthesender. As shown in the Figure 2, crowd workers begin the pro- cess by reviewing the original email and context from the Crowdcoordination—phase2:Consensus,choosingthe perspective of its recipient. Acting from the recipient’s bestamongtheimprovedemailsfromlaststage viewpoint, the crowd workers attempt to identify the cur- After getting three input-phase responses for each email, rent tone using primary and secondary tone options. The CrowdTone coordinates the crowd to get consensus on and two primary options are formal or informal. The ten then output the most improved email. Figure 3 shows an secondary options are — appreciative/thankful, confident, overviewoftheconsensusworkflow. courteous/respectful/polite,emotional/persuasive,enthusias- tic/cheerful,light/humorous/friendliness,regretful/sorrowful, PhaseA:Selectingtwoemailversionsforconsensus serious,cold/unfriendly,andenraged. After three input-phase responses are generated for each email,thenextstepistoselecttwoofthethreerevisedemails These options were developed after reviewing multiple arti- forthefinalconsensusround. cles on tone [28, 26, 36] and running multiple pilot studies. Whilenotexhaustive,theyhelpthecrowdworkersmakede- If “yes” or “no” were selected twice by two crowd-workers cisionsmorequickly. Asoneworkerdescribesit: (66.66%),thetwo“yes”ortwo“no”emailsaresentforward forfurtherconsensusevaluation. “Combiningtheprimaryandsecondarytonesincurrentand correct makes it easier to quickly bring the ideas together, If an email received three “yes” or three “no” responses enjoyedit” (100%),i.e. ifanemailwasunanimouslyfoundtobeofap- propriateorinappropriatetonebythreecrowdworkers—the After identifying the current primary and secondary tone, two emails to be forwarded for further consensus evaluation each crowd worker has to decide whether that tone is right are chosen based on the similarity of attributes such as pri- or not. Based on their yes-or-no decision, the workflow di- maryandsecondarytone. vergesinthefollowingmanner: PhaseB:Selectingoneemailversionforfurtheriteration 1. If the crowd worker chooses “Yes”, he or she takes these Inthesecondpartoftheconsensusphase,fromthetwoemail actions: versions (a and b) forwarded from Phase A, three different (a) Step 1: Identify scope of improvement and at least crowd workers choose the version that — in their view — oneinstanceoftextthatcouldstillbeimproved. is the best in terms of tone, and do another iteration to im- provethetoneofthatemail. Astherearetwoemailversions, (b) Step 2: Use the suggestions from Step 1 as self- andthreedifferentcrowd-workerstomakethedecision—the instruction and make the improvements. Here the final email selected will either enjoy a majority of 66.66%, crowdworkerdirectlyeditsandimprovestheemail. or100%. Additionaliterationforimprovementalsobringsa 2. If the crowd worker chooses “No”, he or she takes these newperspectivefromanothercrowdmember. actions: (a) Step 1: Identify the right tone by choosing the ideal Output Figure 4 shows a “before” and “after” example of an email primaryandsecondarytoneoptionsfortheemailand the appropriate intensity option — very, quite close, that went through the CrowdTone process. By the time somewhat. Forexample, onecanidentifythetoneto eachemailhascompletedthetone-scaffoldingandconsensus beveryformalandappreciative. phases, six crowd workers have helped produce this output. However, depending on whether the tone of email sent was (b) Step 2: Identify and list what needs to be improved correct,theoutputwouldvary: toachievetheidealtone. Herethecrowdworkersare encouraged to be specific and list as many instances 1. If the original email tone was deemed to be correct, the aspossible. outputprovides: 3 Max assignment = 3 per HIT Max assignment = 3 per HIT Launch a survey with Response 1 top 2 options (A and If 100% (on the basis Response 1 B) with matching Survey to of agreement -yes/ primary tone (cluster) choose Final no) between A response Response 2 If > 66% (on the and B, and Response 2 > 66% (A basis of agreement - Launch a survey with draw or B) yes/no) top 2 options (A and consensus B) that caused Response 3 Response 3 agreement – yes/no Final iteration Consensus phase, 3 responses to improve From tone scaffolding phase, by 3 workers for 2 options – over A or B – 3 responses by 3 workers for A and B the winner 1 original email Figure3.Workflowoftheconsensusapproachthathelpsselectthebestimprovedemail,andfacilitatesadditionalroundofiterationtoimprovethebest onefurther Figure4. Beforeandafterexample: OntheleftweseeinputgiventoCrowdTonewithadditionalcontextaboutsenderandrecipient’srelationship, hierarchyandthenatureofemail.Ontherightweseetheimprovedemailthatwaslabeledasveryappropriatetonebyboth,recipientsandprofessionals (a) Originaltoneandintensity expressedaneedforhelpfulfeedbacktoimprovetheiremail tone. Ofthe92participants, 84%assessedthemselvestobe (b) Improvedemailwithadditionalnotes fluentandexpertinEnglishand71%ofparticipantswerefre- 2. If the original email tone was deemed to be incorrect, the quentemailusers,sendingmorethanfiveemailsperday. On outputprovides: further investigation, we found that a substantial number of respondents (74%) felt the need for help in instances when (a) Originaltoneandintensity theywereprofessionallyemailingpeoplemoreseniortothem (b) Righttoneandintensityfortheemail thattheydidnotknow. Theexpressedneedforhelplessened somewhat(59%)withregardtoprofessionallyseniorpeople (c) Improved email with additional notes about changes theydidknow. Overall,mostpeople(75%)reportedneeding andsuggestions themosthelpwhentheyweresending“cold”emails,thatis, CrowdTonesupportsgettinginputfromaRESTrequest,with emailstostrangers,especiallywhentheyneededtoaskafa- eachemailprocessinggeneratingataskid,buttherequester vor or make a request. Figures 5 and 6 give an overview of orsendercanalsogetaJSONformatoutput. thesesurveyresponsesand,takentogether,reflecttheimpor- tanceofandneedforappropriatetoneinemails. CROWDTONEEVALUATION ToassesswhetherCrowdToneimprovesemailtone,wecon- Step2:Collectemailsandinitialinformationfromthepar- ductedthefour-phaseevaluationdescribedbelow. ticipatingrecipients After understanding the use cases when tone-fixing is most Step 1: Understanding email usage, types and tone re- helpful,wegatheredemailsreceivedbysomeofouremploy- latedusecases eesalongwithadditionalinformationoneachemailviaasur- To begin our evaluation, we ran a formative study within vey. We focused on the email recipients rather than senders ourorganizationtobetterunderstandpeople’sneedswhenit becauseitistherecipients’assessmentofwhetheramessage comesto“tonefixing”inemail. Wegathered92responsesto isappropriateorinappropriateintonethatmattersmost[15, anonlinesurveyadministeredtostudents. Ofthese, 94%of 20,11,5]. Weaskedemployeesinourorganizationtoshare respondentsthoughttonetobeimportantinemail,while60% anemailtheyhadreceivedthattheyconsideredtobedeficient 4 Seeking request or favor 75% Very Appropriate 4% Declining a request or favor 50% Appropriate 22% Scheduling a meeCng 27% Neutral 39% Applying for a job posiCon 54% Not Appropriate 26% Apologizing for a mistake 60% Not at all Appropriate 9% Reprimanding an employee 28% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Figure 7. Appropriateness of the tone of original emails, assessed by Others 13% participatingrecipients 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure5.Emailtypesthatmostneedhelpwithtone Upon asking about the senders, we learned that 45% were strangerstotherecipients(coldemails),whiletheyknew52% ofthem.And41%wereprofessionallyjuniortotherecipient, Professionally senior (familiar) 59% while 38% were professionally same level, and 17% were professionally senior. Based on the recipient’s perception, Professionally same level (familiar) 23% 31% of the senders appeared to be native English speakers Professionally junior (familiar) 15% and 43% seemed to be male, while 32% appeared female. Overall,wereceivedabalancedsetofemails. Professionally senior (cold email) 74% Onascaleof1to5,with1being“notatallappropriate”and Professionally same level (cold email) 58% 5being“veryappropriate”,wefoundthat7%werereported Professionally junior (cold email) 32% bytherecipientstohavea“notatallappropriate”tone;24% to have a “not appropriate” tone; while 41% were felt to be Family or friends 8% neutral. Theremaining 24% and3% emailswere rated“ap- Others 10% propriate” and “very appropriate” by the recipients of these emails. WecanseethedetailsinFigure7. 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure6. Categoryofrecipientsforwhichsendersneedthemosthelp Foreachsampleemail,theparticipatingrecipientswerepaid withtone $10 after they gave feedback on the CrowdTone-improved version. intone. Inresponse,wereceived29emailssentto22recip- Step 3: Run HITs (Human-Intelligent Tasks) on Mechani- ients by people they knew or did not know. Before sending calTurktogetoutputfromCrowdTone these emails to us, the participants first made them anony- CrowdToneispoweredbycrowdworkersfromAmazonMe- mousbyremovingallidentifyinginformation. chanicalTurk. Inthethirdstepofourevaluation,eachemail Basedonapreliminarystudyconductedearlierwith92em- receivedthefollowingtwoCrowdTonecontexttreatments: ployees, we learned about the type of emails where people 1. Minimalcontext: mandatoryopen-endedcontext. need most help with tone fixing/improvement — therefore, weaskedourrecipientstosendusemailsthat: 2. Maximum context: additional context with information • askedforafavorormadearequest; such as: gender, native language, hierarchy relationship andtypeofrelationshiptype. • constituted a job application or a query regarding job op- Foreachofthesecontexttreatments,eachemailwentthrough portunities;or two iteration related treatment — where, half of the time, email went through two iterations of improvement; while • wereapologeticorregretful. other half of the time, email went through three iterations ofimprovement. Thethirditerationwasexecutedduringthe Among these 22 recipients, 72% were male, while 28% fe- consensus phase and caused negligible effect. Overall, for male.35%werenativeEnglishspeakers,while86%assessed each treatment, the email went through two phases — tone themselvestobefluentorexpertinEnglishlanguage. Also, scaffolding and the consensus process — and received revi- 55% participants were full time employees, while the rest sioninputfromsixcrowdworkers. wereinternsortemporaryworkers. Finally,67%participants were active email users and sent more than five emails per To filter out non-English speakers from our crowd workers, day. we chose only individuals located in the US. We also only 5 chose individuals with a 95% approval rating. Upon inquir- Original Email - Recipient CrowdTone Email - Recipient ingabouttheselectedworkers,wefoundthat58%weremale, 99%werenativeEnglishspeakers,and100%assessedthem- 70% selves to be expert and fluent in English. Of these workers, 60% 31% had a high school degree, 51% had an undergraduate collegedegree,and17%hadagraduatedegree. 50% 40% 4% 39% Step 4: Evaluate CrowdTone output through people who 30% 52% sharedemails(recipients) 4% ToassesstheabilityofCrowdTonetoimproveemailtone,we 20% 39% reached out to our participating recipients again, and asked 26% them to assess the CrowdTone-generated emails in terms of 10% 22% thetoneappropriatenessandimprovementinqualityoftone. 9% 4% 0% Inthissecondroundofinteraction,wereceivedresponsesfor Not at all Not Appropriate Neutral Appropriate Very 24 of the original 29 emails. Five participants in the initial Appropriate Appropriate group were not available to provide feedback on the newer Figure8.AppropriatenessoftoneforemailsgeneratedfromCrowdTone versionsoftheemailstheysubmitted. comparedtooriginalemails—asratedbytherecipients Step 5: Evaluate CrowdTone output through profession- alsforfurthervalidation Original Email - Professional CrowdTone Email - Professional Tomakeupfortheparticipantslostinthesecondroundofin- 100% teractionwiththerecipients,andtogetasecondperspective, 90% werecruitedthreeprofessionalwritersfromUpwork. These 80% individuals had ratings of 4.5 or higher, were from North 70% America,hadcompletedmorethan100hoursofworkonthe 41% 60% Upwork platform, and assessed themselves to be experts in English. 50% 10% 40% The professionals were asked to fill in a similar survey as 30% participating recipients, where they were shown an original 45% 48% email with its context, followed by newer versions. Pro- 20% 38% fessionals were asked to rate the tone appropriateness and 10% 14% whether the newer versions were improved, and by how 0% 3% Not at all Not Appropriate Neutral Appropriate Very much.Wederivedconsensusfromtheirresponsesthathelped Appropriate Appropriate us make decisions about the quality of work produced by Figure9.AppropriatenessoftoneforemailsgeneratedfromCrowdTone CrowdTone. comparedtooriginalemails—asratedbytheprofessionals RESULTSANDDISCUSSION Inthissection,wediscusstheresultsofourCrowdToneeval- Overall, we found CrowdTone produced emails that were uation. Asthegoalforthisstudywastoevaluatethequality bothappropriateintoneandimprovementsovertheoriginals. ofCrowdToneoutput—foreachsampleemail,wereceived feedbackfromrecipientsandprofessionalsviasurvey. These individuals compared this version to the original. After re- CrowdToneproduceshigh-qualityemailswithorwithout viewingthisfeedback,wecametothefollowingconclusions. additionalcontext Besides extra work to provide additional context, a lot of CrowdTonegeneratesemailswithappropriatetone timesusersmaynothavemorecontextualinformationofthe According to the recipients’ feedback, the original emails recipient — such as: hierarchy relationship, gender and na- weremostlyinappropriate. Only26%werejudgedtobe“ap- tive language. In this light, we wanted our system to be propriate” or “very appropriate”. As Figure 8 shows, after able to produce high-quality results, with or without addi- theCrowdToneprocess,thepercentageof“appropriate”and tional context — and hence to test whether additional con- “veryappropriate”emailsroseto91%,withthemajoritycon- text substantially improves the quality. That is, whether the sideredtobe“veryappropriate”. NoemailsfromCrowdTone CrowdTone results are substantially affected by the amount wereassessedtobe“notatallappropriate.” ofcontextprovided. According to the professionals’ feedback the percentage of AsFigure10shows,theevaluationrecipientsdesignated29% “appropriate” and “very appropriate” emails rose from 45% ofemailswithminimalcontext(mandatory)inputtobe“ap- fortheoriginalsto90%fortheCrowdToneemails.Noemails propriate”and46%oftheseemailstobe“veryappropriate”, fromCrowdTonewerejudgedbytheprofessionalstobe“in- totaling 75%. At the same time, the recipients judged 50% appropriate”(seeFigure9). of the CrowdTone emails with maximum context (including 6 Recipient - Min Context Recipient - Max Context Recipient Professional 90% 140% 80% 120% 70% 100% 60% 25% 55% 50% 50% 80% 40% 60% 45% 30% 21% 40% 46% 20% 63% 29% 20% 10% 4% 17% 29% 8% 0% 0% 4% 4% Not at all Not Appropriate Neutral Appropriate Very Not at all No Same Some Substan=al Appropriate Appropriate Improvement improvement improvement improvement Figure10. AppropriatenessoftoneforemailsgeneratedfromCrowd- Figure12.CrowdToneimprovementovertheoriginalemail—asrated Toneusingminimalandmaximumcontext—asratedbytherecipients bytherecipientsandprofessionals Overall, we learned that adding additional context does not Professional - Min Context Professional - Max Context createasubstantialdifferenceinoutput—CrowdTone’stone 120% scaffoldingandconsensusworkflowisrobustwithregardto contextualinformation. 100% 80% 45% CrowdTone substantially improves the quality of tone overtheoriginalemails 60% As part of the same survey, participating recipients were 40% 28% askedtoquantifytheimprovementintheneweremail’stone on a Likert scale from “not at all improved” to “significant 55% 20% improvement”. 24% 31% 3% 10% As Figure 12 shows, the recipients observed significant im- 0% 3% Not at all Not Appropriate Neutral Appropriate Very provement in 63% and some improvement in 29% of the Appropriate Appropriate emails, totaling 92%. Recipients observed no improvement Figure11. AppropriatenessoftoneforemailsgeneratedfromCrowd- for4%ofemailsthathadalreadybeendeemed“appropriate”. Toneusingminimalandmaximumcontext—asratedbytheprofes- sionals The averaged responses of the three professionals indicated that 55% of the emails showed significant improvement and that45%showedsomeimprovement,totalingto100%. additional information like hierarchy and relationship) to be CrowdTone generates a professional-quality tone that “appropriate”and25%tobe“veryappropriate”emails,also matchesthewritingexpectationsofrecipient totaling75%. Fromthis,welearnedthatthetotalpercentage To understand whether CrowdTone produces high-quality of CrowdTone emails judged to be “appropriate” and “very emails,thatis,thosethatmeetthewritingexpectationsofre- appropriate”isthesameforminimalandmaximuminputs. cipientsorare“professional”inquality,weaskedthesurvey As Figure 11 illustrates, the professional writers found 55% respondents whether the newer emails were of expert qual- ofCrowdToneemailswithminimalcontexttobe“appropri- ity or would match the quality of an email they had written. ate”and31%tobe“veryappropriate”,totaling86%. Incon- They responded on a Likert scale that ranged from “totally trast,theyfound45%ofemailswithmaximumcontextinput disagree”to“totallyagree”. tobe“appropriate”and28%tobe“veryappropriate”, total- As Figure 13 illustrates, the recipients responded that they ing72%. “totally agreed” regarding 21% of the emails and “agreed” After conducting a blind ranking of emails produced using regardingthehighqualityof54%oftheCrowdToneemails, minimal and maximum context input, we found that partici- totaling 75%. The professionals indicated that they “totally patingrecipientsandprofessionalsrankedoutputfrommaxi- agreed” regarding high quality for 38% of the emails and mumcontextemailshigher—54%and61%ofthetime,re- “agreed” for 45%, totaling 83%. Together, the profession- spectively. Thisisnotsubstantiallydifferentfromtheemails als and recipients found that the quality of the newer emails withminimalcontextinputs. matchedthatofprofessionalwork. 7 The crowd is essential to CrowdTone. To help us get bet- Recipient Professional ter results and to optimize the crowd workers’ experience, 120% we incorporated tone scaffolding and consensus workflow into CrowdTone. After recruiting crowd workers that satis- 100% fiedminimalrequirements—a95%approvalratingandgeo- graphicallocationintheUnitedStates—wegothigh-quality 80% 45% resultsthatpleasedtheemailrecipientsandprofessionalwrit- ers. Tohelpusunderstandthecrowd’sexperience,weasked 60% theworkerstogiveusfeedbackaspartoftheHIT. Through a preliminary qualitative study, we found that the 40% 38% crowddeemedtheCrowdToneprocesstobeextremelyeffec- 54% tive. Ithelpedthemidentifyandimproveemailtonethrough 20% 17% multiple iterations. Most crowd workers also acknowledged 21% 0% 4% 13% 8% their enjoyment of the HITs and asked to do more, as the quotesbelowdemonstrate. Totally disagree Disagree Neutral Agree Totally agree Figure13. Agreementtohighqualitymatchingthatofprofessionals— “The HIT was easy enough. I understood what was being asratedbytherecipientsandprofessionals askedofme,andIwasabletodothetaskquickly.” “HITwasveryeasy. Idon’tseeanywayyoucouldhelpme makeatonerelateddecisionfasterwiththisspecificHIT.” No Yes “The HIT was far more easy and straightforward than I ex- 120% pected. Pleasepostmore. ” 100% CONCLUSIONSANDFUTUREWORK This paper presents CrowdTone, a crowd-powered system 80% thatreceivesemailcontentandcontext,andoutputsthesame 59% emailwithimprovedtone. CrowdToneutilizescrowdwork- 60% ersfromAmazonMechanicalTurk,whogothroughspecifi- 92% callydesignedtonescaffoldingandconsensusphases—from 40% self-identifyingtonerelatedrequirementstofixingit—with- outuserinstructions. Toevaluatethequalityoftheenhanced 20% 41% emailsproducedbyCrowdTone,wecollected29emailsfrom 22participants,andhadthemreviewedbysaidparticipating 8% 0% recipientsandthreeprofessionalwritersfromUpwork.Over- Original Email CrowdTone Email all,CrowdTone’scorecontributionsare: Figure14.Chancesofrespondingback—asratedbytherecipients • Robustnesswithregardtothepresenceorabsenceofcon- textprovided. • Substantial improvements in the quality of tone over the CrowdTone increases the chances of hearing back from originalemail. Basedontheparticipatingrecipients’eval- therecipients uation,60%ormoreoftheemailsprocessedexhibitedsub- One aspect that often determines the “success” of an email stantialimprovement. iswhetheritelicitsaresponse. Althoughrecipientscanhave • Generation of professional-quality tone that in the evalu- multiple reasons for replying to an email, we focused here ation matched the writing expectations of the recipients. on tone as the primary motivator. Specifically, we investi- Whensurveyed,75%oftheseparticipants“agreed”or“to- gatedwhetherrecipientsrespondedtotheoriginalemailsand tally agreed” when asked whether the CrowdTone emails whether they would respond to the newer CrowdTone ver- wereofhighquality. sions. AsFigure14shows,wefoundthatrecipientsdidnotrespond • Improved chances of getting a response back from email to 41% of the original emails. With the emails generated recipients. Theparticipatingrecipientsindicatedthatthey byCrowdTone,therecipientsjudgedthattheywouldnotre- wouldrespondtomorethan90%oftheemails,asubstan- spond to just 8% of the emails. In other words they would tialimprovementfromtheoriginal59%. respondto92%oftheCrowdToneemails. • Tone-scaffolding process was reported as easy and effec- tivebycrowdworkers. The crowd behind the CrowdTone found the tone- ThroughCrowdTone,weextendexistingcrowd-poweredsys- scaffolding process easy and effective in accomplishing tems research for efficient writing solutions, and present a theirtask novel process that self-identifies and improves email tone. 8 Thatsaid,thesystemalsoshowedseverallimitationsthatare WordProcessorwithaCrowdInside.InProceedingsof seedsforfuturework: the23NdAnnualACMSymposiumonUserInterface SoftwareandTechnology(UIST’10).ACM,NewYork, • Privacy. Participantsraisedconcernsabouttheprivacyim- NY,USA,313–322.DOI: plications of sharing private communication with crowd- http://dx.doi.org/10.1145/1866029.1866078 workers without all the parties being aware of it. Future work might explore ways of addressing this through ob- 5. KristinByron.2008.Carryingtooheavyaload? The fuscationandexplicitnormsetting,andothermechanisms communicationandmiscommunicationofemotionby exploredinsimilarpriorwork[19] email.AcademyofManagementReview33,2(2008), 309. • Delay. Having to wait for the process of tone setting to gothroughintroducesdelaysthatmightnotbesuitablefor 6. CarrieJCai,ShamsiTIqbal,andJaimeTeevan.2016. someusers,andsomescenarios. Optimizingforreal-time ChainReactions: TheImpactofOrderonMicrotask responsesthroughcrowd-retainersasitwasdoneinsimilar Chains.InProceedingsofCHI. research[3]issomethingworthexploring. 7. StevenDow,JulieFortuna,DanSchwartz,Beth • Deepintegration.Atthemoment,CrowdToneisnotdeeply Altringer,DanielSchwartz,andScottKlemmer.2011. PrototypingDynamics: SharingMultipleDesigns embeddedintheregularworkflowofsendingemails. Cre- ImprovesExploration,GroupRapport,andResults.In ating a plugin could be one way of addressing this, how- ProceedingsoftheSIGCHIConferenceonHuman ever,pluginsarenotuniversallyused,forexample,mobile FactorsinComputingSystems(CHI’11).ACM,New email clients rarely allow for plugins. However, deeper York,NY,USA,2807–2816.DOI: integration can provide access to conversation history — makingCrowdTonemoreintelligentinitsabilitytodeter- http://dx.doi.org/10.1145/1978942.1979359 minedeepercontextandmeaning. Moreworkisneededto 8. StevenDow,AnandKulkarni,ScottKlemmer,and understand the best mechanisms for integration into peo- Bjo¨rnHartmann.2012.ShepherdingtheCrowdYields ple’sworkstyles. BetterWork.InProceedingsoftheACM2012 ConferenceonComputerSupportedCooperativeWork • Restricted applications. Though CrowdTone can be uti- (CSCW’12).ACM,NewYork,NY,USA,1013–1022. lized to improve the tone of any type of written commu- DOI:http://dx.doi.org/10.1145/2145204.2145355 nication, the current paper focuses on its application for professionalemails. Infuture,weplantoapplythecoreof 9. StevenP.Dow,AlanaGlassco,JonathanKass,Melissa our existing approach and expand it to different use cases Schwarz,DanielL.Schwartz,andScottR.Klemmer. beyondemails: toothertypesofwrittencommunications, 2010.ParallelPrototypingLeadstoBetterDesign domainsormedia. Results,MoreDivergence,andIncreasedSelf-efficacy. ACMTrans.Comput.-Hum.Interact.17,4,Article18 Weseealotofpotentialforintroducingthird-partyfeedback (Dec.2010),24pages.DOI: andfixingintoourprofessionalcommunication. Largecom- http://dx.doi.org/10.1145/1879831.1879836 panies often have marketing groups in charge of corporate communication,butweenvisionthislevelofattentiontode- 10. StevenP.Dow,KateHeddleston,andScottR.Klemmer. tail could be affordable for individuals. More generally we 2009.TheEfficacyofPrototypingUnderTime see this work as yet another example of a potential for hy- Constraints.InProceedingsoftheSeventhACM bridintelligencewherecrowdsandAIcancometogetherto ConferenceonCreativityandCognition(C&C’09). optimizeprocessesandleveragethebestofbothworlds. ACM,NewYork,NY,USA,165–174.DOI: http://dx.doi.org/10.1145/1640233.1640260 REFERENCES 11. NicholasEpleyandJustinKruger.2005.Whenwhat 1. IBMAlchemy.2015.AlchemySentimentAnalysisAPI. youtypeisntwhattheyread: Theperseveranceof (2015).http: stereotypesandexpectanciesovere-mail.Journalof //www.alchemyapi.com/api/sentiment-analysis. ExperimentalSocialPsychology41,4(2005),414–422. 2. MicrosoftAzure.2015.LexiconBasedSentiment 12. Google.2015.SentimentAnalysisModel.(2015). Analysis.(2015).https://azure.microsoft.com. https://cloud.google.com/prediction/docs/ 3. MichaelS.Bernstein,JoelBrandt,RobertC.Miller,and sentiment_analysis. DavidR.Karger.2011.CrowdsinTwoSeconds: 13. NickGreer,JaimeTeevan,andShamsiTIqbal.2016. EnablingRealtimeCrowd-poweredInterfaces.In Anintroductiontotechnologicalsupportforwriting. Proceedingsofthe24thAnnualACMSymposiumon TechnicalReport.TechnicalReport.MicrosoftResearch UserInterfaceSoftwareandTechnology(UIST’11). TechReportMSR-TR-2016-001. ACM,NewYork,NY,USA,33–42.DOI: http://dx.doi.org/10.1145/2047196.2047201 14. CrystalProjectInc.2015.Crystal.(2015). https://www.crystalknows.com/. 4. MichaelS.Bernstein,GregLittle,RobertC.Miller, Bjo¨rnHartmann,MarkS.Ackerman,DavidR.Karger, 15. EricJaffe.2014.WhyIt’sSoHardToDetectEmotionIn DavidCrowell,andKatrinaPanovich.2010.Soylent: A EmailsAndTexts.(2014). 9 http://www.fastcodesign.com/3036748/evidence/ 24. KurtLuther,Jari-LeeTolentino,WeiWu,AmyPavel, why-its-so-hard-to-detect-emotion-in-emails. BrianP.Bailey,ManeeshAgrawala,Bjo¨rnHartmann, andStevenP.Dow.2015.Structuring,Aggregating,and 16. JoyKim,JustinCheng,andMichaelS.Bernstein.2014. EvaluatingCrowdsourcedDesignCritique.In Ensemble: ExploringComplementaryStrengthsof Proceedingsofthe18thACMConferenceonComputer LeadersandCrowdsinCreativeCollaboration.In SupportedCooperativeWork&SocialComputing Proceedingsofthe17thACMConferenceonComputer (CSCW’15).ACM,NewYork,NY,USA,473–485. SupportedCooperativeWork&SocialComputing DOI:http://dx.doi.org/10.1145/2675133.2675283 (CSCW’14).ACM,NewYork,NY,USA,745–755. DOI:http://dx.doi.org/10.1145/2531602.2531638 25. Lymbix.2015.ToneCheck.(2015). http://tonecheck.com/. 17. JoyKimandAndresMonroy-Hernandez.2016.Storia: 26. MailChimp.2015.Voice&Tone.(2015). SummarizingSocialMediaContentBasedonNarrative http://voiceandtone.com/. TheoryUsingCrowdsourcing.InProceedingsofthe 19thACMConferenceonComputer-Supported 27. MichaelNebeling,AlexandraTo,AnhongGuo, CooperativeWork&SocialComputing(CSCW’16). AdrianAdeFreitas,JaimeTeevan,StevenPDow,and ACM,NewYork,NY,USA,1018–1027.DOI: JeffreyPBigham.2016.WearWrite: Crowd-Assisted http://dx.doi.org/10.1145/2818048.2820072 WritingfromSmartwatches.InProceedingsofCHI. 18. AniketKittur,BorisSmus,SusheelKhamkar,and 28. LisaPearlandMarkSteyvers.2013.Cmon YouShould RobertE.Kraut.2011.CrowdForge: Crowdsourcing ReadThis: AutomaticIdentificationofTonefrom ComplexWork.InProceedingsofthe24thAnnualACM LanguageText.InternationalJournalofComputational SymposiumonUserInterfaceSoftwareandTechnology Linguistics(IJCL)4,1(2013),12–30. (UIST’11).ACM,NewYork,NY,USA,43–52.DOI: 29. CraigSmith.2016.BytheNumbers: 60Incredible http://dx.doi.org/10.1145/2047196.2047202 EmailStatistics.(2016).http://expandedramblings. 19. NicolasKokkalis,ThomasKo¨hn,CarlPfeiffer,Dima com/index.php/email-statistics/. Chornyi,MichaelS.Bernstein,andScottR.Klemmer. 30. YlaR.TausczikandJamesW.Pennebaker.2010.The 2013.EmailValet: ManagingEmailOverloadThrough PsychologicalMeaningofWords: LIWCand Private,AccountableCrowdsourcing.InProceedingsof ComputerizedTextAnalysisMethods.Journalof the2013ConferenceonComputerSupported LanguageandSocialPsychology29,1(2010),24–54. CooperativeWork(CSCW’13).ACM,NewYork,NY, DOI:http://dx.doi.org/10.1177/0261927X09351676 USA,1291–1300.DOI: http://dx.doi.org/10.1145/2441776.2441922 31. JaimeTeevan,ShamsiIqbal,andCurtisvonVeh.2016. 20. JustinKruger,NicholasEpley,JasonParker,and SupportingCollaborativeWritingwithMicrotasks.In Zhi-WenNg.2005.Egocentrismovere-mail: canwe ProceedingsofCHI. communicateaswellaswethink? Journalof 32. IBMWatson.2015.ToneAnalyzer.(2015). personalityandsocialpsychology89,6(2005),925. http://www.ibm.com/smarterplanet/us/en/ 21. AnandKulkarni,MatthewCan,andBjo¨rnHartmann. ibmwatson/developercloud/tone-analyzer.html. 2012.CollaborativelyCrowdsourcingWorkflowswith 33. LeaWinerman.2006.E-mailsandegos: Aninabilityto Turkomatic.InProceedingsoftheACM2012 stepoutsideofone’sownheadmaybebehinde-mail ConferenceonComputerSupportedCooperativeWork miscommunication,accordingtorecentresearch. (CSCW’12).ACM,NewYork,NY,USA,1003–1012. AmericanPsychologicalAssociation37,2(2006),16. DOI:http://dx.doi.org/10.1145/2145204.2145354 34. IncWordzen.2015.Wordzen.(2015). 22. WalterS.Lasecki,KyleI.Murray,SamuelWhite, http://www.wordzen.com/. RobertC.Miller,andJeffreyP.Bigham.2011. 35. BusinessCoachingWorldwide.2007.DidYouKnow: Real-timeCrowdControlofExistingInterfaces.In Proceedingsofthe24thAnnualACMSymposiumon Thatemailrecipientscorrectlyinterprettheintended UserInterfaceSoftwareandTechnology(UIST’11). toneofanemailmessageonly50%ofthetime? (2007). ACM,NewYork,NY,USA,23–32.DOI: http://www.wabccoaches.com/bcw/2007_v3_i1/ didyouknow.html. http://dx.doi.org/10.1145/2047196.2047200 36. WriteExpress.2015.Howtousetoneinyourwriting. 23. WalterS.Lasecki,RachelWesley,JeffreyNichols, (2015).http://www.writeexpress.com/tone.html. AnandKulkarni,JamesF.Allen,andJeffreyP.Bigham. 2013.Chorus: ACrowd-poweredConversational Assistant.InProceedingsofthe26thAnnualACM SymposiumonUserInterfaceSoftwareandTechnology (UIST’13).ACM,NewYork,NY,USA,151–162. DOI:http://dx.doi.org/10.1145/2501988.2502057 10

ebook img

CrowdTone: Crowd-powered tone feedback and improvement system for emails PDF

0.65 MB·
by  Rajan Vaish
#journals #arxiv
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview CrowdTone: Crowd-powered tone feedback and improvement system for emails

CrowdTone: Crowd-powered tone feedback and improvement system for emails RajanVaish Andre´sMonroy-Herna´ndez StanfordUniversity MicrosoftResearch Stanford,CA,USA Redmond,WA,USA [email protected] [email protected] ABSTRACT give people access to pools of professional writers on- In this paper, we present CrowdTone, a system designed to demand. However, these services can be costly and time- help people set the appropriate tone in their email commu- consuming (e.g. finding the right people, setting a fair con- 7 nication. CrowdTone utilizes the context and content of an tract,spendingtimeinbackandforthcommunication,etc.) 1 0 emailmessagetoidentifyandsettheappropriatetonethrough Previoussystemsresearchhaslookedintomoreefficientwrit- 2 aconsensus-buildingprocessexecutedbycrowdworkers.We ing solutions that rely on crowds. Work in this space has evaluated CrowdTone with 22 participants, who provided a n proven that, with proper scaffolding, crowds can do expert- totalof29emailsthattheyhadreceivedinthepast, andran a qualityworksuchasshorteningtext,correctinggrammar,and J themthroughCrowdTone.Participantsandprofessionalwrit- finding and formatting citations [4, 16, 18, 24]. Despite all ersassessedthequalityofimprovementsfindingasubstantial 7 thiswork,nosystemsresearchhasfocusedontone,andmore increaseinthepercentageofemailsdeemed“appropriate”or ] “veryappropriate”—from25%tomorethan90%byrecipi- specificallyemails’tone. C ents,andfrom45%to90%byprofessionalwriters.Addition- In order to address the need for tone improvement in email H ally,therecipients’feedbackindicatedthatmorethan90%of communication, we created CrowdTone, a crowd-powered theCrowdToneprocessedemailsshowedimprovement. tone-improvement system. CrowdTone first receives an . s email’smaincontent,alongwithbasicinformationaboutthe c [ sender, the receiver, and open-ended context elicited by the AuthorKeywords user. Then, CrowdTone outputs an improved version of the 1 Crowdsourcing,AmazonMechanicalTurk,Tone email, with a tone appropriate to the context and receiver. v improvement,Emailwriting CrowdTone uses crowd workers from Amazon Mechanical 3 Turk to take emails through a step-by-step tone-scaffolding 9 process that identifies the original tone, improves it through 7 1 INTRODUCTION consensusworkflow,andoutputsthebestresult. 0 Settingtherighttone—attitudeexpressedthroughwords— WeevaluatedCrowdTonewith22participantsrecruitedfrom . inwrittencommunicationsoftendetermineswhetherareader 1 our own organization. We asked participants to provide willinterpretthemessageasthewriterintended[13]. 0 emails they had received that they had perceived as prob- 7 Despite changes in communication practices, email is still lematicwithregardstotone. Forexample,someparticipants 1 one of the most popular forms of communication in profes- shared emails they received from students being rude or in- v: sional settings. Every day, over 200 billion emails are writ- appropriate. Participants were asked to remove the name of i ten worldwide [29]. Despite its popularity, email is prone thesenderandotheridentifiableinformationtomaintaintheir X tomisunderstandingsjustlikeothertypesofwrittencommu- anonymity. Based on the participants’ feedback, more than r nication [33, 15, 35, 20, 11, 5]. For example, researchers 90%oftheemailswereimprovedintone,whilethepercent- a havefoundthatemailsendersoftenhavetheerroneousbelief ageofemailmessagesdeemed“appropriate”or“veryappro- thattheirrecipientswillidentifytheintendedemotionintheir priate” rose from about 25% to more than 90%. When sur- messages [15, 20, 11], whiletherealityisthatclosetohalf veyed,75%oftheseparticipants“agreed”or“totallyagreed” (44%)ofemailrecipientsfailtoidentifytheintendedtoneof when asked whether the CrowdTone emails were of high anemailmessage. Furthermore, thesameresearchersfound quality — matching their writing expectations. Besides, the that when the tone of an email is unclear, recipients tend to tone-scaffolding process was reported as easy and effective interprettheemailbasedontheirstereotypesandexistingas- bythecrowd-workers. sumptionsofthewriter. The core contribution of this work is the design, imple- Thereareafewtoolsthatautomaticallydetecttone[32, 25] mentation, and evaluation of a crowd-powered process that and sentiment [12, 1] in writing, but in order to actually set self-identifies — without user instructions — and improves theappropriatetone,peoplehireprofessionalcopywritersand the email tone with basic context and information provided. proofreadersfortheirimportantcommunications. Though outside the scope of this paper, we believe that our approachandfindingscanbeappliedtodifferentdomainsand Only few people and organizations can afford their own ex- mediaoranyformofwrittencommunications. pert writers, so platforms like Upwork.com and Wordy.com 1 RELATEDWORK five North American writers from Upwork, an online free- The need to communicate a message over email in the in- lancing platform. We selected only writers with a rating of tended way is non-trivial [15, 20, 11, 5]. In fact, several at least 4.5 / 5 that had completed more than 100 hours of online services, such as Wordzen [34] and Crystal [14], are writing-relatedworkonUpwork. Wegavethesewritersfive tryingtoaddressthissameneed. Wordzenfocusesongram- emailseach,andaskedtoimprovethetoneandtoformulate mar improvements, while Crystal makes useful suggestions and document a step-by-step process that begins with tone based on the recipient’s personality, however none of these identificationandendswithtoneimprovement. systemsfocusesonimprovingthetoneperse. Only two of the five writers could formulate a process. The The importance of tone [13], along with the need to under- otherthreefoundthistaskextremelydifficulttoexecute. One standsentiment,hasattractedtheattentionofcompanieslike ofthesethreedescribedtheproblemthisway: Google [12], IBM [32, 1] and Microsoft [2]. These com- “Thereverseprocessisacomplicatedone. Therewritingof panies have public APIs (Application Programming Inter- theemailswasnotasdifficultasbreakingeachbeatdownas faces) for analyzing sentiment. Although those APIs and yourequested. Kindofnewterritoryforanyone” othermachine-basedanalysistools[30]areextremelyuseful forparsinglargedatasets,theyarenotgenerallydesignedfor By translating our learning from this exercise and other re- end-users, and are not specifically designed to help improve sources [28], we explored a few pilot approaches to design- emailtone. ingaworkflowthatcanproducehighqualityoutputusingba- siccontext. Ourapproachutilizedthestep-by-stepworkflow Crowdsourcing is another domain being explored for writ- thatUpworkprofessionalsformulatedbasedontheirexperi- ing assistance. Crowd-powered projects like Soylent [4], ence and expertise. The rest of this section will provide a Legion [22] and Chorus [23] have encouraged researchers detaileddescriptionoftheprocess—frominputthroughthe to harness the crowd for complex tasks — and have even tonescaffoldingphasetooutput. been implemented on different interfaces, including smart- watches[27]. ProjectslikeTurkomatic[21],Ensemble[16], Input Crowdforge[18],andothers[31]havesucceededinconvert- CrowdTonesupportsGUIandRESTbasedinput,andaccepts ing macro tasks into micro chunks that makes it easier for a set of mandatory and optional information to process the crowdworkerstoaccomplishexpert-leveltaskssuchaswrit- email. Primarily intended for a sender of an email, Crowd- ingarticlesandstories. Toneacceptsthefollowinginformation: To accomplish those efforts, it is important for crowd- 1. MandatoryInformation: providesminimalcontext powered systems to focus on task sequencing [6], scaffold- ing,andcrowdcoordination. Crowdcoordinationtechniques (a) Sender: relationship(e.g. intern,student) suchasiterativeandparallelcontributions [10,9,8,7]pro- (b) Recipient: relationship(e.g. adviser,professor) ducevaluableresultsthathavebeenusedtobuildprojectslike (c) Emailsubject CrowdCrit [24]andStoria [17]. CrowdCritusesscaffolding and crowd coordination to provide expert-level critiques to (d) Emailcontent designers. (e) Open-ended context short-description to provide in- formationon,orthestorybehindtheemail IndevelopingCrowdTone,weusedexistingtechniquesfrom the literature, and developed our own sequencing and scaf- 2. OptionalInformation: providesmaximumcontext foldingapproachestoidentifyandimproveemailtone. (a) Gender: senderandrecipient THECROWDTONESYSTEM (b) Nativelanguage: senderandrecipient As shown in Figure 1, CrowdTone inputs email information (c) Hierarchy relationship (e.g. professionally senior, and context, and produces an improved email via a tone- samelevel,orjunior),ifapplicable scaffoldingandconsensusworkflowprocess. (d) Relationshiptype: friendsandfamily,acquaintances, strangers(coldemails) Input: Email subject and Crowd coordination If the sender decides to input mandatory information only, cinofnotremnat, tSioenn, dCeo rn atenxdt Receiver Tpornoece sscsa ffolding Conse nsus Output: Impro ved email tthoepyroavreidperoovpitdioinngalmininfoimrmaaltcioonn,tetxhte.yHwoiwllebveerp,riofvtihdeiyngdemcaidxe- imumcontextthatCrowdTonecansupport. Figure1. CrowdTonesystemoverview: Inputsemailinformationand context,tonescaffoldingandconsensushelpscoordinatecrowdworkers fromMTurktoproduceimprovedemail,improvedemailisproducedas Crowdcoordination—phase1:Thetonescaffoldingpro- anoutput cess,fromtoneidentificationtoimprovement Identifyingandimprovingthetoneofawrittentextisacriti- calandexpert/professional-leveltask. Inthissection,wede- Designingtheworkflowprocess scribeourdesignprocessandthesystemthatproducesthree The design of the process behind CrowdTone was informed improvedversionsoftheoriginalemail,bythreecrowdwork- byaformativestudywithprofessionalwriters. Werecruited ers.CrowdTonedoesnotaccepttone-relatedinstructionasan 2 input. Theprocessfocusesinsteadonenablingcrowdwork- (c) Step 3: Use the list and suggestions from Step 2 as erstoidentifyandimprovethetoneofanemail. self-instruction to revise and improve the email di- rectly. Identify current tone (d) Step4: IteratetheoutputfromStep3tofinetunethe email’stone. Herethecrowdworkersareencouraged Right No Identiftyo nthee right tomakefurtherimprovementsthroughdirectediting. tone? Identify things to Yes improve Overall, each email undergoes and is reviewed by three crowd-workersforconfidentconsensus. Emailsdeemedini- Identify things to Fix the tone, edit improve directly tially to have correct tone goes through two crowd-worker Improve the tone, Polish the tone once 2nd phase of consensus Polish the tone once steps,whileone’swithincorrecttonegothroughfourcrowd- edit directly more, continue editing more, continue editing workersteps. Eventually,producingthreeimprovedversions Figure2. Toneidentificationtoimprovementprocess: Crowdworkers peroriginalemail. gothroughthisprocesstoeliminatetheneedfortone-relatedinstruc- tionsfromrequester This process helps improve the tone without requiring tone- relatedinstructionsfromthesender. As shown in the Figure 2, crowd workers begin the pro- cess by reviewing the original email and context from the Crowdcoordination—phase2:Consensus,choosingthe perspective of its recipient. Acting from the recipient’s bestamongtheimprovedemailsfromlaststage viewpoint, the crowd workers attempt to identify the cur- After getting three input-phase responses for each email, rent tone using primary and secondary tone options. The CrowdTone coordinates the crowd to get consensus on and two primary options are formal or informal. The ten then output the most improved email. Figure 3 shows an secondary options are — appreciative/thankful, confident, overviewoftheconsensusworkflow. courteous/respectful/polite,emotional/persuasive,enthusias- tic/cheerful,light/humorous/friendliness,regretful/sorrowful, PhaseA:Selectingtwoemailversionsforconsensus serious,cold/unfriendly,andenraged. After three input-phase responses are generated for each email,thenextstepistoselecttwoofthethreerevisedemails These options were developed after reviewing multiple arti- forthefinalconsensusround. cles on tone [28, 26, 36] and running multiple pilot studies. Whilenotexhaustive,theyhelpthecrowdworkersmakede- If “yes” or “no” were selected twice by two crowd-workers cisionsmorequickly. Asoneworkerdescribesit: (66.66%),thetwo“yes”ortwo“no”emailsaresentforward forfurtherconsensusevaluation. “Combiningtheprimaryandsecondarytonesincurrentand correct makes it easier to quickly bring the ideas together, If an email received three “yes” or three “no” responses enjoyedit” (100%),i.e. ifanemailwasunanimouslyfoundtobeofap- propriateorinappropriatetonebythreecrowdworkers—the After identifying the current primary and secondary tone, two emails to be forwarded for further consensus evaluation each crowd worker has to decide whether that tone is right are chosen based on the similarity of attributes such as pri- or not. Based on their yes-or-no decision, the workflow di- maryandsecondarytone. vergesinthefollowingmanner: PhaseB:Selectingoneemailversionforfurtheriteration 1. If the crowd worker chooses “Yes”, he or she takes these Inthesecondpartoftheconsensusphase,fromthetwoemail actions: versions (a and b) forwarded from Phase A, three different (a) Step 1: Identify scope of improvement and at least crowd workers choose the version that — in their view — oneinstanceoftextthatcouldstillbeimproved. is the best in terms of tone, and do another iteration to im- provethetoneofthatemail. Astherearetwoemailversions, (b) Step 2: Use the suggestions from Step 1 as self- andthreedifferentcrowd-workerstomakethedecision—the instruction and make the improvements. Here the final email selected will either enjoy a majority of 66.66%, crowdworkerdirectlyeditsandimprovestheemail. or100%. Additionaliterationforimprovementalsobringsa 2. If the crowd worker chooses “No”, he or she takes these newperspectivefromanothercrowdmember. actions: (a) Step 1: Identify the right tone by choosing the ideal Output Figure 4 shows a “before” and “after” example of an email primaryandsecondarytoneoptionsfortheemailand the appropriate intensity option — very, quite close, that went through the CrowdTone process. By the time somewhat. Forexample, onecanidentifythetoneto eachemailhascompletedthetone-scaffoldingandconsensus beveryformalandappreciative. phases, six crowd workers have helped produce this output. However, depending on whether the tone of email sent was (b) Step 2: Identify and list what needs to be improved correct,theoutputwouldvary: toachievetheidealtone. Herethecrowdworkersare encouraged to be specific and list as many instances 1. If the original email tone was deemed to be correct, the aspossible. outputprovides: 3 Max assignment = 3 per HIT Max assignment = 3 per HIT Launch a survey with Response 1 top 2 options (A and If 100% (on the basis Response 1 B) with matching Survey to of agreement -yes/ primary tone (cluster) choose Final no) between A response Response 2 If > 66% (on the and B, and Response 2 > 66% (A basis of agreement - Launch a survey with draw or B) yes/no) top 2 options (A and consensus B) that caused Response 3 Response 3 agreement – yes/no Final iteration Consensus phase, 3 responses to improve From tone scaffolding phase, by 3 workers for 2 options – over A or B – 3 responses by 3 workers for A and B the winner 1 original email Figure3.Workflowoftheconsensusapproachthathelpsselectthebestimprovedemail,andfacilitatesadditionalroundofiterationtoimprovethebest onefurther Figure4. Beforeandafterexample: OntheleftweseeinputgiventoCrowdTonewithadditionalcontextaboutsenderandrecipient’srelationship, hierarchyandthenatureofemail.Ontherightweseetheimprovedemailthatwaslabeledasveryappropriatetonebyboth,recipientsandprofessionals (a) Originaltoneandintensity expressedaneedforhelpfulfeedbacktoimprovetheiremail tone. Ofthe92participants, 84%assessedthemselvestobe (b) Improvedemailwithadditionalnotes fluentandexpertinEnglishand71%ofparticipantswerefre- 2. If the original email tone was deemed to be incorrect, the quentemailusers,sendingmorethanfiveemailsperday. On outputprovides: further investigation, we found that a substantial number of respondents (74%) felt the need for help in instances when (a) Originaltoneandintensity theywereprofessionallyemailingpeoplemoreseniortothem (b) Righttoneandintensityfortheemail thattheydidnotknow. Theexpressedneedforhelplessened somewhat(59%)withregardtoprofessionallyseniorpeople (c) Improved email with additional notes about changes theydidknow. Overall,mostpeople(75%)reportedneeding andsuggestions themosthelpwhentheyweresending“cold”emails,thatis, CrowdTonesupportsgettinginputfromaRESTrequest,with emailstostrangers,especiallywhentheyneededtoaskafa- eachemailprocessinggeneratingataskid,buttherequester vor or make a request. Figures 5 and 6 give an overview of orsendercanalsogetaJSONformatoutput. thesesurveyresponsesand,takentogether,reflecttheimpor- tanceofandneedforappropriatetoneinemails. CROWDTONEEVALUATION ToassesswhetherCrowdToneimprovesemailtone,wecon- Step2:Collectemailsandinitialinformationfromthepar- ductedthefour-phaseevaluationdescribedbelow. ticipatingrecipients After understanding the use cases when tone-fixing is most Step 1: Understanding email usage, types and tone re- helpful,wegatheredemailsreceivedbysomeofouremploy- latedusecases eesalongwithadditionalinformationoneachemailviaasur- To begin our evaluation, we ran a formative study within vey. We focused on the email recipients rather than senders ourorganizationtobetterunderstandpeople’sneedswhenit becauseitistherecipients’assessmentofwhetheramessage comesto“tonefixing”inemail. Wegathered92responsesto isappropriateorinappropriateintonethatmattersmost[15, anonlinesurveyadministeredtostudents. Ofthese, 94%of 20,11,5]. Weaskedemployeesinourorganizationtoshare respondentsthoughttonetobeimportantinemail,while60% anemailtheyhadreceivedthattheyconsideredtobedeficient 4 Seeking request or favor 75% Very Appropriate 4% Declining a request or favor 50% Appropriate 22% Scheduling a meeCng 27% Neutral 39% Applying for a job posiCon 54% Not Appropriate 26% Apologizing for a mistake 60% Not at all Appropriate 9% Reprimanding an employee 28% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Figure 7. Appropriateness of the tone of original emails, assessed by Others 13% participatingrecipients 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure5.Emailtypesthatmostneedhelpwithtone Upon asking about the senders, we learned that 45% were strangerstotherecipients(coldemails),whiletheyknew52% ofthem.And41%wereprofessionallyjuniortotherecipient, Professionally senior (familiar) 59% while 38% were professionally same level, and 17% were professionally senior. Based on the recipient’s perception, Professionally same level (familiar) 23% 31% of the senders appeared to be native English speakers Professionally junior (familiar) 15% and 43% seemed to be male, while 32% appeared female. Overall,wereceivedabalancedsetofemails. Professionally senior (cold email) 74% Onascaleof1to5,with1being“notatallappropriate”and Professionally same level (cold email) 58% 5being“veryappropriate”,wefoundthat7%werereported Professionally junior (cold email) 32% bytherecipientstohavea“notatallappropriate”tone;24% to have a “not appropriate” tone; while 41% were felt to be Family or friends 8% neutral. Theremaining 24% and3% emailswere rated“ap- Others 10% propriate” and “very appropriate” by the recipients of these emails. WecanseethedetailsinFigure7. 0% 10% 20% 30% 40% 50% 60% 70% 80% Figure6. Categoryofrecipientsforwhichsendersneedthemosthelp Foreachsampleemail,theparticipatingrecipientswerepaid withtone $10 after they gave feedback on the CrowdTone-improved version. intone. Inresponse,wereceived29emailssentto22recip- Step 3: Run HITs (Human-Intelligent Tasks) on Mechani- ients by people they knew or did not know. Before sending calTurktogetoutputfromCrowdTone these emails to us, the participants first made them anony- CrowdToneispoweredbycrowdworkersfromAmazonMe- mousbyremovingallidentifyinginformation. chanicalTurk. Inthethirdstepofourevaluation,eachemail Basedonapreliminarystudyconductedearlierwith92em- receivedthefollowingtwoCrowdTonecontexttreatments: ployees, we learned about the type of emails where people 1. Minimalcontext: mandatoryopen-endedcontext. need most help with tone fixing/improvement — therefore, weaskedourrecipientstosendusemailsthat: 2. Maximum context: additional context with information • askedforafavorormadearequest; such as: gender, native language, hierarchy relationship andtypeofrelationshiptype. • constituted a job application or a query regarding job op- Foreachofthesecontexttreatments,eachemailwentthrough portunities;or two iteration related treatment — where, half of the time, email went through two iterations of improvement; while • wereapologeticorregretful. other half of the time, email went through three iterations ofimprovement. Thethirditerationwasexecutedduringthe Among these 22 recipients, 72% were male, while 28% fe- consensus phase and caused negligible effect. Overall, for male.35%werenativeEnglishspeakers,while86%assessed each treatment, the email went through two phases — tone themselvestobefluentorexpertinEnglishlanguage. Also, scaffolding and the consensus process — and received revi- 55% participants were full time employees, while the rest sioninputfromsixcrowdworkers. wereinternsortemporaryworkers. Finally,67%participants were active email users and sent more than five emails per To filter out non-English speakers from our crowd workers, day. we chose only individuals located in the US. We also only 5 chose individuals with a 95% approval rating. Upon inquir- Original Email - Recipient CrowdTone Email - Recipient ingabouttheselectedworkers,wefoundthat58%weremale, 99%werenativeEnglishspeakers,and100%assessedthem- 70% selves to be expert and fluent in English. Of these workers, 60% 31% had a high school degree, 51% had an undergraduate collegedegree,and17%hadagraduatedegree. 50% 40% 4% 39% Step 4: Evaluate CrowdTone output through people who 30% 52% sharedemails(recipients) 4% ToassesstheabilityofCrowdTonetoimproveemailtone,we 20% 39% reached out to our participating recipients again, and asked 26% them to assess the CrowdTone-generated emails in terms of 10% 22% thetoneappropriatenessandimprovementinqualityoftone. 9% 4% 0% Inthissecondroundofinteraction,wereceivedresponsesfor Not at all Not Appropriate Neutral Appropriate Very 24 of the original 29 emails. Five participants in the initial Appropriate Appropriate group were not available to provide feedback on the newer Figure8.AppropriatenessoftoneforemailsgeneratedfromCrowdTone versionsoftheemailstheysubmitted. comparedtooriginalemails—asratedbytherecipients Step 5: Evaluate CrowdTone output through profession- alsforfurthervalidation Original Email - Professional CrowdTone Email - Professional Tomakeupfortheparticipantslostinthesecondroundofin- 100% teractionwiththerecipients,andtogetasecondperspective, 90% werecruitedthreeprofessionalwritersfromUpwork. These 80% individuals had ratings of 4.5 or higher, were from North 70% America,hadcompletedmorethan100hoursofworkonthe 41% 60% Upwork platform, and assessed themselves to be experts in English. 50% 10% 40% The professionals were asked to fill in a similar survey as 30% participating recipients, where they were shown an original 45% 48% email with its context, followed by newer versions. Pro- 20% 38% fessionals were asked to rate the tone appropriateness and 10% 14% whether the newer versions were improved, and by how 0% 3% Not at all Not Appropriate Neutral Appropriate Very much.Wederivedconsensusfromtheirresponsesthathelped Appropriate Appropriate us make decisions about the quality of work produced by Figure9.AppropriatenessoftoneforemailsgeneratedfromCrowdTone CrowdTone. comparedtooriginalemails—asratedbytheprofessionals RESULTSANDDISCUSSION Inthissection,wediscusstheresultsofourCrowdToneeval- Overall, we found CrowdTone produced emails that were uation. Asthegoalforthisstudywastoevaluatethequality bothappropriateintoneandimprovementsovertheoriginals. ofCrowdToneoutput—foreachsampleemail,wereceived feedbackfromrecipientsandprofessionalsviasurvey. These individuals compared this version to the original. After re- CrowdToneproduceshigh-qualityemailswithorwithout viewingthisfeedback,wecametothefollowingconclusions. additionalcontext Besides extra work to provide additional context, a lot of CrowdTonegeneratesemailswithappropriatetone timesusersmaynothavemorecontextualinformationofthe According to the recipients’ feedback, the original emails recipient — such as: hierarchy relationship, gender and na- weremostlyinappropriate. Only26%werejudgedtobe“ap- tive language. In this light, we wanted our system to be propriate” or “very appropriate”. As Figure 8 shows, after able to produce high-quality results, with or without addi- theCrowdToneprocess,thepercentageof“appropriate”and tional context — and hence to test whether additional con- “veryappropriate”emailsroseto91%,withthemajoritycon- text substantially improves the quality. That is, whether the sideredtobe“veryappropriate”. NoemailsfromCrowdTone CrowdTone results are substantially affected by the amount wereassessedtobe“notatallappropriate.” ofcontextprovided. According to the professionals’ feedback the percentage of AsFigure10shows,theevaluationrecipientsdesignated29% “appropriate” and “very appropriate” emails rose from 45% ofemailswithminimalcontext(mandatory)inputtobe“ap- fortheoriginalsto90%fortheCrowdToneemails.Noemails propriate”and46%oftheseemailstobe“veryappropriate”, fromCrowdTonewerejudgedbytheprofessionalstobe“in- totaling 75%. At the same time, the recipients judged 50% appropriate”(seeFigure9). of the CrowdTone emails with maximum context (including 6 Recipient - Min Context Recipient - Max Context Recipient Professional 90% 140% 80% 120% 70% 100% 60% 25% 55% 50% 50% 80% 40% 60% 45% 30% 21% 40% 46% 20% 63% 29% 20% 10% 4% 17% 29% 8% 0% 0% 4% 4% Not at all Not Appropriate Neutral Appropriate Very Not at all No Same Some Substan=al Appropriate Appropriate Improvement improvement improvement improvement Figure10. AppropriatenessoftoneforemailsgeneratedfromCrowd- Figure12.CrowdToneimprovementovertheoriginalemail—asrated Toneusingminimalandmaximumcontext—asratedbytherecipients bytherecipientsandprofessionals Overall, we learned that adding additional context does not Professional - Min Context Professional - Max Context createasubstantialdifferenceinoutput—CrowdTone’stone 120% scaffoldingandconsensusworkflowisrobustwithregardto contextualinformation. 100% 80% 45% CrowdTone substantially improves the quality of tone overtheoriginalemails 60% As part of the same survey, participating recipients were 40% 28% askedtoquantifytheimprovementintheneweremail’stone on a Likert scale from “not at all improved” to “significant 55% 20% improvement”. 24% 31% 3% 10% As Figure 12 shows, the recipients observed significant im- 0% 3% Not at all Not Appropriate Neutral Appropriate Very provement in 63% and some improvement in 29% of the Appropriate Appropriate emails, totaling 92%. Recipients observed no improvement Figure11. AppropriatenessoftoneforemailsgeneratedfromCrowd- for4%ofemailsthathadalreadybeendeemed“appropriate”. Toneusingminimalandmaximumcontext—asratedbytheprofes- sionals The averaged responses of the three professionals indicated that 55% of the emails showed significant improvement and that45%showedsomeimprovement,totalingto100%. additional information like hierarchy and relationship) to be CrowdTone generates a professional-quality tone that “appropriate”and25%tobe“veryappropriate”emails,also matchesthewritingexpectationsofrecipient totaling75%. Fromthis,welearnedthatthetotalpercentage To understand whether CrowdTone produces high-quality of CrowdTone emails judged to be “appropriate” and “very emails,thatis,thosethatmeetthewritingexpectationsofre- appropriate”isthesameforminimalandmaximuminputs. cipientsorare“professional”inquality,weaskedthesurvey As Figure 11 illustrates, the professional writers found 55% respondents whether the newer emails were of expert qual- ofCrowdToneemailswithminimalcontexttobe“appropri- ity or would match the quality of an email they had written. ate”and31%tobe“veryappropriate”,totaling86%. Incon- They responded on a Likert scale that ranged from “totally trast,theyfound45%ofemailswithmaximumcontextinput disagree”to“totallyagree”. tobe“appropriate”and28%tobe“veryappropriate”, total- As Figure 13 illustrates, the recipients responded that they ing72%. “totally agreed” regarding 21% of the emails and “agreed” After conducting a blind ranking of emails produced using regardingthehighqualityof54%oftheCrowdToneemails, minimal and maximum context input, we found that partici- totaling 75%. The professionals indicated that they “totally patingrecipientsandprofessionalsrankedoutputfrommaxi- agreed” regarding high quality for 38% of the emails and mumcontextemailshigher—54%and61%ofthetime,re- “agreed” for 45%, totaling 83%. Together, the profession- spectively. Thisisnotsubstantiallydifferentfromtheemails als and recipients found that the quality of the newer emails withminimalcontextinputs. matchedthatofprofessionalwork. 7 The crowd is essential to CrowdTone. To help us get bet- Recipient Professional ter results and to optimize the crowd workers’ experience, 120% we incorporated tone scaffolding and consensus workflow into CrowdTone. After recruiting crowd workers that satis- 100% fiedminimalrequirements—a95%approvalratingandgeo- graphicallocationintheUnitedStates—wegothigh-quality 80% 45% resultsthatpleasedtheemailrecipientsandprofessionalwrit- ers. Tohelpusunderstandthecrowd’sexperience,weasked 60% theworkerstogiveusfeedbackaspartoftheHIT. Through a preliminary qualitative study, we found that the 40% 38% crowddeemedtheCrowdToneprocesstobeextremelyeffec- 54% tive. Ithelpedthemidentifyandimproveemailtonethrough 20% 17% multiple iterations. Most crowd workers also acknowledged 21% 0% 4% 13% 8% their enjoyment of the HITs and asked to do more, as the quotesbelowdemonstrate. Totally disagree Disagree Neutral Agree Totally agree Figure13. Agreementtohighqualitymatchingthatofprofessionals— “The HIT was easy enough. I understood what was being asratedbytherecipientsandprofessionals askedofme,andIwasabletodothetaskquickly.” “HITwasveryeasy. Idon’tseeanywayyoucouldhelpme makeatonerelateddecisionfasterwiththisspecificHIT.” No Yes “The HIT was far more easy and straightforward than I ex- 120% pected. Pleasepostmore. ” 100% CONCLUSIONSANDFUTUREWORK This paper presents CrowdTone, a crowd-powered system 80% thatreceivesemailcontentandcontext,andoutputsthesame 59% emailwithimprovedtone. CrowdToneutilizescrowdwork- 60% ersfromAmazonMechanicalTurk,whogothroughspecifi- 92% callydesignedtonescaffoldingandconsensusphases—from 40% self-identifyingtonerelatedrequirementstofixingit—with- outuserinstructions. Toevaluatethequalityoftheenhanced 20% 41% emailsproducedbyCrowdTone,wecollected29emailsfrom 22participants,andhadthemreviewedbysaidparticipating 8% 0% recipientsandthreeprofessionalwritersfromUpwork.Over- Original Email CrowdTone Email all,CrowdTone’scorecontributionsare: Figure14.Chancesofrespondingback—asratedbytherecipients • Robustnesswithregardtothepresenceorabsenceofcon- textprovided. • Substantial improvements in the quality of tone over the CrowdTone increases the chances of hearing back from originalemail. Basedontheparticipatingrecipients’eval- therecipients uation,60%ormoreoftheemailsprocessedexhibitedsub- One aspect that often determines the “success” of an email stantialimprovement. iswhetheritelicitsaresponse. Althoughrecipientscanhave • Generation of professional-quality tone that in the evalu- multiple reasons for replying to an email, we focused here ation matched the writing expectations of the recipients. on tone as the primary motivator. Specifically, we investi- Whensurveyed,75%oftheseparticipants“agreed”or“to- gatedwhetherrecipientsrespondedtotheoriginalemailsand tally agreed” when asked whether the CrowdTone emails whether they would respond to the newer CrowdTone ver- wereofhighquality. sions. AsFigure14shows,wefoundthatrecipientsdidnotrespond • Improved chances of getting a response back from email to 41% of the original emails. With the emails generated recipients. Theparticipatingrecipientsindicatedthatthey byCrowdTone,therecipientsjudgedthattheywouldnotre- wouldrespondtomorethan90%oftheemails,asubstan- spond to just 8% of the emails. In other words they would tialimprovementfromtheoriginal59%. respondto92%oftheCrowdToneemails. • Tone-scaffolding process was reported as easy and effec- tivebycrowdworkers. The crowd behind the CrowdTone found the tone- ThroughCrowdTone,weextendexistingcrowd-poweredsys- scaffolding process easy and effective in accomplishing tems research for efficient writing solutions, and present a theirtask novel process that self-identifies and improves email tone. 8 Thatsaid,thesystemalsoshowedseverallimitationsthatare WordProcessorwithaCrowdInside.InProceedingsof seedsforfuturework: the23NdAnnualACMSymposiumonUserInterface SoftwareandTechnology(UIST’10).ACM,NewYork, • Privacy. Participantsraisedconcernsabouttheprivacyim- NY,USA,313–322.DOI: plications of sharing private communication with crowd- http://dx.doi.org/10.1145/1866029.1866078 workers without all the parties being aware of it. Future work might explore ways of addressing this through ob- 5. KristinByron.2008.Carryingtooheavyaload? The fuscationandexplicitnormsetting,andothermechanisms communicationandmiscommunicationofemotionby exploredinsimilarpriorwork[19] email.AcademyofManagementReview33,2(2008), 309. • Delay. Having to wait for the process of tone setting to gothroughintroducesdelaysthatmightnotbesuitablefor 6. CarrieJCai,ShamsiTIqbal,andJaimeTeevan.2016. someusers,andsomescenarios. Optimizingforreal-time ChainReactions: TheImpactofOrderonMicrotask responsesthroughcrowd-retainersasitwasdoneinsimilar Chains.InProceedingsofCHI. research[3]issomethingworthexploring. 7. StevenDow,JulieFortuna,DanSchwartz,Beth • Deepintegration.Atthemoment,CrowdToneisnotdeeply Altringer,DanielSchwartz,andScottKlemmer.2011. PrototypingDynamics: SharingMultipleDesigns embeddedintheregularworkflowofsendingemails. Cre- ImprovesExploration,GroupRapport,andResults.In ating a plugin could be one way of addressing this, how- ProceedingsoftheSIGCHIConferenceonHuman ever,pluginsarenotuniversallyused,forexample,mobile FactorsinComputingSystems(CHI’11).ACM,New email clients rarely allow for plugins. However, deeper York,NY,USA,2807–2816.DOI: integration can provide access to conversation history — makingCrowdTonemoreintelligentinitsabilitytodeter- http://dx.doi.org/10.1145/1978942.1979359 minedeepercontextandmeaning. Moreworkisneededto 8. StevenDow,AnandKulkarni,ScottKlemmer,and understand the best mechanisms for integration into peo- Bjo¨rnHartmann.2012.ShepherdingtheCrowdYields ple’sworkstyles. BetterWork.InProceedingsoftheACM2012 ConferenceonComputerSupportedCooperativeWork • Restricted applications. Though CrowdTone can be uti- (CSCW’12).ACM,NewYork,NY,USA,1013–1022. lized to improve the tone of any type of written commu- DOI:http://dx.doi.org/10.1145/2145204.2145355 nication, the current paper focuses on its application for professionalemails. Infuture,weplantoapplythecoreof 9. StevenP.Dow,AlanaGlassco,JonathanKass,Melissa our existing approach and expand it to different use cases Schwarz,DanielL.Schwartz,andScottR.Klemmer. beyondemails: toothertypesofwrittencommunications, 2010.ParallelPrototypingLeadstoBetterDesign domainsormedia. Results,MoreDivergence,andIncreasedSelf-efficacy. ACMTrans.Comput.-Hum.Interact.17,4,Article18 Weseealotofpotentialforintroducingthird-partyfeedback (Dec.2010),24pages.DOI: andfixingintoourprofessionalcommunication. Largecom- http://dx.doi.org/10.1145/1879831.1879836 panies often have marketing groups in charge of corporate communication,butweenvisionthislevelofattentiontode- 10. StevenP.Dow,KateHeddleston,andScottR.Klemmer. tail could be affordable for individuals. More generally we 2009.TheEfficacyofPrototypingUnderTime see this work as yet another example of a potential for hy- Constraints.InProceedingsoftheSeventhACM bridintelligencewherecrowdsandAIcancometogetherto ConferenceonCreativityandCognition(C&C’09). optimizeprocessesandleveragethebestofbothworlds. ACM,NewYork,NY,USA,165–174.DOI: http://dx.doi.org/10.1145/1640233.1640260 REFERENCES 11. NicholasEpleyandJustinKruger.2005.Whenwhat 1. IBMAlchemy.2015.AlchemySentimentAnalysisAPI. youtypeisntwhattheyread: Theperseveranceof (2015).http: stereotypesandexpectanciesovere-mail.Journalof //www.alchemyapi.com/api/sentiment-analysis. ExperimentalSocialPsychology41,4(2005),414–422. 2. MicrosoftAzure.2015.LexiconBasedSentiment 12. Google.2015.SentimentAnalysisModel.(2015). Analysis.(2015).https://azure.microsoft.com. https://cloud.google.com/prediction/docs/ 3. MichaelS.Bernstein,JoelBrandt,RobertC.Miller,and sentiment_analysis. DavidR.Karger.2011.CrowdsinTwoSeconds: 13. NickGreer,JaimeTeevan,andShamsiTIqbal.2016. EnablingRealtimeCrowd-poweredInterfaces.In Anintroductiontotechnologicalsupportforwriting. Proceedingsofthe24thAnnualACMSymposiumon TechnicalReport.TechnicalReport.MicrosoftResearch UserInterfaceSoftwareandTechnology(UIST’11). TechReportMSR-TR-2016-001. ACM,NewYork,NY,USA,33–42.DOI: http://dx.doi.org/10.1145/2047196.2047201 14. CrystalProjectInc.2015.Crystal.(2015). https://www.crystalknows.com/. 4. MichaelS.Bernstein,GregLittle,RobertC.Miller, Bjo¨rnHartmann,MarkS.Ackerman,DavidR.Karger, 15. EricJaffe.2014.WhyIt’sSoHardToDetectEmotionIn DavidCrowell,andKatrinaPanovich.2010.Soylent: A EmailsAndTexts.(2014). 9 http://www.fastcodesign.com/3036748/evidence/ 24. KurtLuther,Jari-LeeTolentino,WeiWu,AmyPavel, why-its-so-hard-to-detect-emotion-in-emails. BrianP.Bailey,ManeeshAgrawala,Bjo¨rnHartmann, andStevenP.Dow.2015.Structuring,Aggregating,and 16. JoyKim,JustinCheng,andMichaelS.Bernstein.2014. EvaluatingCrowdsourcedDesignCritique.In Ensemble: ExploringComplementaryStrengthsof Proceedingsofthe18thACMConferenceonComputer LeadersandCrowdsinCreativeCollaboration.In SupportedCooperativeWork&SocialComputing Proceedingsofthe17thACMConferenceonComputer (CSCW’15).ACM,NewYork,NY,USA,473–485. SupportedCooperativeWork&SocialComputing DOI:http://dx.doi.org/10.1145/2675133.2675283 (CSCW’14).ACM,NewYork,NY,USA,745–755. DOI:http://dx.doi.org/10.1145/2531602.2531638 25. Lymbix.2015.ToneCheck.(2015). http://tonecheck.com/. 17. JoyKimandAndresMonroy-Hernandez.2016.Storia: 26. MailChimp.2015.Voice&Tone.(2015). SummarizingSocialMediaContentBasedonNarrative http://voiceandtone.com/. TheoryUsingCrowdsourcing.InProceedingsofthe 19thACMConferenceonComputer-Supported 27. MichaelNebeling,AlexandraTo,AnhongGuo, CooperativeWork&SocialComputing(CSCW’16). AdrianAdeFreitas,JaimeTeevan,StevenPDow,and ACM,NewYork,NY,USA,1018–1027.DOI: JeffreyPBigham.2016.WearWrite: Crowd-Assisted http://dx.doi.org/10.1145/2818048.2820072 WritingfromSmartwatches.InProceedingsofCHI. 18. AniketKittur,BorisSmus,SusheelKhamkar,and 28. LisaPearlandMarkSteyvers.2013.Cmon YouShould RobertE.Kraut.2011.CrowdForge: Crowdsourcing ReadThis: AutomaticIdentificationofTonefrom ComplexWork.InProceedingsofthe24thAnnualACM LanguageText.InternationalJournalofComputational SymposiumonUserInterfaceSoftwareandTechnology Linguistics(IJCL)4,1(2013),12–30. (UIST’11).ACM,NewYork,NY,USA,43–52.DOI: 29. CraigSmith.2016.BytheNumbers: 60Incredible http://dx.doi.org/10.1145/2047196.2047202 EmailStatistics.(2016).http://expandedramblings. 19. NicolasKokkalis,ThomasKo¨hn,CarlPfeiffer,Dima com/index.php/email-statistics/. Chornyi,MichaelS.Bernstein,andScottR.Klemmer. 30. YlaR.TausczikandJamesW.Pennebaker.2010.The 2013.EmailValet: ManagingEmailOverloadThrough PsychologicalMeaningofWords: LIWCand Private,AccountableCrowdsourcing.InProceedingsof ComputerizedTextAnalysisMethods.Journalof the2013ConferenceonComputerSupported LanguageandSocialPsychology29,1(2010),24–54. CooperativeWork(CSCW’13).ACM,NewYork,NY, DOI:http://dx.doi.org/10.1177/0261927X09351676 USA,1291–1300.DOI: http://dx.doi.org/10.1145/2441776.2441922 31. JaimeTeevan,ShamsiIqbal,andCurtisvonVeh.2016. 20. JustinKruger,NicholasEpley,JasonParker,and SupportingCollaborativeWritingwithMicrotasks.In Zhi-WenNg.2005.Egocentrismovere-mail: canwe ProceedingsofCHI. communicateaswellaswethink? Journalof 32. IBMWatson.2015.ToneAnalyzer.(2015). personalityandsocialpsychology89,6(2005),925. http://www.ibm.com/smarterplanet/us/en/ 21. AnandKulkarni,MatthewCan,andBjo¨rnHartmann. ibmwatson/developercloud/tone-analyzer.html. 2012.CollaborativelyCrowdsourcingWorkflowswith 33. LeaWinerman.2006.E-mailsandegos: Aninabilityto Turkomatic.InProceedingsoftheACM2012 stepoutsideofone’sownheadmaybebehinde-mail ConferenceonComputerSupportedCooperativeWork miscommunication,accordingtorecentresearch. (CSCW’12).ACM,NewYork,NY,USA,1003–1012. AmericanPsychologicalAssociation37,2(2006),16. DOI:http://dx.doi.org/10.1145/2145204.2145354 34. IncWordzen.2015.Wordzen.(2015). 22. WalterS.Lasecki,KyleI.Murray,SamuelWhite, http://www.wordzen.com/. RobertC.Miller,andJeffreyP.Bigham.2011. 35. BusinessCoachingWorldwide.2007.DidYouKnow: Real-timeCrowdControlofExistingInterfaces.In Proceedingsofthe24thAnnualACMSymposiumon Thatemailrecipientscorrectlyinterprettheintended UserInterfaceSoftwareandTechnology(UIST’11). toneofanemailmessageonly50%ofthetime? (2007). ACM,NewYork,NY,USA,23–32.DOI: http://www.wabccoaches.com/bcw/2007_v3_i1/ didyouknow.html. http://dx.doi.org/10.1145/2047196.2047200 36. WriteExpress.2015.Howtousetoneinyourwriting. 23. WalterS.Lasecki,RachelWesley,JeffreyNichols, (2015).http://www.writeexpress.com/tone.html. AnandKulkarni,JamesF.Allen,andJeffreyP.Bigham. 2013.Chorus: ACrowd-poweredConversational Assistant.InProceedingsofthe26thAnnualACM SymposiumonUserInterfaceSoftwareandTechnology (UIST’13).ACM,NewYork,NY,USA,151–162. DOI:http://dx.doi.org/10.1145/2501988.2502057 10

See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users