Communications of the ACM
Home/Magazine Archive/March 2021 (Vol. 64, No. 3)/Informing California Privacy Regulations with Evidence.../Full Text
Privacy
Informing California Privacy Regulations with Evidence from Research
Exercising privacy choices is akin to a scavenger hunt: information about available choices is hard to find and mechanisms can be difficult to use. My research group has been examining ways to improve privacy user experiences (UX).12 We started exploring website privacy "nutrition labels"9,10 a decade before Apple introduced them in their app store in December 2020, and recently we proposed a privacy and security label for IoT devices.5 When the State of California passed the California Consumer Privacy Act (CCPA) mandating a "Do Not Sell My Personal Information" website opt-out link and optional icon, we developed and evaluated icon designs and submitted recommendations in response to the Office of the Attorney General (OAG) call for public comments.a After several twists and turns, in December 2020 the OAG issued proposed regulations with our recommended icon.
Icon Development and Evaluation
In fall 2019, our team of researchersb began brainstorming possible icon designs. We developed 11 icons that could represent one of three concepts: choice, opting out, and do not sell personal information. We focused on representing these concepts rather than on representing privacy itself, as privacy is difficult to visualize and popular privacy visualizations (locks, shields, keys, masks, eyes) are already used in Web security and privacy tools.
We conducted an initial evaluation of our 11 icons as well as the green "privacy rights" icon promoted by the Digital Advertising Alliance industry group for use as a CCPA icon. We recruited participants from Amazon's Mechanical Turk (MTurk) and showed one randomly selected icon to each participant. Half the participants saw the icon with the text "Do Not Sell My Personal Information" and half saw the icon alone. We asked participants to tell us what they thought the icon communicated and what they thought would happen if they clicked on it. Then we showed them all 12 icons, shown in Figure 1a, and asked them to select the icons that best conveyed the presence of privacy choices and do-not-sell choices.
Figure 1a. Icons evaluated in first user study.
Based on the results of the initial evaluation, we refined five of the icons (see Figure 1b) and conducted another MTurk evaluation. The slash-dollar icon was misinterpreted as relating to money when it appeared without link text, but it was most preferred by participants as an icon for representing do-not-sell. The DAA icon was often misinterpreted as a play button or an information button. The stylized-toggle led to the fewest misconceptions.
Figure 1b. Refined icons evaluated in second user study. The three on the left were also evaluated in the combined study with link texts.
Our initial evaluations demonstrated the importance of placing link text next to the icons, and our prior research showed the specific wording of this text can have a large impact.6 We brainstormed possible link texts and evaluated 16 of them (see Figure 1c), including "Do Not Sell My Personal Information" and "Do Not Sell My Info," which were in the CCPA legislation. Our evaluation identified three new promising link texts: "Privacy Choices," "Privacy Options," and "Personal Info Choices."
Figure 1c. Link texts evaluated in user study. The five shown in bold were also evaluated in the combined study with icons.
Our next step was to evaluate three icons and five link texts together in the context of a fictitious shoe retailer website. We tested 23 icon-link text combinations, including link texts without an icon and the icons without a link text. We recruited 1,468 MTurk participants and randomly assigned them to view the shoe website with one icon-link text combination shown in the footer (see Figure 2).
Figure 2. In the combination icon and link text study participants were shown this screen shot of a fictitious footwear website with an icon and link text highlighted.
We found the link texts had more of an impact on participant expectations than the icons, and the icons continued to convey misconceptions. The combination of stylized-toggle and the "Privacy Options" link text best conveyed choices about personal information. The CCPA link texts best conveyed do-not-sell choices. In February 2020, we sent a detailed report to the OAG and recommended adoption of the stylized-toggle icon with either the "Privacy Options" link text or the CCPA link texts.2
More Research Needed
Shortly after receiving our report, the OAG released the first set of modifications to the CCPA regulations with an icon that was similar to our stylized-toggle but differed in significant ways. While our icon was blue and contained both a checkmark and an X arranged to convey choices without suggesting a toggle in a particular state, the OAG's icon was red, contained only an X, and strongly resembled an actual toggle button. Comments on Twitter raised concerns that the OAG's icon might be misinterpreted as representing the state of a user's opt-out selection.
We quickly ran another MTurk study to compare our stylized toggle icon with the OAG's toggle icon and a variant of the OAG's toggle icon with a larger X—each tested in both red and blue (see Figure 1d). We found our stylized toggle better conveyed do-not-sell choices than the OAG's icon and led to fewer misconceptions. The larger X and the color had minimal impact. After we submitted a report on these results to the OAG,3 they released their second set of modifications to the CCPA regulations, removing their recommendation for an icon altogether.
Figure 1d. Toggle icon variants evaluated in user study. All icons were tested in both red and blue.
Later the OAG asked us if we would evaluate a set of four new icons (see Figure 1e) with 1,000 California residents. Besides evaluating each icon's ability to communicate the presence of do-not-sell choices, they asked us to test the ability of each icon to stand out on websites and motivate users to click. This necessitated some changes to our study protocol.
Figure 1e. OAG icons evaluated in final user study.
To ensure participants viewed the area of the fictitious shoe store website where the CCPA link appeared, we showed the website with the CCPA link text and one of the four icons or no icon and asked participants to find a link where they could get information about shipping shoes overnight. We then hid the shoe store website image and instructed participants to imagine they were concerned about an online store selling their personal information. We then asked, "Do you remember seeing any feature in the screenshot that you could use to prevent this from happening?" Next, we showed the screenshot again, calling attention to the icon and link text. We instructed participants to imagine this was the first time they had noticed the icon and link text on a website, and we asked how likely they would be to click on them. We followed up with questions about what would happen if they clicked and then showed them all four icons and asked them how well each conveyed the presence of do-not-sell choices.
This story provides a case study of how academic researchers can refocus their research to answer policymakers' questions.
Our results showed the icons successfully increased users' attention to the link text but did not create a significantly higher motivation to click. Interestingly, we found participants who were not shown any icon were most likely to have correct expectations about what would happen if they clicked; all four icons introduced misconceptions. Furthermore, participants did not rate any of the icons well. We submitted our report to the OAG in May 2020 and recommended they evaluate other icons and conduct public education to increase awareness of do-not-sell choices.1
Informing Policymaking with Research
Over six months passed before the OAG released their fourth set of modified regulations in December 2020, this time recommending the optional use of our blue stylized-toggle icon.
This story provides a case study of how academic researchers can refocus their research to answer policymakers' questions. When our team realized the OAG had a need for a specific privacy icon, we quickly pivoted from studying website privacy choices generally, to designing and evaluating a privacy icon to meet this need. After a three-month sprint to meet the public comment deadline we turned our attention to writing a research paper on this project. However, the OAG's recommendation of an untested icon triggered more quick action from our team, and we conducted another study to demonstrate that small changes in the icon could make a big difference in how it would be interpreted. Just when we thought we were done, the OAG reached out to us again and we put other work on hold so that we could redesign our experimental protocol and perform another evaluation.
In a world where privacy is increasingly threatened by online trackers and ubiquitous sensors, how much can a little blue privacy icon accomplish, especially when its use is entirely optional?
The combined expertise of our research team, the availability of flexible research funding, and the ability to conduct studies quickly and inexpensively using crowd workers allowed us to provide timely research that informed public policymaking. While crowd working platforms that do not offer demographically representative samples have their limitations, they are useful inexpensive tools for carrying out studies like these where the focus is on comparing alternatives.10
In the end, this project has resulted in a forthcoming CHI 2021 paper,8 a case study I will use in my usable privacy and security class, and an icon that may soon appear on websites. Moving forward, I am hopeful that websites will adopt the stylized-toggle icon not only for CCPA compliance, but also to point users toward a "Privacy Options" page with all of their privacy choices and settings in one place.
In a world where privacy is increasingly threatened by online trackers and ubiquitous sensors, how much can a little blue privacy icon accomplish, especially when its use is entirely optional? While an icon alone will not protect privacy, it can make it easy for users to find information about their privacy choices. We have seen in our research that Internet users are not always aware they have privacy choices, and they struggle to figure out how to exercise them.7 A standardized icon is a good first step toward increasing the discoverability of privacy choices and raising awareness about them. Ultimately, the use of standardized protocols interfacing with usable "personal privacy assistants"4 will allow users to make flexible fine-grained privacy choices that adjust according to each user's preferences and context across all websites, apps, and devices.
References
1. Cranor, L.F. et al. CCPA Opt-out icon testing—phase 2. May 28, 2020; https://oag.ca.gov/sites/all/files/agweb/pdfs/privacy/dns-icon-study-report-052822020.pdf
2. Cranor, L.F. et al. Design and Evaluation of a Usable Icon and Tagline to Signal an Opt-Out of the Sale of Personal Information as Required by CCPA. February 4, 2020; http://cups.cs.cmu.edu/pubs/CCPA2020Feb04.pdf
3. Cranor, L.F. et al. User Testing of the Proposed CCPA Do-Not-Sell Icon. February 24, 2020. http://cups.cs.cmu.edu/pubs/CCPA2020Feb24.pdf
4. Das, A. et al. Personalized privacy assistants for the Internet of Things: Providing users with notice and choice. IEEE Pervasive Computing 17, 3 (Jul.–Sep. 2018), 35–46; DOI:10.1109/MPRV.2018.03367733
5. Emami-Naeini, P. et al. Ask the experts: What should be on an IoT privacy and security label? In Proceedings of the 2020 IEEE Symposium on Security and Privacy (San Francisco, CA, USA, 2020), pp. 447–464. DOI:https://doi.ieeecomputersociety.org/10.1109/SP40000.2020.00043
6. Giovanni Leon, P. What do online behavioral advertising privacy disclosures communicate to users? In Proceedings of the 2012 ACM workshop on Privacy in the electronic society (WPES '12). ACM, New York, NY, USA, 2012, 19–30; DOI:https://doi.org/10.1145/2381966.2381970
7. Habib, H. It's a scavenger hunt: Usability of Websites' opt-out and data deletion choices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, USA, 2020, 1–12; DOI:https://doi.org/10.1145/3313831.3376511
8. Habib, H. et al. Toggles, dollar signs, and triangles: How to (in)effectively convey privacy choices with icons and link texts. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '21) (May 2021, Yokohama, Japan). ACM, New York, NY; https://doi.org/10.1145/3411764.3445387
9. Kelley, P.G., Cranor, L.F., and Sadeh, N. Privacy as part of the app decision-making process. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2013, 3393–3402; DOI:https://doi.org/10.1145/2470654.2466466
10. Kelley, P.G. et al. Standardizing privacy notices: an online study of the nutrition label approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2010, 1573–1582. DOI:https://doi.org/10.1145/1753326.1753561
11. Redmiles, E.M., Kross, S., and Mazurek, M.L. How well do my results generalize? Comparing security and privacy survey results from mturk, web, and telephone samples. In Proceedings of the 2019 IEEE Symposium on Security and Privacy, 1326–1343; DOI:10.1109/SP.2019.00014.
12. Schaub, F. and Cranor, L.F. Usable and useful privacy interfaces. In An Introduction to Privacy for Technology Professionals, Travis D. Breaux, Ed., IAPP (2020), 176–238; https://iapp.org/media/pdf/certification/IAPP-Intro-to-Privacy-for-Tech-Prof-SAMPLE.pdf
Footnotes
a. All documents pertaining to the CCPA rule-making activities can be found at https://oag.ca.gov/privacy/ccpa/current
b. Members of our team included Alessandro Acquisti, Michelle Chou, Lorrie Cranor, Hana Habib, Norman Sadeh, and Yaxing Yao from Carnegie Mellon University; Florian Schaub and Yixin Zou from the University of Michigan School of Information; and Joel Reidenberg from Fordham University School of Law.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.
No entries found