Usability Evaluation of an Automated Access Control Policy Generation System
Study Overview
There are several key steps to this study.
Think-aloud Study [25 minutes]
First, you will be using a new semi-automated access control policy generation system, AGentV to generate access control policies from a organization's high-level requirement specification document, by thinking aloud. [15 minutes][Go to the task]
Then you will need to answer a few questions based on the generated access control policies and make necessary changes (e.g., add new policies, remove existing policies, etc.) to the policies when needed. [10 minutes][Go to the task]
During the think-aloud study, we expect you to,
- Perform the given task: Perform the given task using the provided input documents according to the given scenario within 25 minutes.
- Speak: As you use AGentV to perform the given task, describe what you’re doing, thinking, and why.
- Share openly: If you’re confused, hesitate, or find something surprising, say it out loud!Share openly: If you’re confused, hesitate, or find something surprising, say it out loud!
- Explain your intentions: E.g., “I clicked this because I thought it would let me translate a requirement into an access control policy”
Fill out the Satisfaction Survey [5 minutes]
After performing the think-aloud study, you will be provided with a survey (System Usability Scale) to rate AGentV from 1 (Strongly disagree) to 5 (Strongly agree) according to 10 statements. [Go to the task]
Interview [30-45 minutes]
Finally you will be participating in a semi-structured interview where you will get a chance to provide feedback on the system, AGentV as well as the automated access control policy generation.
Introduction to AGentV
Scenario
You are the system administrator for HealthStar General Hospital, a medium-sized urban hospital with around 300 employees. The hospital recently transitioned to a new digital health records system.
You've been provided with the,
- Hospital's organization hierarchies that defines roles/subjects, resources, and actions.
- Hospital's high-level requirement specification document that explains who can access what information under what conditions in markdown format.
Your goal is to translate the access control requirements in the provided high-level requirement specification document into formal access control policies using AGentV.
You have 15 minutes to complete the task.
IMPORTANT: Assume that the hospital's authorization system, AGentV's policy database, and PDP (Policy Decision Point) are based on XACML and operate under the default deny principle. They use the deny-overrides policy, combining algorithm, and first-applicable rule combining algorithm to resolve conflicts.
Downlod the Documents
Before starting the policy generation, you are required to download two crucial documents:
- High-level requirement specification document of the hospital that mentions the hospital's access requirements (Requirements.md).
- Organization hierarchies of the hospital that are used to provide the context for policy generation (Hierarchies.yaml).
Download both documents below. Once downloaded, they will be automatically checked off. After both documents are downloaded, you'll be able to proceed to the next step.
Access AGentV and Perform the task
After you downloaded the necessary document, click the button below to open the AGentV application in a new tab. Complete the policy generation task and come back for Step 3.
IMPORTANT Please do NOT close the AGentV window as you have to perform few more generations in Step 3.
Answer the Questions and Perform the Sub-tasks
After performing the main policy generation task with the hospital's high-level requirement specifications, open the following questionnaire to answer a few questions on the generated policies and make necessary changes to the policies (e.g., Add/remove policies into the system).
Fill out the Satisfaction Survey
Once you completed the think-aloud study, please fill out the user satisfaction survey, where you can rate AGentV according to 10 criteria from Strongly Disagree (1) to Strongly Agree (5).