Problem

Individual characteristics including user preferences and perceptions are important to designing effective autonomous vehicles (AV).

Adapting AV behavior to various situations will require understanding the many factors that influence a driver to retain or relinquish control of an AV. What one driver deems a critical situation for the AV to relinquish control, another driver may be perfectly comfortable with the vehicle remaining in control. Furthermore, a general public perception study found 44% of people would feel uncomfortable using an AV [1].

However, to date, the focus of AV development has been mainly technological. A user-centered orientation in AV design is critical in increasing driver satisfaction and experience.

 

Research goals

  • To illustrate the importance of the user (i.e., driver) in a technologically-driven context of AV development

  • To map driver preferences around AV decision-making for a variety of scenarios

  • To build an understanding of how much impact drivers’ preference should have on AV design

 

Research questions

  • RQ1: How do individual differences affect perceptions of AVs?

  • RQ2: How do contextual factors contribute to AV decision-making preferences?

  • RQ3: How can aligning individual preferences with appropriate contexts optimize perceptions of AV performance?

 

Method

We designed a survey-based study and recruited N = 300 participants (from Amazon Mechanical Turk) to evaluate their perceptions and preferences for AVs in various scenarios.

Why this method—i.e., survey?

  • To garner wide range of driver perceptions

  • To evaluate various real-life AV scenarios with nuances in situations (i.e., experiment various treatments)

  • To quantify perceptions for each scenario with a range of AV expertise

  • To show with statistical power that driver preferences matter

Scenario generation

Natural inter-vehicle and pedestrian interactions in downtown San Francisco, California were captured using a forward-facing GoPro camera. From the raw video, we selected 18 short clips that portrayed various decision-making points in driving.

The clips were trimmed (averaged 9 seconds long) such that only the relevant part of the vehicle decision-making was visible to participants; this also emulated a situation where the human driver does not pay attention to the road until a critical event.

 

Above: Scenario where the vehicle is running a yellow light (i.e., “runYellow” scenario). Below: 18 scenarios selected including vehicle running a yellow light (image above), making a double lane change, and maneuvering around a truck parked on the side of the road; of these 12 were distinct, and 6 were similar to others.

 
 

Survey design

We designed the survey with largely 4 parts:

  1. Introduction to the term “semi-autonomous vehicle” for consistency of understanding and to ease understanding of SAE Levels 3 and 4 vehicles, which have self-driving capabilities, but at times may handover the controls to the human driver

  2. General baseline perceptions of AVs and preferences of the AV for a “critical event”

  3. Randomly-selected and evenly-presented a subset of 6 of 18 scenarios (to produce an aggregate of 100 responses per clip without causing survey fatigue), with evaluations after each clip consisting of the following:

    (a) Descriptions of what the AV did in the clip
    (b) Preference ratings for how participants wanted AVs to act and inform them
    (c) Perception ratings of the AV with respect to 12 characteristics

  4. Demographics questions

 

Combining different options for how an AV can act and inform led to 9 possible driver preferences. Implausible options are labeled “N/A”.
*From preliminary tests, we found that participants considered informing before and during the decision to be similar, and therefore in this study we only considered informing before the event.

Driver preferences

We considered preferences with regards to 2 dimensions—Informing and Acting—as these are the 2 rudimentary methods in which AVs can interact with drivers; “Automation Functions” also described that an airplane communicates with a pilot through both acting and informing.

Considering these 2 dimensions resulted in 5 options that we expanded upon as answer choices for preferences.

 

Core results

Participants perceived events differently + Some events varied more than others

📊 Analysis: Distribution (Descriptive statistics)

Participants’ interpretation of the events differed at varying degrees; this variance in interpretations led to different perceptions of and preferences for the AVs. We quantified this variance by coding participant descriptions by level of alignment with the pre-study criteria created based on SA Level 2 (i.e., manipulation check).
*Validity of participant responses was checked by examining text responses for relevance.

List of 18 scenarios ordered by the percentage of responses that met criteria (“Met criteria”) from total number of participants’ text-based scenario descriptions (“Total”). The “Pre-study criteria” were articulated prior to data collection, and this determined participants’ correct comprehension of scenario.

Percentages of participant interpretations for the 4 high-variance scenarios after manipulation check. For these scenarios in particular, low percentages of participant descriptions matched the Pre-study criteria for correct comprehension of scenario (green column).

 

Correspondence analysis of participant preference of the AV for each scenario

Action emerges as the first dimension for explaining variance in preferences of AV decisions

📊 Analysis: Correspondence analysis (CA)

To grasp how driver preferences of AV action and inform are affected by each scenario, we conducted correspondence analysis. The CA showed that AV action is the more important dimension (Dim 1) that explains 61% of the variance in preferences.

Clusters of scenarios represent perceived similarities in how AVs should behave. For example, truckComing, turnRight, carComing, and 4wayStop scenarios were regarded similar to each other. Conversely, pigeonCrossing is farthest from aroundCar, which suggested that these scenarios are considered least like each other. Semantic interpretations align with the plot: Act-inform and inform-act are close together, while inform-consent is least similar to act-inform.

 

Change in perception of AV impression with respect to alignment of AV decision and driver preference by scenario.

Aligning individual preferences with appropriate contexts optimizes perception of performance

📊 Analysis: Mean, Logistic regression

Overall, 40% of the vehicle’s decisions (all action-noInform) aligned with participants’ preference for the vehicle. When the AV acted in accordance with preferences, participants had a more positive perception of the vehicle.

In particular, changes in perception of the AV being competent and aggressive were significantly indicative of preference alignment (p < 0.05, GLM) as well as changes in the AV’s communicative and knowledgeable characteristics (p < 0.05, GLM), and to a lesser extent trustworthiness and speed of response (p < 0.1, GLM).

 

Impact

✨ Highlighted the importance of users in AV design

We demonstrated how user characteristics (e.g., AV experience, expectation) can impact their perceptions and evaluations of AVs. In doing so, we have started an important conversation around user-centered AV design. Even if AVs act “flawlessly” (in terms of their technological capabilities), our results show that their perceived performance is highly dependent upon user perceptions and preferences. In order for AVs to gain trust, they must adhere to users’ (i.e., drivers’) expectations of them.

✨ Developed ontology of real-world AV decision-making scenarios

I have shown the various real-world scenarios that drivers may have opinions and expectations around when in an AV in terms of their decisions to take “action” (i.e., autonomy) and to “inform” (e.g., alert).

✨ Introduced a new method of garnering user insights for AV design

Design insights for developing AVs have been captured primarily through Wizard-of-Oz or simulator studies. Even with greater access to AVs today, it is difficult to attain insights at scale. Through our study, we offer another method of garnering user insights. Using surveys, we can glean nuanced insights quickly across multiple scenarios from numerous participants, which we could not achieve with the other customary methods of AV research.

✨ Identified design principles to consider in designing a positive AV experience

  • Ambiguous interpretation can lead to bad outcomes for the AV

  • Prioritized alerts are necessary for the driver

  • Individual differences will influence mental models of appropriate AV decision-making

  • Drivers may desire handover even in situations where the AV would make appropriate decision

  • Undesirable AV decisions can lead to more harmful consequences even if drivers have had positive AV experiences

✨ Visualized driver preference among various factors in AV decision-making

Building upon the Situation Awareness models of Endsley [2] and Matthews et al. [3], our influence diagram visualizes how factors in the AV’s decision making process are interconnected with a distinct emphasis on the user. Here, a designer can easily see where the user is positioned with regards to the aspects of the AV decision-making process that are affected.

 

Influence diagram that positions “preferences” amongst other factors that influence ultimate AV action. There is uncertainty embedded in every node. (*Other indicates the other entity, as the diagram applies to both vehicle and driver perspectives.)

 
 

Reflection

🤔 Select scenarios with more consistent parameters

While our selection of 18 clips was representative of a typical drive, we could have chosen scenarios with a priori parameters that would help AV companies consider all parts of the driving experience from beginning to end including parking. We could choose to consider the same number of types of actions with similar factors involved such that more rigorous scenario comparisons could be made (e.g., changing lanes vs stopping due to sedan, truck, and person).

🤔 Present preference questions with dimensions separated

We presented 5 choices for participants to answer their preferences for AV action and inform. Asking participants to rate their preference in terms of action (e.g., take action on its own, take action after receiving consent, do not take action) and inform (e.g., inform before, inform during, inform after, do not inform) would have allowed us to capture more nuance as well as enable us to consider more than these 2 dimensions.

🤔 Consider other individual factors

While we collected demographics as well as experience levels with AVs, other individual characteristics could affect user preferences not considered in the current analysis. Collecting additional data such as driving styles (for oneself vs preference when driven) could help us to better understand the various elements that contribute to driver preference.

 
 

Results & Analyses +

In case you wanted to see more :)

Perceptions of AVs differ with level of experience with AV systems

📊 Analysis: Mean, Standard deviation, Analysis of variance (ANOVA)

There was great variation in participants’ general baseline (before seeing any clips) perceptions of AVs; SD ranged up to 1.16 on a 5-point Likert-type scale). Overall, perceptions biased towards neutral to moderately positive, but we found experience with AVs to be a significant predictor of certain AV attributes: The greater the level of AV experience, the more positive their baseline perceptions of semi-AVs. In particular, perceptions of friendliness increased significantly with greater experience with AV systems (pFDR < 0.05, ANOVA), and characteristics of assertiveness, pleasantness, and trustworthiness were marginally significant (pFDR < 0.1, ANOVA).

General baseline semi-AV perceptions with means of Likert-scale responses, colored by level of experience with vehicle autonomous systems, across all participants before seeing any scenarios (left) and changes in Likert-scale responses from baseline after watching runYellow scenario (right).

 

Level of AV experience does not affect absolute AV perceptions, but affects relative perceptions

📊 Analysis: Analysis of variance (ANOVA)

Analyzing relative perceptions for each scenario (i.e., absolute perception for the scenario subtracted by baseline perception for each participant), we found that level of AV experience influenced relative perceptions of the AV in different scenarios. For instance, in the runYellow scenario, more experience tended to reduce perceived assertiveness, competency, and responsibility (pFDR < 0.05, ANOVA), and to a lesser extent trustworthiness, aggressiveness, and response speed (pFDR < 0.1, ANOVA).

 

Bin plot of each participants’ preferences for AV inform and action across all scenarios (e.g., a participant with preference for being informed in 4 of 6 scenarios would have a “Prefer inform” percentage of 67%). The highest frequencies are represented by dark blue hexagons.

Preferences for action and inform differ between individuals

📊 Analysis: Exploratory (Hexagonal bin plot)

3 groups emerged from plotting a bin plot with each participant’s preference for AVs across the 6 scenarios they watched.

  • Those in the top-right corner desired the AV to act with complete autonomy 100% of the time and also desired to always be informed of the AV’s decision

  • Those in the top-left corner preferred automation to take action approximately 80% of the time with minimal informing (about 20%)

  • Those in the bottom-right corner preferred the AV to act autonomously approximately 30% of the time, while always wanting to be informed

 

Proportion of participant preference for AV inform and action per scenario as a scatter plot, including participants’ preference for a “critical” situation.

Scenarios affect preference for AV action and inform

📊 Analysis: Exploratory (Scatter plot), Linear regression

We plotted the proportions of participants wanting AV to inform (y-axis) and act autonomously (x-axis), thereby contextualizing aggregate preference for the 18 scenarios and a “critical” event. The 3 scenarios closest to the “critical” situation—pigeonCrossing, passPedestrian, and changeLanes—have the highest percentage of participants preferring that the AV inform them. Nuanced variance in participant responses between similar scenarios were also observable (e.g., aroundCar vs aroundTruck, changeLanes vs doubleLaneChange).

Conducting linear regression shows significant prediction of action from inform (F(1,17)=37.21, p<0.001) and, depending on the scenario, the degree of inform varies inversely with the degree of autonomy desired by the participants.

 

Scenarios affect relative perception of AV characteristics

📊 Analysis: Principal component analysis (PCA), Cronbach’s alpha

We examined how participants’ perceptions of each scenario changed relative to their baseline perception of AVs (i.e., relative perceptions) using PCA and found that AV characteristics could be separated out into 2 groups: Passivity (α = 0.77) and Impression (α = 0.96).

Plotting means of relative changes in perceptions, I found scenarios could be separated into 3 types: (1) +Impression & +Passivity (6 scenarios), (2) +Impression & -Passivity (11 scenarios), and (3) -Impression & -Passivity (i.e., passPedestrian).

Plotting PCA results showed that AV characteristics could be separated out into 2 groups: Passivity (passive and timid; α = 0.77) and Impression (friendly, sensible, responsible, trustworthy, knowledgeable, competent, pleasant, intelligent; α = 0.96). Arguably another group of Responsiveness can be considered, but we decided to focus my analysis on Passivity and Impression.

Overall perceived impression and passivity of AV varied significantly across scenario. Points to the right of center indicate more positive impression of vehicle (blue) or perception of vehicle as more passive (yellow). Points to the left of center indicate more negative impression of car or perception of car as more aggressive.

 

Select references

[1] Tennant, Chris, et al. 2016. “Autonomous vehicles: Negotiating a place on the road.” Technical report, London School of Economics, 2016.

[2] Endsley, Mica R. 1988. Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society annual meeting, Vol. 32. SAGE Publications Sage CA: Los Angeles, CA, SAGE Publications, 97—101.

[3] Matthews, Michael L, et al. 2001. Model for situation awareness and driving: Application to analysis and research for intelligent transportation systems. Transportation research record 1779, 1 (2001), 26—32.