By Anya Zivanov
Around early December of every year, millions of Spotify users share their Spotify Wrapped: an annual data-driven feature and marketing campaign by Spotify offering users a personalised summary of their listening behavior from the past year. Beyond revealing our most-played songs and favourite artists, these summaries have become a way of expressing our identity. The widely shared summaries resemble social media stories and are often perceived as reflections of personal identity. But how accurately can listening data represent someone’s preferences and their identity?
Historically, preferences have been shaped largely by social settings, family dynamics, and cultural narratives. Identity was determined in large by our engagement with others, cultural practices, and periods of self-reflection. Today, Large Language Models (LLMs) are becoming increasingly influential in that role. Recommendation algorithms, predictive language models, and behavior-monitoring AI now present the capability to alter what we ought to feel and think. Li et. al. developed a framework for personalising LLMs using over 1.3 million constructed preference samples. They demonstrated that information about users’ behaviours helps AI systems generate responses better aligned with their inferred preferences.
An important distinction remains between predicting what people want and understanding why they want it. While algorithms are increasingly personalising the information and recommendations that users encounter, a broader question emerges: are these systems merely learning our preferences and adjusting to them, or are they also influencing how those preferences develop?
Algorithms learn what we want
Recommendation systems typically rely on two forms of information: explicit preferences, such as ratings or reviews, and implicit preferences, inferred from browsing histories, purchases, and other online behaviour. But behavioural data is not necessarily a reliable representation of actual preferences. For example, purchasing a product may not guarantee that someone will enjoy using it. Listening to a particular song may not reflect a preference for the music but rather familiarity with it. Algorithms can observe what individuals do, but identifying why they do it is considerably more difficult.
In their research on preference alignment, Li et. al. tested how accurately LLMs could predict users’ preferences from different types of information. Their best model reached around 91% accuracy when it was given descriptive personas (self-reported demographics and interests), but only around 58% when given behavioural personas (past comments and pair-wise comparative feedback). State-of-the-art universally-aligned LLMs (those aligned with general preferences) performed at about 50% (chance level) when predicting a user’s choice between two responses. When the researchers gave their own model ground-truth preference directions rather than having it infer them, the accuracy rose from 71% to 91%. This indicated that inferring preferences, rather than generating responses, is the main bottleneck in alignment accuracy. Their results illustrate the difficulty of AI determining user preferences based on user behaviour alone.
An important distinction is to be made between revealed preferences, those inferred from observed choices, and normative preferences, those reflecting an individual’s actual interests. When decisions are affected by limited information or inattention, the two preference models do not coincide. For instance, someone repeatedly watching short videos may continue engaging with content despite wishing to spend less time online. An algorithm might predict their next interaction and recommend another video based on their viewing history but not recognise that the user would actually prefer to spend that time reading. This raises the question of whether algorithms are actively shaping user preferences if recommendations themselves influence what users choose.
From precision to influence
Consider a music streaming platform like Spotify that repeatedly recommends artists based on a user’s listening history. If the user engages with the recommendations, their subsequent listening activity generates new data which informs future recommendations. This creates a feedback loop in which algorithms learn from behavior they have already influenced.
This phenomenon is also known as algorithmic confounding and demonstrates how recommendation systems trained on data influenced by previous recommendations produce homogeneous consumer behavior without corresponding movements in utility. Rather than merely identifying existing interests, recommendation systems may reinforce particular patterns of consumption and limit exposure to alternative choices.
Another phenomenon is that of preference pollution, whereby recommendations influence users’ subsequent ratings of products or experiences. Experiments have shown that an alteration of one star by the system prediction leads to a mean user rating shift of 0.35 stars on a five-star scale. This suggests the potential of algorithmic predictions to influence expressed preferences (although it does not necessarily demonstrate permanent changes in personal tastes).
The concept of hypernudging also arises in recommendation systems. Personalised digital environments can steer individuals’ decisions without explicitly restricting their available options. Similarly, AI systems are increasingly taking over the role of choice architecture by which algorithms determine how choices are presented to users while pursuing objectives they’ve been programmed to maximise. Consequently, the question is not simply whether algorithms can predict what users want but how their influence over available choices affects what users subsequently choose.
Can personalization and autonomy coexist?
The influence of recommendation systems does not necessarily mean that personal autonomy is lost. Personalisation can make decision-making easier by reducing the number of options users need to consider. Recommendation systems can help users navigate an otherwise overwhelming information environment when they have limited time or resources. Now the question is whether algorithms influencing our decisions undermines our ability to make decisions independently.
In the analysis of AI-powered recommendation systems, two dimensions of autonomy were identified: competence and authenticity, where competence is the ability to understand available options and authenticity is the ability to reflect on those options and identify with your own preferences. The dimensions introduce an important distinction between algorithms that aid decision-making and algorithms that do the decision-making. For example, Spotify recommending unfamiliar artists may expand the user’s musical interests, but consistently prioritising the same content could limit the range of music the user encounters. In the latter, the user may become accustomed to the recommendations without considering other alternatives.
But greater reliance on algorithms may not necessarily imply greater dependence on them. Across three online experiments involving vacation choices, results showed that participants were significantly more likely to accept algorithmic recommendations when they were presented with multiple options rather than a single recommendation. Additionally, allowing participants to determine how many recommendations they received strengthened this relationship. The findings suggested that giving users greater control over the decision-making process increases their willingness to use algorithmic assistance.
Recommendation systems can make decisions more efficiently, but we should be wary of using efficiency alone to determine whether a system is benefitting its users. As AI becomes increasingly capable of predicting what we are likely to choose, preserving autonomy may require ensuring that it leaves room for what we have not yet considered wanting. Perhaps then the greatest risk is that we gradually stop questioning whether algorithmic predictions accurately represent our preferences, choices, and identity.
The views expressed in this article are the author’s own and may not reflect the opinions of The St Andrews Economist.
Image Credit: Unsplash