News organisations expect a lot from automated recommendation systems, says Sanne Vrijenhoek, researcher at the UvA's Institute for Information Law (IViR). "A good recommendation system can be a competitive asset." Nevertheless, there's resistance to the use of recommendation systems within many news organisations, she observes. "Editorial teams want to retain control over the content presented to users, not delegate control to an algorithm." What's more, editors and readers are worried about automated recommendations leading to filter bubbles: people seeing only content that matches their established interests and opinions. The concern is that bubble formation may undermine democracy.
According to Vrijenhoek, however, the fear of recommendation systems is not entirely justified. "The problem is that recommendation systems usually work on the basis of what people have previously shown an interest in. That inevitably leads to people seeing more and more of the same. But a recommendation system doesn't have to work that way. It's perfectly possible to build in diversity." The UvA's Institute for Information Law (IVIR) has therefore developed a diversity toolkit. The kit can make recommendations measurably more diverse, while also allowing a news organisation to add its own 'signature' to its content offering.
Underpinning the diversity toolkit is previous research by Natali Helberger with support from SIDN Fund. Helberger developed a normative framework covering various forms of democracy and their translation to operational recommendation systems. "Democracy isn't a rigid universal concept; it's a complex concept that's interpreted in different ways," explains Vrijenhoek.
In the 'liberal model', the individual is centre stage.
With the 'participative model', the community comes first.
The 'deliberative model' emphasises the importance of plurality of opinion.
In the 'critical model', the focus is on minorities and highlighting injustice.
By measuring a number of parameters – calibration, fragmentation, activation, representation and alternative voices – it's possible to gauge the extent to which a recommendation system supports the various forms of democracy. "Calibration depends on factors such as the degree of personalisation that's permitted," Vrijenhoek continues. "A news organisation that favours the liberal model of democracy would want to emphasise that aspect of their recommendation system. Then you've got fragmentation, which is about how much overlap there is between what individual users see. That's an important parameter if you want to promote participative democracy. And so on." Anyone interested in the technical side of the diversity toolkit can find details on GitHub.
Measuring complex social concepts
Using the diversity toolkit, a news organisation can create a nuanced recommendation system accurately aligned with the organisation's signature. That's the theory, at least. But does it work in practice? "We've had a lot of interest from news organisations," says Vrijenhoek. "But the coronavirus hit just when we were making plans to test the toolkit with two Dutch news providers." For a long time, the pandemic dominated the news, making it hard to test the team's assumptions. Nevertheless, the toolkit was tested on one provider's historical datasets and found to work well. The findings also led to a number of refinements. A study involving national and international news organisations is planned for 2021.
Vrijenhoek has also taken her research forward on the conceptual level. "Making diversity in recommendation algorithms measurable is extremely complex," she says. "The work touches on various academic disciplines, including information law, media law, communication science, political philosophy and ICT. That creates all sorts of complications: political philosophers see the parameters as over-simplifying complex concepts, while computer scientists are inclined to say they're too vague."
However, Vrijenhoek believes that, within all the relevant disciplines, there is growing interest in the measurement of social concepts. "The US elections and Brexit have underlined the social impact of the way that the information supply is organised," she argues. "When I presented the normative framework at a computer science conference recently, half the room got out their phones to get a snap." Vrijenhoek will be presenting her research at the upcoming CHIIR 2021 conference.
