Precision, Recall, and Inclusivity

[This is a guest post by Talia’s friend Annie, who is maintaining this blog while Talia is away at Middlebury Language Schools]

In my last post on here I mentioned that my job involves writing programs that identify important information from text documents. When testing these programs, there are two main ways of evaluating the accuracy of their output: precision and recall. Put simply, precision refers to what fraction of the retrieved information is relevant to what we’re looking for, while recall refers to what fraction of the relevant information in the original text is retrieved by the program. Ideally, one would want a program to have both high precision and high recall – that is, for it to return most or all of the information the user is looking for and little or no irrelevant information – but this isn’t always possible. More realistically, you’ll often face a tradeoff between precision and recall. You can optimize for precision, and make sure that everything in your output is the sort of thing the user is looking for, but then you run the risk of overlooking other information that might be slightly less obviously relevant, but still within the scope of the user’s query. You can optimize for recall, and make sure to return every piece of information that could possibly be relevant, but then there’s the chance you’ll also turn up a lot of junk data along with the useful stuff. Or you can try to strike a balance between precision and recall, getting each one high enough to be useful without sacrificing the other.

Which of these evaluation metrics is most important depends on what task a program is intended to accomplish, and what the end user’s particular needs are. For instance, suppose you’re writing a program to automatically filter out offensive content on social media. (Some of my classmates and I wrote a program like this for a class project once, although it didn’t end up working very well.) If a social media company is going to be using your program as the first line of moderation for everything that gets posted on their platform, you likely don’t want it to become a heavy-handed censor. In this case you’ll probably want to err on the side of optimizing for precision, filtering out only the things that are well and truly beyond the pale and letting human moderators make the call on the edge cases. On the other hand, if this program is intended as an optional add-on that users of a social media platform can choose to enable, you’ll probably want to err on the side of optimizing for recall. The people who are likeliest to use such an add-on tend to have a very strong need to filter out certain content – say, people with anxiety or trauma who are trying to avoid seeing content that is triggering for them – so you want to make sure all of this content gets filtered, even if that means blocking some harmless stuff as well. High precision isn’t inherently better than high recall, or vice versa; it all depends on the specific goal you’re trying to achieve.

I’m now going to completely switch gears for a minute and move from this relatively dry, technical subject to one with much more emotional heft: inclusivity in the LGBTQ+ community. (I’m focusing on this community because I’m a part of it, but I doubt it’s the only community this discussion will be relevant for.) If you’ve spent any time discussing LGBTQ+ politics – especially on the internet, where political discourse is a full-contact sport – you’re no doubt aware of the frequent heated debates about where exactly to draw the boundaries of this community. For instance, I once saw a post on Facebook proposing the acronym SAGA (Sexuality And Gender Acceptance) as an alternative term to LGBTQ+ that could include everyone without making the acronym cumbersomely long, and one of the comments was arguing against this idea, pointing out that such imprecise wording could allow kinky straight people to elbow their way into the community. Or on a more serious note, back in June when Pride was going on, I saw rather a lot of posts on Tumblr arguing about whether asexuals should be included in Pride, with several people arguing that asexuals hadn’t experienced the same oppression that gay, lesbian, bi, and trans people had, and so Pride wasn’t for them.

When I come across arguments like this, all I can think is: this is clearly a situation in which recall matters a whole lot more than precision. Remember how I said that whether it’s better to optimize for recall or precision depends on your particular goal? Well, what is the goal of building an LGBTQ+ community? There are obviously many goals, but as far as I’m concerned, the primary goal is to provide a space for people who are marginalized by the heteronormativity and cissexism of mainstream culture, where they can be safe and free to live their authentic lives, and where they find the support and solidarity they need in order to overcome the obstacles that the rest of society has placed in their way.

Personally, I care much more about making sure that everyone who needs such a space has access to it than I do about keeping out people who don’t meet some set of entry requirements. Sure, there might be a few straight kinksters who will see inclusive language around sexuality and assume it’s talking about them, but there are also countless queer kids who are just figuring out their sexuality or gender and aren’t sure what label – if any – to claim. Don’t we want them to know they have a place in our community, even if they don’t have the precise words for what they are yet?

For every asexual person who doesn’t face much discrimination for their orientation, there’s another one who’s facing ostracism from their family for rejecting their “sacred duty” of getting married and having children. Don’t they need our love and support too? By focusing on precision instead of recall – that is, focusing on who we need to exclude instead of who we need to include – we run the risk of pushing away some of the people who our community could do the most good for.

In other words, when the machine revolution comes and our new robot overlords implement Fully Automated Luxury Gay Space Communism, I know what sort of information retrieval algorithms I want them to be running.