The anatomy of an Airbnb review

As someone who has completed many a pragmatics problem set in undergrad and later in grad school, I know the makings of a good pragmatics problem set when I see one. And Airbnb reviews are basically a giant real-life pragmatics problem set. 

For those not familiar, pragmatics is the area of linguistics that deals with meaning in context. What did a speaker mean? What did they communicate to a listener? How does the meaning of a message change across settings?

Outside the hallowed halls of learning, it’s known as reading between the lines.

I’m the designated Airbnb chooser in our house, and I study the words of those reviews like a detective (or someone doing a pragmatics problem set). “Host resolved issues promptly” – issues, well that’s not great. “Front door was a bit of a challenge” – what the…? “Nice apartment” – yeah, I don’t think so.

What features of a home or experience are important enough to get commented on? How do reviewers communicate problems while still maintaining a super-polite and positive tone? What delights people, outweighing any possible negatives of a stay?

You would think that, if distilled and shared publicly, these reviews could be a treasure trove to 1) help hosts improve people’s experience, and 2) get sliced and diced into cool insights and visualizations à la Zillow. (For example, I’ll always remember the finding that—at least back in 2018—homes with black doors tended to sell for more money. Unexpected and cool.)

However, as far as I can tell, no such thing is happening at Airbnb. Their data science blog is focused on the technical nitty-gritty, while their news section features stories like “8 Ways to Tap Into Your Inner Chess Champion on Airbnb” and “Airbnb and Bumble Help Guests Celebrate Love This Valentine’s Day”. 

It’s really too bad! 

Fortunately, there’s Inside Airbnb, a truly Herculean activism effort that continually scrapes publicly available data from the website and makes it available for visualization (e.g., Montréal here) or download. The data includes full written reviews, some going back more than a decade. 

By using Inside Airbnb, or by using publicly available information directly from the Airbnb website, a number of not-affiliated-with-Airbnb groups have conducted research on these reviews.

Today, I want to share a few of these research projects, each using different techniques and increasingly large data sets, that begin to reveal the anatomy of Airbnb reviews.

Airbnb reviews are overwhelmingly, stiflingly positive.

One study, in-depth but with a relatively limited dataset, examined guest and host reviews to understand how they were typically structured and what they contained. The researchers manually collected 100 reciprocal guest and host reviews in four US cities (so 400 in total) in different regions – Portland (OR), Albuquerque, Atlanta, and Philadelphia. 

If it seems like Airbnb reviews all kind of sound the same, you’re not wrong. The reviews were largely similar, much more so than typical written language. In particular, hosts often fell back on a review “template”. Around one in ten used identical or nearly-identical ways to describe guest after guest (some actual examples: “Stephanie was a great guest. She is welcome back any time,” “Scott was a great guest. He is welcome back any time.”)

The guest reviews had commonalities, too. For one, they were almost all positive. Around 93% had unequivocally positive language. Let that sink in. (Another study, one that included researchers from Airbnb for once, found that 94% (!) of reviews rated the stay 4 or 5 stars. So, the positivity bias is out of control.)

Guests conveyed their sentiment by using positive adjectives (‘great’), intensifying adverbs (‘seriously’) and extra punctuations for emphasis (‘!!!’).

The reviews also tended to follow a similar structure. First, people mentioned the host (around 4 in 5 even mentioned the host by name), then they described the home, and then summarized the overall experience. 

Perhaps due to the pressure of reciprocity (after all, hosts were also reviewing guests), hosts got a hell of a break no matter what happened. Within the entire set of reviews used for this study, there was not a single case where the host was described poorly. Even in negative reviews, the host somehow got away scot-free, and problems were attributed to external circumstances or even Airbnb itself. 

Here’s an actual example: “Lane, who is the host, could not have been more pleasant. A wonderful guy. However, this rental is a terrible place… Cozy it is not, loft, it is not. I have to say that I feel ripped off and tricked. It is not Lane’s fault. The owner is a slum lord that refuses to do the repairs. I would not recommend this to my worst enemy, or maybe I would. Airbnb should not list this hole.” 

Can you believe what Lane gets away with here?

The rare negative reviews were almost never entirely negative. Instead, they typically followed the sandwich feedback technique, beginning and ending with positive comments. The complaints were also hedged (e.g., ‘just,’ ‘pretty,’ ‘somewhat’) to soften their force and maintain a generally cheery attitude.

Then, there were the lukewarm reviews, my favorite category. These were positive but not as positive as typical reviews. They didn’t use any negative language but still clearly signaled a guest’s true opinion to future readers. These reviews make me pay attention when scrolling through a listing and are the cause of 100% of Airbnb booking disagreements in our household.

Lukewarm reviews were usually shorter, didn’t use intensifiers, and used weak positive descriptors like ‘good,’ ‘ok,’ ‘nice,’ ‘as expected,’ or ‘as advertised.’ They sometimes ended with a qualified recommendation (‘recommended for a short stay’).

The contrast was especially clear when comparing different reviews from a single guest. Can you guess which place was likely terrible?

Image source, emoji added.

So yes, a place described as ‘good’ or ‘nice’ in an Airbnb review is most likely not good or nice at all.

What’s good? Location. Amenities. Host.

Moving up in dataset size and forward in the future, another study mined Airbnb reviews to understand what factors determine the quality of a person’s experience during their stay. 

This study used Inside Airbnb data from Sydney up through 2016, at which time the city was the #4 most popular destination in the world. The analysis included over 170 thousand English-language reviews.

The researchers uncovered themes in the reviews by applying a framework from the hotel literature and then adapting it based on the findings. This was followed by a sentiment analysis to understand the probability that a mention of a given theme was positive or negative.  

There were three general themes covered by reviews: location, amenities, and host. 

There’s no arguing with a good location. Convenience to local attractions, restaurants, shops, and transport was seen positively. In fact, as long as the location was convenient, people seemed willing to overlook accompanying inconveniences like noise at night, safety concerns, or difficulty parking.

Second most important were the amenities. These included the facilities of the building, the room environment—which included a description of the space, its design and decoration, and its cleanliness—and the nighttime environment—which mostly dealt with the quality of sleep and privacy. People tended to write positive reviews if they felt at home in a space. Though not expected by guests, food and drinks left by the host were seen as a delight, and people said so. 

As an aside, searching through Inside Airbnb reviews for ones that mention “delight” reveals some real winners. For example, this listing in SF has been described as “delightful” countless times, and this one in Montréal got similar shout-outs. If you really want to optimize when we can travel freely again, you could Command/Control F for “delight” in those mega-spreadsheets. And if you work at Airbnb and are reading this, then I really hope you’re using these kinds of keywords to surface delightful locations to bookers! Also, it would be pretty cool to be able to search through reviews in a city for  keywords (yes, “delightful” but also “kids” or “playground”). These reviews are freaking goldmine.

Image source. A consistently-delightful SF listing.

Finally, there was the host—their helpfulness, their flexibility in accommodating requests, and their communication, including the instructions they left guests. Supporting the findings of the study we talked about earlier, hosts tended to be described very positively. Two-thirds of mentions of a host were positive, and only 1% were negative. Everyone gets treated like Lane.

So, what made for negative reviews? There were a few key topics that tended to be harbingers of trouble: noise, floor, shower, and parking. Let’s expand on those and other causes of less-than-ideal experience by moving to another part of the commonwealth on the other side of the world.

Indoor environmental irritants are no good.

Over in Canada, a recent study examined indoor environmental quality issues in Airbnb reviews. Using data from Inside Airbnb for six Canadian cities up through 2019, it included nearly 1.2 million English-language reviews.

Search terms for four aspects of environmental quality – air (e.g., smell, musty), acoustic (e.g., noise, loud), visual (e.g., glare, dim), and thermal (e.g., cold, hot)—were used to identify reviews with environmental issues. The researchers also tried to capture the cause of the issues by identifying other words that frequently reccured with each one. Finally, sentiment analysis was used to estimate the likely effect of environmental issues on the overall experience.

Almost 5% of the reviews mentioned at least one indoor environmental quality issue, and about a quarter of a percent mentioned more than one. Reviews that specified environmental issues also tended to be more negative.

As you may recall from the first study we talked about, around 93% of Airbnb reviews are positive. So, the small proportion that are negative in tone could very well be attributed to environmental issues that guests encountered.

What were these issues? In order of frequency: problems with acoustic, thermal, air, and visual quality. This order was also stable across building types (e.g., true for both apartments and cabins). 

Noise can be miserable and feels like something you can (and should!) complain about in reviews. To a lesser extent, so is being too hot or too cold. Meanwhile, bad lighting might be annoying, but perhaps it doesn’t seem like something you have a “right” to expect in a short-term rental as much as a quiet place to sleep.

The suspected causes of environmental issues (i.e., the words that occurred with each one) varied. For example, for acoustic quality they included things like ‘street,’ ‘traffic,’ or ‘floor’ (if you recall, ‘noise’ and ‘floor’ were two of the big complaint drivers in the Sydney study described earlier, too—so noise is an important issue all around the world).  For air quality issues, causes included things like ‘bathroom,’ ‘cigarette,’ or ‘kitchen’.

Image source. Identified causes of indoor environmental quality (IEQ) issues.

A charming tidbit was that some causes of environmental issues were city-specific. For example, people complained about noisy seagulls in Victoria, BC, while those in Toronto were irritated by the smell of fish from nearby fish shops (e.g., “the room is small and stinks of fish,” a truly damning review).

Another cool finding was the seasonality of thermal and acoustic issues. As you would expect, people complained more about the cold in the winter and the heat in the summer. Meanwhile, acoustic issues were more frequent in the summer and fall than in the other seasons. The causes of acoustic issues also varied throughout the year. In the summer, the causes included ‘street,’ ‘outside,’ and ‘construction,’ while those in the winter included ‘upstairs’ and ‘downstairs.’ What does this mean? Outdoor noises are a bigger pain in the summer, when we’re all outdoors and keep the windows open. In the winter, when everyone is indoors, it’s the people in the building who are at the root of noise issues.

Finally, if you’re looking for the most comfortable (or the least indoor-environment-complaining) city to visit in Canada, this study has an answer. It’s… Toronto. 

Grab yourself the full Inside Airbnb Toronto review CSV and find yourself a ‘delightful’ spot (here’s one apparently). After your stay, you know what to say: ‘Kevin was the best. What a host, what a guy. Great location! The place was beyond beautiful, looks EXACTLY like the pictures and I must say, smells great too. It’s clean, quiet and fully equipped!!! 6 stars. Highly, highly, HIGHLY recommend. Will be sure to stay again on our next visit to Toronto!!”

Anything less would basically be an insult.