I recently did some A/B testing work through the Facebook advertising platform, and gave a quick presentation on the pros and cons of the platform. Here’s a summary.
PRO
- Microtargeting
- Optimization
- Inexpensive, low ceilings
- Demonstrated to work at scale, sophisticated distribution
CON
- Click bots
- Opaque
To clarify my perspective on the platform, some background on the work we did:
We ran some A/B tests through the platform targeting a specific population, evaluating different levels of resulting engagement for statistical significance. I assure you, nothing fancy.
PRO: MICROTARGETING
In my book, this is the biggest pro. When creating a campaign (FB has its own hierarchy and nomenclature describing ads, ad sets, ad runs, etc.) you have the option of customizing your target audience. At this point you’re able to winnow down your target audience based on age, gender, language, location (country, state, or city), page likes, political stances, education, relationship status, device, finances, job title, interests, favorite music…the list goes on. Essentially, any and all information about you on Facebook can be used to microtarget you. This is ideal if you’re running A/B tests and want to maximize experimental control – we were able to target the same tiny subpopulation when running our ads, minimizing the chance that any resulting signal got buried in the noise of a large and diverse sample population. In addition, the audience manager allows you to select audience by email, so if users have granted you access to their information then you have a platform for retargeting.
The microtargeting is a little bit too good: go to facebook.com/ads/manager and create an audience. Find a friend’s Facebook profile and start narrowing your audience based on his/her profile information. With a little effort, you can actually narrow your target audience down to just your friend by stacking more obscure interests in AND relationships. The audience size never seems to dip below 20, but at some point its clear that there is an audience of 1, and you can run ads accordingly: “Jeff, have you seen my grey hoodie? I think I left it at yours.” There’s an urban legend out there about a startup that was able to secure VC funding after targeting and blasting ads towards all partners at a specific fund. All in all a very effective tool, but extremely problematic from a privacy standpoint.
PRO: OPTIMIZATION
When creating an ad you’re presented with an optimization target (see list in the image above). For off-platform optimization targets like conversion, Facebook offers a “pixel,” or what basically amounts to a tracking snippet that embed on your site to communicate analytics back to Facebook. The flexibility of goals is quite useful, and based off of A/B testing against optimization targets, your choice of target does in fact make a substantial difference in the performance of your ad.
PRO: INEXPENSIVE, LOW CEILINGS
I don’t have extensive experience with other ad platforms, but exposure per dollar on Facebook seems quite reasonable. On one trial with exposure as optimization target, we reached 6,000 people for about $13.00 USD. With engagement (clicks, likes) as an optimization target, cost was around $1/engagement during the run, with some residual engagement occurring after the ad was finished but had gained traction in some communities. Additionally, if your ad fares poorly and has little chance of reaching respectable optimization targets, Facebook will usually take it out of circulation prior to your ad budget hitting $0.
By very low ceilings, I mostly mean low barrier to entry and low buy-ins. If you already have a Facebook account then you already have access to the ad platform, and creating an ad takes less than an hour (assuming your content is ready). So far as I know there’s no minimum budget, and as mentioned above even $10 can buy you a pretty sizable audience.
PRO: DEMONSTRATED TO WORK AT SCALE, SOPHISTICATED DISTRIBUTION
The fact that a large number of very big companies continue to use Facebook advertising signals that, in at least some use cases, it is an effective platform. You can speculate about what those particular use cases are (demographics, product vertical) and in most cases you’ll be correct.
More interesting than that is the suite of distribution services. It’s really easy to get lost in the developer docs and marketing tutorials – there are APIs and microservices left and right for pretty much any custom marketing scenario you can think up (split testing, exporting analytics, rule-based distribution, etc). Crucially, it looks like large scale, refined, programmatic distribution (I haven’t used it myself) is well-documented and fairly easy to implement. There’s plenty more to say on the subject, but I don’t feel qualified to discuss it in depth, and you’re better off browsing the developer docs yourself to see what’s possible.
CON: CLICK BOTS
The bot story goes something like this:
- Rick’s Auto Dealership hires a clickfarm to get their page to 10,000 likes so that Rick can please his investors, or whatever.
- Facebook notices 10,000 new accounts all created in Bangladesh, all of whom engaged in just one activity: liking Rick’s Auto Dealership in New Jersey.
- These obviously fraudulent accounts inflate the number of users. This concerns potential ad-buyers who have to start questioning the market size and validity of the Facebook ad platform.
- Facebook automatically scans its user base and purges these fraudulent accounts.
- The clickfarm from the beginning of our story needs to stay in business, so now each fraudulent account likes not only Rick’s Auto Dealership, but an additional 500 other randomly selected pages on Facebook. This effectively buries the signal and makes it harder for Facebook to detect fraudulent accounts.
- Now, when you go and advertise your page through the ad platform, you’ll receive likes and engagement from fraudulent accounts. Your budget is used up on worthless engagements, and only a fraction of it gets spent on meaningful advertisement. Your retargeted and renewed content is afterwards distributed to a diluted audience, which leads to more dilution per $.
Bots are easy to spot, but time-consuming to spot. It’s not until you look at the user’s likes that you can determine who is and is not a real, sane, human being. Here’s a screencap from the “Interests” section of a bot that liked one of our ads:
As you can see this person likes everything from auto repair to erotic jewelry to CitiBank careers…and then about 2000 other things.
It will be interesting to see how Facebook handles this balance in the future. On the one hand, bots increase user growth (tbh I don’t know how important that metric is anymore) and make the Facebook ad platform look good: you can get 200 likes on your page for just $20! On the other hand people who start to dig deeper and notice that only 40/200 of those likes really matter will pull away from the ad platform. I speculate that this is a balancing act for two reasons: 1) Facebook has some incentive to, at the very least, let click bot behavior slide and 2) I think that if they wanted to, they wouldn’t have too much trouble purging the bots from their system. There are very strong indicators of (the current cycle of) bot behavior that seem difficult to hide: lots of page engagement, minimal user engagement, incredibly distant and disparate likes and interests, geolocation indicators, etc.
CON(S): OPAQUE
I originally listed optimization as a con because of its opacity, but decided to make opacity itself the con since it exists in a few different parts of the platform, though the optimization aspect is a good place to start. The optimization engine is opaque in a stronger sense than the sense in which large systems with complex operations and lots of data are hard to understand.
Experimental control: When you buy billboard space on the side of the highway, there’s no question about who sees it and for how long it runs: it’s seen by everyone who passes the highway for as long a period as you’ve paid for. The downside of optimization is there are no such guarantees. If you wanted to show that ad A is significantly better than ad B, you might find that when you run both of them, ad B gets pulled out of circulation long before you can gather any kind of statistically significant insight. In essence, positive feedback gets through the system while negative feedback is stunted. For these reasons experimental design becomes tricky.
For instance, part of our experiments involved showing Big-Endian articles to Little-Endian audiences, and in this case ad optimization is about the worst thing you can ask for – what isn’t buried by the platform somehow finds its way to the exact people you don’t want to reach: Big-Endians. That’s a somewhat unique example, but something very similar must occur when attempting to expand your market; your ads end up finding their way to your existing market and you don’t learn about the markets for which your ad is not appealing. Yes, a great deal of useful information flows through the platform – where else do you find immediate, at-scale, and high-fidelity feedback on ad performance? – but unless you work at Facebook you won’t see it.
If you already know what ad you want to run and want to get as many relevant eyeballs on it as possible, then optimization offers great impact/$. If you want to understand the details of ad performance from an experimental design perspective, you’ll likely be frustrated by optimization and will need to design effective countermeasures like microtargeting, minimal ad variations, and A/B testing.
Customer service: Following one of our Big-Endian Little-Endian experiments, we received notification that our ad had been flagged by the community and our account was suspended. The language for suspension was fairly obscure and lent itself to any number of interpretations: copyright violations (we had a wrapper URL on another domain), audience targeting violation, under-representation of advertising source…If you review the advertising policy guidelines in detail, it will become clear that they’re well within their rights to shut down an advertiser for any one of a hundred reasons.
Unfortunately, if your account runs into trouble you might never find out why because Facebook offers no customer service. Perhaps if you spend a good chunk of money and have a sales rep you might get answers, but in general there is no substantial contact with account services that I came across. In our case, it only appeared that we had done one of any N things incorrectly. In order to resume our ad tests we had to set up a new account and begin ruling out all of these N hypotheses successively, all the while aware that whatever our first ad had done to result in suspension was in part out of our hands and due to the behavior of the audience. In that respect, an ad run today might not be flagged while the same ad run one week from now could be flagged due to some whim of the audience.
As it later turned out, we got a hold of someone at Facebook working close to or in the relevant department (in-person, not via customer service). Our ad was run around the time (December 2016) that Facebook came under fire for the proliferation of fake news, and was filtered out via the newly minted fake news detection algorithm (I’m sure it has a sexier name internally). Presumably, this filtering algorithm looks at content, metadata, derives some reputation proxies about advertiser, URL, etc., so given our goal and Facebook presence (small), it’s easy to see why the system suspended our account.
Regardless, without insight into the causes or sensitivity thresholds of policy violations or problem X related to your account, you might find yourself in a difficult situation and unable to interpret your results.
Optimization: Facebook has little incentive to share the details of how and why it distributes and redistributes content on a user’s wall in the way that it does; their data moat informs these complex decisions. The interesting question is what objective does the optimization engine that makes these decisions have? Is Facebook more interested in maximizing the time you spend browsing per day, the number of peer-peer interactions per day, the number of ad clicks…?
We know that it’s not based purely on a consumption metric like clicks and engagement – otherwise every ad and article on your wall would fall into what I imagine is the internet’s local optimum: kittens, clickbait, Kardashians, and celebrity nudity (inclusive of Kardashians). There’s some kind of experience/culture/reputation curation built into the system, but it’s hard to say what it is or how resilient it is to the exposure of non-obvious advertising targets or, more generally, divergent ideas.
This relates directly back to the objective of the company itself. Facebook’s responsibility to its shareholders likely demands revenue above all else, while messaging from the company seems to indicate an increasing awareness of itself as an important information distribution platform and of the responsibilities that should attend a product that has come to replace the newspaper for quite a lot of people. (You are free to interpret communication from Facebook execs about corporate mission as cynically as you like.)
If the shareholder model dominates, you can expect to see more resilient information “bubbles” and possible difficulty as an advertiser reaching new audiences. I’ve heard some strongly argue that the shareholder model will dominate, subgroup “bubbles” will further polarize, fake news will become abundant, etc. though I suspect that the company is now and will continue to toe the lines.
SUMMARY
Despite its problems, we found Facebook to be a good platform for ad testing, mostly due to low costs, microtargeting, and ease-of-use. I’m not sure that the platform is as well-suited for large-scale ad testing as it is for pure eyeballs-on-my-product advertising. If you’re not sure and have $20 lying around somewhere, I encourage you to head over to the ad manager site and try it yourself. Maybe you’ll get that hoodie back from Jeff’s place.
Nick Ryan 