Invalid traffic & fraud · also called IAB bots list, IAB/ABC International Spiders and Bots List
IAB/ABC Spiders & Bots List
The IAB/ABC International Spiders & Bots List is an industry-maintained list of known crawler and bot user agents, plus valid browser patterns, used to filter general invalid traffic from ad measurement.
Many automated visitors announce themselves: search engine crawlers, SEO tools, uptime monitors and feed readers include identifying text in their user agent string. The IAB/ABC International Spiders & Bots List, run by IAB Tech Lab together with ABC UK (the Audit Bureau of Circulations), collects these identifiers so measurement systems can recognise and exclude them.
The list is licensed to ad servers, verification vendors, analytics companies and publishers, and updated regularly. It includes patterns for known bots and spiders to exclude, and patterns of valid browsers, so that user agents matching neither can be treated with suspicion. Using it is a standard part of GIVT filtration expected by the MRC IVT guidelines.
It has clear limits. It only catches bots that tell the truth about themselves; any bot that copies a normal Chrome or Safari user agent passes straight through. So the list is a hygiene baseline, not fraud detection. SIVT requires behavioural, network and device analysis on top.
Think of it like this
It is like a guest list at the door of a club that names the known photographers and delivery staff, so they are not counted as paying customers; anyone lying about who they are still gets in.
An example
A publisher's ad server matches 1.4% of daily impressions to user agents on the list, including search and SEO crawlers, and excludes them from the counts reported to advertisers.
Related terms
GIVT (general invalid traffic)
GIVT (general invalid traffic) is invalid traffic that can be identified with routine, list-based or parameter-based filtration, such as known crawlers, data-center traffic and non-browser user agents.
User agent
A user agent is the text string a browser or app sends with each request describing its browser, version, operating system and device, used for compatibility, analytics and fraud detection.
Bot traffic
Bot traffic is visits, ad impressions or clicks generated by automated software rather than people, ranging from honest search crawlers to fraudulent bots built to look human.
MRC IVT guidelines
The MRC IVT guidelines are the Media Rating Council's standards for detecting and filtering invalid traffic in ad measurement, defining the GIVT and SIVT categories that accredited measurers must handle.
IAB Tech Lab
IAB Tech Lab is the non-profit standards body for digital advertising technology, founded in 2014, which develops specifications such as OpenRTB, VAST, ads.txt, sellers.json and the Open Measurement SDK.
Sources: IAB Tech Lab / ABC International Spiders & Bots List, MRC Invalid Traffic Detection and Filtration Guidelines Addendum (2020 update)