July 2nd-July 8th

Below is a roundup of last week’s notable industry news, with summaries and our opinions. AI is coming, quick, someone put out the c̶l̶o̶u̶d̶ flare for publishers!

Due to popular demand, U of Digital’s AI ACCELERATOR is coming BACK for round 2 in August! Space is limited. Use EarlyBirdsAreSmart promo code for 20% off. Email [email protected] for team discounts. ENROLL NOW!

Top Story 👁️

How brands can get AI bots to pay for scraping their websites🔒
Source: Ad Age
July 7, 2025

Summary: Cloudflare made headlines last week when it introduced a new "pay per crawl" product that promises to transform how AI platforms access website content. This new type of marketplace allows publishers and brands to charge fees when AI bots scrape their sites to train their models, addressing a fundamental imbalance in the digital economy.

Pay per crawl, currently in private beta testing, distinguishes between different types of crawlers, blocking training-focused bots while preserving access for search-related bots that help consumers find real-time information. Website owners can choose to block all bots, let the bots crawl their sites for free, or scrape their sites for a set fee.

Research reveals that AI platforms take far more than they give back—some services crawl thousands of times more than they send visitors to publisher and brand websites. When consumers get information from AI bots, publishers and brands miss out on traffic to their sites. That means lost revenue for publishers, which is damaging to the open web and open web advertising

Opinion: Bravo Cloudflare for trying to help publishers recoup some revenue and opening up a new revenue stream for themselves. But this whole thing feels like slapping a Band-Aid on a gunshot wound. Publishers are stuck choosing between three crappy options: 

1. Block AI crawlers and become completely invisible

2. Let them crawl for free and watch traffic disappear

3. Charge a few bucks and still watch traffic disappear

Option 3 is probably the least of all evils.

For brands, it's a little trickier. Sure, you want to be visible in AI. But when people just ask ChatGPT about your product instead of visiting your site, you're losing way more than traffic. You're losing the chance to build an actual relationship with your customers, collect data that actually matters, and control how your story gets told. You might make a few bucks from crawl fees, but you're basically selling out your entire data-driven marketing strategy. Still, it’s probably the right move.

The bottom line is this: people are using AI because it’s better and faster and easier than clicking through five different websites to get to content or compare products. AI usage is going to continue to grow, and soon it’ll become the entire consumer experience. So publishers and brands can either figure out how to work with that reality or become irrelevant.

Some companies will reluctantly make the obvious, net-negative trade-offs, like taking the Cloudflare revenue. And they’ll do little else. They won’t win. The companies that figure out how to make AI work for them instead of against them will win. That might mean playing a scale game in order to dictate their terms to AI companies. That might mean building their own AI toolset or “owning” the AI layer. That might mean innovating totally new ways to connect with customers that AI can't disintermediate. These companies will be in the minority. And they WILL win.

Everyone else is just trying to monetize their own obsolescence.

Paid Subscribers See ALL The Digital Advertising News.

Thank you for reading the U of Digital Newsletter.
Continue reading by upgrading.

That’s It For This Week 👋

The U of Digital Weekly Newsletter is intended for subscribers, but occasional forwarding is okay!

To subscribe visit Uof.Digital/Newsletters or contact us directly for group subscriptions.

And remember, U of Digital helps teams drive better outcomes through structured education on critical topics like programmatic, privacy / identity, CTV, commerce media, AI, and more. Interested in learning more about how we can supercharge your team?