I wanted to scrape the Facebook Ad Library for my SaaS, SpreshApp .

On paper, the plan was simple: launch Puppeteer, open the Ad Library, scroll until nothing new loads, collect the data. Done.

Reality had other plans.

Facebook’s Ad Library behaves less like a webpage and more like a suspicious nightclub bouncer. The moment it senses automation, it quietly changes behavior. No dramatic CAPTCHA, no big red error screen. Just silence. Empty results. Missing ads. Inconsistent data. The kind of bugs that make you question whether your code is broken or whether the internet itself has developed trust issues.

Attempt #1: Cloudflare Workers

The scraper technically ran, but the results were garbage. Missing ads and incomplete payloads. It looked successful until you compared the output against the actual Ad Library.

Attempt #2: AWS Lambda

Things looked promising. I started getting real results.

Until I tested multiple pages.

Some returned empty datasets. Others returned ads that didn’t match what I saw manually in the UI. It felt like Facebook had spun up parallel universes for different server regions.

Then I noticed: my Lambda in the US region worked fine for US-targeted ads, but EU ads went wrong. That was the first real clue. Facebook didn’t just care about the request. It cared about where the request came from.

You could technically fix this by deploying Lambdas across multiple regions and building a routing layer to pick the right one based on geography. Possible? Yes. Fun? Absolutely not.

The proxy rabbit hole

So I tried residential proxies. The logic made sense, if Facebook trusts residential IPs more than datacenter IPs, a good proxy should fix everything.

Instead I got empty JSON responses. The behavior got more inconsistent, not less. At this point debugging felt like paranormal investigation.

Lambda had another problem

Some advertiser pages have thousands of ads. I found pages running over 9,000 simultaneously, which is honestly its own form of performance art. Full pagination sometimes took over 30 minutes. AWS Lambda has a 15-minute timeout. So even if I solved the fingerprinting problem, Lambda itself was the bottleneck.

Attempt #3: VPS + residential proxy

I moved everything to a VPS. The Lambda code worked perfectly on my local machine, so I was convinced the right proxy setup would solve it. I converted the logic into a standalone Node.js script, deployed it, attached the proxy, and hit run.

Nothing. No data.

The exact same proxy worked fine on my local machine. Same IP. Same code. Different outcome.

That’s when things got weird.

The real problem wasn’t the proxy

After way too many hours debugging, I started suspecting the browser fingerprint.

My local machine had a normal Chrome install with years of accumulated “human-ness” baked in: a real browser profile, real fonts, a real graphics stack, actual browsing history. The VPS had a freshly downloaded Chromium binary that basically announced itself the moment it connected.

My first instinct was to add advanced human emulation. Mouse movements, random clicks, typing delays, scroll physics. Probably emotional damage next.

Turns out none of that was necessary.

The actual fix

After all of that. the region routing, the proxy experiments, the fingerprint rabbit hole, the mouse movement theatrics, I finally figured out what was actually happening.

Facebook sometimes just refuses to load data when it suspects automation. No errors, no warnings. The page renders perfectly. Looks completely normal. But the underlying API requests silently return nothing.

That’s it. That was the whole thing.

I sat there staring at the screen for a moment. Weeks of debugging, three infrastructure rewrites, a growing collection of proxy subscriptions, and the answer was: the page didn’t load the data, so… reload the page.

Detect when the page came up empty. Reload. Retry until valid payloads came through. Done.

Once the data loaded correctly, I captured the authenticated session and hit the internal paginated API directly instead of scrolling the UI like a caffeinated intern.

I could pull structured data straight from the network layer. Scraping got faster, results got consistent, and pages with 9,000 ads stopped being a problem. The whole thing felt almost too simple after everything it took to get there.

What I actually learned

Scraping modern platforms isn’t just about parsing HTML. You’re fighting browser fingerprints, regional routing, behavioral detection, infrastructure reputation, and dynamic APIs buried behind frontend state machines.

The hardest bugs are the ones where everything looks fine. No crashes, no exceptions. Just subtly wrong data that consumes your entire weekend.

Final thoughts

Facebook scraping is hard not because the data is unreachable, but because the platform is designed to distrust automation by default.

The funny part is that the breakthrough wasn’t some advanced anti-bot technique. It was realizing: if the page loads empty, just reload it.

The same service costs $40/month on searchapi.io. Apify charges $0.75 per 1,000 ads, which gets painful at scale. I do it with a $3 proxy, which is why our pricing is so much lower than the alternatives .

This is technically a business secret, but honestly it’s cool to share. If you want help with scraping, hit me up on X or at [email protected] .