Rank trackers, content-gap tools and AI agents all want the same raw material: what Google shows for a query in a given market. Teams usually call the job web scraping Google, then reach for a headless browser and a residential proxy pool. That order skips the deciding step: picking a channel that Google or a contract permits.
This guide covers the official API options for rank tracking, what Google’s robots.txt and terms say, volume planning and country targeting, drawing on Google’s documentation and on notes from EZXLabs, a residential proxy provider.
What collecting Google search data covers
Two different jobs hide behind the phrase. One is reading the results page Google returns for a query: organic listings, ads, People Also Ask boxes, local packs and AI Overviews. Using Google Sheets to pull data from other websites is a separate job with looser rules.
Each stored result needs more than a URL and a position. Save the query, country, language, device, timestamp and an access status next to every row, or two snapshots of the same keyword cannot be compared. Keep the raw response beside the parsed rows by default, so a layout change means re-parsing old data rather than collecting it again.
Official channels and where each one stops
Older tutorials mostly point to Google’s Custom Search JSON API. Google’s documentation now says the API is not available for new customers, and that pricing applies to existing customers until the service is discontinued on January 1, 2027. Existing projects get 100 search queries per day free, then pay $5 per 1,000 queries, up to 10,000 queries per day. Results come from a Programmable Search Engine you configure, which is not the same as google.com rankings.
Depth is capped too: at most 10 results per request and no more than 100 per query, so positions 1 to 100 for one keyword cost ten requests. Search Console sits at the other extreme. Its searchAnalytics.query method reports clicks, impressions, CTR and average position, with a row limit between 1 and 25,000 per request, but only for properties you have verified, never for a competitor.
| Channel | Who can use it | Where it stops |
|---|---|---|
| Custom Search JSON API | Existing customers only | Ends January 1, 2027; 10 results per request |
| Search Console API | Owners of verified properties | Your own site only |
| Third-party SERP API | Anyone with a contract | Permission depends on how the vendor collects |
| Own collector | Sources you own or may collect | Not for google.com results pages |
For google.com rankings across markets, most teams use a third-party SERP API. Ask each vendor in writing how it obtains results, and prefer an official channel whenever it returns the fields you need.
What robots.txt and Google’s terms actually say
Most confusion comes from treating public as permitted. Google’s robots.txt opens with a group for all user agents that disallows the /search path, while allowing /search/about and /search/howsearchworks. That Disallow line covers exactly the results pages a scraper would request.
Google’s Terms of Service, effective July 30, 2026, list under abuse of services “using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages,” naming robots.txt files as the example. Read together, the two documents put automated collection of google.com results pages outside what Google permits.
Courts are still sorting out whether breaching those terms also breaks a law. In July 2026 a federal court dismissed Google’s anti-circumvention claims against SerpApi in part, at the pleading stage, and the EZXLabs write-up of the order stresses that it did not make automated collection of Google results permitted. Plan around the terms and leave legal questions to counsel.
Choosing a channel: a decision order
Work down this list and stop at the first match:
- Rankings for your own site: the Search Console API.
- An existing Custom Search project: keep it and plan the migration early.
- Competitor rankings on google.com in several countries: a third-party SERP API with written answers on sourcing.
- Cited answers for an AI agent: Grounding with Google Search, which returns an answer with source links.
- Any index will do: an independent search API with its own crawl.
- Rendering or internal search checks on your own site: your own collector.
Each step down weakens the permission trail that a client or auditor asks about first. For rank tracking on your own pages, start with the official API: Search Console holds real clicks rather than a sampled position.
Planning request rate and volume
Volume planning is arithmetic, so do it before signing anything. One worked example in the EZXLabs post: 1,000 keywords checked once a day for 30 days at first-page depth come to 30,000 requests a month, which on an existing Custom Search project means 27,000 billable queries, or $135 a month at list price. Depth 100 multiplies the requests by ten, to 10,000 a day, exactly the daily cap.
Pagination has shifted as well. Since September 2025, Google’s num=100 URL parameter no longer loads 100 results at once, so tools reading Google’s pages directly need more page loads for deeper positions. Price only the depth you report on: track page one daily and deeper positions on a slower cycle.
Rate matters even on permitted sources. Cap concurrency per job and write a stop condition into the scheduler, so errors or challenge pages halt the job instead of triggering harder retries.
Country, language and device targeting
Rankings differ by market, so localisation belongs in the request rather than in a guess afterwards. On Custom Search, the gl parameter takes a two-letter country code and boosts results whose country of origin matches, while hl sets the interface language. Search Console reports the same split from the other side, by country and by device.
When a team runs its own collector on a permitted source, such as checking how its localized landing pages render abroad, the network exit becomes part of the location signal. EZXLabs lists 195+ supported countries and regions with country-level targeting on every plan, and advises customers to bind the selected country to the job so retries never drift into another market. Its sticky session has a fixed 10-minute TTL for multi-step checks; for independent requests, the backconnect gateway assigns a new IP to each request.
Keep one fixed query set per country, language and device, and record the country code in every row even for a single-market project.
An empty result is not a rank drop
Here is what surprises most dashboard owners: many apparent ranking losses turn out to be collection failures. Challenge pages, consent screens and empty responses replace the results you asked for, and stored as position zero they look exactly like lost keywords.
EZXLabs states the rule plainly for collectors: mark queries with no retrieved results separately rather than recording a ranking drop, then check access status, parser version and conditions, and repeat on the same route. After the num=100 change, Search Engine Land reported that 87.7% of 319 properties lost impressions in Search Console; falling impression counts alone, the post argues, do not establish lost positions.
Give every row a status value such as returned, empty, challenge or error, and let reports read returned rows only.
Where proxies fit, and where they stop
Think of a residential proxy as network plumbing. It routes a request through a household IP in a chosen country; it does not parse pages, schedule jobs or grant permission. EZXLabs is explicit that its service is not a crawler, scraper, extraction API or managed scraping service, and that it does not return search results, rankings or parsed SERP records.
Limits work the same way. Connections use proxy.ezxlabs.com on port 39080, and once plan traffic runs out the gateway answers HTTP 429 with EZP_QUOTA_DENIED, which should stop the job rather than prompt a workaround. None of this changes what robots.txt says about /search: sending a disallowed request through a different IP leaves it disallowed.
For channel-by-channel prices, the full field list for a clean SERP record and the court rulings so far, the EZXLabs guide to web scraping Google sets each option side by side; confirm prices on each provider’s page before budgeting.
Checklist before the first request
- Write down which channel you use and why it is permitted.
- Archive a dated copy of the target’s robots.txt and terms on day one.
- Calculate monthly requests from keywords, depth and frequency before choosing a plan.
- If a residential proxy serves permitted sources, log which country exit produced each row.
- Add stop conditions for errors, challenge pages and quota responses.
- Keep personal data out: names or emails in snippets stay subject to privacy law even when public.
Teams that can answer every line in writing have a pipeline they can defend; the rest are borrowing risk.
Sandeep Kumar is the Founder & CEO of Aitude, a leading AI tools, research, and tutorial platform dedicated to empowering learners, researchers, and innovators. Under his leadership, Aitude has become a go-to resource for those seeking the latest in artificial intelligence, machine learning, computer vision, and development strategies.
