Skip to content
/ serp-api Public

An enterprise-grade Bright Data SERP API for large-scale, real-time search engine results extraction.

brightdata.com/products/serp-api
0 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

luminati-io/serp-api

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
Images
Images
 
 
free_serp_scraper
free_serp_scraper
 
 
serp_api_outputs
serp_api_outputs
 
 
serp_scraper_api
serp_scraper_api
 
 
README.md
README.md
 
 
setup_serp_api.md
setup_serp_api.md
 
 

Repository files navigation

SERP API

Promo

This repository provides two approaches for collecting Search Engine Results Page (SERP) data:

  1. A free, small-scale Google scraper suitable for basic data collection
  2. An enterprise-grade API solution for large-scale, real-time data collection from major search engines

Table of Contents

Free SERP Scraper

The free scraper allows small-scale Google SERP data collection.

google-search

Input Parameters

  • File: Text file containing search terms (required)
  • Format: One search term per line

Implementation

Modify these parameters in the Python file:

# free_serp_scraper/google_serp.py
HEADLESS = False
MAX_RETRIES = 2
REQUEST_DELAY = (1, 4)

with open("search_terms.txt", "r", encoding="utf-8") as file:
    pass

Sample Output

google-serp-data

Limitations

Google implements several anti-scraping measures:

  1. CAPTCHAs: Used to differentiate between humans and bots
  2. IP Blocks: Temporary or permanent bans for suspicious activity
  3. Rate Limiting: Quick detection and blocking of unidentified requests
  4. Geotargeting: Results vary by location, language, and device
  5. Honeypot Traps: Hidden elements to detect automated access

Bright Data SERP API

Bright Data's SERP API offers a robust solution for reliable SERP data collection.

Key Features

  • Pay-per-successful-request model
  • Fast response times
  • Location-specific targeting
  • Support for multiple device types and search parameters
  • Coverage of major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu, Yahoo, Naver)
  • Built-in anti-bot solutions
  • Real-time results with city-level accuracy
  • Structured data output (JSON/HTML)

Note: The SERP API is part of Bright Data’s Web Scraping Suite and includes full proxy management, unblocking, and parsing capabilities.

Getting Started

  1. Prerequisites:
  2. Setting Up SERP API: Follow the step-by-step guide to set up the new SERP API in your Bright Data account.
  3. Implementation Methods:
    1. Direct API Access
    2. Native Proxy-Based Access

Direct API Access

The simplest way to use the API is by making a direct request.

cURL Example

curl https://api.brightdata.com/request \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer API_TOKEN" \
  -d '{
        "zone": "ZONE_NAME",
        "url": "https://www.google.com/search?q=ollama&brd_json=1",
        "format": "raw"
      }'

Python Example

import requests
import json

url = "https://api.brightdata.com/request"

headers = {"Content-Type": "application/json", "Authorization": "Bearer API_TOKEN"}

payload = {
    "zone": "ZONE_NAME",
    "url": "https://www.google.com/search?q=ollama&brd_json=1",
    "format": "raw",
}

response = requests.post(url, headers=headers, json=payload)

with open("serp_direct_api.json", "w") as file:
    json.dump(response.json(), file, indent=4)

print("Response saved to 'serp_direct_api.json'.")

👉 See the full JSON output

Note: Use brd_json=1 for parsed JSON or brd_json=html for parsed JSON + full nested HTML.

Learn more about parsing results: SERP API Parsing Guide

Native Proxy-Based Access

An alternative method using proxy routing.

cURL Example

curl -i \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<CUSTOMER_ID>-zone-<ZONE_NAME>:<ZONE_PASSWORD> \
  -k \
  "https://www.google.com/search?q=ollama"

Python Example

import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

host = "brd.superproxy.io"
port = 33335
username = "brd-customer-<customer_id>-zone-<zone_name>"
password = "<zone_password>"
proxy_url = f"http://{username}:{password}@{host}:{port}"

proxies = {"http": proxy_url, "https": proxy_url}

url = "https://www.google.com/search?q=ollama"
response = requests.get(url, proxies=proxies, verify=False)

with open("serp_native_proxy.html", "w", encoding="utf-8") as file:
    file.write(response.text)

print("Response saved to 'serp_native_proxy.html'.")

👉 See the full HTML output

SSL Certificate: Load Bright Data’s SSL certificate for production. Learn more: SSL Certificate Guide

Query Parameters Overview

Bright Data SERP API lets you tailor requests for multiple search engines—including Google, Bing, Yandex, and DuckDuckGo—using query parameters for localization, pagination, device emulation, and more. This overview provides a high-level look at the API’s capabilities.

For a complete list and detailed explanation of all query parameters, please refer to the Detailed Query Parameters Documentation.

Google

SERP API supports various Google services, including Search, Maps, Trends, Reviews, Lens, Hotels, and Flights. Below are key configuration parameters for each service:

1. Google Search

Customize your search results with options for localization, search type, pagination, geolocation, and device targeting.

Localization

  • gl: Country code for the search location (e.g., gl=us).
  • hl: Language code for the results (e.g., hl=en).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/search?q=pizza&gl=us&hl=en"

Search Type: Use the tbm parameter to specify the search type:

  • Images: tbm=isch
  • Shopping: tbm=shop
  • News: tbm=nws
  • Videos: tbm=vid
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/search?q=pizza&tbm=shop"

Pagination:

  • start: Result offset (0 for the first page, 20 for the second, etc.).
  • num: Number of results per page (default is 20).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/search?q=pizza&start=20&num=50"

Geolocation:

  • uule: Encoded location string for geo-specific results
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/search?q=pizza&uule=w+CAIQICINVW5pdGVkK1N0YXRlcw"

Device Targeting: Use the brd_mobile parameter:

  • 0: Desktop (default)
  • 1: Random mobile
  • Specific values: ios (or iphone), ipad, android, android_tablet
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/search?q=pizza&brd_mobile=1"

2. Google Maps

Customize maps queries by specifying coordinates and filtering by accommodation type.

Coordinates:

  • Format: @latitude,longitude,zoom (e.g., zoom from 3z to 21z).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/maps/search/restaurants/@47.30227,1.67458,14.00z"

Accommodation Search:

  • brd_accomodation_type:
    • hotels (default)
    • vacation_rentals
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/maps/search/hotels+new+york/?brd_accomodation_type=vacation_rentals"

3. Google Trends

Retrieve trend data with customizable time ranges and widget options.

Required Parameters:

  • brd_json=1: Return parsed JSON results.
  • brd_trends: Specify widgets (e.g., timeseries,geo_map).

Time Range:

  • date: Defines the time range (e.g., now 1-d for the past day).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://trends.google.com/trends/explore?q=pizza&date=now+1-d&brd_trends=timeseries,geo_map&brd_json=1"

4. Google Reviews

Fetch reviews using a feature ID and sort them as needed.

Key Parameters:

  • fid: Feature ID from search results.
  • sort: Sorting order (e.g., newestFirst, ratingHigh).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user "brd-customer-<id>-zone-<name>:<pass>" \
  "https://www.google.com/reviews?fid=0x808fba02425dad8f&sort=newestFirst"

5. Google Lens

Search by image using a URL or file upload.

Image Search:

  • url: The image URL to search.
  • brd_json=1: Return results as JSON.
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://lens.google.com/uploadbyurl?url=https://example.com/image.jpg&brd_json=1"

6. Google Hotels

Customize hotel searches with booking dates and currency options.

Booking Parameters:

  • brd_dates: Check-in and check-out dates (YYYY-MM-DD,YYYY-MM-DD).
  • brd_currency: Currency code (e.g., USD, EUR).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/travel/hotels?q=hotels+new+york&brd_dates=2022-01-20,2022-02-05"

7. Google Flights

Search for flights using similar localization parameters.

curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.google.com/travel/flights?q=flights+new+york&gl=us&hl=en"

Bing

Configure Bing queries with options for localization, geo-targeting, pagination, device and browser targeting, and output formats. Check out the dedicated Bing API.

Localization

  • setLang: Language for the interface (e.g., setLang=en-US).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&setLang=en-US"

Geo-Location

  • location: Search origin (e.g., location=New+York).
  • cc: Country code (e.g., cc=us).
  • mkt: Market code (e.g., mkt=en-US).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&location=New+York&cc=us&mkt=en-US"

Pagination

  • count: Number of results (e.g., count=50).
  • first: Offset for pagination (e.g., first=11 for the second page).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&count=50&first=11"

Filters

  • safesearch: Adult content filter (e.g., safesearch=off).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&safesearch=off"

Device Targeting

  • brd_mobile: Specifies the device type (e.g., brd_mobile=1 for mobile, or brd_mobile=ios).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&brd_mobile=1"

Browser Targeting

  • brd_browser: Specifies the browser (e.g., brd_browser=chrome).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&brd_browser=chrome"

Parsing

  • brd_json: Returns parsed JSON (e.g., brd_json=1).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.bing.com/search?q=pizza&brd_json=1"

Yandex

Briefly configure Yandex queries with parameters for localization, pagination, time range, and device/browser targeting. Check out the dedicated Yandex API.

Localization

  • lr: Specifies the region (e.g., lr=84 for the USA).
  • lang: Page language (e.g., lang=en).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.yandex.com/search/?text=pizza&lr=84&lang=en"

Pagination

  • p: Result page number (e.g., p=2 for the second page).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.yandex.com/search/?text=pizza&p=2"

Time Range

  • within: Specifies the time range (e.g., within=1).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.yandex.com/search/?text=pizza&within=1"

Device Targeting

curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.yandex.com/search/?text=pizza&brd_mobile=1"

Browser Targeting

curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://www.yandex.com/search/?text=pizza&brd_browser=chrome"

DuckDuckGo

A quick overview of DuckDuckGo search customization using localization, safe search, time range, and device/browser targeting. Check out the dedicated DuckDuckGo API.

Localization

  • kl: Country and language (e.g., kl=us-en).
  • kad: Defines the language for interface elements.
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://duckduckgo.com/?q=pizza&kl=us-en"

Safe Search

  • kp: Enables safe search (e.g., kp=1).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://duckduckgo.com/?q=pizza&kp=1"

Time Range

  • df: Specifies the time range (e.g., df=d).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://duckduckgo.com/?q=pizza&df=d"

Device Targeting

  • brd_mobile: For mobile device emulation.
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://duckduckgo.com/?q=pizza&brd_mobile=1"

Browser Targeting

  • brd_browser: For specifying a browser (e.g., chrome).
curl \
  --proxy brd.superproxy.io:33335 \
  --proxy-user brd-customer-<customer-id>-zone-<zone-name>:<zone-password> \
  "https://duckduckgo.com/?q=pizza&brd_browser=chrome"

Other Settings for SERP API

Asynchronous Requests

  • Sync (default): Get a real-time response immediately.
  • Async: Retrieve results later (ideal for high-volume requests).

Learn more: How Async Works

Multi-Query Requests

Send parallel queries in one API call, sharing the same IP and session using the multi parameter.

multi:[
  {"keyword":"shoes","num":50},
  {"keyword":"shoes","num":200}
]

Learn more: Multiple Queries Guide

Support & Resources

About

An enterprise-grade Bright Data SERP API for large-scale, real-time search engine results extraction.

brightdata.com/products/serp-api

Topics

python html api web-scraping google-scraper serp-api serp-scraping

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

1 fork
Report repository

Languages