The official Python SDK for ScrapeBadger - async web scraping APIs for Twitter, Google, Vinted, Reddit, and more.
- Async-first - Built with
asynciofor high-performance concurrent scraping - Type-safe - Full type hints and Pydantic models for all responses
- Automatic pagination - Iterator methods with smart rate limit handling
- Resilient retries - Exponential backoff on transient errors
- 37+ Twitter endpoints - Tweets, users, lists, communities, trends, geo, real-time streams
- 20 Google product APIs - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, Local Pack, Shorts, Flights, Products, Ads Transparency Center
- Vinted scraping - Search items, item details, user profiles, brands, colors, markets
- Reddit scraping - Search posts/subreddits/users/domains, subreddit posts, post comments, user profiles, trophies, wiki pages, moderators
- Web scraping - Anti-bot bypass, JS rendering, screenshots, and selector/AI data extraction
pip install scrapebadgerOr with uv:
uv add scrapebadgerimport asyncio
from scrapebadger import ScrapeBadger
async def main():
async with ScrapeBadger(api_key="your-api-key") as client:
# Get a user profile
user = await client.twitter.users.get_by_username("elonmusk")
print(f"{user.name} has {user.followers_count:,} followers")
# Scrape a website
result = await client.web.scrape("https://scrapebadger.com", format="markdown")
print(result.content)
# Search tweets
tweets = await client.twitter.tweets.search("python programming")
for tweet in tweets.data:
print(f"@{tweet.username}: {tweet.text[:100]}...")
asyncio.run(main())Get your API key from scrapebadger.com and pass it to the client:
from scrapebadger import ScrapeBadger
client = ScrapeBadger(api_key="sb_live_xxxxxxxxxxxxx")You can also set the SCRAPEBADGER_API_KEY environment variable:
export SCRAPEBADGER_API_KEY="sb_live_xxxxxxxxxxxxx"The etsy, aliexpress, target, asos, and bestbuy clients expose 44 dedicated ecommerce endpoints. These generated methods return response dictionaries; consult the endpoint documentation for response fields, supported filters, pagination and current pricing.
async with ScrapeBadger(api_key="your-api-key") as client:
etsy = await client.etsy.search_listings(query="ceramic mug")
aliexpress = await client.aliexpress.search_products(query="wireless mouse")
target = await client.target.search_products(query="coffee")
asos = await client.asos.search_products(query="shoes", market="uk")
bestbuy = await client.bestbuy.search_products(query="laptop")| Client | Endpoints | Documentation |
|---|---|---|
client.etsy |
8 — listing search/detail/reviews, shop profile/listings/reviews, categories, autocomplete | Etsy |
client.aliexpress |
8 — product search/detail/reviews, store profile/products/categories, category navigation, autocomplete | AliExpress |
client.target |
11 — search, category browse/navigation, autocomplete, product detail/reviews/questions/availability, nearby stores/store detail, markets | Target |
client.asos |
7 — product search/detail, stock/prices, category navigation, search filters, autocomplete, markets | ASOS |
client.bestbuy |
10 — search, category browse/navigation, deals, autocomplete, product detail/reviews/questions/availability, markets | Best Buy |
Etsy and AliExpress category navigation is separate from search: their search methods do not offer a category filter. ASOS search follows its upstream semantic matching. Target and Best Buy serve the US market. The linked endpoint references describe further measured limits.
| API | Description | Documentation |
|---|---|---|
| Web Scraping | Scrape any website with JS rendering, anti-bot bypass, screenshots, and selector/AI extraction | Web Scraping Guide |
| 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | Twitter Guide | |
| 20 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products, Ads Transparency | Google Guide | |
| Vinted | Search items, item details, user profiles, brands, colors, statuses, and markets | Vinted Guide |
| Search posts, subreddits, users, and domains; fetch post comments, subreddit rules, moderators, wiki pages, user trophies | Reddit Guide | |
| User profile/about/related/posts/videos/reels/tagged/pinned/followers/following/stories/highlights, media detail/comments/replies/likers/oEmbed, search (users/hashtags/places/top/reels/music/autocomplete), hashtag/location/audio feeds | Instagram Guide | |
| Amazon | 14 endpoints — search, autocomplete, product detail, offers, reviews, bestsellers, new releases, deals, category browse, seller profile/products/feedback, markets, categories | Amazon Guide |
| eBay | 13 endpoints across 18 markets — search, search by image (visual search), completed/sold search, item detail, item reviews, seller profile/items/feedback, category browse, categories, autocomplete, markets | eBay Guide |
| YouTube | 39 endpoints — search, autocomplete, video detail/related/comments/replies/transcript/captions/streams/live-chat/batch, channel detail + videos/shorts/streams/playlists/community/about/subscriber-count/in-channel-search/resolve, playlists/items/mixes, trending/hashtag/home, shorts, community post/comments, music search, oembed, categories/languages/regions | YouTube Guide |
| TikTok | 25 endpoints — user profile/videos/followers/following/liked/reposts, video detail/comments/replies/related/transcript/oEmbed, search (general/videos/hashtags/users), music detail/videos, hashtag detail/videos, trending videos/hashtags/songs, ad library, regions | TikTok Guide |
| Immobiliare | 8 endpoints across immobiliare.it, indomio.es, indomio.gr, immotop.lu — autocomplete, search, listing detail, agency profile/listings, price stats, markets, reference | Immobiliare Guide |
| LoopNet | 5 endpoints across loopnet.com/.ca/.co.uk/.fr/.es — commercial-real-estate search (for-lease/for-sale/auctions), listing detail, broker profile, markets, property types | LoopNet Guide |
| Apartments.com | US rental listings with UNIT-LEVEL pricing — search by location with bed/price filters, plus per-unit rent, beds, baths, sqft and availability for every rentable unit | Apartments Guide |
| Wallapop | 10 endpoints across Spain, Italy and Portugal — listing search with Wallapop's own filters (incl. cars: year, km, gearbox, horsepower), listing detail, user profile/listings/reviews, autocomplete, categories, search filters and filter options, markets | Wallapop Docs |
| Walmart | 11 endpoints (US-only) — search, category browse, deals feed, autocomplete, product detail, reviews, seller profile/products, store detail, markets | Walmart Guide |
| Gumtree | 9 endpoints across UK, Australia and South Africa (market="uk" default, "au", "za") — search (category, location/postcode radius, price, seller type, urgent/photos, category-specific filters), search filters with live counts, listing detail (full vehicle spec, seller rating/verification, masked phone), seller listings, keyword and location autocomplete, location and category trees, markets. E.g. client.gumtree.search(market="au", category="18320", location="3003435", distance=10) or client.gumtree.search(market="za", category="9077", location="3100003") |
Gumtree Docs |
| Baidu | 4 endpoints — web search (language + date filters), news vertical, image search, autocomplete. Results carry the real target URL, not just Baidu's tracking redirect | Baidu Docs |
| Bing | 6 endpoints — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | Bing Docs |
| DuckDuckGo | 7 endpoints — web search (with abstract box), image/news/video search, autocomplete, instant answers, regions | DuckDuckGo Docs |
| Yahoo | 6 endpoints across 35 markets — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | Yahoo Docs |
| Yandex | 4 endpoints across 6 markets (tr/com/ru/by/kz/uz) — web search (organic + ads + sitelinks + inline media), image search, reverse-image (CBIR) search, markets | Yandex Docs |
| Naver | 55 endpoints — integrated & vertical search (news, blog, cafe, kin, image, video, clip, web), autocomplete, local & Place detail/reviews/photos, Shopping rankings/insight/finder & catalog search/price comparison, SmartStore & Brand stores, commerce products with reviews & Q&A, and Shopping Live | Naver Docs |
| ChatGPT | Prompt the real chatgpt.com anonymously — structured answer with citations anchored to character offsets, the full retrieved search trail, and AEO/GEO brand-visibility analysis | ChatGPT Guide |
| Gemini | Prompt the real gemini.google.com anonymously — structured answer with cited web sources, the full retrieved search trail, and AEO/GEO brand-visibility analysis | Gemini Docs |
from scrapebadger import (
ScrapeBadger,
ScrapeBadgerError,
AuthenticationError,
RateLimitError,
InsufficientCreditsError,
NotFoundError,
ValidationError,
ServerError,
)
async with ScrapeBadger(api_key="your-key") as client:
try:
user = await client.twitter.users.get_by_username("elonmusk")
except AuthenticationError:
print("Invalid API key")
except RateLimitError as e:
print(f"Rate limited. Retry after {e.retry_after} seconds")
print(f"Limit: {e.limit}, Remaining: {e.remaining}")
except InsufficientCreditsError:
print("Out of credits! Purchase more at scrapebadger.com")
except NotFoundError:
print("User not found")
except ValidationError as e:
print(f"Invalid parameters: {e}")
except ServerError:
print("Server error, try again later")
except ScrapeBadgerError as e:
print(f"API error: {e}")from scrapebadger import ScrapeBadger
client = ScrapeBadger(
api_key="your-key",
timeout=120.0, # Request timeout in seconds (default: 300)
max_retries=5, # Retry attempts (default: 10)
)from scrapebadger import ScrapeBadger
from scrapebadger._internal import ClientConfig
config = ClientConfig(
api_key="your-key",
base_url="https://scrapebadger.com",
timeout=300.0,
connect_timeout=10.0,
max_retries=10,
retry_on_status=(500, 502, 503, 504),
headers={"X-Custom-Header": "value"},
)
client = ScrapeBadger(config=config)The SDK automatically retries requests that fail with 500, 502, 503, or 504 status
codes, as well as transport-level failures (timeouts, network errors, dropped
connections), using exponential backoff (1s, 2s, 4s, 8s, ...). Retryable HTTP
responses with a valid Retry-After header wait at least that long, whether
given as seconds or an HTTP-date, without shortening exponential backoff.
Missing or invalid headers keep the backoff, and max_retries still limits
retry attempts. Each retry logs a warning:
⚠ 503 Service Unavailable — retrying in 4s (attempt 3/10)
To see these warnings, configure Python logging:
import logging
logging.basicConfig(level=logging.WARNING)When using *_all pagination methods, the SDK reads X-RateLimit-Remaining and
X-RateLimit-Reset headers from each response. When remaining requests drop below
20% of your tier's limit, pagination automatically slows down to spread requests
across the remaining window — preventing 429 errors. A warning is logged when
throttling activates:
⚠ Rate limit: 25/300 remaining (resets in 42s), throttling pagination to ~0.6 req/s
This works transparently with all tier levels (Free: 60/min, Basic: 300/min, Pro: 1000/min, Enterprise: 5000/min).
# Clone the repository
git clone https://github.com/scrape-badger/scrapebadger-python.git
cd scrapebadger-python
# Install dependencies with uv
uv sync --dev
# Install pre-commit hooks
uv run pre-commit install# Run all tests
uv run pytest
# Run with coverage
uv run pytest --cov=src/scrapebadger --cov-report=html
# Run specific tests
uv run pytest tests/test_client.py -v# Lint
uv run ruff check src/ tests/
# Format
uv run ruff format src/ tests/
# Type check
uv run mypy src/
# All checks
uv run ruff check src/ tests/ && uv run ruff format --check src/ tests/ && uv run mypy src/Contributions are welcome! Please read our Contributing Guide for details.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run tests and linting (
uv run pytest && uv run ruff check) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- Documentation: docs.scrapebadger.com
- Issues: GitHub Issues
- Email: support@scrapebadger.com
- Discord: Join our community
Made with ❤️ by ScrapeBadger