ScrapeGraphAI SOP

Appearance

Usage ​

Cloud SDK (Python) - Minimal example ​

Requires pip install "scrapegraph-py>=2.1.0" and SGAI_API_KEY set in the environment.

python
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()

result = sgai.scrape("https://example.com")

if result.status == "success":
    print(result.data.results["markdown"]["data"])
else:
    print(result.error)

Every method returns an ApiResult wrapper with four fields: status, data, error, and elapsed_ms. No exception handling is needed for API-level errors.

Cloud SDK - Structured extraction with a prompt ​

python
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()

result = sgai.smartscraper(
    url="https://news.ycombinator.com",
    prompt="Extract the top 5 story titles and their URLs"
)

if result.status == "success":
    print(result.data)
else:
    print(f"Error: {result.error}")

Open-source library example (self-hosted, Ollama LLM) ​

Requires pip install scrapegraphai && playwright install and a running Ollama instance with llama3.2 pulled.

python
from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "model_tokens": 8192,
        "format": "json",
    },
    "verbose": True,
    "headless": False,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="Extract useful information from the webpage",
    source="https://scrapegraphai.com/",
    config=graph_config
)

result = smart_scraper_graph.run()
print(result)

REST API example (curl) ​

No SDK install required.

bash
curl -X POST https://v2-api.scrapegraphai.com/api/extract \
  -H "Content-Type: application/json" \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -d '{
    "url": "https://example.com",
    "prompt": "Extract product name, price, and description"
  }'

LangChain integration ​

The platform supports LangChain and LlamaIndex integrations for chaining extractions into larger agent workflows. See https://docs.scrapegraphai.com/ for integration guides.

Disabling telemetry (open-source library) ​

bash
export SCRAPEGRAPHAI_TELEMETRY_ENABLED=false