[Extract] Add extract, map and crawl commands for the Extract API by strzibny · Pull Request #21 · serpapi/serpapi-cli

strzibny · 2026-10-22T13:21:44Z

This PR adds Extract API support. I wanted to add specific extract subcommand but though it's actually useful for CLI to also add map and crawl aliases for developer experience.

serpapi extract https://example.com output=md
serpapi map https://books.toscrape.com --jq '.urls[].url'   # = extract URL mode=map
serpapi crawl https://books.toscrape.com max_pages=5        # = extract URL mode=crawl

Done with Opus 5.5 High and Max. Some futher notes:

The Extract API is engine=extract on /search.json, so these commands reuse search's request and output code: --fields, --jq, output=md, --debug, --timeout and archive work as in search.
Unlike search, they need an API key up front; the API returns 401 without one.
Bad input fails before any request is sent, so it costs nothing: no URL or several URLs, a URL that isn't absolute http(s), an engine= other than extract, a mode= that contradicts map/crawl, a parameter missing its =, and search-only flags such as --all-pages. An argument counts as the URL when it has :, /, . or ? before any =.
A timed-out Extract request no longer gets search's "slow engines such as google_ai_mode" advice, and crawl suggests --timeout 300. This changes shared code: networkError in pkg/api/client.go now picks the advice from the request's engine and mode.
crawl costs one search per extracted page (default max_pages=10). The help and README say so.

Note Extract API can already be used by passing engine=extract, but this makes things nicer.

`serpapi extract URL [key=value...]` sends an engine=extract search, so --fields, --jq, output=md, --debug and --timeout work as in search. `serpapi map URL` and `serpapi crawl URL` are shortcuts for `serpapi extract URL mode=map` and `mode=crawl`: same request, parameters and flags. The URL may be given bare or as url=, and must be one absolute http or https URL; anything else is a usage error before any request, as are engine= other than extract, a mode= that contradicts map or crawl, a word without "=", which search would take as q=, and search's --all-pages and --image. Unlike search, the commands need an API key up front, since the Extract API answers nothing without one. A timed-out extraction no longer blames slow engines such as google_ai_mode: extract and map say only how to retry, and crawl suggests --timeout 300 and that each page counts. The README documents the three commands, says map and crawl are shortcuts for mode=map and mode=crawl, and lists crawl among the requests that can outlast the default --timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

trusche

Thanks @strzibny I don't know Go, so this is a fully automated AI review with my usual coding agents. Let me know if that's of any use or you'd rather not get reviews like this.

Everything below and inline is verbatim from Opus 5.5. High

Claude Opus 5.5 AI-assisted review: read the code, ran the tests and checked the CLI by hand.

Approve with comments. Nothing blocks the merge. Please add the release:minor label before you merge: this PR adds three new commands, and without a label the merge releases only a patch version (see AGENTS.md, "Releases").

QA: go test ./pkg/..., go vet and the offline e2e tests pass. I ran extract, map and crawl against the live API (3 searches), and 14 usage-error and timeout cases offline. The help text matches the Extract API docs. Details are in the inline comments; all of them are non-blocking.

trusche · 2026-10-23T10:46:22Z

+	if raw {
+		body, err := withSpinner(label, func() ([]byte, error) { return client.SearchRaw(ctx, p) })
+		if err != nil {
+			return err
+		}
+		return outputError(output.PrintRaw(cmd.OutOrStdout(), body))
+	}
+	result, err := withSpinner(label, func() (json.RawMessage, error) { return client.Search(ctx, p) })
+	if err != nil {
+		return err
+	}
+	if restricted && isEmptyObject(result) {
+		warnNothingMatched(cmd)
+	}
+	return handleOutput(cmd, result, searchHint(p))


These lines are a copy of sendSearch (search.go:180-201). Only the spinner text is different. The extract commands do not have --all-pages, so allPagesFlag is always false here. Add a label argument to sendSearch and call it from here. That removes about 15 lines.

The output= and --jq check at lines 117-125 is now in 3 places (also search.go:130-138 and archive.go:47-53). Maybe move it into one helper.

trusche · 2026-10-23T10:46:22Z

+		case strings.EqualFold(query.Get("engine"), "extract") && strings.EqualFold(query.Get("mode"), "crawl"):
+			message = timedOut + ". A crawl extracts every page it visits, so it takes longer than one page: " +
+				retry(max(suggestedTimeout(c.http.Timeout), 300)) +
+				". The crawl may still finish on SerpApi's side, count one search per page and be available in your search archive"
+		case strings.EqualFold(query.Get("engine"), "extract"):
+			message = timedOut + ": " + retry(suggestedTimeout(c.http.Timeout)) + ". The extraction" + onSerpApi
+		default:


The HTTP client now reads engine=extract and mode=crawl from the request URL, and 300 is a fixed number here. The /image and google_ai_mode cases already do something similar, so this is OK. But this is the second engine-specific branch in the client. Maybe give 300 a name.

A question: when a crawl times out on the CLI side, it can still finish on the server and charge one search per page. This advice arrives only after that cost. Should crawl default to a 300s timeout when the user does not set --timeout or SERPAPI_TIMEOUT? Then this branch may not be needed.

trusche · 2026-10-23T10:46:22Z

+  mode=crawl         every page of the website it visits; same as 'serpapi crawl'
+  include_tags=, exclude_tags=   CSS selectors to keep or drop
+  include_markdown=true          add the main content as Markdown to JSON
+  output=md                      only the main content, as Markdown


"only the main content" is not quite correct: the Markdown starts with a YAML block with the search metadata (live: serpapi extract https://example.com/ output=md). Maybe: "the main content as Markdown, after the search metadata".

trusche · 2026-10-23T10:46:22Z

+			continue
+		}
+		// search takes a bare word as q=; here it is a parameter missing its "=".
+		if !strings.Contains(a, "=") {


Good. But the root help at root.go:60-61 still says "words without "=" are joined into q" for all commands. That is not true for extract, map and crawl now. Maybe add "in search" there.

trusche · 2026-10-23T10:46:22Z

+// as the URL, so it is reported as an invalid one.
+func isURLArg(arg string) bool {
+	key, _, _ := strings.Cut(arg, "=")
+	return arg == "" || strings.ContainsAny(key, ":/.?")


An empty argument counts as the URL. When it comes with a real URL, the error has an empty first item:

$ serpapi extract "" https://example.com extract takes one URL, got 2: , https://example.com; ...

Maybe report the empty URL first, before the CLI counts the URLs.

trusche · 2026-10-23T10:46:22Z

+	}
+	msg := fmt.Sprintf("Invalid URL %q: the Extract API takes an absolute http or https URL", raw)
+	if !strings.Contains(raw, "://") {
+		if u, err := url.Parse("https://" + raw); err == nil && strings.Contains(u.Hostname(), ".") {


Small: serpapi extract mailto:me@x.com suggests e.g. https://mailto:me@x.com. url.Parse reads mailto:me as a user and password, so the hostname is x.com. Maybe suggest only when u.User == nil.

trusche · 2026-10-23T10:46:22Z

+		if isExtractCommand(cmd) {
+			// search's --all-pages and --image have no Extract API parameter;
+			// its --max-pages does, as crawl's max_pages.
+			if name != "max-pages" && searchCmd.Flags().Lookup(name) != nil {
+				return msg + "; --" + name + " works only with search; " + params
+			}
+			// Extract API parameters use underscores, as in max_pages.
+			name = strings.ReplaceAll(name, "-", "_")
+		}


The change from - to _ applies only to the extract commands. search --no-cache still suggests no-cache=VALUE. That bug is not from this PR, but the same fix applies there. Maybe apply it to all commands, or leave it for a separate PR.

Also, serpapi extract URL --max-pages 3 suggests max_pages=VALUE, but only crawl uses that parameter.

trusche · 2026-10-23T10:46:22Z

+serpapi extract https://example.com --jq '.inline_links[].link'
+```
+
+It counts one search. The Extract API runs no JavaScript and follows each site's `robots.txt`. Search pages that have their own engine, such as Google or Bing, are redirected to it and return that engine's JSON. Unlike `search`, `extract`, `map` and `crawl` need an API key.


"It counts one search" comes right after the table row for mode=crawl, which costs one search per page. The help at extract.go:37-38 has the correct sentence: "scrape and map count one search each; crawl counts one per page it extracts".

trusche · 2026-10-23T10:46:22Z

+
+`serpapi crawl URL` is a shortcut for `serpapi extract URL mode=crawl`: it sends the same request and takes the same parameters and flags.
+
+Starts at the URL, follows links on the same website and extracts every page it visits, one entry each in `pages`. `max_pages` caps the pages, the first included (default 10, at most 100), and `max_depth` how many links deep it goes (default 2, at most 3; `0` is only the first page).


The help (extract.go:81) and the API docs list include_subdomains=true for crawl, but this section does not mention it.

trusche · 2026-10-23T10:46:22Z

+func TestExtractCommandsTimeOut(t *testing.T) {
+	const extraction = "Request to /search.json timed out after 50ms: retry with a higher --timeout, such as --timeout 120"
+	const crawl = "Request to /search.json timed out after 50ms. A crawl extracts every page it visits, so it takes longer than one page: " +
+		"retry with a higher --timeout, such as --timeout 300"
+	for _, tc := range []struct {
+		name, env, want string
+		args            []string
+	}{
+		{"extract --timeout", "", extraction, []string{"--timeout", "0.05", "extract"}},
+		{"map --timeout", "", extraction, []string{"--timeout", "50ms", "map"}},
+		{"extract mode=crawl", "", crawl, []string{"--timeout", "50ms", "extract", "mode=crawl"}},
+		{"crawl --timeout", "", crawl, []string{"--timeout", "0.05", "crawl"}},
+		{"crawl SERPAPI_TIMEOUT", "0.05", crawl, []string{"crawl"}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			fakeAPI(t, hang)
+			t.Setenv("SERPAPI_TIMEOUT", tc.env)
+
+			start := time.Now()
+			res := execute(t, "", append(append([]string{"--api-key", extractKey}, tc.args...), "https://example.com/")...)
+
+			wantError(t, res, 1, "network_error", tc.want)
+			if strings.Contains(res.stdout+res.stderr, "Slow engines") {
+				t.Errorf("advice blames slow engines:\n%s%s", res.stdout, res.stderr)
+			}
+			if elapsed := time.Since(start); elapsed > 2*time.Second {
+				t.Errorf("took %v, want the 50ms timeout to stop it", elapsed)
+			}
+		})
+	}
+}


Some tests check the same thing more than once:

The timeout messages are checked here (5 cases) and also in client_test.go TestExtractTimeoutAdvice.

The debug trace runs 3 times (lines 209-245).

"Needs an API key" is in lines 118-124 and 197-204, and in 2 rows of tests/e2e_test.go.

One case for each is enough. Not blocking.

strzibny requested review from ilyazub, jvmvik and trusche October 22, 2026 13:24

trusche approved these changes Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

[Extract] Add extract, map and crawl commands for the Extract API - #21

[Extract] Add extract, map and crawl commands for the Extract API#21
strzibny wants to merge 1 commit into
mainfrom
extract-api

strzibny commented Oct 22, 2026 •

edited

Loading

Uh oh!

trusche left a comment

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

trusche Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		`serpapi crawl URL` is a shortcut for `serpapi extract URL mode=crawl`: it sends the same request and takes the same parameters and flags.

		Starts at the URL, follows links on the same website and extracts every page it visits, one entry each in `pages`. `max_pages` caps the pages, the first included (default 10, at most 100), and `max_depth` how many links deep it goes (default 2, at most 3; `0` is only the first page).

Conversation

strzibny commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trusche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

strzibny commented Oct 22, 2026 •

edited

Loading