Service eligibility and regional restrictions
PuppyIP serves only compliant overseas businesses and their authorized personnel. Proxy services are not available in mainland China. The service may only be used for lawful business activities outside mainland China. Use of this service within mainland China is prohibited.
Hosting a proxy IP or server overseas does not change these restrictions. The service must not be provided to end users in mainland China through relaying, forwarding, sharing or resale. Before use, read the Terms of Service.
Key Takeaways
- The POST response gives you a job ID; retrieve the records with GET after the asynchronous job finishes.
- Job status describes the crawl run, each record's URL status describes that URL, and outcome gives the processing result; older records may omit outcome.
- The robots array reports the first 50 origins encountered by the crawl, not a complete inventory of every origin.
- Check robots.txt, Content Signals, crawl filters, and account limits without trying to bypass CAPTCHA, Turnstile, or site eligibility rules.
Keep the three status layers separate
Browser Run /crawl is asynchronous. Its creation POST responds immediately with a job ID for later requests; that acknowledgement does not mean the crawler has fetched page content or that every URL will return content. A GET for the job ID reports the crawl's overall status: running, completed, errored, cancelled_by_user, cancelled_due_to_limits, or cancelled_due_to_timeout.
After retrieving results, inspect each URL's status in records. Documented URL states include queued, completed, disallowed, skipped, errored, and cancelled. Job status describes the whole job, URL status describes one address, and outcome explains the processing result. A completed job does not establish that every page succeeded.
Retrieve the complete result before diagnosing missing pages
For polling, add limit=1 to the GET request to receive a lightweight status response. Once the job reaches a terminal state, request the results again without that parameter. If a response exceeds 10 MB, Cloudflare returns a cursor for pagination; continue with cursor, use limit to control the number of records, or filter by status to inspect a URL state.
The documentation says a crawl can run for up to seven days and completed results remain available for 14 days. When checking an earlier job, verify the job ID, terminal state, and pagination before concluding that pages were missed. Results cannot be retrieved after their retention period, and the initial POST acknowledgement is not a substitute for the final result.
Use outcome to identify why an individual URL was processed
URL status is the state; outcome is the processing reason. The current documentation lists success, robots_disallowed, crawl_purpose_disallowed, content_use_disallowed, connection_error, timeout, browser_unavailable, format_generation_error, unsupported_content_type, unclassified_error, page_error, and too_large. These point to success, robots or Content Signals denials, connection or timeout problems, unavailable browsers, format-generation failures, unsupported response types, page errors, or oversized content.
The November 3, 2026 update added an optional outcome field to URL records, but older records may not contain it. A missing outcome is not proof of success; use the available URL status, response information, and returned content to assess older results. Filter HTTP-error records with status=errored, then review the response information included with each record.
Read the robots array as origin-level diagnostics
The top-level robots array summarizes robots.txt checks for the first 50 origins encountered by the job. Each entry includes an origin and a state: available means usable rules were obtained, absent means usable rules were absent, and fetch_failed means fetching the rules failed. The array is bounded and may not cover every site encountered later in the crawl.
An entry includes status when the crawler received an HTTP response while requesting robots.txt. That field is omitted when no HTTP response was received. fetch_failed and absent do not mean the same thing as an explicit rule denying a URL. Use the individual URL's disallowed status or robots_disallowed outcome to investigate a rule-based denial; do not equate a failed fetch with a robots prohibition.
Respect robots.txt, Content Signals, and site protections
Browser Run follows a target site's robots.txt directives, including crawl-delay, and lists URLs it is directed not to crawl with disallowed status. It also respects Content Signals in robots.txt. By default, crawlPurposes includes search, ai-input, and ai-train, while contentUse defaults to full. A crawl can be rejected with a 400 response at creation if the site rejects a declared purpose or permits a less permissive use level than full.
Declare a narrower crawlPurposes value only when that genuinely reflects how you will use the content, and set contentUse at or below the site's declared level. Content Signals are a declaration to the site owner, not a way around its terms. Browser Run does not bypass CAPTCHAs, Turnstile, or other bot protections; the target site's controls still apply.
Check current account limits when a job is cancelled
Workers Free has additional /crawl limits: up to five crawl jobs per day and 100 pages per crawl. Browser Run usage is also subject to a 10-minute daily browser-time limit on Free, resetting at the UTC day boundary. Job count, pages per crawl, and browser time are separate limits. If you see cancelled_due_to_limits or a rate limit, compare the plan, job usage, and relevant limit instead of relying on one error code.
The pricing page charges Quick Actions, including /crawl, for browser hours; browser-hour usage is shared across Browser Run methods. Browser Sessions, which provide direct browser control, also have concurrent-browser charges. Workers Paid limits differ from Free, so check the current plan and usage. JSON extraction uses Workers AI by default and incurs separate AI usage. No real crawl or billing test was performed for this guide, so it does not establish the cost of a particular job.
Sources
Frequently Asked Questions
What should I keep when opening a troubleshooting ticket?
Keep the job ID, query time, relevant URL status and outcome, the origin-level robots state, and any HTTP status actually returned. State whether all cursor pages were retrieved and which result filters were applied. Remove API tokens and authorization headers before sharing records so missing results, processing failures, and policy denials can be discussed separately.
Does setting the creation limit to 100000 let Free crawl 100,000 pages?
No. The general job-creation limit parameter defaults to 10 and allows a maximum of 100000, but it does not expand plan entitlements. Workers Free still has limits of 100 pages per crawl, five jobs per day, and 10 browser minutes per day. The creation page limit is also different from the limit used to paginate GET results.
Does one timeout or page_error prove that the exit IP caused it?
No. These outcomes describe processing results for a URL; they do not establish the network path or exit IP as the root cause. Review the same record’s response information, site rules, and job configuration before investigating further. Changing an IP does not override site eligibility or bot protections.
Are all failed URLs free of charge?
Do not assume that. The pricing page specifically says a browser session is not charged when a Quick Actions request fails with a waitForTimeout error. That statement does not establish that every timeout, page_error, or failed outcome is free. Check actual usage and current billing terms; this guide did not verify a real bill.