Scrape.do plugin

• Every scrape spends credits. Only some statuses are charged: 2xx and a target 404, 400 or 410 cost credits, while 429, 401 (no credits or a suspended subscription), a Scrape.do 400, 502 and 510 are free. A plain datacenter request is 1 credit; residential proxies and rendering cost more. account.info and account.asyncInfo are free.
• A target 404 is a result, not an error. scrape.html reports the target’s status in status_code and still returns the page, because the credit was spent. A target 410 behaves the same way.
• 400 means two different things. One is Scrape.do refusing the request (a bad parameter, no credit spent); the other is the target site answering with a bad request (a credit spent). A target’s 400 comes back from scrape.* as a result with status_code: 400, like a 404, so check status_code rather than catching. A Scrape.do refusal throws ScrapeDoAPIError with the provider’s message; its isTargetRejection is true only for a 400 that arrived without Scrape.do’s error envelope.
• A bad token answers HTTP 200 on /info. It returns IsActive: false with zeros rather than an error status, so account.info turns that into an error instead of reporting a dead account as healthy.
• /info is limited to 10 calls a minute and answers 429 past that. Scrape.do’s own retry mechanism is separate and on by default; disableRetry turns it off.
• A screenshot needs the browser and the JSON answer. scrape.screenshot requires render: true and always requests returnJSON, because the API refuses a screenshot without it. The images come back as base64 PNGs.
• Browser-only options are rejected here, not ignored. waitUntil, customWait, waitSelector and blockResources do nothing without render, so sending one with render: false fails before the request.
• The async API is a second host. jobs.* uses https://q.scrape.do with its own concurrency pool and its own account view in account.asyncInfo. A job takes up to 100 URLs, and its results are held until they expire, so read them with jobs.result before then. jobs.start declares a job, so an agent can read it back with jobs.get instead of polling the scrape.
• Writes are not retried. A POST through the proxy forwards its body to the target site, so a failure is reported rather than replayed: the write may have landed.
• No webhooks. Scrape.do does not send them to this plugin; its own async API can POST results to a URL you set with webhookUrl at job creation.

• 11 typed API operations

​

Setup

npm install corsair @corsair-dev/scrapedo better-sqlite3
npm install --save-dev @types/better-sqlite3

yarn add corsair @corsair-dev/scrapedo better-sqlite3
yarn add --dev @types/better-sqlite3

pnpm install corsair @corsair-dev/scrapedo better-sqlite3
pnpm install --save-dev @types/better-sqlite3

bun add corsair @corsair-dev/scrapedo better-sqlite3
bun add --dev @types/better-sqlite3

import Database from 'better-sqlite3';
import { createCorsair } from 'corsair';
import { scrapedo } from '@corsair-dev/scrapedo';

export const corsair = createCorsair({
	plugins: [
		scrapedo(),
	],
	database: new Database('corsair.db'),
	kek: process.env.CORSAIR_KEK!,
	multiTenancy: true,
	hub: {
		projectApiKey: process.env.CORSAIR_API_KEY!,
		signingSecret: process.env.CORSAIR_SIGNING_SECRET!,
	},
});

scrapedo()

const { connectUrl } = await corsair.manage.connect.createLink({
	plugin: 'scrapedo',
	tenantId: 'acme',
});
// redirect the user's browser to connectUrl

​

Example API calls

const tenant = corsair.withTenant('acme');
await tenant.scrapedo.api.scrape.markdown({ url: 'https://example.com/' });

const tenant = corsair.withTenant('acme');
await tenant.scrapedo.api.jobs.start({
	targets: ['https://example.com/', 'https://example.org/'],
});

​

What’s next