What is the callback functionality?

While some customers prefer to poll our web servers to check the status of a job after adding it, a less memory intensive option is to supply a callback URL when you add a job, which we automatically POST the same response to upon completion of the job. This avoids you having to poll our system, and you get the result as soon as it's complete.

You can also set the external_identifier parameter, which is included in the POST that we send you to help you identify which resource the payload is associated with.

Note: callbacks are only sent when jobs become complete or invalid_url, and not for other statuses (eg. maintenance).

Using the callback functionality is also helpful in cases when our team detects an anomaly with certain jobs and re-sync those - it allows our system to POST updates for a specific job to your system automatically. 

As an example of an anomaly, consider that we're scraping reviews from Yelp. We run your jobs through and sync its reviews, but then notice that there has been an update in Yelp markup, so we issue a fix. Depending on the severity of the update, our team may decide to re-sync affected jobs to ensure data integrity.

Failed callbacks

If the first callback fails (ie. you don't respond with a 2xx HTTP code), our system will automatically retry the callback 2, 6 and 24 hours later. No further retries are made after these 3 attempts.