All posts
EngineeringProductATS

How We Crawl 500+ Company ATS Systems Every Day

The technical reality behind keeping a job board genuinely up-to-date

The Remotato TeamSeptember 30, 20265 min read

Every morning, Remotato runs a crawl across 500+ company ATS systems. By the time you open the jobs board, every listing you see was verified live — and anything that was taken down overnight is already gone.

Most job boards don't work this way. They scrape each other, buy feeds from aggregators, or rely on companies to manually post. That's how you end up with listings that are three weeks old, already filled, or outright fabricated. It's not anyone's fault — it's just what happens when you're one step removed from the source.

We decided early on that going directly to the source wasn't optional. Here's how it works.

The three ATS platforms that matter

The majority of tech and remote-first companies use one of three platforms to manage their hiring: Greenhouse, Lever, or Ashby. Each has a structured API or a consistent public-facing format for job listings.

Greenhouse exposes jobs at /careers endpoints with consistent JSON structure. Lever has a public API at api.lever.co/v0/postings. Ashby publishes structured listings through their jobs board infrastructure. All three give us reliable, machine-readable data — title, department, location, remote type, posted date, and the direct application URL.

For companies using other platforms — Workable, SmartRecruiters, and others — we use normalised crawlers that extract the same fields from the public-facing careers page HTML.

What we do with each listing

Raw ATS data is messy. A job might be listed as "Remote, USA" or "Remote - US Only" or "United States (Remote)" or just "Remote." We normalise all of that into a consistent remote_type: fully_remote, remote_us, remote_eu, hybrid, or unknown.

Similarly, department names vary wildly by company. "Product & Design," "Product Management," "PM," and "Product Team" all become Product. This normalisation is what makes cross-company filtering reliable.

We also detect and suppress noise: talent pool submissions, general applications, and expression-of-interest forms. If it's not a real, open role with a discrete hiring process, it doesn't appear.

Keeping listings fresh

The hardest part isn't fetching listings — it's knowing when they're gone. Companies don't announce when they fill a role; they just close the listing. Our crawl checks every active listing against the source system on every run. If the original URL returns a 404, a redirect to the careers homepage, or a "this position is no longer available" response, we set removed_at and the listing disappears from search results immediately.

This is why the jobs you see on Remotato are live. Not "last verified 3 weeks ago" live — actually live, checked today.

What this means for applicants

Two things, practically speaking:

  • No ghost listings. If you see it, the position exists. The company is actively hiring. You won't spend 40 minutes on an application form for a role that was filled before you started.
  • Direct application links. Every "Apply" button goes to the company's own ATS, not an intermediary. Your application goes straight into the recruiter's queue — not a third-party inbox that may or may not forward it.

The job search experience is already draining enough. We can at least make sure the listings are real.