Case Study: Building a Browser-Backed Careers Crawler for Beyond Tabs
Beyond Tabs is not trying to index every job on the internet.
It is intentionally narrower than that: a curated site for niche programming-language work and adjacent technical communities. That makes the crawling problem more interesting. A useful result is not merely “this company has jobs.” It is “this source might contain jobs relevant to the kinds of people who come to Beyond Tabs.”
The current challenge is the careers pipeline. Hundreds of companies have career portals. Some use familiar applicant tracking systems. Some expose a branded page over a shared platform. Some are custom sites with just a few links. Some are aggregators that should be skipped. Some only reveal the real structure after JavaScript runs, a consent banner is dismissed, an age gate is passed, or a country-specific redirect finishes.
That is the kind of work Guida was built for.
The problem was not just scraping
If every company used the same ATS, the solution would be straightforward. Detect the platform, call its API, normalize the results, and move on.
The real web is not like that.
In one triage run, a careers report contained hundreds of portals split across broad categories: known ATS coverage, dedicated company workflows, generic unknown sites, and aggregators to ignore. The “generic unknown” bucket was the uncomfortable one. It contained everything from tiny static career pages to global enterprise recruiting platforms hidden behind branded URLs.
A URL alone was not enough. A few anonymized examples from the triage work:
- A consumer brand looked like a first-party careers URL, but after an age gate it rendered a SuccessFactors/jobs2web board.
- An industrial technology company rendered an Oracle HCM career site under a branded careers domain.
- A research organization showed a polished company page, but the job links went to Lever.
- A market intelligence company embedded a BambooHR jobs widget.
- An energy-trading company listed jobs on its own page, but detail links went to HR-ON.
- A large enterprise exposed a custom careers search with thousands of results and product-specific detail pages.
Those distinctions matter because they decide what kind of crawler should be written. A shared platform should become reusable infrastructure. A first-party custom site needs a company-specific worker. An aggregator should be ignored. A source-data issue should be fixed before anyone writes code.
Why WebView2 changed the workflow
The tempting shortcut is to classify everything with HTTP requests and URL heuristics.
That would have been faster. It also would have been wrong often enough to waste time later.
Guida uses WebView2 as the automation surface, so the agent and the user inspect the same browser reality. Pages render with a normal browser engine. JavaScript runs. Cookie banners appear. Redirects happen. The user can allow domains as needed. The agent can inspect the rendered DOM, visible links, scripts, and final URL from the live tab rather than guessing from raw HTML.
For this case study, the working loop became:
- Start from the crawler gap report.
- Pick the next unclassified portals from the workspace store.
- Reuse four browser tabs.
- Apply quad layout and Full HD viewports.
- Navigate each tab to a real careers page.
- Inspect rendered content, links, scripts, iframes, and platform hints.
- Store a structured finding keyed by the portal key.
- Continue with the next batch.
The point was not full autonomy. The point was supervised automation. The user could see exactly which pages were open, where navigation stalled, and whether the agent was classifying the right thing.
The store became the working memory
The first version of the triage notes lived in text files. That did not last.
As soon as the work spanned hundreds of portals and multiple sessions, the notes needed to become queryable state. Guida’s workspace store became the working memory for the process.
The crawler report was imported into a collection keyed by a stable portal identifier. Browser-backed findings were stored in a separate collection, using the same identifier and a consistent schema. That made it easy to answer:
- Which portals are still untriaged?
- Which ones are shared platforms?
- Which ones need custom crawlers?
- Which findings came from live browser evidence?
- What did the agent see when it made the decision?
This changed the character of the work. Instead of asking the agent to remember where it left off, the process could resume from data.
The ledger matters because crawling is operational work
Career crawling is not a one-shot script. It is a workflow.
Portals enqueue job URLs. Fetch workers visit pages. Parse workers extract and index listings. Some items complete. Some skip because a known artifact already exists. Some fail transiently. Some become dead letters. Some leases expire and need recovery.
That is why the workflow ledger became important. A crawler pipeline without operational visibility leaves users guessing. The ledger gives Guida a place to explain what happened: runs, items, states, stages, events, retries, failures, and artifacts.
That pressure later drove the dedicated Workflow Ledger Console work. For Beyond Tabs, the user should not have to wonder whether the crawler is done, stuck, retrying, or silently broken. The application should show that directly.
What Guida did well
Guida made the work inspectable.
The quad layout let the user watch four real sites at a time. The MCP history pane showed tool calls. The Store pane showed structured findings. The browser tabs showed the actual rendered pages. That combination is more useful than a headless log stream because the user can intervene when the web behaves like the web.
Guida also made the work resumable. Once findings moved into the workspace store, batches could continue across sessions without relying on conversational memory or manually edited notes.
Most importantly, Guida kept the agent grounded. The agent was not classifying pages from vibes or stale assumptions. It was looking at rendered evidence and writing down why the classification was made.
What still needs work
This was not frictionless.
Navigation can still report failure even when the page actually loaded, so the workflow needs follow-up checks with the current URL and DOM. The four-tab triage loop also deserves a higher-level control: “prepare this batch, keep these tabs in quad view, and keep the layout stable” should be a first-class operation rather than a sequence of lower-level tab and layout calls.
Consent prompts and allowlisting are human-paced, which is correct, but the UI needs to make pending approvals and blocked pages clear. That product lesson has already started to show up in Guida: the MCP Approval Center makes pending approval requests visible, and the dedicated workflow and queue consoles move operational recovery out of cramped side panes.
Those are product problems, not reasons to abandon the approach. The core bet still looks right: a real browser, visible state, structured workspace data, and auditable AI tool use.
The lesson
The Beyond Tabs crawler is exactly the kind of problem that sits between manual browsing and industrial scraping.
It needs automation, but not blind automation. It needs AI assistance, but not an invisible agent roaming the web. It needs data pipelines, but also human judgment. It needs to scale, but only after the classifications are trustworthy.
Guida’s role is to make that middle ground practical.
For Beyond Tabs, that means using a browser-backed workflow to decide which crawlers to build, which shared platforms to support, which sources to skip, and which failures need attention. For Guida, it is a real case study in why visible, local, supervised browser automation is different from another headless scraper.
The long-term goal is simple: users should not feel clueless while automation runs. They should be able to see the browser, inspect the evidence, trust the stored state, and understand the pipeline.
That is the product direction this case study is helping to validate.