Menu

Null means "not stated": four bugs from extracting visa, remote and hybrid flags out of 13,000 Gulf jobs
📰

Null means "not stated": four bugs from extracting visa, remote and hybrid flags out of 13,000 Gulf jobs

DEV Community: python·Rowharbor Data·1 day ago
#aKl7Wpis
#devtopython#devto#dev#visa#jobs
Reading 0:00
15s threshold

In my last post I pulled Gulf jobs from employer job feeds. The feed gives you a title, a location and a description. Everything a job seeker actually filters on (visa sponsorship, remote or on-site, years of experience) is buried in free text.

So I wrote rules to pull those fields out. The index now holds about 12,000 to 13,700 open jobs from 742 employers, refreshed daily. Here is what broke along the way.

1. "We do not offer visa sponsorship" matches "visa sponsorship"

My first visa rule was a single regex looking for phrases like visa sponsorship or sponsor your visa . It flagged jobs as sponsoring visas when the posting said the exact opposite:

Please note we do not offer visa sponsorship for this role.

The fix was ordering. I check the negative patterns first and only look for the positive ones if no negative matched:

def flag ( text , pos , neg , low , hints ) : if not any ( h in low for h in hints ) : return None # the post never mentions it if neg .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.