Menu

My duplicate filter decided 166 of 174 job posts were the same post
📰

My duplicate filter decided 166 of 174 job posts were the same post

DEV Community: webdev·Ruturaj Sonkamble·9 days ago
#K4NI8JOA
#devtowebdev#devto#dev#bits#hash
Reading 0:00
15s threshold

I'm building GigsPaid , a board for paid coding gigs. It pulls listings from Freelancer, RemoteOK, We Work Remotely, Remotive, HN "Who is hiring" and GitHub bounties every minute. The same job often shows up on two sites, so there's a near-duplicate filter in front of the database.

After launch the board looked dead: 18 listings. Each scan was fetching around 340 posts.

So I replayed one scan into an empty database. 174 tech gigs went in. 8 came out.

The hash

Here's the simhash I shipped. I'll own it:

function simhash ( text ) { const tokens = text . toLowerCase (). split ( / \s +/ ); const v = new Array ( 64 ). fill ( 0 ); for ( const tok of tokens ) { let h = 0 ; for ( let i = 0 ; i < tok . length ; i ++ ) h = ( h * 31 + tok . charCodeAt ( i )) >>> 0 ; for ( let i = 0 ; i < 64 ; i ++ ) v [ i ] += ( h & ( 1 << ( i % 31 ))) ? 1 : - 1 ; } return v . map (( x ) => ( x >= 0 ? ' 1 ' : ' 0 ' )).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.