I'm building GigsPaid , a board for paid coding gigs. It pulls listings from Freelancer, RemoteOK, We Work Remotely, Remotive, HN "Who is hiring" and GitHub bounties every minute. The same job often shows up on two sites, so there's a near-duplicate filter in front of the database.
After launch the board looked dead: 18 listings. Each scan was fetching around 340 posts.
So I replayed one scan into an empty database. 174 tech gigs went in. 8 came out.
The hash
Here's the simhash I shipped. I'll own it:
function simhash ( text ) { const tokens = text . toLowerCase (). split ( / \s +/ ); const v = new Array ( 64 ). fill ( 0 ); for ( const tok of tokens ) { let h = 0 ; for ( let i = 0 ; i < tok . length ; i ++ ) h = ( h * 31 + tok . charCodeAt ( i )) >>> 0 ; for ( let i = 0 ; i < 64 ; i ++ ) v [ i ] += ( h & ( 1 << ( i % 31 ))) ? 1 : - 1 ; } return v . map (( x ) => ( x >= 0 ? ' 1 ' : ' 0 ' )).…