Menu

27.9KB language-agnostic WebGPU syntax highlighter

gpu-lexer.vercel.app·Shu Ding at Vercel Labs·about 2 months ago
#gPg2snIR
#language#lexer#returned#code#photo
Reading 0:00
15s threshold

import { parse } from 'gpu-lexer'

const spans = await parse('source code') // { // type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator' // start: number // end: number // }[]

gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language : instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the syntax spans returned to your code.

This is an experiment , not a grammar-equivalent highlighter (see the agreement chart below). The chart measures agreement with Shiki—not objective correctness—and results may vary on unseen languages or real-world code.

Highlight 10× three.min.js warmed browser time · lower is better Median of three samples on October 22, 2026.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.