Menu

I merged attention and MLP processing into a single gated Transformer block. GAP [R]

Reading 0:00
15s threshold

Hi everyone! I've been experimenting with an alternative Transformer architecture called GAP (Gated Attention Projection) . The main idea is to combine attention and MLP-style processing into a single gated block, instead of using the standard separate…

Anonymous readers can preview up to 1024 characters here. Log in to unlock the full article once ingest succeeds.