Hi everyone! I've been experimenting with an alternative Transformer architecture called GAP (Gated Attention Projection) . The main idea is to combine attention and MLP-style processing into a single gated block, instead of using the standard separate…
Anonymous readers can preview up to 1024 characters here. Log in to unlock the full article once ingest succeeds.