Sketch Engine is the ultimate corpus tool to create and search 500+ text corpora in 90+ languages. Try 30-day free trial.

This post relates an observation I've made in my work with GPT-2, which I have
not seen made elsewhere.

IMO, this observation sheds a good deal of light on how the GPT-2/3/etc models
(hereafter just "GPT") work internally.

There is an accompanying Colab notebook which will let you interactively explore
the phenomenon I describe here.

[Edit: updated with another section on comparing to the inputs, rather than the
outputs. This arguably resolves some of my confusion at the end. Thanks to
algon33 and Gurkenglas for relevant suggestions here.]

[Edit 5/17/21: I've recently written a new Colab notebook which extends this
post in various ways:

* trying the "lens" on various models from 125M to 2.7B parameters, including
  GPT-Neo and CTRL
* exploring the contributions of the attention and MLP sub-blocks within
  transformer blocks/layers
* trying out a variant of the "decoder" used in this post, which dramatically
  helps with interpreting some models

]

OVERVIEW
* GPT's probabilistic predictions are a…

Привет! Меня зовут Екатерина Ванская, и я занимаюсь data science в компании Prequel. В этой статье я продолжу обзор рекомендательных алгоритмов, который мы начали в предыдущем материале(ссылка на...