I wanted to understand vision language models (VLMs), the AI that looks at a photo and answers questions about it. So I started with a tiny grid of numbers and ended up quizzing an AI about two sleeping cats.

Everything below runs free in Google Colab.

An image is just numbers

import numpy as np

tiny = np . zeros (( 8 , 8 )) tiny [:, 4 :] = 9 # left dark, right bright print ( tiny )

Enter fullscreen mode

Exit fullscreen mode

Output:

[[0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.] [0. 0. 0. 0. 9. 9. 9. 9.]]

Enter fullscreen mode

Exit fullscreen mode

Every row is 0 0 0 0 9 9 9 9 . There's an edge, but the computer needs a way to find it.

A filter finds the edge

kernel = np . array ([ - 1 , 0 , 1 ]) # right pixel minus left pixel out = np . array ([[ np .…