Denver Data Design home (opens in a new tab)Denver Data Design home (opens in a new tab)

AI Impact Lab

Understand AI through probability, sampling, and experiments

AI systems generate responses one prediction at a time. Every word in a prompt shifts the probabilities behind those predictions. Learn the core ideas, then run three experiments on a real model to measure how prompts steer responses: which words carry weight, how specificity changes consistency, and which lines carry weight.

Learn the patterns behind AI responses

Explore prediction, uncertainty, sampling, and word weight before running the experiments.

Key idea 1

Tokens: the pieces a model reads and writes

A model doesn't see words. It splits text into tokens: common words stay whole, and uncommon words split into common pieces. Every prediction in this lab is a prediction of the next token.

Learning objective

A model writes one token at a time. Each token is a separate prediction. Uncommon words split into several tokens.

Tokens count how much text there is.
Bits measure how undecided the model is about which token comes next.

7 words → 10 tokens. summit is 1 token, and breathtakingly is 3. A dot (·) marks a space.

Each bar shows bits (Key idea 3): how undecided the model was before picking that token.
summit (8.2) could be almost any noun; ly (0.2) is nearly forced.

Illustrative: splits and bits vary by model. English averages about ¾ of a word per token.

Calculating bits for the token after breath

Surprise is how unexpected an option would be: the rarer the token, the higher the value. Add up surprise × probability for every option to get the bits.

Next tokenpSurprisep × surprise
taking80%0.3220.8×0.322 = 0.258
less15%2.7370.15×2.737 = 0.411
ing5%4.3220.05×4.322 = 0.216
Bits for this choice (sum)0.885 ≈ 0.9 bits

More on these calculations below.

In the lab: Experiment 1 shows which words of your prompt steer the model's predictions, one token at a time.

Key idea 2

Prediction and probability

When completing a sentence, the model gives every possible next word a probability (0–100%, summing to 100%). It picks one, adds it to the text, and predicts again. A response is a chain of these predictions. A setting called temperature controls how adventurous each pick is.

In the lab: Experiment 1 runs at T = 1 and shows the spread of your responses. Experiment 2 measures how much a crafted prompt steers them.

Key idea 3

Entropy and uncertainty

Entropy is how undecided the model is: low when one word clearly wins, high when many words are about equally likely.
Measured in bits: 1 bit is a coin flip (0 or 1), and each extra bit doubles the number of equally likely options.

Learning objective

Bits count the model's indecision. 0 bits means one clear answer, and each extra bit doubles the options in play.
Brainstorming wants more bits. Facts and code want fewer.

Choices = 2bits. At 0 bits, one word has 100%: 1 × log₂ 1 = 0, and 2⁰ = 1 choice.

Formula: H = − Σ p · log₂ p. Multiply each word's probability by how surprising it would be, then add them up: entropy is the average surprise.

Calculating entropy

Entropy is the average number of yes/no questions needed to make a prediction.

Next wordChanceQuestions asked#Chance × #
cold50%cold?10.5×1 = 0.5
quiet25%cold? quiet?20.25×2 = 0.5
still12.5%cold? quiet? still?30.125×3 = 0.375
misty12.5%cold? quiet? still? (no → misty)30.125×3 = 0.375
Entropy (sum)1.75 bits

When is higher or lower better?

Higher is better
Brainstorming, poems, ideas
Key objective: variety
Lower is better
Facts, instructions, code, a set format
Key objective: precision and consistency

In the lab: Experiment 1 estimates entropy from your own samples. Low entropy means the model is confident, not correct: a vague prompt can get a confident, generic response (see Experiment 2).

Key idea 4

Sampling and variation

Hosted models like Claude and ChatGPT don't show their probabilities. The experiments sample instead: ask the same thing several times and look at the spread. Each sample is one draw from the model's hidden prediction.

Temperature matters: at T = 0 every sample is nearly the same, and there's almost no spread to see. The experiments use T = 1, the usual default, to show the model's natural spread.

In the lab: every experiment sends the same prompt several times and compares the samples. Experiment 1 turns their spread into an entropy estimate.

Key idea 5

Weight and influence

A word's weight is how much it shapes the response. Removing a word shifts the model's hidden probabilities. To see the shift, we sample the prompt with and without the word and compare the responses. In the experiments, weight is drawn as font weight: the bolder the word, the more it steered.

Formula: weight = consistency(with the word) − consistency(without it). Removing a heavy word changes the responses a lot and consistency drops. A filler word barely moves them.

In the lab: Experiment 3 removes one line at a time to rank your lines by weight.

Start the AI Impact Lab

Each experiment follows the same steps: Set up → Predict → Run → Observe → Reflect.
Lab takes approximately 30 minutes; longer if you repeat experiments with different prompts and models.

Step 0

Load your model

Choose a live provider to run the experiments with your own API key, or choose DEMO to explore a guided example with pre-recorded sample data and no API calls.

Responses run at T = 1, the usual default, to show the model's natural spread. The graders run at T = 0 for consistent scores.

Skip current events. The model is limited to training data within the lab and does not have web search capabilities.

Learning objective

The model you choose sets the price. Providers bill per token, and rates differ significantly between models.

Cost to run all three experiments

  • Anthropic
  • OpenAI
    Pricing as of November, 2026
    1 · Choose a provider
    Experiment 1

    Every word is a prediction

    The model predicts one word at a time, and the tokens of your prompt steer those predictions. Goal: visualize which words carry weight and see the spread of the model's predictions.

    1 · Write your prompt
    2 · Predict
    3 · Run
    Make your prediction above to unlock.

    Each experiment result is a new draw from the AI model; results will vary.

    Make your prediction to start.
    4 · Explore

    Which words steered the model

    After the run, bolder words show which words changed the responses most.

    How much the responses varied

    After the run: how the responses began, measured by the first three words of each.

    5 · Observe

    Run the experiment to see your result.

    6 · Reflect
    1. Which words in your prompt are the boldest? Are they the ones you expected to matter?
    2. How many different ways did the model begin its response? What does that tell you about how undecided it was?
    3. If every word is a prediction with other options close behind, what does that suggest about asking the same question twice?
    Experiment 2

    Steering the response: quick vs. crafted

    Write the same request two ways. Goal: measure the effect of carefully crafted prompts.

    1 · Write your prompts
    2 · Predict
    Where will each prompt's responses land on the chart?

    Left to right: consistency. Bottom to top: fit to your intent.

    3 · Run
    Make your prediction above to unlock.
    Set your prediction, then run.
    4 · Explore

    Consistent doesn't mean correct. A vague prompt can get the same generic response every time (bottom right). The ideal (top right) is responses that agree and match your intent.

    Consistency vs. fit to intent: where each prompt landed

    5 · Observe

    Run the experiment to see your result.

    6 · Reflect
    1. How consistent was the quick, generic prompt?
    2. Open the responses. Can you trace the crafted responses' details back to something you wrote?
    3. Rewrite your crafted prompt to be even more specific. Does fit to intent go up? Does consistency?
    Experiment 3

    Which lines of your prompt carry the weight?

    Remove one line at a time and grade the new responses. If fit drops, that line was steering.
    Goal: rank your lines by importance, then test your ranking.

    1 · Set up your lines

    Your lines come from your Experiment 2 crafted prompt, or create a different prompt. Long paragraphs are split into sentences automatically.

    2 · Predict: drag your lines into order of importance

    Most important on top. Drag a card, or use its ↑ ↓ buttons. Click any card's text to fix it if it was split oddly.

    Choose where your lines come from above.
    3 · Run
    Load your lines above to unlock.
    Rank your lines, then run.
    5 · Reflect
    1. Compare your order with the measured order. Where did they disagree most, and why might the model weigh that line differently than you did?
    2. Did any line come out red, beyond the noise? Open its responses: how did that line pull the model away from what you meant?

    Wrap-up

    Save your predictions, results, charts, and notes as a PDF, for example to hand in or discuss. In the print dialog, choose Save as PDF.

    • Results come from a few samples per test and vary from run to run. More samples give steadier results but cost more. Run experiments again to observe spread.
    • The experiments measure how responses change, not what happens inside the model. A heavy word changed the responses; the why in AI is hidden.
    • Try running the lab again with a different model selected to see how responses vary.
    The math

    The patterns underneath

    Every measurement in this lab compares responses from the outside. Hosted models don't expose their token probabilities or attention, so the results are behavioral estimates, not readings of the model's internals.

    Symbols used on this page

    πpi · true probabilityThe population parameter: the true probability inside the model. It’s hidden: we can’t see it directly.
    p̂p-hat · our estimateThe sample statistic: our estimate of π, from 0 to 1, measured by sampling. More samples bring p̂ closer to π.
    ℙ (A | B)conditional probabilityThe chance of A, given that B already happened.
    ΣsigmaAdd up everything that follows.
    ∏productMultiply everything that follows.
    eEuler’s number≈ 2.718, used by softmax to turn scores into probabilities.
    lnnatural logThe inverse of eˣ: ln(eˣ) = x. ln e = 1; ln 1 = 0.
    ∝proportional toGrows in step with.
    Sources

    Citations and further reading

    This project's design and code are original. It builds on the published ideas and research below. Each entry notes where it's used.

    Foundations

    1. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3). Entropy and bits (Learn, Experiment 1).
    2. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. Softmax with temperature (Learn: prediction and probability).
    3. Penn State Department of Statistics. STAT 200: Elementary Statistics, Lesson 8.1.1 "Confidence Intervals" (one-sample proportion). Sampling variability and the margin of error for a proportion (cost estimation).

    Methods used in this lab

    1. Zeiler, M. D., & Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. ECCV. Occlusion: remove part of the input and measure the change (Experiments 1 and 3).
    2. Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024). ContextCite: Attributing Model Generation to Context. Attributing a model's response to parts of its input by removing them (Experiments 1 and 3).
    3. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630. Estimating a model's uncertainty by sampling many responses (Learn, Experiment 1).
    4. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS. Using a model as a grader, and its limits (Experiments 2 and 3).

    Inspiration

    Tools and data

    Further learning

    AI Impact Lab

    lab.denverdatadesign.com

    AI Impact Lab · runs entirely in your browser; your API key never leaves this tab except to call your chosen provider.