Applied GenAI Curriculum for AI PMs · Depth Layer · Optional Background
Training-data realities and pre-training vs. post-training at a conceptual level — enough to explain to executives why models behave as they do, and why "just train it on our data" isn't one sentence.
This unit is optional — you can operate without it. But two conversations get much easier once you have it: explaining to a stakeholder why a model behaves the way it does (why it's confidently wrong, worse in some languages, or oddly moralistic), and gently correcting the executive who says "let's just train it on our data" as if that were a Tuesday task. Both trace back to where the model came from: the data it ate, and the two-stage process that shaped it.
Most foundation models are trained on Common Crawl — a massive scrape of the internet — or cleaned subsets of it. That means their raw diet includes "clickbait, misinformation, propaganda, conspiracy theories" and every sketchy corner of the web. Teams filter with heuristics (GPT-2 used only Reddit links with 3+ upvotes — hardly a bar for good taste), but the core truth holds: a model inherits the biases, gaps, and errors of its training data.
English dominates the internet — nearly 46% of Common Crawl, eight times the next language. "Low-resource" languages are badly under-represented, and it shows:
The consequences compound: models perform worse (GPT-4 solved English math problems 3× more often than Armenian/Farsi), "just translate to English and back" loses information (Vietnamese pronouns encoding social relationships collapse into "I/you"), and non-English text is more expensive and slower — the same meaning takes ~10× more tokens in Burmese than English, so it costs ~10× more on a per-token API. Useful to know before promising global parity.
General models are broad because their training data was broad — but they're unlikely to do well on tasks they never saw. Genuinely specialized domains (drug discovery on protein/DNA data, cancer screening on X-rays/fMRI) need curated, hard-to-acquire datasets that aren't on the public internet at all. That's why domain-specific models like AlphaFold (trained on ~100,000 protein structures) or Med-PaLM exist as distinct efforts. "Train it on our data" means acquiring, cleaning, and curating that data first — the expensive part.
A raw pre-trained model has two problems: it only knows how to complete text, not converse, and — fed the raw internet — its outputs "can be racist, sexist, rude, or just wrong." Post-training fixes both, in two steps. Read pre-training as reading to acquire knowledge, post-training as learning how to use it.
Self-supervised learning on the raw internet. Produces a model that's knowledgeable but untamed — great at predicting the next token, with no sense of conversation or appropriateness.
Finetune on demonstration data — high-quality (prompt, response) pairs written by skilled humans — so the model learns to answer "how to make pizza" rather than continue the sentence. This is "behavior cloning": show good behavior, the model copies it.
Align outputs with human preference — via RLHF (train a reward model to score responses, then optimize for high scores), DPO, or RLAIF. SFT teaches it to have a conversation; this teaches it what conversations it should have.
Demonstration data teaches the model to converse but not what to say on contested questions — abortion, gun control, immigration. The goal of preference finetuning is to make models "behave according to human preference," which quietly assumes a universal human preference exists and can be embedded in a model. It can't, fully. Respond to a controversial issue and you upset some users; over-censor and the model gets boring and drives users away. This is why models feel cautious or inconsistent on hot topics — it's an unsolved balancing act, not an oversight.
One optional-background unit remains: foundation-model evaluation concepts (Unit 24) — for parsing vendor benchmark claims when eval talk with researchers goes over your head.