ZETTELKASTEN

News: EMNLP 2026 Presentation

Thrilled to share that our paper, “LLMs Learn Better In-Context from Rules than from Examples,” will appear at EMNLP 2026, and we’ll be presenting it in Budapest!

When you want a language model to pick up a new task from the prompt alone, you can describe the task’s rules in words (instruction following) or show it input–output examples (few-shot prompting). We ask which of the two works better when both specify exactly the same task.

Learning modes at test timeThe same task is given to a model in three ways: as rules (Rule Mode), as input–output examples (Example Mode), or as both (Combined Mode). Each mode leads to the same problem, f(−19, −14, −7), and the model outputs 259.Rule Mode (R)Model learns from explicit rules.RULESf(x) is composed of twoarithmetic operators.Let x, y, z be inputs.1. First multiply x and y.2. Then add z to the result.Example Mode (E)Model learns from input–output examples.EXAMPLESf(−19, 6, −13) = −127f(−19, −13, 6) = 253f(−13, 6, −19) = −97f(6, −19, −13) = −127f(6, −13, −19) = −97Combined Mode (R+E)Model learns from rules and examples.RULES & EXAMPLESf(x) is composed of twoarithmetic operators…f(−19, 6, −13) = −127f(−19, −13, 6) = 253…Problemf(−19, −14, −7) =OUTPUT259
The same task can reach the model as rules, as input–output examples, or as both, and all three lead to the same test problem.

We test this on five tasks spanning games, arithmetic, and linguistic inference: Set Game, Tapatan, Operator Function, Noun Class Agreement, and Lexical Category Inference. Each task comes in easy, medium, and hard versions, and we evaluate seven base and instruction-tuned model pairs from Gemma 3, Qwen 2.5/3, and OLMo 3/3.1 (7–32B).

The five tasksSet Game: pick the cards that share an attribute. Tapatan: predict the outcome from a final board, here A wins on the diagonal. Operator Function: f(x, y, z) = (x × y) + z, so f(−19, −14, −7) = 259. Noun Class Agreement: a sentence is grammatical only when every word agrees in noun class. Lexical Category Inference: with AND a candidate must meet every condition, with OR at least one.1. Set GameCard selection taskbased on attribute comparison🐇🐇🐇🐱🐢🐢🦊🐱🐢🐇🐇🐇2. TapatanTapatan game outcome predictionfrom a final board stateBABABAA winB wincontinue3. Operator FunctionArithmetic problem with a newlydefined operator function.f(x, y, z) = (x × y) + zx = −19, y = −14, z = −7f(−19, −14, −7)?259(−19 × −14) + (−7) = 2594. Noun Class AgreementGrammaticality judgment (Yes/No)in an artificial languagewith noun-class agreement.kafepdaxtizeshYesclassaaaakabimmivkasularNoclassabaa5a. Lexical Category Inference(AND)A candidate is acceptedonly if it satisfies all conditionsItem=roundANDmetallist itemroundmetalcoinYesballNo5b. Lexical Category Inference(OR)A candidate is acceptedif it satisfies at least one conditionItem=animalORvehiclelist itemanimalvehicleturtleYestrainYesrockNo
The five tasks: picking cards that share an attribute, predicting the outcome of a Tapatan board, applying a newly defined operator, judging noun-class agreement, and testing AND and OR category conditions.

What we find:

  • Rules outperform examples. Models learn more reliably from rules than from examples alone. Adding examples on top of the rules, or simply scaling up the number of examples, does not give consistent or significant gains.
  • Instruction tuning amplifies the rule advantage. It improves learning from rules (and from rules plus examples) while leaving learning from examples intact. Base models show no privileged effect of examples either, and they still benefit from rules on algebraic tasks.
  • The task matters. The gap is largest when a task recruits algebraic abstraction and computation, and smaller when it rewards distributional cues or draws on parametric knowledge.

Joint work with Seungmin Cho, Yukyung Lee, and Najoung Kim at tinlab.

To find out more:

#EMNLP2026

Backlinks

Local Graph