Embedded Agency: Reasoning About Intelligence That Can't Step Outside Itself

Decision Points

When designing an agent system:

1. Self-reference check:
   IF agent must reason about systems containing itself
   THEN → Use embedded frameworks, expect logical paradoxes
   ELSE → Standard dualistic models (AIXI, Bayesian) may work

2. Optimization pressure assessment:
   IF pressure will be low/moderate
   THEN → Simple proxies likely stable
   IF pressure will be high  
   THEN → Plan for extremal Goodhart (edge case exploitation)
   IF pressure will be extreme
   THEN → Plan for adversarial Goodhart (active gaming) + mesa-optimizers