The Missing Half of Agent Memory