This paper discusses some aspects of the controversies regarding the operation of the agent intellect on sensory images. I selectively consider views developed between the 13th century and the beginning of the 17th century, focusing on positions which question the need for a (distinct) agent intellect or argue for its essential “inactivity” with respect to phantasms. My aim is to reveal limitations of the Peripatetical framework for analyzing and explaining the mechanisms involved in conceptual abstraction. The first section surveys developments of Aristotelian noetics and abstraction in Ancient and Arabic philosophy. The second section presents a discussion of some “positive” accounts on abstraction and the agent intellect, and some “negative” accounts.