Very good point. My gut feeling is that before settled agriculture, population density was so low that it wouldn’t have been an issue. Afterwards, of course, you need armies to defend your grain.