Is nominalism self-defeating?

Here's an objection to nominalism I've heard a few times.  Sometimes the 'access problem' to abstract objects is motivated by the idea that embodied creatures adapted to a particular environment, such as ourselves, need to interact with the world in order to gain knowledge of what it's like.  If we're to learn something about a given domain of objects, at some point we will require some kind of causal interaction with at least some members of that domain of objects.  Abstract objects, such as mathematical objects, are not like this: there is no method by which we could interact with anything in a domain of abstract objects at any time.  As a result, even if abstract objects exist, there is no means by which we could come to gain knowledge of which abstract objects exist or what properties they have.  This kind of minimal causal condition for knowledge is sometimes called the (or a) "eleatic principle".  But it's sometimes said that this eleatic principle is self-defeating.  Here's Sorin Bangu in his nice book The Applicability of Mathematics in Science (pp.18-9):

My naturalist’s reaction to the reformulated [eleatic principle] challenge is to point out that it is ultimately self-defeating.  That is, the naturalist notes that one cannot even formulate the challenge without actually making appeal to mathematics: one simply can’t grasp what the new naturalized [eleatic principle] actually says unless one understands the physical theories describing the abovementioned types of interactions.  But these theories are, of course, thoroughly mathematical!  So, anyone attempting to advance a challenge of the [eleatic principle] type in naturalistically acceptable terms finds herself engaged in the self-undermining enterprise of rejecting the very (mathematical) terms which allow the (acceptable naturalistic version of the) challenge to be meaningfully formulated in the first place.

This criticism seems wrong to me, on two counts.  Firstly, it ignores responses to the indispensability argument.  The nominalist will need some response to the indispensability argument.  If this response doesn't work then the nominalist is in trouble anyway.  If it does work—whether it involves doing without reference to or quantification over mathematical objects in scientific theories, like Field, Chihara etc., or offering some account of why it's acceptable for the nominalist to continue to refer to or quantify over mathematical objects in scientific theories, like Leng—then it will work here too: that we give mathematical models of how we (concrete) creatures interact with (concrete) parts of the world will pose no special problems.  Secondly, even in lieu of a response to the indispensability argument, the eleatic principle can be used to give a sort of reductio of mathematical platonism: (i) assume mathematical platonism is true, (ii) motivate the eleatic principle, (iii) our own mathematicized theories which describe how we interact with the world show that we cannot have knowledge of mathematical objects. So the assumption we began with is unknowable and rationally self-defeating.