The natural numbers are often defined in terms of their role as ordinal numbers. This is, for example, how ZFC defines them. In earlier work I have suggested that we may have a notion of cardinal number that cannot be reduced to that of ordinal number in this way (Buijsman, forthcoming). I here extend these findings by presenting two new arguments in favour of the claim that the natural numbers are not fundamentally ordinals. I hope to thereby establish, on the best available empirical evidence, that a philosophy of mathematics which aims to describe our actual use of numbers has to make room for an autonomous notion of cardinal number. In the second half of the paper I consider what implications this has for prominent platonist and nominalist accounts, finding a number of constraints on which accounts can accurately describe our use of number. This extends the earlier work, which only considered structuralism and neo-logicism.