The point is that Wittgenstein is not interested in explaining the "source" of regularities in nature. It's taken as a brute fact that they exist. Not only he's not interested in such "explanations", he seems to think that it's not philosophy's business in general. If the source of natural regularities is an empirical question, it's science's business; if it's not an empirical question, most probably it falls within the domain of a mystery which many try to explain by another mystery or: "a nothing could serve just as well as a something about which nothing could be said".

I know this is highly contrary to Wittgenstein, but that is my point. Math works not because it "has to work" in the internal logic, when it is applied to empirical evidence and technology. It works, because there is something about how it is describing the very patterns that we initially used to recognize more practical or immediate situations in our development. — schopenhauer1

I don't think this is contrary to Wittgenstein. Natural regularities constrain human behaviour, so human behaviour presents its own regularities. Math rests on these. But it's of a different order than these: "Here we see two kinds of responsibility. One may be called "mathematical responsibility": the sense in which one proposition is responsible to another. Given certain principles and laws of deduction, you can say certain things and not others. - But it is a totally different thing if we ask, "And now what's all this responsible to?"

The last part of this quote seems like an allusion to another distinction: "We must distinguish between a necessity in the system and a necessity of the whole system".

There are necessities within our mathematical system, but the system itself is not necessary. I don't see him as a conventionalist either regarding math or logic (or, I should say, especially regarding logic): "it has often been put in the form of an assertion that the truths of logic are determined by a consensus of opinions. Is this what I am saying? No". Given the world we live in, that's the math we can have. There's not a whole lot to say about "why this world" though. I take him to hold that (in its metaphysical depth (or shallowness rather)) this is a nonsensical question which produces nonsensical explanations.

Oskari Kuusela gives his interpretation of Wittgenstein in his latest book. My understanding is as follows. Based on Wittgenstein's own remark that he had made contributions to logic, he tries to show that Kantian readings of him* can't explain this fact, even if they avoid the problem of armchair empiricism of which Wittgenstein was accused of but he himself denied that he was undertaking. Wittgenstein's preoccupation with logic was based on the work of Russel, Frege and his own TLP. Later, he rejected calculus-based methods, but it's not clear what his contributions to logic are if his work on it amounts to this rejection. Also, according to Kuusela, under kantian readings of Wittgenstein, grammatical statements constitute philosophical theses, which Wittgenstein famously denied.

So, Kuusela argues that Wittgenstein's contribution wasn't just a negative one. He takes Wittgenstein to hold, throughout his life, a conception of philosophy as a logical investigation. TLP, while preserving the basic assumptions of Frege's and Rusell's approach to logic, was trying to fill the gaps and solve the difficulties that this approach faced. Later on, he came to see as problematic the notion of a universal logical calculus or that logic's non-empirical status could be explained by thinking of propositions as being abstract entities. Kuusela claims that Wittgenstein continued to hold logic as a non-empirical discipline, even though it (i.e. logic) is able to take into account empirical facts about language users and their environment.

He sees Wittgenstein as trying to extend logic beyond calculus-based methods, by introducing alternative logical methods such as grammatical rules, language-games and a "quasi-ethnology". Kuusela takes Wittgenstein's late contributions to be a hybrid between "ideal" and "ordinary" language philosophies. In the sense that he still maintained a basic article of Russel's approach, according to which philosophical problems are primarily logical and are solved by logical investigations, while, at the same time, extending logic beyond calculus-based methods and into "ordinary" language-games, grammar etc. In that, Kuusela argues against Russell and others who viewed Wittgenstein's later thought as a curious kind of empirical linguistic anthropology and as an abandonment of his work on logic.

So, ultimately, under Kuusela's reading, Wittgenstein's late philosophy tries to fill in the gaps in what he took to be an impoverished conception of logic. Logical calculi preoccupied with grammatical form are useful in certain contexts but may not be as useful in others and certainly they are not all there is to logic. They are part of it. Wittgenstein's own methods, such as language-games, are different parts of it and the bulk of PI's investigations are examples of cases where the employment of logical methods such as language-games might be preferable to calculus-based methods.

There might be problems, philosophical or otherwise, where the idealizations of logical calculi do a good enough job. But ideal languages with fixed and precise rules are simplified descriptions of something far more complex and open-ended. Natural languages are not as simple as ideal languages, nor are they governed by such fixed and precise rules. Wittgenstein argues that some philosophical dead-ends are reached precisely because we're not using good enough logical methods and we end up describing our concepts as simpler than they really are.

Logical necessity, which makes no exceptions, is not explained solely by rules and conventions, they are not the source of necessity, even though language is an evovled spatio-temporal phenomenon for Wittgenstein. Kuusela names Wittgenstein's attempt to do justice both to empirical facts and logical necessity, non-empiricist naturalism. Empirical generality cannot account for logical necessity and universality, but empirical facts are nevertheless not irrelevant to logic. Regarding Wittgenstein's discussion of pain expression in the PI, Kuusela writes:

"Instead of using a rule or a set of rules as a mode of representing language use, §244 describes an aspect of language use by means of a natural historical picture or model. Importantly,this involves construing the notion of language use more broadly than as rule-governed use—which also throws light on the sense in which Wittgenstein’s methods do not involve a commitment to a theory or thesis about language use as rule-governed, or that it is always possible to describe language in terms of rules. Rather, the notion of rule-governed use is merely one of several related notions of the use of language that Wittgenstein employs"

With respect to how the use of "natural historical pictures" or "empirical facts" establishes logical necessity without collapsing into empiricism, Kuusela explains that:

"Accordingly, insofar as the employment of calculi and grammatical rules consistently with Wittgenstein’s method does not involve a collapse into empiricism, neither does the employment of natural historical pictures. The explanation why is now easy to state: none of these different kinds of clarificatory devices is used to make empirical statements, when employed for the purpose of logical clarification. The difference of the use of natural historical pictures in Wittgensteinian logic from empirical assertions can be further clarified with reference to certain formal features of the use of natural historical pictures, namely their manner of justification and their generality."

Wittgenstein's model is justified, not because it corresponds with empirical facts, but due to its clarificatory power, which makes comprehensible the object of inquiry (pain) without producing conundrums that other models produce. For example, Wittgenstein's model does not have to deny the possibility of knowing other people's sensations, like the model which takes sensation-language as naming inner states.

Since Wittgenstein is not concerned with empirical facts, he does not need to refer to a certain space or time when he brings up his examples. As devices of logical clarification, these examples are universal and necessary, just like logic. The gain here is that it manages to clarify language without having to postulate abstract entities (e.g. ideal languages) to which our natural language must conform to get it right. Wittgenstein turns the classical account on its head. The classical account just ignored the way language is actually used and sought to find the ideal which would dictate proper usage. Wittgenstein takes into account the way we talk in order to show the logic behind it, its grammar, by comparing language with calculi or games according to fixed and exact rules.

By employing these means of idealization in logic, we bring into focus and clarify certain aspects and uses of language which account for specific problems. But what these idealizations help us figure out does not hold just for this calculus or that language-game, it is true of our natural language which is the object of inquiry. That's a difference between logical and scientific modelling. In science, models are approximations of its object (nature or reality) in a way that logical models are not such approximations of its object (language). As Wittgenstein puts it:

"But if you say that our languages only approximate to such calculi you are standing on the very brink of a misunderstanding. For then it may look as if what we were talking about were an ideal language. As if our logic were, so to speak, a logic for a vacuum.—Whereas logic does not treat of language—or of thought—in the sense in which a natural science treats of a natural phenomenon, and the most that can be said is that we construct ideal languages. But here the word "ideal" is liable to mislead, for it sounds as if these languages were better, more perfect, than our everyday language; and as if it took the logician to shew people at last what a proper sentence looked like"

When the scientist abstracts away certain features of reality to build her model, she leaves something out. What she presents to us now is not reality, she's not making ontological claims, anymore than cartographers do. Are maps ontological statements? Idealizations in science are methodological choices. Ideally, science would like to produce ever more accurate approximations of reality until they're not approximations (not that this is possible though). On the other hand, there's no such need or aim in logical modeling. According to Kuusela:

"For the descriptions of logic in idealized terms are not merely approximate clarifications in the absence of more proper clarifications. Rather, clarification by means of ideal languages constitutes a particular method for resolving philosophical problems"

"As outlined, unlike science logical clarification does not ultimately aim at a comprehensive non-idealized account of its objects of study. Due to their problem-relativity logical clarifications can remain idealizations, as long as they account for whatever is relevant for the problems at hand."

Or, in the words of Wittgenstein:

"Just as a judge treats certain cases as paradigms, so to speak as ideal cases, so too we construct ideal cases, grammatical pictures, in order to secure different perspectives in cases of philosophical dispute and to settle the conflict. We wish to investigate language solely from the point of view of a procedure governed by definite rules, under such an aspect. To a certain extent the method is similar to the one proposed by Boltzmann: describing a physical model, for instance a model of Maxwell’s equations, without making any claim that it conforms to something else. Rather, it is simply described, and then the resemblance will become evident to us. The model is none the worse for this. It is a thing in its own right, and it serves a purpose as well as it can. What Boltzmann accomplished by this means was a kind of safeguarding of the purity of his explanations. There is no temptation to falsify reality, but the model is, so to speak, given once and for all, and it will itself show to what extent it is correct. And even where it does not do so, it does not thereby lose its value.It is in this sense that one can say: We have no system. That is, there is no possibility of another’s agreeing or disagreeing with us; for we really indicate only a method. It is as if Boltzmann’s model were simply placed beside the phenomenon of electricity and someone said: ‘Just look at that!’."

---

* according to which "grammatical statements articulate conditions of intelligibility for the employment of concepts, clarifying what is necessarily assumed in their use and what their possible uses are."