My notes on the Definition of Mathematics. it depends on what kind of arithmetic you have in mind. Anyhow you can of course list a complete set of axioms for arithmetic that is not recursively enumerable, this is easy take for example PA+omega rule.
As regards provability and logical truth, one needs to be careful, what Godel has proved is that some theories cannot prove all statements made in their language, so there would be statements that the theory neither denies nor proves. That's all, this is about completeness of theories. From the perspective of this account this would be phrased as: some rule following games can have statements written in their language that has no consequential truth from the rules of those games nor do their negation has, because they are neither consequences of those rules nor is their negations. Godel demonstrated that for some theory T there can be a sentence S written in the language of T such that S is equivalent to statement "S is not provable in T", so obviously if T is consistent, then T cannot prove S. Be aware that this doesn't entail that S is a consequential truth of T, no! For S can be false and T be inconsistent! When we say that S is a logical truth, this is actually mean that S is a consequential truth of the theory T + Con(T), where "Con(T)" is the statement "T is consistent". In other words S is provable in T+Con(N), that's why its said to be a logical truth, in reality it means that it is a consequential truth from the rule following game T+Con(N), notice that it is not a consequential truth of T itself. "Logical truth" is provability in some system. So it is a relative concept. In other words S is not a logical truth of T, the logical truths of T are the theorems of T only.
I don't know what you mean that all propositions other than tautologies and contradictions has Correspondence truth. Correspondence with what? with Reality? what's Reality? I think this is mistaken, those propositions has no innate truths in them, rather truth is assigned to them by the system by dictation (if they are axioms) or by being consequential truths following from the rules of the game.
As regards your second reply, my answer is YES, inconsistent rule following games are definitely mathematical as far as our study of them is about the consequential truth of them.
I didn't get your question about how to create new rules, you simply stipulate them or they are consequences of newly stipulated rules? what's the problem?