Mathematics is the part of physics where experiments are cheap Mathematics is the natural language physics is written in. The validity of a physical theory is an experimental matter - nothing to do with the math.
Here is an interesting example:
https://arxiv.org/abs/1507.06393
It's mostly just math - but there is that word - almost - in there. That almost is the existence of magnetic mono-poles which is an experimental matter - and far from easy to experimentally look for. None have yet been found - but may in the future in which case Maxwell's equations are not quite correct - however its easily accommodated in the math of the linked paper.
Another example is Lovelock's Theorem that shows in dimension 4, the Einstein equations are the unique second-order field equations generated by an action. i.e., you can derive Einstein's equations by requiring that they are generated by an action and that they are of second order in the derivatives of the metric. Normally actions contain only first order derivatives but its impossible to construct a first order one in GR, so you go to second order. The interesting thing is it turns out when you calculate the GR equations the second order derivatives in the action do not matter, and you get normal field equations that second order actions would not usually give. Now the question is why do we have actions (I will not go into why you would usually only want first order derivatives in the action)? That requires QM to explain (it follows from Feynman's path integral approach). Without the supporting experiments who would come up with QM?
The relation of math and physics is in fact quite subtle.
I do however find the ending of the link in the first post amazing - its true - but virtually never pointed out except by those like me that have read it:
'A teacher of mathematics who has not got to grips with at least some of the volumes of the course by Landau and Lifshitz will then become a relic like the person nowadays who does not know the difference between an open and a closed set.'
The first book in the series, mechanics, is simply beauty beyond compare as reviews on Amazon attest to:
https://www.amazon.com/Mechanics-Course-Theoretical-Physics-Landau/dp/0750628960
BTW it explains, amongst other things why actions only (usually) contain first derivatives.
Could this be taught at HS? Well its deep and requires multi-variable calculus but IMHO it can - generally where I am in Australia calculus is taught later than it should (for good students its taught by some schools in grade 10 - so it is possible for them - normally one waits until grade 11 and 12). Then books that authors think well prepared HS students can handle like Morin's is possible:
https://www.amazon.com/Introduction-Classical-Mechanics-Problems-Solutions/dp/0521876222
Thanks
Bill