I see, I guess the problem is within me, mainly that I have an issue with the fact that I'm doing something (in this case choosing to not serve in the military) but I don't want others to do it too (which would result in everyone not serving and thus - a really bad outcome for everyone) and that "seems" very morally wrong to me but since I identify as a rule-utilitarian I should just invalidate that argument because it has nothing to do with the consequences or my initial intention with not serving (preventing unjustified harm to the world caused by me serving).

That's a good point, but I've thought of two counter points to this argument.
Firstly, how do you know that a military where people suddenly stop serving would equate that to the fact that people don't want unjustified harm to be caused?
Secondly, what if the military genuinely believes that the harm they're causing is entirely justified and ethical so they don't cooperate? You could argue that it's their fault, but from a consequentialist perspective if everyone doesn't serve knowing that the military might not agree to their "terms" they would still be in the wrong because the outcome of doing that would be enormous harm due to the lack of military protection.