Your question as to why the rehabilitation model which is adopted with people who self harm is not applied to those who harm others touches upon debate within forensic psychiatry. I have worked in the mental health system and it is a complex matter differentiating between whether a person has harmed others as a result of underlying mental illness or not.
In many cases, it is clear that a person who has killed another has done so on account of being unwell mentally. In particular, if there are obvious signs of psychosis, such as the person hearing voices. The person may have command hallucinations to kill someone. Or, they might have paranoid delusions which lead to the person to attack and kill another.
In some cases, the signs of mental illness may not be picked up initially. For this reason, there is usually some input from psychiatrists within prisons to try to ensure that the underlying signs of mental illness are not missed. This is about looking for psychosis but other mental illnesses too.
One important area within psychiatry relevant to your question is the whole idea of personality disorders,The label is in itself contentious and can be analyzed critically from a sociological or philosophical angle. However, for the current purpose I am speaking of the idea of personality disorders as depicted within the psychiatric model. There are various recognised personalities disorders or traits, and it could be seen as a spectrum. However, two differences are between the idea of borderline personality disorder and antisocial personality disorder.
The main difference between the borderline personality disorder and antisocial personality disorder is based on the extent to which the person has difficulty with the self or others. The individual with a borderline personality disorder often has difficulty with emotional regulation, suicidal and self harm intentions, and difficulties with impulsive behaviour. These people are often the ones who present in acute psychiatric admissions.
The people with antisocial personality disorder are often those within the forensic psychiatric institutions, such as Broadmoor. I have a little experience of working within forensic psychiatry, but more in step down services for individuals at a later stage of recovery but I would not feel that I have sufficient knowledge to speak with any authority on the subject of antisocial personality disorder. The one thing which I would say is that even within forensic psychiatry is that there is a lot of overlap between recognised disorders in the persons who present. Some people have a history of self harm and harming others. Also, some people have a mixture of psychosis, alongside other difficulties.
One main element which is of critical importance in determining whether a person should be in prison or in a mental health establishment is whether or not they are seen as suffering from a treatable mental illness. Usually, this involves medication alongside psychological therapies. I do not think that the differentiation between mental illness, or lack of it, in a person who harms others is absolute and I have simply tried to sketch an outline of this very complex area within psychiatry.