My point is that to take "experience" as "primordial" is already to frame it as the noumenal thing-in-itself. It is already going beyond expererience to make a claim about the ontology of experience. So your claim that experience is primordially affective is a modelling claim about the structure of experience - even if a claim about some kind of lack of structure in just being this vague thing of "primordially affective".
The problem for your position is that it is incoherent if you try to eliminate the affective experiential dimension since you cannot coherently explain how there can be interpretation without some prior, or at least present, experience. — Janus
So your claim here is that there can't be conscious experience without affective experience. But note how this unpacks.
What matters is some distinction between an interpreting selfhood and then the signs of being that self. You are claiming that experience has this essential structure of a witnessing ego living in some internal world of recalcitrant affect - all these affective events that come and go of their own accord and speak to the third thing of whatever reality they in turn reflect. Some generalised realm of spirit or idea or mind, I guess.
So you speak of the "me" who stands witness, doing the interpretation. You speak of the affects, which are the percepts of stuff happening - look, there a flash of pain, there a flicker of anxiety, there a jolt of surprise. And to the degree that the witnessing self and the perception of the passing, uncontrolled or "primordial", affects are distinct from each other as interpreter and sign, together they must speak to the deeper reality that could be the intelligible cause of a play of affects. Your framing certainly points attention towards this further thing-in-itself - some noumenal world of generalised mind - even if you prefer to remain diplomatically vague about that implication.
Now if you can explain your position differently, go for it. But it seems clear that you are adopting the basic structure of a triadic semiosis, as all good psychological models will wind up doing. And so your (implied) claims about the noumenal - a realm of mind that is the cause of the affects - have to be judged on the usual pragmatic grounds. They are justified to the degree that the signs are measurements which reliably support a habit of interpretance.
So sure, if I have a pain, or hesitation, or a startle, these are all signs of something happening "out there" which have a habitual meaning for "me in here". Affect is a form of perception that feels very intimately connected to the self as a thing. It is the level of sensation that builds the sharp distinction between self and world. It is the construction of a boundary to the ego.
But that is the sensible psychological model of what an organism does to be a body with intentionality in a physical environment. And you are instead cutting away that actual world by saying "we" only have our experience. And that experience is primordially composed of affect. The physical world has now vanished from sight (well, it was always phenomenal). But in its place - because you still speak in terms of an interpretant and its signs - there still has to be a ghostly beyond of some (now "experiential") kind. The mind or spirit as a general noumenal thing-in-itself that is the ultimate ground of being.
Where does all this confusion start? Note how asserting anything about "experience" at all is already to assert a structure, as now there is also everything that experience isn't. So as soon as you talk about being "inside" the phenomenal, the noumenal "exterior" comes into play.
This is the bind that led Peirce to the logic of Firstness or Vagueness (although I agree, he didn't always quite stick to it himself). If you are going to have some foundational notion - a starting point for a developmental tale - then it can't come pre-loaded with distinctions like inside vs outside. It has to be conceived of as a "state" to which the principle of non-contradiction fails to apply.
So when it comes to the question of how "we" could have developed, we find ourselves already in that developed state - the one where there must have been a first accidental or fortuitious division followed by its hierarchical stabilisation as now a habit of interpretance.
In biology, that first primordial division would be affectless. It would simply be some biosemiotic distinction that starts a division between an organism and its world. A spherical membrane separating an inside and outside is a definite start. Membrane pores to regulate a difference between the chemistry inside and outside would be a next step. A beating flagella to mark a "conscious" difference between the direction a cell wants to go in, vs those it doesn't, is yet another.
So a lot of semiotic distinctions would get built up before we start to reach your world of a "self separated from the affects that are signs of a more generalised realm of mind". For an organism to develop the semiotic sophistication of seeing itself as a self in a world because it can sense its own boundaries as a general thing is hardly a primordial state of being. It is hierarchically complex, or recursive, already. The modeller modelling itself now.
Thus again, note how you attempt to slap the label of "experience" across everything in sight as a way to flatten all the semiotic complexity. Sure, the Peircean approach agrees that we start stuck inside our own highly developed phenomenology. We can't escape our "experience". In that sense, it seems a primordial condition.
But we can pay attention to the logical structure of that experience. And we can see that the perception of affects requires a higher order of recursive modelling complexity than the perception of "the world outside". To see our affects as affects requires that we don't just hold a model of the world, we hold a model of us as selves in that world having an experience of ourselves as phenomenologically bounded beings. We are so aware of the modelling relation that we feel aware of having to make a choice about which of two worlds - the ideal or the real - that we actually exist "primordially" in.