Well, I wouldn't get too upset about it. Generally, to rate well on IRC, your boat needs to sail like a barge. Not a lot of sailors really enjoy sailing barges, so they get a boat that is sweet to sail, but doesn't rate that well on IRC.
There is a narrow exception, and that is designs that are optimized to IRC, and are still kinda nice to sail. I do mean 'designs' and not 'boats' that are optimized to the rule. The Sunfast is almost definitely a good example of a design that would do not bad under IRC.
As you said, you've been doing a lot of things to make your boat go faster and be nice to sail. IRC penalises that. Mainly, because it is a handicapping system. By logic, if you change your boat to go faster, the handicapping system should address that so that all boats in the fleet finish in a dead heat. i.e. if you tweak your boat to go faster, IRC will penalise you, so you still finish in a dead heat with a 4 knot sh*t box.
The idea with IRC is that a more skillful sailor (not the fastest boat) will win. If you are a skillful sailor, well prepared, and get a little bit of good luck with conditions etc, you 'should' be able to win under IRC. The logic is a gun sailor on a 4 knot sh*t box can still win a major race, so you don't need an arms race on design and sails etc. The reality is mildly different though.