What makes you say no to the other is your need to take a break from what you got involved with.
Some of your noes are just temporary. What is a no to someone in a certain moment might as well as be a yes in another moment.
Any noes of yours reflect a lack of will to accomplish what proves your value on the material plane, or what expresses your loyalty to the ones you care for.
What feeds your willpower is people's massage, in terms of emotional support, encouragement or gratefulness.
People's massage which turns into a trap when what your body needs is rest, whereas it's great when what it needs is the will to go after a goal and/or to keep a promise.
Saying no is an art when what you're rejecting, rather than the person itself, is the request or demand thrown at you at the wrong time, i.e. when you have no will for it.
On the other hand, saying no is a vital necessity when a certain request or demand comes from someone who shouldn't be in your life at all, because of their negative impact on you.