It is really, but so is gaining.  I'm sure for most of the population, you never notice you're gaining weight.  Once you do it's 'oh i've put on a few pounds, no big deal'..  but then it keeps coming.  Then one day you see a picture of yourself, or read the scale and say "holy crap, wtf happened to me'???  hopefull you catch it before it's too late.
I don't know when I noticed I had gained what I perceived as 'too much'.  (I say perceived, as I was not unhealthy or even overweight for my build).  I noticed a gradual gain, but I was ok with it.  Then I tried on a bridesmaid dress.  It was a teeny bit loose, but I don't like to be squeezed.  It was in my closet for 3 months until the wedding.  I tried it on a few weeks before the wedding and oh shit, it's verging on *tight*.  It still fit, and I filled it out quite nicely.  However, getting on the scale showed I had gained 5-10 pounds in 3 months.  
At first I attributed the weight gain to a particular prescription I had started to help my IBS.  It was known to increase food cravings and people gained weight from that.  But I was on a super low dose.  When I started watching calories, I noticed that no, I wasnt really craving things any more than I ever had.  However, the drug, added with a vitamin supplement I needed to take as well,  put my body in a better state than I could ever remember it being in.  I could eat food and not worry.  I didn't have to be afraid of every thing I ate.  So I ate more, but I'm also fairly sedentary, so I gained.
That's my rational anyway.  In about 2 months, I've lost close to 6 pounds.  No, it's not much,  but I still notice the difference.  My waist started to feel thinner.  I thought it was just my mind wanting the weight loss.  I waited a few more weeks and got on the scale.  Confirmed, I lost a bit of weight, like magic. It's satisfying for sure.  I'm hoping to lose another 10 - 15 pounds, but we'll see how that goes.