Loan Refinance and when it's a bad idea

Earlier in the summer I started getting calls from our mortgage regarding loan refinance.  We hadn't even had our loan for a year yet!  I grew suspicious when they tried to increase the pressure, and they kept calling at weird hours, or when I was on a business call, so after 7 or 8 missed calls they finally stopped pestering me.  Curious though, I contacted the mortgage broker who set up our original loan, which differs from our actual loan company.  I trust this guy as he's always been straight forward with us.  We started by saying we want to sell in 2-3 years.  He immediately said have a nice day!  He said it was useless to continue because we'd spend more by adding 4K to the back end of our loan and the $100 or $150 savings per month on loan rate reduction would not pan out. 

I've learned this in business - ask questions, especially when you don't understand something and if someone won't answer the questions find someone who will.  The lady from my loan company wanted to make it sound like we needed this, when in the end, we didn't.  I'm glad I never returned her calls!