By Diana Olick, CNBC Real Estate Reporter

NEW YORK (CNBC) — As we edge ever closer to D-Day (default day, debt ceiling day, however you choose to see next Tuesday, August 2nd), those of us who live and breathe the housing market are trying to figure out what this will mean to mortgage interest rates.

They are currently bouncing around historic lows and have been for some time. Refinances are surging, as the seven people left who haven’t yet refied are scrambling to do so.

But are we all worried over nothing?

“The debt crisis is probably the biggest factor hanging over mortgage rates at the moment,” says Guy Cecala of Inside Mortgage Finance. “Once investors feel there is any uncertainty about the U.S. government’s ability to guaranty its debt, Treasury rates and mortgage rates will start to rise — probably by at least 25-50 basis points.”

That’s the clearest answer I’ve gotten from the many experts we’ve had discussing this on CNBC today. Most just say, “We can’t know.” I’ve been arguing that the debt crisis is not as big a deal as the scheduled drop in the conforming loan limits at Fannie Mae, Freddie Mac and the FHA. Experts say the change from the $729,750 limit in the highest priced markets to $625,000 and the drop back to $417,000 in lower-priced markets will really only affect 5 % of homes nationally, but the percentage is far higher in certain local markets.

“But some FHA borrowers will be pushed towards the Fannie/Freddie market and higher down payments since the FHA loan limits don’t bottom out at $417k (like the GSE limits do),” reminds Cecala.

While we all worry about what that lowest rate can be and where, the fact is that while the “average rate” on the 30-year fixed is very low, today’s buyers are not all eligible, especially as those rates require big down payments and super-clean credit.

“Those people are not qualifying for the super low rates now, and 30% of our sales now are to cash buyers, investors and foreigners, anyway,” notes Shari Olefson of Fowler, White, Boggs, who argues we’re missing the point.