Only?
By that measure pretty much everything is "only".
We're talking here about an annual bill on the order of about USD 800 Bn (actually, more than 900 if you count the hidden positions in non-DoD budgets)!
http://en.wikipedia.org/wiki/File:U...._-_FY_2007.png

Halving the military expenditures (including some coast guard, nuclear "energy" budget and DHS budget positions) would in itself suffice to eliminate the deficit in a few years.

http://en.wikipedia.org/wiki/File:20..._Increases.png


"Foreign aid (reportedly the favourite target for fiscal conservative rhetoric) is "only" and "small" and unable to contribute significantly to any cost savings.
So-called "Defense" is a huge chunk and deserves to be called the reason for the federal budget deficit.