I have an iPhone 3G that is my Phone and my Dev Phone. Then I test it on my wifes 3G before I submit to apple. NEVER trust your dev machine, there is a saying in the dev world " It always works on the developers machine", apple devices are no different.
Also because it works fine on 3.1.3 doesn't mean it will work the same on ios4.X,
I have see my app work fine on 3.1.3 and then ios4 has issues, have to tweak it some more, so not only do you need to test on multiple devices but you need to test on multiple versions of iOS.
You talking about iPad and iPod, thats even more of a difference. The screen size for starters, and the features available on each device will be different and performance as well.. your iPad probably screams you app but I bet it runs slower on a 3g Phone, which will make you look at the way your coding a little different.
If you're planning to release iPad and iPhone/iPod Touch apps, I'd probably want one of each to test on. Universal apps can get confusing, and testing on each device separately is quite helpful.
Also, in hand, both devices are quite different. Testing on hardware has definitely helped me fine tune my interfaces for each.
There's also conditional hardware across iOS devices that could cause problems if overlooked or not tested properly. Cameras, graphics hw, retina display etc. If you write an app that can use the camera, what happens on a device without it? It's probably something you want to test on an actual device since you may overlook it if you only test on a device that supports it.
There's also the issue of performance on older hardware, which is something you probably need a device for if you want to test.
I think having an iPad and the new iPod Touch would give you a good range of device features to hardware test. And if you get hardcore about wanting to test performance on older devices, you might later invest in a second hand something on ebay.
I have an iPad, iPhone 3GS, iPod Touch 1G and am picking up an iPod Touch 4G. I test my apps generally on the iPad/3GS and use my 1G iPod Touch for legacy testing (<4.0, slowest processor). I'll use the 4G for the retina display & additional testing.
That said, a good cheap option for testing everything is to have a beta test. I had nearly all devices accounted for in a recent beta test I ran.