There were so many steps, I'm not sure I remember them all, but here are the parts I do remember, in case this helps anybody:
- I had two "System" keychains, so I deleted one of them
- I set the "login" keychain to be the default, so that new keys would be added there (it should show up bold when it's the default)
- deleted the old public and private keys for my developer identity, and the wwdr and iphone developer certificates (in both the login and system keychains)
- quit keychain access and restarted it
- generated completely new keys, certificates, app id, provisioning profile
- made sure everything got added to the login keychain, not the system keychain
- in xcode deleted the old provisioning profiles from the Organizer and the device
- added the new profile to xcode and the device
- updated the info.plist (since I'd changed the com.yourcompany part of the app id in the new profile)
After verifying it worked with beta 4, I upgraded again to beta 5, did a clean rebuild, and everything still works fine.