The hard part is how do we verify that the records were actually deleted. Is the info still sitting on random backups somewhere. It seems that once it is out there, it will be very hard to put back in the box again. Personally that concerns me.

Yes, that's a real problem. I'd wager that many companies have no idea whatsoever where all the copies of your data have gone.