If this is a scenario that happens often, you probably should invest in a portable method to hold your ssh keys, such as a smart card. I have only used YubiKey for this purpose, but I'm sure others, like the Nitrokey, work similarly.