Why? Your GPG key is airgapped and inextricable (except possibly via something like a DPA-style attack), and you can set a PIN required to perform any private key operations, complete with a configurable number of attempts before the device wipes itself.
There are DPA attacks to be worried about, but if you set a decent PIN and limit the number of failed attempts your GPG keys will probably be safe if your Yubikey is ever lost/stolen.
I have a 6 digit pin (8 digit admin pin) with 3 retries before the device wipes itself. The chances of an attacker cracking that are negligible - I'm more concerned about someone observing my typing (applies to passphrase as well) or actual physical attacks on the yubikey (stolen, decapped, ...) but I'm not the target demographic where such effort would be warranted (or at least I hope so). An attacker could backdoor my PC and intercept the passphrase dialog - but in that case he'd still need my yubikey which makes remote attacks impractical and makes me notice the theft. This is not the case with a passphrase on a secret key - a machine compromise may silently take everything without you ever noticing.
There are DPA attacks to be worried about, but if you set a decent PIN and limit the number of failed attempts your GPG keys will probably be safe if your Yubikey is ever lost/stolen.