A deterministic wallet derives keys from a single master key. It is a random 128 to 256 bit value that is completely unique. This master key generates an infinite number of other keys/addresses as described in the BIP32. This allows you to only ever need a single backup for every address you ever use. You can use a different address for every transaction, increasing your privacy on the blockchain.Even if you need more addresses in the future, the backup you made when first setting up your wallet will work forever.
One practical way to display this key is by transforming it into a series of words in a specific order. This set of words is commonly called "recovery phrase" or "mnemonic seed". The seed allows a user to easily back up and restore a wallet without needing nothing but pen and paper. This is described on the BIP39.
By using the BIPs above along with BIP44, HD wallets can be interoperable without moving any files around. It's possible to use the same seed on multiple different apps and correctly see the same funds (as long as the standards are implemented correctly).
Early clients such as the Satoshi client generate a list of fresh random keys/addresses to be used as receiving and change addresses in the future. When this list was used up, the client generated a new list. On every such event, the wallets had to be backed up again. Failing to do so would mean funds would be lost forever. HD wallets don't have that weak point, their single backup is valid for as long as you keep using the same seed.
Here is a great write up on HD Wallets and here is another.