The GNOME keyring is very convenient; it figures out what keys you need to unlock and pops up the relevant dialogs to do so at the right times. But by default it caches them until you logoff. You can have caches of PGP passphrases expire:

gsettings set org.gnome.crypto.cache gpg-cache-ttl 300
gsettings set org.gnome.crypto.cache gpg-cache-method 'timeout'

but per this bug you can’t do the same for SSH keys.[1] An alternative is to check for X11 activity using the xprintidle utility, and clear all keys when the user has been idle for five minutes. This crontab entry does that:

#!/bin/sh

while true; do
    if [ $(xprintidle) -ge 300000 ]; then
        ssh-add -D 2>/dev/null
    fi
    sleep 300
done

I’ve got Xfce running pkill -u $USER /path/to/this/script; /path/to/this/script & as part of its startup sequence.

Notes

[1] You can just turn off the SSH key handling of gnome-keyring-daemon though I’m not sure this works in all versions of gnome-settings-daemon in circulation. The gconf boolean key might be /apps/gnome-keyring/daemon-components/ssh.