Add your ssh key with password and manage password for key over keychain.

ssh-add --apple-use-keychain ~/.ssh/id_rsa

Also edit your ~/.ssh/config and add::

Host *
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_rsa

or add to your ~/.bash_profile:

eval "$(ssh-agent -s)"
ssh-add --apple-use-keychain ~/.ssh/id_rsa