What I do:

augroup Misc autocmd! " Only create a swap file when the file is modified autocmd CursorHold,BufWritePost,BufReadPost,BufLeave * \ if !$VIMSWAP && isdirectory(expand("<amatch>:h")) | let &swapfile = &modified | endif augroup END

What Tim Pope does https://github.com/tpope/tpope/blob/master/.vimrc#L232-L235

You're welcome.