It's a series of commands. In the Audio window's Edit menu there is a command "Select unused". Invoke this. Then you will see which audio files get selected. Audiofiles that have the actual filename selected (not just some of the regions -pointing- to that audio file) are the ones that are not used at all in the arrange window.

Next, without deselecting any of the audio files that just got selected, go to the window's File menu and invoke the "Delete Audiofiles" command. You will then get a warning message saying "xx audiofiles to be deleted from disk. Cannot undo this operation." or something like that.

If you click OK, only the audiofiles that actually have their filenames selected will be deleted from your song and off the hard drive completely.

You should not just use the DEL-key for deletion but the "Delete Audio Files" command instead, since the DEL-key just removes them from Logics pool (i.e. from the Audio window) but not from your harddisk. Once you mess this up (read: use the DEL-key when you would like to completely erase files) you will have a hard time sorting out used/unused files on your harddisk.

Additional possibility for Windows PCs: If you however just have recorded files in that particular directory (meaning that there are no further files such as "to-be-imported" ones or ones that you might plan to use in other tunes) you could also select all of them one by one and press delete (in Explorer). Windows will then complain that some of your selected files are currently used by a running application (Logic) and won't let you delete them.