The large audio files are all kept in a folder called Ivory Items. Simply deleting this folder will remove most of Ivory's data. The location of each plugin is:
OS X VST: /Library/Audio/Plug-Ins/VST/Ivory (OS X).vst
OS X AU: /Library/Audio/Plug-Ins/Components/Ivory AU.component
OS X RTAS: /Library/Application Support/Digidesign/Plug-Ins/Ivory RTAS (OS X)
OS 9 VST: usually /Cubase/VstPlugIns/Ivory VST (OS 9)
OS 9 RTAS: /System Folder/DAE Folder/Plug-Ins/Ivory RTAS (OS 9)
Finally, you may discard the Ivory Preferences folder located in OS 9: /System Folder/Preferences/
in OS X: /Users/{Home}/Library/Preferences/