Deleting an item in Outlook sends it to the Deleted Items folder. If you're sure that you want to eradicate the note, you can hold the Shift key when you delete an item.
You will receive a dialog asking if you are sure you want to permanently delete it.
To avoid the nagging you can turn off "Warn before permanently deleting items" under Tools>Options>Other Advanced Options.
Be careful because you can delete items that might have benefited from one more moment's consideration.
(If you're using an Exchange server, you can probably get it back)
