When you select elements, you can remove individual elements from your selection set. You can also deselect everything that is currently selected.
To deselect individual elements:
Hold down your Shift key.
Click the element you want to deselect.
To deselect everything in your selection set:
Select Edit > Deselect All, or right-click in the drawing area and select Deselect All, or simply click in a blank spot somewhere else in the drawing area.