If being on a deserted island is your thing, Gramvousa Island is the place for you. One of two uninhabited islands located off Amorgos, Gramvousa can be accessed via boat from Kalotaritissa. There are two main beaches and lots of other spots to swim in (the water is to die for).
BYO everything as there are no shops. I spent the day swimming and reading. Paradise.
Visit here for more details on why I love Amorgos so much.