2016 update:
USB 3 vs Thunderbolt interface
HDD (Hard Disk Drive) vs SSD (Solid State Drive)

What do you need:
The main need for high speed external storage is for video editors working on large files requiring a lot of data transfer.
See Hard-Drive Solutions for Video-Editing Studios | B&H Explora
I suppose if you think you might need to boot and run off of an external backup drive it would be important also, but failure rates of internal drives are so low now most people are not worried about this.

A note about speeds and testing.
Many specs list burst data rates. Sustained Data Transfer (SDT) rates.

Actual speeds will be slower than maximum interface specs because of the protocol overhead and extra bits (start bit, stop bit, ...) required in serial transfer. USB has more overhead than firewire, so a USB 2 (480 Mbs) will actually be slower than a firewire 400 (400Mbs).

USB 3 vs Thunderbolt 2:
In 2013 tests How fast is USB 3.0 really? | Macworld they found the following:
USB 3 and Thunderbolt 2 were about the same (115 MB/s) with a HDD (Hard disk)
With a SSD (Solid State Disk) Thunderbolt was faster (188 MB/s vs 167 MB/s Read)
and 370 vs 167 with the Aja Video Systems benchmark.

Thunderbolt should be a lot faster (1.2 GB/s max) than USB 3 (640 MB/s max), but in real world testing there is not much difference.
One explanation I saw was that the power provided on the thunderbolt cable was not enough to optimally power the electronics in the drive.

HDD vs memory stick:
With 128 GB flash drives claiming to have 300 MB/sec transfer rates for $80 what is the best choice? My 2017 tests reading a 18 GB file and USB 3 on a 2.89 GHz MacBook Pro with SSD Storage
64 GB SanDisk Ultra Fit memory stick USB 3.0 - 152 MB/s
1 TB WD Passport HDD USB 3.0 - 94 MB/s

Links:
Best hard drives and storage devices of 2016 - CNET
The Best External Desktop Hard Drive | The Wirecutter Dec. 2016

Typical Sustained Data Transfer Rate (STR):

USB-C and Thunderbolt 3
USB-C and Thunderbolt 3: Here's everything you need to know - CNET Oct 2016
Thunderbolt 3 and USB C use the same cable.
USB 3 devices can use up to 900mA of power (compared to 500mA used by USB 2 devices).

Total Nightmare: USB-C and Thunderbolt 3 - Stephen Foskett, Pack Rat

See also Data Comm Interfaces
FireWire 800: 40-50MB/s (2.5" 5400 rpm drive) 75-85MB/s (3.5" 7200 rpm drive)
FireWire 400: 30-40MB/s
USB 2.0: 30MB/s
  I got 11 MB/s from my 1GHz macBook G4 to an iomega LDHD250 on a mac formatted partition
  and 8 MB/s to a FAT 32 formated partition
eSATA II 120 MByte/sec
USB 3.0 - max. 625 MB/s
Thunderbolt 2- max. 2,500 MB/s
Note: Actual thruput is lower than the maximum spec.
Source: How fast is USB 3.0 really? | Macworld
Time to write 20GB - Thunderbolt 23 sec on SSD, 55 sec on HDD; USB 3 110 sec on HDD
Source: Thunderbolt vs. USB 3.0: | gizmodo.com

Formated as FAT 32 vs OS X Extended journaled USB 3.0
Copy of 2.45 GB - Speed is Bytes/sec so bits/sec is about 8 times more. <

Format	2.5 disk		3.5 disk
		write	read	write	read
FAT 32	18MBs	66MBs
OS X	62MBs	77MBs	62MBs	81MBs

Time includes 3 sec. of initialization before copy starts

Standard external drives use 3.5 " drives and with case are about 6.3 x 6.8"
Compact external drives use 2.5 " drives and with case are about 3.2" x 5"

Prices Apr 2017

Media and Data Servers:
LaCie Ethernet Disk Mini - Home Edition (500GB) $110-200 [6.7]
Iomega StorCenter NAS drive (250GB) $200 [6.3]
Media Only:
Apple TV (160GB) $320 [8]
Data Servers only:
Iomega StorCenter Pro NAS 150d (2TB) $750-800 [7.8]
Western Digital Essential Netcenter Network Hard Drive (250GB) [6.8]

Prices 2007:

Item	eBay new	Amazon	ClubMac	MacConnection	Costco
LaCie 250GB Mar '09	$83	$90	$106	$110
Seagate FreeAgent™ Go 250GB	$75	$79	$88	$90	$80
Shipping	free	free	$5	$10	$5
Shipped from	??				S. Calif.

Consumer Reports Store Ratings Aug., 2006:
Amazon (90), MacConnection (90), CDW (87), Costco (87), MacMall (84)
Stores: Price: Better.: Amazon, Costco, MacConnection, MacMall; Average: Office Depot, Staples, Best Buy, Fry; Worse: Apple Store, Office Max, Circut City

Terms:
ATA - Advanced Technology Attachment EDSA - Enhanced Datamover and Storage Accelerator eSATA - External SATA 3Gbit/sec (150 MByte/sec) ESDI - Enhanced Small Disk Interface HDD - Hard Disk Drive IDE - Integrated Drive Electronics later renamed to ATA NAS - Network attached storage RAID - Redundant Array of Independent Disks SAN - Storage area network SATA - Serial ATA SSD - Solid State Disk or Solid State Storage Device (e.g. flash or thumb drive) STR - Sustained Data Transfer Rate USB - Universal Serial Buss

Links:
See 2016 links above
PC Hardware
Mac Hardware
External Hard Drive Reviews at ZDnet
NAS - Network attached storage at ZDnet
Data Comm interfaces
Flash Memory Cards and drives
Hard Disks at wikipedia
Hard Disk Drives: 3.5" or 2.5"?