This subject has been discussed at great length on all the BMW message boards, with many opinions expressed, but here is the last known SIB from BMW on the subject.

BMW Service Information bulletin 11 04 00 (October 2001)

Engine Oil Recommendations: 8 Cylinder Engine (S62)

M5 produced up to 3/00:
Castrol Formula RS SAE 10W-60 or Castrol TWS Motorsport SAE 10W-60 Synthetic Oil
BMW Part Number: 07 51 0 009 420

M5 produced from 3/00, Z8 all:
BMW High Performance Synthetic Oil SAE 5W-30
BMW Part Number: 07 51 0 017 866

In the transmission I use Red Line D4ATF in the gearbox, and Red Line 75-90 in the differential. I change these once evey 12 months or so, just to keep an eye on it all.