Recently two mysticete related papers have been published - Erich Fitzgerald's monograph on the truly bizarre Mammalodon colliveri, which I'll cover later, and M.E. Steeman's (2009) thought provoking paper naming the new "cetothere" Uranocetus from the Miocene of Denmark and its implications for mysticete hearing.
First off, "cetotheres" are a wastebasket group of generalized archaic baleen whales that don't fit nicely in modern families, although Bouetel and Muizon (2006) have redefined the Cetotheriidae sensu stricto as a small group with some very strange cranial features, including my personal favorite, Herpetocetus. Most other cetotheres (Cetotheriidae sensu lato) were placed into newly named families (Pelocetidae, Aglaocetidae, and Diorocetidae) which were sister taxa to the Balaenopteridae all included in her concept of the Balaenopteroidea (but not in the concept of the Balaenopteroidea advocated by Demere et al. 2005, which is Eschrictiidae + Balaenopteridae). Bottom line is Uranocetus is some kind of stem baleen-bearing mysticete, no matter whose phylogeny you use. The dentary of Uranocetus (from Steeman, 2009).
The fact that most basal mysticetes have an enlarged mandibular foramen suggests that this is a feature inherited from basilosaurid ancestors. Interestingly, modern mysticetes are adapted for hearing low frequency sounds, which pass through dense bone without significant volume loss. While Uranocetus has a large mandibular foramen, the lateral wall is too thick to be useful for hearing anything aside from low frequency sounds (which Uranocetus is adapted to hear based on its cochlear structure; Steeman 2009). The exact same thing is seen in Herpetocetus, which is also adapted for low frequency hearing, but has a large foramen with a thick lateral wall. This suggests that at least in these later diverging taxa, that the large mandibular foramen was a vestigial feature perpetuated by phylogenetic inertia.
Lateral aspect of a (not so typical) mysticete (Eshrichtius robustus, the Gray Whale) skull and dentary in articulation, from Johnston et al. (2009).
Steeman (2009) reasoned that the mandibular foramen decreased in size to strengthen the dentary due to the intense forces involved during feeding. Above shows a gray whale skull and mandible in articulation, just to give you an idea of how strange the mysticete feeding apparatus is (exclusive of baleen). In any event, I've been thinking about this quite a bit recently, and got to add a (very short) synopsis of this in my manuscript revisions, but you'll hear about that soon enough.
Deméré, T.A. and A. Berta (2008). Cranial anatomy of the toothed mysticete Aetiocetus weltoni and its implications for aetiocetid phylogeny. Zoological Journal of Linnean Society, 154(2): 308-352. PDF
Deméré, T.A., A. Berta, and M.R. McGowen. 2005. The taxonomic and evolutionary history of fossil and modern balaenopteroid mysticetes. Journal of Mammalian Evolution 12:99-143.
Fitzgerald, E.M.G. 2009. The morphology and systematics of Mammalodon colliveri (Cetacea:Mysticeti), a toothed mysticete from the Oligocene of Australia. Zoological Journal of the Linnean Society 110p.
Johnston, C., T. Deméré, A. Berta, J. St. Leger and J. Yonas. 2009. Observations on the musculoskeletal anatomy of the head of a neonate gray whale (Eschrichtius robustus). Marine Mammal Science PDF
Nummela, S., J.G.M. Thewissen, S. Bajpai, T. Hussain, and K. Kumar. 2007. Sound transmision in archaic and modern whales: anatomical adaptations for underwater hearing. Anatomical Record 290:716-733.
Steeman, M.E. 2009. A new baleen whale from the late Miocene of Denmark and early mysticete hearing. Palaeontology 52 :1169-1190.