Gnathostomes comprise over 99% of extant vertebrate diversity. Enormous progress has been made in understanding the major skeletal evolutionary transformations within this group. However, the deep-time evolution of their muscular system remains elusive. Part of this problem stems from poor knowledge of muscle homologies across large groups (1, 2). Our study markedly illustrates a second problem: the lack of reliable information about musculature in the literature. Despite being one of the most iconic living vertebrates, we found a plethora of errors in the identification of cranial muscles in the African coelacanth (Latimeria chalumnae), some of which have been replicated for nearly 70 years. Given the key position of coelacanths in the vertebrate tree of life, correcting these errors has profound implications for understanding the early gnathostome evolution.

Despite the existence of previous studies on coelacanth cranial muscles (3–9), many of them contradict each other. In addition, no serious attempt has been made to homologize most of these muscles with those of other jawed vertebrates. Here, we address these contradictions, report previously unknown muscle subdivisions and connections, and homologize coelacanth muscles with those of other jawed fishes. Last, we infer the probable anatomy of these muscles in major fossil lineages and uncover the early evolutionary changes that characterized the largest radiations of jawed fishes.

The cranial muscles may show different patterns of subdivision in different gnathostome lineages (fig. S1). The constrictor mandibularis dorsalis (CMD) connects the neurocranium to the palatoquadrate and is absent in the gnathostome fishes where these elements are fused—chimaeras (holocephalans) and lungfishes (dipnoans). CMD also often attaches to spiracle-associated structures. Coelacanths have three muscles presumably derived from CMD: adductor palatoquadrati, levator palatoquadrati, and spiracularis (Fig. 1 and fig. S2A). The last muscle has been overlooked in all previous studies (3, 4, 7). Although coelacanths lack a spiracular opening, the spiracularis is present and runs from the palatoquadrate to the spiracular ossicle and a connective tissue fold just posterior to it. The coelacanth spiracularis is only partially separated from the levator palatoquadrati, as in polypteriforms, acipenseriforms, and many elasmobranchs (10, 11). However, the muscle in coelacanths has an important change in its origin, which is from a concavity on the dorsoposterior profile of the palatoquadrate, rather than the postorbital fossa as in actinopterygians and elasmobranchs. Given the position of the spiracular ossicle relative to the postorbital fossa and the usual presence of a dorsoposterior concavity on the palatoquadrate in extinct crown sarcopterygians (12–14), they probably had the same configuration of the spiracularis as coelacanths. We cannot infer whether these conditions were shared with stem sarcopterygians, since data on their palatoquadrate and spiracular series are largely incomplete. Therefore, it is currently impossible to indicate at which node of the sarcopterygian tree the attachment of the spiracularis to the palatoquadrate evolved. In any case, the presence of the spiracularis is probably primitive for gnathostomes, since acanthodians, placoderms, and some extinct agnathans apparently had functional spiracles (Fig. 2 and fig. S1) (15, 16).

The levator palatoquadrati in coelacanths extends from a shallow medial depression on the posterodorsal border of the palatoquadrate to a deep postorbital (=temporal) fossa located between the postorbital and transverse otic processes [sensu (17); Fig. 3]. Similar fossae and depressions are reported for several extinct sarcopterygian fishes (7, 12, 18), indicating that they probably had a similar levator palatoquadrati.

Elasmobranchs (sharks and rays) primitively have a levator palatoquadrati with attachment sites similar to those of coelacanths (Fig. 4). Acanthodians also had a recognizable postorbital fossa, bounded anteriorly by the postorbital process and dorsally by a ridge (Fig. 5) (19, 20). This fossa and a depression along the dorsomedial region of the palatoquadrate (21) suggest that acanthodians had a levator palatoquadrati similar to that of elasmobranchs (Fig. 4) and sarcopterygians (Figs. 1 and 3 and fig. S2). Placoderms also had a postorbital (=mandibular) fossa above the superior border of the palatoquadrate (22–25), suggesting a comparable levator palatoquadrati in the group (Fig. 6). This pattern is therefore primitive for gnathostomes (Fig. 2 and fig. S1).

Most extant actinopterygians have a levator palatoquadrati differing from that primitive pattern. Originating from a short postorbital fossa, it subdivides into a levator arcus palatini and a dilatator operculi (Fig. 7). The latter inserts on the anterodorsal border of the opercle, which has an anteromedial depression that forms a spheroid articulation with the opercular (=posterior) process of the hyomandibular (10). This configuration allows abduction of the opercle when the dilatator operculi contracts. Although some stem actinopterygians had an opercular process, an associated anteromedial depression on the opercle was invariably absent (26–31). This is the same condition of coelacanths (Fig. 1), which suggest that stem actinopterygians also lacked a differentiated dilatator operculi (Figs. 2 and 7F and fig. S1). Therefore, the subdivision of the levator palatoquadrati into levator arcus palatini and dilatator operculi is an evolutionary novelty of crown actinopterygians (see Figs. 2 and 8, character 1, and the Supplementary Materials).

The levator arcus palatini of crown actinopterygians typically inserts on the lateral faces of the palatoquadrate and hyomandibular (Fig. 7B) (10). Suggestions of a similar configuration in stem actinopterygians (32) seem unlikely because their palatoquadrate had a laterally curved posterodorsal margin adhered to the jugal and preopercle, forming the maxillary-palatoquadrate chamber that encloses the adductor mandibulae (AM) (Fig. 7G) (28–30). This makes insertion of the levator arcus palatini on the lateral face of the palatoquadrate virtually impossible. Moreover, the dorsal margin of the palatoquadrate of stem actinopterygians had a reinforced lateral expansion (28, 29) that most likely served as the insertion site for their levator arcus palatini or levator palatoquadrati (Fig. 7, E and G). Consequently, the insertion of the levator arcus palatini on the lateral faces of the palatoquadrate and hyomandibular is unique to crown actinopterygians (see Figs. 2 and 8, characters 2 and 3, and the Supplementary Materials).

Reconstruction of the CMD in gnathostomes, thus, shows that the levator arcus palatini is unique to crown actinopterygians (Fig. 2 and fig. S1). The muscle identified by the same name in placoderms (33), sarcopterygians (3, 7), and most stem actinopterygians (30, 32) actually corresponds to the levator palatoquadrati of elasmobranchs. Some studies have reported that coelacanths would have one (7) or two (6) muscles connecting the prootic to the lateral face of the hyomandibular. This muscle could arguably be comparable to the levator arcus palatini. However, our observations indicate that these muscles do not exist in coelacanths. Previous studies probably misidentified the flattened oto-hyoid ligaments interconnecting those structures as muscles (Fig. 1). Both gross morphology and histology confirm the ligamentous nature of these structures and the absence of any muscular tissue (Fig. 9).

The coelacanth adductor palatoquadrati runs from the suprapterygoid fossa to the elongated suprapterygoid (=ascendens) process of the palatoquadrate (Fig. 3A and fig. S2). Enlarged suprapterygoid processes are also found in many extinct crown sarcopterygians (Fig. 3, B and D) (7, 12, 18, 34). Accordingly, despite the secondary loss of the entire CMD in lungfishes (fig. S1), the adductor palatoquadrati was likely widespread among crown sarcopterygian fishes (Figs. 2 and 3). A comparably elongated suprapterygoid process is absent in other gnathostomes, including stem sarcopterygians (35), and no other extant fish has a CMD division anterior to the postorbital process. Thus, the adductor palatoquadrati is an evolutionary novelty of the crown sarcopterygians (see Figs. 2 and 8, character 4, and the Supplementary Materials).

The AM connects at least the palatoquadrate to the lower jaw and has multiple subdivisions in vertebrates (fig. S1). In the coelacanth, it has been reported to have two (4) or three (3, 7) major subdivisions located in the cheek region. We identified four major, incompletely separated cheek subdivisions plus one in the lower jaw (Fig. 1 and fig. S2). The outermost cheek segment, originating from the palatoquadrate, corresponds to the facialis segment of actinopterygians (10). It partially subdivides into rictalis and stego-malaris sections.

The origin of the facialis from the palatoquadrate in coelacanths is shared with extant elasmobranchs (Fig. 4 and fig. S2). In both taxa, the muscle attaches primarily to the anterior face of the extrapalatoquadrate ridge and adjacent portions of the palatoquadrate commissural lamina (Figs. 3A and 4A). A similar configuration is inferred in extinct gnathostomes by the presence of the same ridge along the posterodorsal border of the palatoquadrate. This ridge is identified in placoderms (Fig. 6B) (22–24, 36), acanthodians (Fig. 5C) (19, 21, 37), stem chondrichthyans (Fig. 4C) (34, 38, 39), and stem actinopterygians (Fig. 7, D and E) (29, 30, 32, 34). Thus, the origin of the facialis from the palatoquadrate is plesiomorphic for gnathostomes (Fig. 2).

In most extant actinopterygians, the facialis origin is expanded posteriorly to include the hyomandibular and preopercle (Fig. 7B) (2, 10, 40). Stem actinopterygians had the posterior margin of the preopercle laterally attached to the extrapalatoquadrate ridge, forming a maxillary-palatoquadrate chamber (41) for the facialis (Fig. 7, D, E, and G) (28–30, 32, 34). This configuration suggests that they already had the lateralmost fibers of the facialis attached to the posteromedial region of the preopercle plate, as in polypteriforms (see Figs. 2 and 8, character 5, and the Supplementary Materials). The attachment of the facialis to the hyomandibular probably arose only in crown actinopterygians, in which the maxillary-palatoquadrate chamber is further expanded or open posteriorly (see Figs. 7B and 8, character 6, and the Supplementary Materials).

The innermost cheek segment of the coelacanth AM originates primarily from a connective tissue enveloping the postotic (=antotic) process of the neurocranium (Figs. 1 and 3A and fig. S2). This origin and medial placement allow us to homologize this muscle division with the buccalis segment of basal actinopterygians (10). This segment further divides dorsally into a lateral orbitalis and a medial sphenoidalis. Although previous studies have reported a buccalis in coelacanths (3, 4, 7), none have noted its dorsal subdivision into two sections, which is curiously one of the most noticeable separations in the entire AM (Fig. 3C).

The origin of the buccalis from the region of the postorbital process in coelacanths is shared with dipnoans, polypteriforms (Fig. 7A), and, with additional modifications, holosts (Fig. 2) (10). In addition to the extrapalatoquadrate crest, acanthodians had a cranial depression at the posterior portion of the postorbital process (19, 34), which has been suggested as a second site of origin for the AM (Fig. 5A) (19). This postorbital region coincides with the origin of the buccalis in coelacanths (Fig. 3A) and polypteriforms (Fig. 7A), so the extrapalatoquadrate crest of acanthodians would be the origin of the facialis only (Fig. 5B). It is unclear whether the acanthodian buccalis was subdivided into orbitalis and sphenoidalis sections. At least some acanthodians had a ramus mandibularis trigeminus (RMT) foramen at the dorsoposterior corner of the palatoquadrate (19, 21), which would then emerge along the lateral separation between the buccalis and facialis (Fig. 5, B and C), exactly as in coelacanths (Fig. 1).

A subdivision of the buccalis passing ventral to the orbit has been suggested in acanthodians (42). This seems unlikely for most acanthodians, which had enlarged eyes and little space below the orbit for accommodating this muscle (Fig. 5B) (19). However, stem chondrichthyans sensu stricto probably had this connection as they had ventrolaterally expanded postnasal walls resembling the ectethmoid process (38, 43), which is the origin site for the anteriormost subdivision of the AM (=preorbitalis) in extant elasmobranchs (see Figs. 4 and 8, character 7, and the Supplementary Materials). This chondrichthyan subdivision is the one with the most anterolateral origin of the AM in the neurocranium and, thus, topologically corresponds to the orbitalis section of the buccalis of bony fishes (42, 44). However, contrary to previous hypotheses (42, 44), the differentiation of the orbitalis (=preorbitalis) is not unique to elasmobranchs, as it is also present in coelacanths (Fig. 1 and fig. S2), polypteriforms, and holosts (Fig. 2 and fig. S1) (10). The entire buccalis, including its orbitalis section, is absent only in teleosts (10).

In stem chondrichthyans, the extrapalatoquadrate ridge terminates before contacting the ventrally expanded postorbital process (Fig. 4C) (34, 38, 39, 43). This postorbital expansion forms a wall that possibly served as site for origin of the sphenoidalis, similarly to the condition of acanthodians (Fig. 5, A and B). In extant elasmobranchs, the extrapalatoquadrate ridge extends anterodorsally, surpassing the postorbital process (Fig. 4A) (11, 45) and thus coopting the origin of the sphenoidalis (see Figs. 2 and 8, character 8, and the Supplementary Materials). The partial separation between the sphenoidalis and facialis in elasmobranchs is often overlooked, as the RMT rests atop this separation, which is only visible when the nerve is removed (fig. S3). This nerve location in elasmobranchs also coincides with that in coelacanths (Fig. 1) and possibly acanthodians (Fig. 5B). In holocephalans, the palatoquadrate is displaced anteriorly relative to the orbit and fuses with the neurocranium. Consequently, the sphenoidalis also originates from the ethmoid region (see Fig. 8, character 9, and the Supplementary Materials), while the facialis retains its origin from a remnant of the extrapalatoquadrate ridge. The RMT in holocephalans also follows the path between the buccalis and facialis segments.

Therefore, the buccalis probably originated from the postorbital region of the neurocranium in basal osteichthyans, acanthodians, and stem chondrichthyans (Figs. 2 to 5 and 7). This differs from placoderms, wherein the entire cheek portion of the AM (buccalis and facialis) appears to originate from the palatoquadrate and possibly adjacent dermal plates (Fig. 6) (22–24, 36, 46). The attachment of the AM to the palatoquadrate was likely along the ventral face of the extrapalatoquadrate ridge. This ridge is also traversed by an RMT foramen (Fig. 6C) (23, 46), which likely marked the separation between the buccalis and facialis segments, as in coelacanths (Fig. 1), elasmobranchs (Fig. 4 and fig. S3), holocephalans, and possibly acanthodians (Fig. 5). Some studies have identified a depression in the orbital region of some arthrodires as a potential site of AM origin (47, 48), but this depression actually corresponds to the postorbital fossa from which CMD originates (Fig. 6A and see above) (24, 25, 33). Hence, the neurocranial origin of the buccalis is an evolutionary novelty of eugnathostomes (see Figs. 2 and 8, character 10, and the Supplementary Materials).

Most fibers of the facialis and buccalis in coelacanths converge to an intersegmental aponeurosis on the medial side of the AM (fig. S2B), except for the lateralmost fibers of the rictalis, which insert directly on the lower jaw (Fig. 1). This aponeurosis splits ventrally into two tendons, corresponding to the meckelian and mandibular tendons of actinopterygians (2) (fig. S2B). The meckelian tendon anchors to the lower jaw, while the mandibular tendon serves as the origin for the fifth AM subdivision. This division inserts onto the angular and Meckel’s cartilage and is homologous to the mandibularis segment of actinopterygians (2, 10).

The actual mandibularis segment in coelacanths has been overlooked in all previous studies (3, 4, 7) [Lauder (42) mistakenly interpreted the portion of the rictalis superficially covered by an aponeurosis as the mandibularis; see Fig. 1). This discovery resolves a controversy regarding the homology of the mandibularis in gnathostomes. In elasmobranchs, a ventral segment of the AM runs from the intersegmental aponeurosis to a depression on the lateral face of Meckel’s cartilage (Fig. 6). Similar depressions on the endoskeletal lower jaw of most placoderms (Fig. 6) (23, 24, 36) and acanthodians (Fig. 5) (21, 37) suggest that they probably shared this condition. In coelacanths and actinopterygians, the mandibularis also arises from the intersegmental aponeurosis but does not insert on the lateral face of the lower jaw (Figs. 1 and 7F and fig. S2B). This difference has led some authors to reject the homology of the muscle between osteichthyans and other gnathostomes (42, 44). However, the medial placement of the mandibularis in bony fishes is not due to a change in muscle insertion but merely to the presence of superficial dermal ossifications on the lower jaw, which are absent in cartilaginous fishes, acanthodians, and most placoderms (22, 49). Coelacanths retain a well-developed Meckel’s cartilage, and their segmentum mandibularis inserts laterally on this cartilage (Figs. 1 and 3A and fig. S2B), as in extant elasmobranchs (Fig. 4B and fig. S3). Therefore, the assumption of nonhomology of the mandibularis across gnathostomes is unjustified, and neither the presence of the muscle nor its medial insertion can be considered synapomorphies for Osteichthyes [contra (42, 44)]. Likewise, the broad insertion of the muscle on the lateral surface of Meckel’s cartilage cannot be synapomorphic for placoderms [contra (50)]. The mandibularis was secondarily lost multiple times independently in jawed fishes, including holocephalans, dipnoans, and several actinopterygians (2, 10, 40) (fig. S1).

The constrictores branchiales superficiales (CBSs) have been reported in branchial arches 1 to 4 of coelacanths, supposedly extending over most of the arches parallel to the gills (8, 9). However, direct dissection (Fig. 10) and microscopy (Fig. 9) failed to find any trace of these muscles in both embryo and adult coelacanths. The presence of flap-like CBSs is primitive for gnathostomes, as they are found in lampreys, elasmobranchs, chimaeras, and even placoderms (33). Among bony fishes, the muscle has been reported in the dipnoan Neoceratodus and the sturgeons Acipenser and Scaphirhynchus (51). We confirmed the presence of the muscle only in Neoceratodus; the three acipenseriform genera examined here (Acipenser, Huso, and Polyodon) lack the muscle, and we were unable to examine a Scaphirhynchus. No other extant bony fish, including other acipenseriforms and dipnoans, have CBSs. Therefore, their absence is probably synapomorphic for Osteichthyes (see Fig. 8, character 11, and the Supplementary Materials), with homoplastic reacquisitions in Neoceratodus and possibly Scaphirhynchus.

Conflicting information has been provided about the muscles that serve the anteroventral portion of the branchial arches in coelacanths. Millot and Anthony (3) described three serial muscles, each running from ceratobranchials 2 to 4 to the urohyal. Wiley (8, 9) reported six muscles: two “transversi ventrales,” from ceratobranchials 2 and 3 to the urohyal, and four “interarcuales ventrales,” each connecting adjacent ceratobranchials. We unequivocally found only two bilaterally paired branchial muscles associated with the second and third arches (Fig. 10). These muscles, subarcuales II and III, run from the urohyal to ceratobranchials 2 and 3, respectively. A few fibers of subarcualis II additionally attach to the basibranchial. The supposed interarcuales ventrales muscles reported in (8, 9) are actually interarcual ligaments that lie just dorsal to the afferent branchial arteries and are commonly found in other sarcopterygians (52) and actinopterygians (53).

The absence of interarcuales ventrales in coelacanths indicates that bony fishes primitively have only one series of intrinsic muscles associated with the ventral portion of the first three branchial arches, the subarcuales (=obliqui, recti, or transversi ventrales). Although the subarcuales have been proposed to be synapomorphically unique to bony fishes (8, 9), they are also present in holocephalans (54). Subarcuales are absent in elasmobranchs and lampreys, and their presence in extinct lineages could not be inferred. Consequently, it is equally parsimonious to postulate that subarcuales I to III evolved independently in osteichthyans and holocephalans or that they evolved in gnathostomes and were lost secondarily in elasmobranchs.

The remaining cranial muscles of the African coelacanth are mostly as described in (3). In addition to the characters discussed above, the coracomandibularis (=coracobranchialis) originating from hypobranchial 3 is confirmed as a synapomorphy for crown actinopterygians as originally proposed (8, 9) (see Fig. 8, character 12, and the Supplementary Materials). Two supposed myological synapomorphies from the literature could not be corroborated or falsified because of the impossibility of unambiguous character polarization: the presence of interbranchiales for actinopterygians (8) and “fan-shaped AM with unspecialized suborbital and postorbital fibers” for eugnathostomes (42).

Our analysis clarifies the homologies of several cranial muscles, their occurrences, and alternative configurations in the major groups of gnathostome fishes—placoderms, acanthodians, chondrichthyans, sarcopterygians, and actinopterygians (Fig. 2 and fig. S1). This allowed us to critically reevaluate previous hypotheses of evolution of these muscles in gnathostomes. As a result, only 3 (16%) of the 22 myological evolutionary transformations (synapomorphies) proposed for the major gnathostome lineages (8–10, 42, 44) proved to be valid (Fig. 8). Some of these errors apparently resulted from assumptions of nonhomology of muscles due to different terminologies applied to different taxa (e.g., preorbitalis in elasmobranchs and multiple synonyms of orbitalis in actinopterygians). Other errors are a consequence of the large number of overlooked muscles and nonmuscular structures misidentified as muscles in coelacanths (Figs. 1, 9, and 10). On the other hand, nine previously unknown evolutionary novelties are proposed here (Fig. 8), including those for groups that were never previously diagnosed by myological changes (e.g., crown sarcopterygians).

The analysis shows that the emergence of the three major groups of living gnathostomes was characterized by varied rates of morphological change in the cranial muscles. Low, moderate, and high amounts of innovations are reported for lobe-finned (sarcopterygian), cartilaginous (chondrichthyan), and ray-finned (actinopterygian) fishes, respectively (Figs. 2 and 8). Of particularly note is the high number of new muscle connections in the AM and CMD that arose in the early radiation of actinopterygians (Fig. 2). As discussed below, some of these previously unknown connections have profound functional implications, as these cranial muscles are involved in vital processes such as feeding and breathing.

Mouth opening in jawed fishes is coupled with the expansion of the pharyngeal cavity (55, 56). In this sense, the muscles associated with the mandibular and hyoid arches can be divided into expansors and compressors of the mouth and/or pharynx (Fig. 2). Primitively in gnathostomes, the coracomandibularis (=branchiomandibularis) and coracohyoideus (=sternohyoideus and rectus cervicis) are the main oropharyngeal expansors (41, 55). Although the spiracularis does not play a substantial role in this mechanism in most fishes, it technically functions as an expansor in most taxa by opening the spiracle. Muscle attachments indicate that all remaining mandibular muscles serve as oropharyngeal compressors in most gnathostomes, including coelacanths. This pattern changes with the attachment of the CMD to the opercle (dilatator operculi) and to the lateral face of the palatoquadrate and hyomandibular (levator arcus palatini) in crown actinopterygians. These created a second mechanism of active oropharyngeal expansion, in addition to the primitive mechanism performed by the coracomandibularis and coracohyoideus. The emergence of this previously unknown mechanism is probably related to the massive predominance in actinopterygians of suction feeding over bite feeding, which is the primitive form of food capture in eugnathostomes (41, 55). This innovation may have played an important role in the evolutionary success of the group, with several teleosts evolving even more sophisticated mechanisms of suction feeding (56, 57).

The discovery that the supposed muscles on the lateral face of the coelacanth hyomandibular [“levatores hyomandibulares” (6) or “adductor operculi” (7)] are actually ligaments (Figs. 1 and 9) invalidates reconstructions that propose an additional active hyoid mechanism for oropharyngeal expansion (6, 7) in this taxon. Evolutionary reconstructions of cranial muscle attachments in early gnathostomes show that the feeding mechanism of coelacanths is largely similar to that of nonactinopterygian fishes (Fig. 2), except for the notable presence of the subcephalicus (=basicranialis) as an important additional compressor muscle (3, 58).

Small- and medium-sized specimens were double stained for the cartilage and bone before dissections following (59). Anatomical drawings were based on photographs and direct stereomicroscopic observations of specimens to capture fine anatomical details. Illustrations were generated in Adobe Illustrator CC 2020 and Adobe Photoshop CC 2020. Three-dimensional segmentation and rendering of microtomographic data were performed with VGStudio MAX 2.2.3 64 bits and MeshLab 2022.02 64 bits.

In the anatomical accounts, the term insertion refers to the attachment of the muscle to the structure (usually a bone) that presumably moves (or moves more intensely) during its contraction; origin is defined as the opposite muscle attachment to the stationary (or less movable) skeletal element (1).

This study was carried out under approval of the Animal Care and Use Committee of the Biosciences Institute, University of São Paulo (project no. 226/2015; CIAEP no. 01.0165.2014). The research used only ethanol-preserved specimens deposited in museums and did not involve animal experimentation. Examined specimens are listed in the Supplementary Materials.