Category: Language

Debugging CUDA indexing errors

Perhaps you’ve seen pages of the following scary error:

../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [99,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

It turns out there is a relatively simple way to figure out what the indexing issue is. The internet suggests prepending

CUDA_LAUNCH_BLOCKING=1

to your command, but this doesn’t seem to help much either. There is a simpler solution: run whatever you’re doing on CPU. It’ll give you much nicer errors.

A note on pure allophony

I have previously discussed the notion of pure allophony, contrasting it with the facts of alternations. What follows is a lightly edited section from my recent NAPhC 12 talk, which in part hinges on this notion.

While Halle (1959) famously dispenses with the structuralist distinction between phonemics and morphophonemics, some later generativists reject pure allophony outright. Let the phonemic inventory of some grammar G be P and the set of surface phones generated by G from P be S. If some phoneme p ∈ P always corresponds—in some to be made precise—to some phone s ∈ S and if s ∉ P then s is a pure allophone of p. For example, if /s/ is a phoneme and [ʃ] is not, but all [ʃ]s correspond to /s/s, then [ʃ] is a pure allophone of [s]. According to some descriptions, this is the case for Korean, as [ʃ] is a (pure) allophone of /s/ when followed by [i].

One might argue that alternations are more entrenched facts than pure allophony, simply because it is always possible to construct a grammar free of pure allophony. For instance, if one wants to do away with pure allophony one can derive the Korean word [ʃI] ‘poem’ from /ʃi/ rather than from /si/. One early attempt to rule out pure allophony—and thus to motivate the choice of /ʃi/ over /si/ for the this problem—is the alternation condition (Kiparsky 1968). As Kenstowicz & Kisseberth (1979:215) state it, this condition holds that “the UR of a morpheme may not contain a phoneme /x/ that is always realized phonetically as identical to the realization of some other phoneme /y/.” [Note here that /x, y/ are to be interpreted as variables rather than as the voiceless velar fricative or the front high round vowel.–KBG] Another recent version of this idea—often attributed to Dell (1973) or Stampe (1973)—is the notion of lexicon optimization (Prince & Smolensky 1993:192).

A correspondent to this list wonders why, in a grammar G such that G(a) = G(b) for potential input elements /a, b/, a nonalternating observed element [a] is not (sometimes, always, freely) lexically /b/. The correct answer is surely “why bother?”—i.e. to set up /b/ for [a] when /a/ will do […] The basic idea reappears as “lexicon optimization” in recent discussions. (Alan Prince, electronic discussion; cited in Hale & Reiss 2008:246)

Should grammars with pure allophony be permitted? The question is not, as is sometimes supposed, a purely philosophical one (see Hale & Reiss 2008:16-22): both linguists and infants acquiring language require a satisfactory answer. In my opinion, the burden of proof lies with those who would deny pure allophony. They must explain how the language acquisition device (LAD) either directly induces grammars that satisfy the alternation condition, or optimizes all pure allophony out of them after the fact. “Why bother” could go either way: why posit either complication to the LAD when pure allophony will do? The linguist faces a similar problem to the infant. To wit, I began this project assuming Latin glide formation was purely allophonic, and only later uncovered—subtle and rare—evidence for vowel-glide alternations. Thus in this study, I make no apology for—and draw no further attention to—the fact that some data are purely allophonic. This important question will have to be settled by other means.

References

Dell, F. 1973. Les règles et les sons. Hermann.
Hale, M, and Reiss, R.. 2008. The Phonological Enterprise. Oxford University Press.
Halle, M. 1959. The Sound Pattern of Russian. Mouton.
Kenstowicz, M. and Kisseberth, C. 1979. Generative Phonology: Description and Theory. Academic Press.
Kiparsky. P. 1968. How Abstract is Phonology? Indiana University Linguistics Club.
Prince, A. and Smolensky, P. 1993. Optimality Theory: Constraint interaction in generative grammar. Technical Report TR-2, Rutgers University Center For Cognitive Science and Technical Report CU-CS-533-91, University of Colorado, Boulder Department of Computer Science.
Stampe, D. 1973. A Dissertation on Natural Phonology. Garland.

Defectivity in Amharic

[This is part of a series of defectivity case studies.]

According to Sande (2015), only Amharic verb stems that contain a geminate can form a frequentative. Since not all imperfect aspect verbs have geminates, some lack frequentatives and speakers must resort to periphrasis. If I understand the data correctly, it appears that the frequentative is a /Ca-/ reduplicant template which docks to the immediate left of the first geminate; the C (consonant) slot takes its value from said geminate. For instance, for the perfect verb [ˈsäb.bärä] ‘he broke’, the frequentative is [sä.ˈbab.bärä] ‘he broke repeatedly’. But there is no corresponding frequentative for the imperfective verb [ˈjə.säb(ə)r] ‘he breaks’ since there is no geminate to dock the reduplicant against; Sande marks as ungrammatical *[jə.sä.ˈbab(ə)r] and presumably other options are out too.

(h/t: Heather Newell)

References

Sande, H. 2015. Amharic infixing reduplication: support for a stratal approach to morphophonology. Talk presented at NELS 46.

Linguistics and prosociality

It is commonly said that linguistics as a discipline has enormous prosocial potential. What I actually suspect is that this potential is smaller than some linguists imagine. Linguistics is of course essential to the deep question of “what is human nature”, but we are up against our own epistemic bounds in answering these questions and the social impact of answering this question is not at all clear to me. Linguistics is also essential to the design of speech and language processing technologies (despite what you may have heard: don’t believe the hype), and while I find these technologies exciting, it remains to be seen whether they will be as societically transformative as investors think. And language documentation is transformative to some of society’s most marginalized. But I am generally skeptical of linguistics’ and linguists’ ability to combat societal biases more generally. While I don’t think any member of society should be considered well-educated until they’ve thought about the logical problems of language acquisition, considered the idea of language as something that exists in the mind rather than just in the ether, or confronted standard language ideologies, I have to question whether the broader discipline has been very effective here getting these messages out.

Online poisoning

One of my working theories for why natural language processing feels unusually contentious at present is, yes, social media. The outspoken researchers speak, more or less constantly, to a large social media audience, and use this forum as the primary way to form and disseminate opinions. For instance, there is a very strong correlation between being an “ACL thought leader”, if not an officer, and tweeting often and aggressively. People of my age understand the addictive and corrosive nature of presenting oneself for online kudos (and jeers), but some people of the older generations lack the appropriate internet literacy to use these tools in moderation, and some people of the younger generations lack the maturity to do the same. Such people have online poisoning. Side-effects include outing oneself as the subject of a subtweet and complaining to a student’s advisor. If you have any of these symptoms, please log off immediately and touch grass.

Defectivity in Turkish; part 2: desideratives

[This is part of a series of defectivity case studies.]

Thanks to correspondence with one of the authors I recently became aware of another possible paradigm gap in Turkish. According to İleri & Demirok (2022), henceforth ID, Turkish speakers are uncertain about the form of 3rd person plural desideratives. In this language, desideratives are deverbal nominals which select for and agree with a genitive subject. The desiderative suffix is /-AsI-/, where the capital letters mark archiphonemes subject to root harmony, and the 3rd person plural (3pl;.) possessive agreement suffix is /-lArI/. However, according to ID’s survey, Turkish speakers rate 3pl. desideratives formed from the root plus /-AsI-lArI/ as quite poor, and 3pl. desideratives are exceeding rare in corpora, even compared to other desiderative forms.

ID relate this observation to something unexpected about the 3rd person singular (3sg.) desiderative. Desideratives select, and agree with, a genitive subject, and the ordinary 3sg. genitive agreement suffix is /-sI/, but the 3sg. desiderative, there is apparently a haplology and we get just /-AsI/ (e.g., yapası, the 3sg. desiderative of ‘do’) instead of the expected */-AsI-sI/. They suggest that speakers may have reanalyzed /-AsI/ as a desiderative allomorph /-A/ followed by a 3sg. agreement suffix /-sI/, and thus predict that the 3pl. desiderative will be expressed by /-A-lArI/, though this is also judged to be quite bad (thus *yapasıları but also *yapaları). However, it is not immediately clear to me why ID expect speakers to hypothesize that the 3sg. desiderative allomorph should generalize to the 3pl.

This has a rather different flavor than the other defectivity case studies I’ve presented thus far. It could be that there simply are not enough desideratives in this person/number slot in the input, but I still don’t see what could be objectionable about /-AsI-lArI/. Another mystery is that their judgment task finds an unexplained very low acceptability for 2nd person plural desideratives (which seem to be of the form /-AsI-n/).

References

İleri, M. & Demirok, Ö. 2022. A paradigm gap in Turkish. In Proceedings of the Workshop on Turkic and Languages in Contact with Turkic 7, pages 1-15.

It’s time to retire “agglutinative”

A common trope in computational linguistics papers is the use of the technical term agglutinative as a synonym for rich inflectional morphology. This is not really what that term means. Properly, a language has agglutinative morphology in the case that it has affixes, each of which has a single syntacto-semantic function. (To really measure this properly, you probably need a richer, and more syntactically-oriented, theory of morphology than is au courant among the kind of linguistic typologist who would think it interesting to measure this over a wide variety of languages in the first place, but that’s another issue.) Thus Russian, for instance, has rich inflectional morphology, but it is not at all agglutinative, because it is quite happy for the suffix -ov to mark both the genitive and the plural, whereas the genitive plural in Hungarian is marked by two affixes.

I propose that we take agglutinative away from NLP researchers until they learn even a little bit about morphology. If you want to use the term, you need to state why agglutination, rather than the mere matter of lexemes having a large number of inflectional variants, is the thing you want to highlight. While I don’t think WALS is very good —certainly it’s over-used in NLP—it nicely distinguishes between isolation (#20), exponence (#21), and synthesis (#22). This ought to allow one to distinguish between agglutination and synthesis with a carefully-drawn sample, should one wish to.

A prediction

You didn’t build that. – Barack Obama, July 13, 2012

Connectionism originates in psychology, but the “old connectionists” are mostly gone, having largely failed to pass on their ideology to their trainees, and there really aren’t many “young connectionists” to speak of. But, I predict that in the next few years we’ll see a bunch of psychologists of language—the ones who define themselves by their opposition to internalism, innateness, and generativism—become some of the biggest cheerleaders for large language models (LLMs). In fact, psychologists have not made substantial contributions to neural network modeling in many years. Virtually all the work on improving neural networks over the last few decades has been done by computer scientists who cared not a whit whether they had anything to do with human brains or cognitive plausibility.1 (Sometimes they’ll put things like “…inspired by the human brain…” in the press releases, but we all know that’s just fluff.) At this point, psychology as a discipline has no more claim to neural networks than the Irish do to Gaul, and in the rather unlikely case that LLMs do end up furnishing deep truths about cognition, psychology as a discipline will have failed us by not following up on a promising lead. I think it will be particularly revealing if psychologists who previously worshipped at the Church of Bayes suddenly lose all interest in mathematical rigor and find themselves praying to the great Black Box. I want to say it now: if this happens—and I am starting to see signs that it will—those people will be cynics, haters, and trolls, and you shouldn’t pay them any mind.

Endnotes

  1. I am also critical of machine learning pedagogy, and it is therefore interesting to see that those same computer scientists pushing things forward don’t seem to care much for machine learning as an academic discipline either.

Noam and Bill are friends

One of the more confusing slanders against generativism is the belief that it has all somehow been undone by William Labov and the tradition of variationist sociolinguistics. I have bad news: Noam and Bill are friends. I saw them chopping it up once, in Philadelphia, and I have to assume they were making fun of functionalists. Bill has nice things to say about the generativist program in his classic paper on negative concord; Noam has some interesting comments about how the acquirenda probably involve multiple competing grammars in that Piaget lecture book. They both think functionalism is wildly overrated. And of course, the i-language perspective that Noam brings is an absolute essential to dialogues about language ideologies, language change, stigma and stratification, and so forth that we associate with Bill.