The role of grammar in transition-probabilities of subsequent words in English text

16774 7 91830 2 265753 document 91830 journal.pone.0240018.pdf application/pdf 54e77be2598d61939c6ac9703cfa57b3 MD5 1343811 2020-10-12 08:43:09 https://pure.iiasa.ac.at/id/eprint/16774/1/journal.pone.0240018.pdf 16774 1 1 application/pdf text en public cc_by_4

journal.pone.0240018.pdf

published 91831 1 265759 document 91831 indexcodes.txt text/x-c++ 187775986c01f50c9f0d7d683a4ef841 MD5 10612 2020-10-12 08:52:39 https://pure.iiasa.ac.at/id/eprint/16774/2/indexcodes.txt 16774 2 2 text/x-c++ other Generate index codes conversion from text to indexcodes en public

indexcodes.txt

lightbox.jpg

preview.jpg

medium.jpg

small.jpg

http://eprints.org/relation/isVersionOf https://pure.iiasa.ac.at/id/document/91830 http://eprints.org/relation/isVolatileVersionOf https://pure.iiasa.ac.at/id/document/91830 http://eprints.org/relation/issmallThumbnailVersionOf https://pure.iiasa.ac.at/id/document/91830 archive 353 disk0/00/01/67/74 2020-10-12 08:44:30 2021-08-27 17:33:42 2020-10-12 08:44:30 article show Hanel R. Thurner S. 1918 The role of grammar in transition-probabilities of subsequent words in English text pub prog_esm Sentence formation is a highly structured, history-dependent, and sample-space reducing (SSR) process. While the first word in a sentence can be chosen from the entire vocabulary, typically, the freedom of choosing subsequent words gets more and more constrained by grammar and context, as the sentence progresses. This sample-space reducing property offers a natural explanation of Zipf’s law in word frequencies, however, it fails to capture the structure of the word-to-word transition probability matrices of English text. Here we adopt the view that grammatical constraints (such as subject–predicate–object) locally re-order the word order in sentences that are sampled by the word generation process. We demonstrate that superimposing grammatical structure–as a local word re-ordering (permutation) process–on a sample-space reducing word generation process is sufficient to explain both, word frequencies and word-to-word transition probabilities. We compare the performance of the grammatically ordered SSR model in reproducing several test statistics of real texts with other text generation models, such as the Bernoulli model, the Simon model, and the random typewriting model. 2020-10-08 published Public Library of Science (PLoS) 10.1371/journal.pone.0240018 307 public PLoS ONE 15 10 e0240018 TRUE 1932-6203 FALSE no info:eu-repo/semantics/article