The Wikipedia Corpus was created by Mark Davies. It contains the full text of Wikipedia, and it contains 1.9 billion words in more than 4.4 million articles. But this corpus allows you to search Wikipedia in a much more powerful way than is possible with the standard interface. You can search by word, phrase, part of speech, and synonyms. You can also find collocates (nearby words), and see re-sortable concordance lines for any word or phrase. Click on any of the links in the search form on the search page for context-sensitive help, and to see the range of queries that the corpus offers. Most importantly, you can create and use virtual corpora from any of the 4,400,000 articles in the corpus. For example, in less than a minute you could create a corpus with 500-1,000 pages (perhaps 500,000-1,000,000 words) related to microbiology, economics, basketball, Buddhism, or thousands of other topics. (More information, with YouTube videos) You can then search within that virtual corpus, compare the frequency of a word, phrase or grammatical construction in your different virtual corpora, and also create "keyword lists" based on the texts in your virtual corpus. Rather than spending hours or days to create a specialized corpus for a particular topic, you can create it and search it within just a minute or two with this Wikipedia corpus. Finally, the corpus is related to other corpora from English-Corpora.org, which are the most widely used corpora of English and which offer unparalleled insight into variation in English.
LIST display
Find single words like
mysterious,
all forms of a word like JUMP,
words matching patterns like
*break*,
phrases like more * than or
rough NOUN.
You can also search by synonyms (e.g.
gorgeous), and
customized wordlists like
clothes. In each case, you see each individual
matching string.
You can use parts of speech as part of your query. For example, ADJ eyes would find a two word string, composed of an adjective followed by the word eyes. Some other examples are: rough NOUN, NAME Smith, VERB * money, TALK ADV, NUM people, LET PRON VERB. An easy way to use part of speech tags is by selecting them from the drop-down list (click on [PoS] to show it). You can also type the part of speech tags directly into the search form. Click here for a list of these part of speech tags. Previously, you had to use the part of speech tag (from the link above) inside of brackets, e.g. [j*]. But that's a bit cumbersome for mobile phones, and there are now different ways of specifying the part of speech -- all of which work equally as well. For example, all of the following would find the same strings: ADJ eyes, [j*] eyes, J eyes, _j eyes.
If you are using Type 1 or Type 4 above, you can use wildcards for the part of speech tag. For example, [nn2*] = plural nouns, [n*] = all nouns, [*n*] = nouns (including ambiguous noun/adj tags), etc. If you are using Type 2 or Type 3, it needs to be upper case: short NOUN. You can also add a part of speech tag to the end of any word, but you need to use either Type 1 or Type 4 above. For example, end would find end with any part of speech, but end.[n*] or end_n would limit it to end as a noun, and end_v or end_v would limit it to end as a verb. Make sure that you separate the word and the part of speech with a period / full stop and bracket (Type 1) or an underscore (Type 4), and that there is no space. Remember also that you can combine these with lemma searches to find all forms of a word with a given part of speech, e.g. END_v or END.[v*]. If you don't know what the part of speech tag is for a given word (or the words in a phrase), just select [OPTIONS] and then [GROUP BY] = [NONE] (SHOW POS). For example, see the PoS tags for light, back, front, or in light of If you capitalize an entire word, it will find all forms of that word. For example, DECIDE would find all forms of decide (decide, decides, decided, deciding), whereas decide would just find the single form decide. Another example: =CLEAN would find all of the synonyms of clean (scour, scoured, polish, polishes, etc), whereas =clean would just find scour, polish, etc. (Notice that we have also added the part of speech _v to the end, to limit these to verbs.) You can search by all of the synonyms of a given word, which provides powerful "semantically-based" searches of the corpus. For example, you can find the synonynms of beautiful, nonsense, or clean (v). Of course you can use the synonyms as part of phrases as well. For example, =CLEAN * NOUN, =clever =man, or =strong ARGUMENT. As the last example shows, synonyms can be very useful when you are a non-native speaker, and you want to know which related words are used in a particular context. As =clean * NOUN shows, not every token will actually be a synonym of a given word in every case. For example, scour may be a synonym of clean in scour the sink, but not in scour the library for good books. Note the it is often useful to limit the synonyms to those with a particular part of speech, as in clean_v. It is often also useful to find all forms of the synonyms, by capitalizing the word: CLEAN. And of course you can combine these as well, for example all forms of all synonyms of clean as a verb. Finally, note that you can click on the [S] to find synonyms for each word in the results set. This allows you to follow a "synonym chain" from one word to another to another...
The Hansard and EEBO corpora have been "semantically tagged", and you can use these tags as part of your search. A few examples are given below.
"User lists" or "customized lists" are word lists that you create -- related to a certain topic (e.g. sports, clothing, or emotions), words that are grammatically related (e.g. a certain subset of adverbs or pronouns), or any other listing that you might want. For example, click here to run a query based on two sample word lists that we created -- one with a list of colors, and the other with a short list of parts of clothing. There
are two ways to create a customized list: You can later view the lists that you have created, and modify the wordlist (add or delete words), or delete a list entirely. Once created, you can re-use a wordlist in queries at any time in the future -- they remain stored in the database on the server. The easiest way to include a list in the main search window is to just select it in the wordlist window. If desired, you can also type it into the search form directly. The format is: @listName 1.Select [SAVE LISTS] = 'YES' in the search form 2. Run a query, such as (finding synonyms of beautiful) 3. Select and de-select words from the list by clicking in the checkbox to the left of each word. Only the words that you select will be saved to the list. You can use the checkbox to the left of the [CONTEXT] button to select or de-select the entire list. 4. Enter the name you want to give to the list (in this case, maybe beautiful-syn). 5. Make sure you really have selected some words (step 2 above), and then click [Submit] to save your list. 6. If you want, select the list that you've saved in the customized wordlists interface. You can add to the list, modify entries (click M), or delete words from the list. 7. Finally, you can then re-use this list as part of subsequent queries. For example, if [mark_davies@byu.edu] has created and stored the list [beautiful-syn] then he could find cases of was ADV followed by one of these adjectives.
Many of the examples shown in the other sections are for individual words. But you can combine the different types of searches to create fairly complex phrases. For example:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
You can now do searches where there are a variable number of "slots". For example, the search: PUT (NOUN){3} away (click to run the query) would find strings with PUT at the beginning and away at the end, with up to three words between, at least one of which has to be a NOUN. In other words, it would do the following seven searches, one right after another, and would then display the results for all of the searches on one page.
In terms of search syntax, note that: 1. {n} indicates the number of words (0 to n) that can be in this "variable length" string. Valid numbers are 1, 2, or 3 (in other words, the longest variable length string is three words) 2. If you don't indicate {n} -- for example (NOUN) -- then it would be just one word -- meaning that it will be either that one word or nothing 3. Any "slot" without parentheses around it is obligatory. For example, put * away would not match put away, since * doesn't have parentheses around it. 4. You can't include multiple "flex" operators in a search. For example, they (VERB+}{2} notice (NOUN){3} would not be possible. The following are some additional searches. They produce interesting results in the one billion word COCA corpus), but the results in other corpora may not be as good. In each case, we show a few sample matching strings, and some strings that would not be generated by the search (and why not).
Some additional notes: 1. Because a "flex search" had involve up to seven different searches (see above), there are some limits on the number of flex searches in a given 24 hour period. For those who do not have a premium or academic license, there is a limit of five flex searches in 24 hours. Those who do have a license can do up to 50 flex searches in a 24 hour period. 2. Again, because of the number of searches that are done in a flex search, it would take a long time to do these searches if all of the "slots" are high frequency. This can be a real limitation in very large corpora like NOW (19+ billion words) or iWeb (14 billion words). So a search like HAVE (ADJ){3} time probably won't work in those corpora -- HAVE and time are too high of frequency. In a case like this, you will probably need to do these as a series of separate searches -- HAVE time, HAVE * time, HAVE * ADJ time, etc. But again, this should be a problem with a small corpus like the BNC.
CHART display
If you are interested in a set of words or a grammatical construction, then the LIST option shows the frequency of each matching form (end up being, ended up saying, etc), while the CHART option shows the total frequency in each section.
COLLOCATES display
See what words occur near other words, which provides great
insight into meaning and usage.
For example, nouns after
thick
or
look into,
verbs before
money, or any word near
crack,
believe,
loud, or
quickly.
The collocates search finds words near another word (i.e. within a "cloud" of nearby words), whereas the
LIST search finds an exact string of words.
For both the WORD and COLLOCATES field, you can include the full range of searches, including words, lemmas, substrings, parts of speech, and synonyms. For example, the following are searches for collocates of gap (n): any word, nouns, adjective, the word fill, synonyms of large.
Select the "span" (number of words to the left and the right) for the
collocates. Use + to search more than four words to the left or right, and 0 to
exclude the words to the left or right. If you don't select a span, it will
default to 4 words left and 4 words right.
You can use collocates to do "variable length" searches, where there might be 0-4 (or more) words between two other sets of words or phrases.
For example, you could find all of the following with one simple search.
COMPARE WORDS display
Compare the collocates of two words, to see how they differ in meaning and usage.
For example,
utter and sheer (note the negative collocates with utter),
warm and hot,
small and little, or adjectives near
boy and girl.
KWIC (Keyword in Context) display
See the patterns in which a word occurs, by sorting the words to the left and/or right. For example:
budge (v),
matter (n),
diametrically,
end up, or
naked eye.
Select the words that you want to sort with. Select L for 1, 2, and 3 words to the left. Select R for 1, 2, and 3 words to the right. You could also, for example, sort by one word to the left, then one and two words to the right. Click * to clear the entries and start over. Use the dropdown list to the left (POS or _pos) to input tags for parts of speech (PoS, e.g. nouns or verbs) into your search string. By default, it will add the PoS as a "full word", as in the searches strong NOUN or ADJ eyes. You can also have the PoS added as a "tag" on the end of a word, to limit the word to that PoS, as in the searches strike_n or and FIND_v. To make it insert PoS tags after words, click on _pos. To change it back to PoS as a separate "word", click on POS. You can find a wealth of information for the top 60,000 words in the corpus. As the following examples with bread show, you can see:
You can find a wealth of information for the top 40,000 words in the corpus, including:
SECTIONS
SHOW Determines whether the frequency is shown for each "section" of the corpus
.
For example, the
synonyms of beautiful in
each section and
overall.
OTHER OPTIONS
# HITS is the number of results. # KWIC is the number of results for a KWIC (concordances) search. GROUP BY determines whether words are grouped by word form (e.g. decide and decided separately), lemma (e.g. all forms of decide together), and whether you see the part of speech for word (e.g. beat as a noun and verb displayed separately). SHOW # TEXTS determines whether you see the number of texts in which a word or phrase occurs, in addition to its frequency. This can be useful in finding words and phrases that are limited just to a few texts in the corpus. (More information) CASE SENSITIVE determines whether She thought and she thought would be two different searches, or The Office, the Office, and the office. DISPLAY shows raw frequency, occurrences per million words, or a combination of these. SAVE LISTS allows you to create a wordlist from the results and then re-use it later in your searches.
SORT / LIMIT
Sort by raw frequency (e.g. hard * ) or by "relevance" ( hard *). Relevance uses the Mutual Information score. It is often useful to specify the minumim frequency when you are sorting by "relevance", to eliminate very low frequency strings. For example, collocates of green where minimum frequency = 1 (strange once-off strings) and where minimum frequency = 20. Note also that when you do a collocates search and you don't specify anything for the collocates field, it will automatically set MINIMUM to MUT INFO = 3 (Mutual Information score). It does this to remove high frequency noise words like the, to, with, etc. If you want to see more of these words, lower the MI score; to see less, increase it.
VIRTUAL CORPORA
Create a "virtual corpus" -- essentially your own personalized corpus within WIKI. You can create the corpus either by keywords in the texts (e.g. texts with the words investments, basketball, or biology), or information about the texts (e.g. date, title, or source), or a combination of keyword and text information. You can then edit your virtual corpora, search within a particular virtual corpus, compare the frequency of a word, phrase or grammatical construction in your different virtual corpora, and also create "keyword lists" based on the texts in your virtual corpus.
Click on any of the links above for more information. To create a virtual corpus by keywords, enter a word or phrase to the left, and then set TEXTS/VIRTUAL to FIND TEXTS (do it / undo; must be logged in first). You might also want to set SORT/LIMIT to RELEVANCE and MINIMUM FREQUENCY to something like 5 (the minimum number of times you want the word to occur in a text) (do it / undo). After clicking SUBMIT, you will see a list of matching texts from the corpus. For example, see matching texts for investment*, rocket, or electron. On the "results" page, choose how many texts you want in your virtual corpus, and then click SAVE LIST. After the virtual corpus is created, you might want to click on FIND KEYWORDS to see whether the corpus is providing the focus that you want. You can create a virtual corpus by selecting texts that match certain criteria -- such as title of the source (e.g. New York Times) or the title of the article, the topic, the date, and so on. Click on CREATE CORPUS to the left to see the interface to select the texts. As an example, this list was created by searching for articles dealing with biology. Note that in that search form, you can also make sure that the texts have certain words in them. If you want more control in finding texts with certain words, you might want to search by keywords. See list that was created by searching WIKI for articles dealing with biology.
See sample editing page (from Wikipedia corpus, but similar to WIKI). Explanation: You can add to or delete texts from your virtual corpus, or move texts from one virtual corpus to another. You can also rename and delete corpora, temporarily "ignore" corpora (for example, when you're comparing corpora. Finally, you can arrange virtual corpora into user-defined categories (science, religion, sports, etc). You can see what words occur much more in a particular virtual corpus than in the corpus overall. For example, see the keywords from the virtual corpus that is composed of articles dealing with biology. Once you have created a virtual corpus (by keyword or by text metadata), then you can search that set of texts as though it were its own corpus. You can search for matching strings, collocates (nearby words), and retrieve re-sortable concordance (KWIC) lines. To search one of the corpora, just select it from your list of virtual corpora, and then fill out the rest of the search form as you normally would. For example, you can search for the word cell in the biology virtual corpus. (Click on the word in the results list, and you will see that all of the occurrences are from your virtual corpus.) If you have created multiple virtual corpora, then you can compare the frequency of a word, phrase, or grammatical construction in these different corpora. Just enter the word or phrase in the search form (as you would do normally), and then select MY CORPORA (try it; must be logged in first -). |
