Text Analysis with AntConc for social media data: Word Lists, Word Frequencies and File View

In the last post, we learn the basics about AntConc. Now we are going to show you how to use AntConc to generate word lists (and frequencies) and the useful File Viewer.

Don’t forget that this post is part of a series of tutorials:

  1. Intro, Opening a File and Settings
  2. Word Lists and File Viewer (we are here)
  3. Concordancer and Concordance Plot
  4. Clusters and N-Grams (soon)
  5. Collocations (soon)

 

AntConc functions are accessed through the seven tabs below:

In that basic tutorial, we are going to follow the steps to produce simple word lists.

Remember to open your file and import the recommended settings for social media research [tutorial].

Generating and navigating in a simple Word List

  1. Open your file(s). In the examples below I’m going to use a dataset with 16k tweets in english containing the word ‘brazil’ (collected through Netlytic). Download the file brazil_tweets_16732tweets_2017_11_30.txt in our folder.

 

2. In the Word List tab, click on Start and wait a few seconds 

3. Now you can explore and navigate the data, scrolling down to find meaningful words, sort by Frequency, Word (alphabetical order) or Word end.

4. You can search for a term on the box at the bottom and click on the button “Search Only”:

 

5. If you click on any word, you’ll be directed to the Concordance tab. You can also read a tutorial on the Concordance tool (soon)

 

6. And if you click in any word on the Concordance tool, you’ll be directed to the File View tool. It functions like a simple text reader, where you can see the full corpora.

 

7. To export the list, just go back to the Word List tab and click on File -> Save Output.

 

8. The output is a .txt file that looks like this:

9. Then you can open or copy-paste the output in a spreadsheet software like Excel or Libreoffice to further analyses.

 

 

Filtering out stopwords

Stopwords are words that you don’t want to count or visualize. Usually, they are the most common words without semantic or topical relevance for your research problem (such as articles, pronouns and some adverbs).

 

  1. First, you need a stopword list! You can produce or edit a list yourself, but let’s start with an example list. You can download it on the lists folder.

 

2. To upload a stopwords lists from a .txt file, go to Tool Preferences -> Word List. There you’ll see the option “Use a stoplist below” in the “Word List Range” section. Click on Open and select your .txt file.

 

If you have done it right, the words will show in the box:

 

Now you just click on “Apply”!

  1. Go back to the Word List tab and click again the button “Start”. Compare the two word lists below. The first one was the original word list and the second one is the list with stopwords filtered out:

 

 

Counting specific words

Other useful Word List option is to count only specific words that you already know or that you just discovered in your corpora/datasets. Follow the steps below:

1. Firstly, you’ll need a word list. In our case, we are going to upload a list of words of brazilian soccer teams like that below:

 

2. Go to Tool Preferences -> Word Lists and open the Words from the file (download it in the lists folder). Click on “Use specific words below” and Apply.

 

3. Go to the Word List tab and click on ‘Start’ to generate the list again. The result will be a list of only the desired words:

 

4. Exporting that list (through File -> Save Output) and you can produce a Treemap like that visualization below with RAW Graphs:

 

 

Counting Lemma Word Forms

This is a optional step, if you want to aggregate the inflected forms of a word. For example, the verb talk may appear as talking, talks, talking and so on. Lemmatization aggregates those inflected forms to talk.

On social media data, this is important to investigate variations of a same root meaning, such as autism, autistic, “autist” related to a search query for vaccines for example.

 

Steps:

  1. In AntConc, the first thing you’ll need is a lemma list. You can download it directly from the AntConc website or in our wordlists folder. The file will look like this:

 

That means you can add or remove lines of lemma correspondences. As you can see above, that list doesn’t include the word autism. We could add the following line:

autism -> autistic, autistically, autist

(even though the word ‘autist’ doesn’t exists, it could be added because it is a common error between portuguese speakers, for example)

 

2. To add a Lemma List you just need go to Tool Preferences -> Word List and click Load on Lemma List options.

 

After you selected your file, AntConc will show you a preview. Click in “OK” and then “Apply”:

 

3. Now you can go back to the Word List tab and generate your list again. As you can see below, now AntConc counts Lemma Types and Lemma Tokens instead of Word Types and Word Tokens:

Lemmatization can greatly improve the preciseness of some claims about your corpora.

We hope Word lists, Word Frequencies, Filtering Stop Words and Lemmatization techniques will help you to explore and analyze your social media datasets.

 

The next AntConc tutorial will focus on Concordancer and Concordance Plot (soon)!

Text Analysis with AntConc for social media data: intro, files and settings

Broadly defined, (computational) text analysis is a set of techniques for automated content analysis. Even without the use of complex statistics or computational analysis, social science researchers can improve their data exploration with techniques involving word counting, co-occurrence and collocations.

AntConc is one of the most easy-to-use and useful tools for text analysis and corpus linguistics. It was developed by Laurence Anthony, Professor in the Faculty of Science and Engineering at Waseda University, Japan. He maintains dozens of tools in his website like TagAnt and FireAnt.

After this intro on AntConc, we are going to see the following posts covering its main functionalities:

  1. Intro, Opening a File and Settings (we are here)
  2. Word Lists and File Viewer
  3. Concordancer and Concordance Plot
  4. Clusters and N-Grams (soon)
  5. Collocations (soon)

 

The following matrix was proposed in the paper Computational text analysis for social science: Model assumptions and complexity and summarizes the possibilities between simple statistics/computation x complex statistics/computation and between weaker and stronger domain assumption. Using Antconc for analyze social media textual data encompasses simple statistics/computation tasks such as word counting and statistics, but can be further applied on dictionary-based word counting by topic experts.

To understand and compare approaches from computer-aided content analysis, computer-aided interpretive textual analysis and corpus linguistics, I recommend the paper Taming textual data: The contribution of corpus linguistics to computer-aided text analysis.

AntConc will allow you to perform the main techniques of corpus linguistics such as Word Frequencies, Collocation, Concordance, N-Grams, Corpora Comparison to any kind of text.

But first things first! Download AntConc and read the following text, which will teach you the basics about the settings and how to open a file.

 

How to collect social media textual data?

There are dozens of social media research tools which allow to extract or monitor textual data on the main platforms. Most of them collect data through keyword/hashtag search and/or from specific pages and websites. The majority of the following tools uses UTF-8 encoding to export files in .csv format. You can open them with Excel or Libreoffice and copy-paste the desired texts to a notepad and save it as a .txt file.

Repositories/curated lists of tools:

If you are entirely new to analyzing social media textual data, I strongly recommend you to try the awesome and user-friendly tool Netlytic and collect some tweets or youtube comments. But don’t worry: I’m going to give you some datasets in the following posts.

 

File Formats

AntConc can read several text formats: .txt, .html, .xml, .ant. The simpler one is the .txt file.

File Format Description
.txt .TXT is the simpler format to store text files. Softwares like Notepad, Notepad++, TextMate, Word and most of the word editing softwares can save your files as .txt.
.html .HTML is the standard format for saving web pages. You can save a webpage and upload it to AntConc.

AntConc has some settings to ignore text between the characters “< and “> used on HTML files.

.xml .XML files: Extensible Markup Language. It is similar to .HTML document, but uses custom tags to define objects and the data within each object. In corpus linguistics/text analysis, it is frequently used to mark each word with word categories in Part-of-Speech Tagging.
.ant .ANT is a file format used by AntConc, interchangeable with txt. It only saves the data on the current screen as an output.

 

  1. Encoding

It is recommended that you save your text files with UTF-8 encoding. A character encoding is a standard on how to process characters and symbols. UTF-8 is defined by the Unicode Standard, which englobes characters used in most Western languages and scripts. Due to that, several data collection tools use UTF-8 encoding as a standard. So, remember to save your files in UTF-8 encoding!

 

Optimal Settings for Social Media Texts

  1. Pre-configured settings

AntConc was not developed just for social media data but, instead, to analyse all sorts of texts, mainly literature, natural language and language corpora. It requires some adjustments on the software settings.

The specifications are listed below, but instead of following each step, you could just import a Settings File with the recommended definitions. Download the file antconc_settings_for_social_media.ant and, on AntConc, go to File -> Import Settings from File…, select and open the file:

 

That’s it! Now AntConc can be more useful for social media analysis. You can skip the following settings description if you have already imported the file.

 

2. Global Settings – Token Definition

In this section, we explain the recommended settings. Remember: you don’t need to follow these steps if you have just uploaded the pre-configured settings file provided above.

Firstly, we configured the token settings. A token is an element (word, character, punctuation, symbol, etc). In the Token Definition Settings, you can define which characters/symbols AntConc will consider when counting and processing your text data.

The default ettings are the following:

Default:

But, when we are working with social media data, there are some special characters used by social media users which represent specific conversation and affiliation practices. Two of them are very important:

The ‘@’ symbol: for Twitter users, the [at] symbol is used to mark user profiles. So, it is important to append the ‘@’ symbol. This will allow us, for example, to count the most mentioned Twitter’s users or the opposite: to filter out the usernames.

The ‘#’ symbol, in its turn, is a type of metadata used on most social media platforms to define hashtags. You need to append ‘#’ in AntConc token definitions to properly count hashtags.

Recommended:

So, we just need to go to Global Settings -> Token Definition, check the box “Append Following Definition” and include the signs ‘#’ and ‘@’.

3. Wildcard definitions

A WildCard is a character that can be substituted by a character, word or symbol during a query. AntConc has seven different wildcards. Below we can see the default settings (Global Settings -> Wildcards).

The problem is that two of these wildcards are attributed to very important signs on social media data: ‘#’ and ‘@’. This result means that AntConc “ignores” these two signs in the results, because they are reserved as wildcards.

So, we recommend to change these two wildcard to other signs. In the example below, we changed them for ‘{ and ‘} .

 

 

Opening your File or Corpora

  1. Opening your File(s)

To open a file or a set of files in AntConc you just need to go to File -> Open File(s)… or File -> Open Dir.

With the option File -> Open File(s) you can select one or more files.

If you open two or more files, AntConc will apply your queries and analyses on all of them at once.

This is very useful for managing datasets/corpora. For example: you could be analysing a year of data and save the texts (comments, posts, tweets) for each month in a different file. Open the 12 files at once allows you to compare things like: countings of specific words in the Concordance Plot; or Range of presence of clusters/n-grams.

 

Now we can talk about counting word frequencies. See you next post: Word Lists, Word Frequencies and File View

Debate Público nas Mídias Sociais: uma questão de letramento midiático digital

Por convite da Revista Rumos, publicação da Associação Brasileira de Desenvolvimento, redigi em dezembro pequeno artigo sobre o debate público nas mídias sociais. Está disponível no site da associação e no Issuu (páginas 26 e 27):

Salvador sediará o IV Simpósio Lavits: vigilância, gênero e raça

A LAVITS é a Rede Latino-Americana de Estudos sobre Vigilância, Tecnologia e Sociedade. De 26 a 27 de junho organizará seu sexto simpósio com o tema Assimetrias e (In)Visibilidades: Vigilância, Gênero e Raça. As chamadas para trabalhos individuais, sessões livres, oficina e mostras artísticas estão abertas até o dia 15 de março. No caso dos trabalhos individuais, o envio no momento é apenas dos resumos (até 300 palavras) e os trabalhos completos aprovados deverão ser enviados até 30 de maio. Confira mais informações:

 

Negação do racismo no TripAdvisor: estratégias discursivas

O racismo, como elemento pervasivo, se manifesta de formas particulares nos campos do turismo e hospitalidade. Viajantes, pesquisadores e ativistas experienciam e analisaram isto em novas plataformas digitais semi-automatizadas por interfaces e algoritmos como Airbnb, levando empreendedores a criar plataformas de hospedagemagências de turismo mais seguras.

Durante viagens para outros locais ou países, os viajantes racializados são mais fragilizados, estando sujeitos a atos racistas que dificilmente serão confrontados imediatamente devido ao risco intensificado. Plataformas de reviews online, entretanto, pode(ria)m ser um dos espaços de reação através do relato das experiências negativas.

Leia texto relacionado “Traveling while black” no blog For Harriet

O artigo Traveling Across Racial Borders: TripAdvisor and the Discursive Stragies Business Use to Deny Racism, entretanto, apresenta resultados assustadores de pesquisa de Heather M. Dalmage da Roosevelt University. Publicado na revista Sociology of Race and Ethnicity, o artigo aplica análise de discurso para entender como os hotéis e restaurantes respondem e negam atitudes racistas especificamente ao receber casais interraciais.

A autora apresenta um histórico sobre os processos de racialização do Outro e como são especialmente intensos em países de maioria branca. Escrevendo do ponto de vista de uma pesquisadora estadunidense, Dalmage revisa as categorias de casais e comunidades inter-, multi- e bi-raciais para, em seguida, adentrar nas particularidades das plataformas digitais. Citando Hughey e Daniels, explica que “raça e racismo persistem online em modos que são tanto novos e específicos na Internet, junto a vestígios de formas seculares que reverberam tanto off quanto online“. Ao analisar a interface da plataforma, guias de como responder a avaliações de hóspedes de consultorias e da própria TripAdvisor, a autora identificou que as recomendações explicitamente invisibilizam a questão, ignorando como lidar com casos de racismo.

O estudo inédito reuniu 233 casos no TripAdvisor, através de buscas por termos como relacionamento interracial. Deste total, apenas 50 foram respondidos e entraram na fase de análise de discurso. A autora identificou que a maior parte seguem um padrão discursivo de negação da raça e do episódio racista, com o objetivo de patologizar os casais avaliadores.

Quatro táticas foram identificadas e são detalhadas na seção seguinte: (a) “Você está errado! Eu não sou racista”; (b) Explicação não ligada à raça; (c) “Nós tratamos todos do mesmo jeito”; (d) “Você entendeu errado e/ou poderia ter evitado a experiência negativa se tivesse agido diferente”. Nas conclusões, a autora explica como, em um mundo racializado e estruturado pela supremacia Branca, as práticas de evasão de debate sobre raça e racismo servem aos ideais neoliberais deste mundo. A análise do discurso é especialmente importante pois “as fronteiras raciais, desenvolvidas através da colonização e capitalismo global e infundidas em identidades, ideologias e instituições agora estão sendo criadas em novos e não tão novos modos online”.

Leia o artigo completo na revista Sociology of Race and Ethnicity.