INS202 is an introductory module on research methods in International Relations. One aim of the module is to introduce undergraduate students to methods that provide low-threshold opportunities for collecting and analyzing their own data. Content Analysis is such a method. While researchers have developed advanced and complex techniques of content analysis, simpler approaches are often sufficient to identify and discover patterns in communications that can help answer students’ research questions in political science and international relations. Data such as media reports or government documents are often readily available on the internet and through library databases. Such documents can subsequently be analyzed through relatively simple coding practices. Thus, content analysis provides undergraduates with a low-threshold method for data collection and analysis in their final-year projects.

 

As a text-oriented method, this module explores the potential benefits of integrating AI language models into content analysis. Maybe the most common form of content analysis is seeking to quantify qualitative data through coding, that is the process of assigning numerical codes to text and other forms of communication-based on certain rules. The coded data can subsequently be analyzed with simple descriptive statistics. In other words, in its simpler forms, content analysis revolves around identifying words, themes, topics, etc., across a set of communications, statistically discovering patterns in their usage that can help answer research questions.

 

Against this background, integrating XIPU AI into my module’s discussion of content analysis serves two purposes: Firstly, a quick classroom survey revealed that despite the buzz, most students in the module have never used XIPU AI. On a more practical level, I thus wanted to provide students with hands-on experience using AI as a research assistant. Secondly, XIPU AI served as a pedagogical tool to highlight the researcher's decision-making process during data coding and the inherent limitations of relying on AI-driven analysis, including practical hurdles and reliability concerns.

 

To these ends, AI was introduced in a 3-step process. Firstly, students were tasked to manually code media reports. In step 2, they asked XIPU AI to assist in coding with simple prompts. In step 3, we tried out more complex prompts for the development of coding manuals and schedules.

 

 

During the COVID-19 pandemic, the communications and policies of the World Health Organization became a point of political contestation between governments. To illustrate this point, students are asked to analyze media reporting revolving around the World Health Organization during COVID-19, to discover patterns and biases in reporting. To operationalize this problem, we want to measure how Chinese English-language media portray the actions of various political actors in the context of the WHO. Which actors are identified, and how are their actions portrayed?

 

To operationalize this task, students receive a coding manual and schedule. They are instructed to code selected newspaper articles along two dimensions: “actors” and “actions”. The categories in the former dimension are undetermined, requiring students to develop and augment categories in the manual as they encounter different actors in the analyzed texts. The dimension “actions” has a fixed set of categories asking students to evaluate how actors’ actions are portrayed (1 positive; 2 negative; 3 neutral; 4 no action).

 

Working in groups, students collaborate on coding tasks. The coding manual and schedule are shared within and across groups and collaboratively edited and appended on XJTLU Box (XJTLU Cloud Storage Platform).

 

This manual coding process confronts students with several problems that they need to reflect on critically. For example, how should actors be categorized? Does each deserve their own category, or should we group them somehow? How do we decide whether an action is portrayed as positive, negative, and so on? Ensuring consistency in interpretation across groups highlights the problem of inter-coder reliability.

 

 

While there is no shortage of powerful programs and programming libraries to assist in content analysis, the complexity of such software can pose an intimidating hurdle for undergraduate students in conducting their research. XIPU AI, relying on natural language input, offers a more accessible and practical alternative to assist in coding large amounts of text.

 

Students are introduced to XIPU AI and develop a simple prompt that asks the AI to apply our two coding dimensions to its analysis of a newspaper article: 1) actors and 2) how their actions are evaluated (screenshot 1).

 

 

Students are then asked, firstly, to compare their interpretations and coding developed in step 1 with that of the AI, and, secondly, to compare AI outputs between groups.  Through this exercise, students gain a deeper understanding of the decision-making process inherent in coding text, including the necessity of establishing precise rules for category development and application.

 

The exercise also raises issues regarding inter-coder reliability: just as humans tend to apply rules in different ways and thus tend to code differently, XIPU AI will produce different outputs based on similar or even the same prompts. For instance, the AI may overlook certain actors initially, requiring students to point out and rectify this error. In some cases, the AI volunteered great detail as to why it evaluated actions in one way or another, while at other times, it just provided a factual “positive” or “negative” evaluation, and so on.

 

An important finding for the students thus is that just like fellow human coders, XIPU AI needs to be supervised and trained (prompted in more detail) to produce more reliable results.

 

Some of these limitations can be addressed by developing more detailed prompts that produce more reliable results.

 

 

More advanced and detailed prompts allow students to not only query the AI for its interpretation of individual texts but also to use it to develop and maintain a coding manual and schedule. Screenshot 2 shows the results produced based on the AI’s coding of four articles, after regularly providing XIPU AI with various prompts to refine its coding methods and output. Here, XIPU AI now understands that it needs to provide a coding manual that it needs to update as new actors are encountered in texts. It also provides an easily readable coding schedule in the form of a table. 

 

 

The process to achieve this output involves an intuitive step-by-step approach that asks XIPU AI to refine its output and produce a manual schedule that resembles those used in manual coding. We reviewed three articles before arriving at the format above, which, at least ostensibly, suited our needs.

 

However, we also found that XIPU AI needs to be reminded frequently of the rules defined in our coding manual. For example, it would not always identify all actors mentioned in an article, or correctly evaluate their actions. Also, even with more advanced and detailed prompts, when using the same or similar prompts, different users would receive different outputs from XIPU AI.

 

 

The integration of XIPU AI for content analysis into INS202 offers several advantages. Firstly, it provides undergraduate students with limited research methods experience with a tool for text analysis, which is easier to use than established software and programming libraries. Secondly, trying out XIPU AI for content analysis can also serve as a pedagogical tool to create awareness of issues commonly encountered with this method. For example, coding “in dialogue” with the AI alerted students to issues commonly encountered when coding text, such as decisions that need to be made when developing and applying categories, or the problem of inter-coder reliability. The experience with XIPU AI thus serves to highlight the importance of a balanced and thoughtful approach to leveraging AI technology in research methods.