# Full Text: Tracking Public Sensemaking through Rhetorical Annotation of Image Memes

> Extracted from `ECOMEME2-v1.0.pdf`

---

## Page 1

Tracking Public Sensemaking through
Rhetorical Annotation of Image
Memes
Mridula Mascarenhas 1, RJ Cordes 2, Bleu Knight 3,4, Sarah Murphy 5, Daniel Friedman 2,3,6
1.
California State University, Monterey Bay, CA. School of Humanities & Communication.
2.
COGSEC
3.
Active Inference Institute
4.
New Mexico State University, Las Cruces, NM. Department of Biology.
5.
Seemeless Solutions
6.
University of California Davis, Davis, CA. Department of Entomology & Nematology.
July 25, 2022
Version 1.0
Abstract:
Political polarization and declining trust in institutions are driving societal destabilization and
radicalization. Recently there has been increased interest in online misinformation intervention
and deterrence, for example through the use of machine learning on language use. We argue
that addressing crises in the information environment will require a sharper situational
awareness and a deeper understanding of how beliefs emerge and crystallize, as well as greater
connectivity in the work of teams and organizations in order to reduce the effects of bias and
partisanship in collection and analysis. Image memes play an increasingly important role in
public sensemaking and discourse and the emergence of public beliefs. Despite their
significance, image memes have proven to be a very difficult category of artifact to collect,
classify, and analyze in aggregate. In this white paper, the function and form of image memes
are discussed, the challenges of performing image meme collection and analysis within the
context
of
emergent,
interdisciplinary
teams
are
detailed,
and
requirements
and
recommendations for alleviating these challenges are offered.

## Page 2

A Case for Rhetorical Annotation of Image Memes
A growing loss of faith in normative institutions and official accounts of historical or current
events is violently destabilizing social bonds. One of the drivers of this destabilization is a
growing divide between those who still grant credibility to institutional narratives and those
who have gravitated via social media toward counter-institutional narratives. The rise of
counter-discursive
online
communities
has
produced
shared
identities
constructed
through shared narratives. Recent years have seen the emergence, among social media
users around the world, of the “truth-teller” or “digital warrior” identities constituted by the
consumption and sharing of counter-institutional narratives [1].
Specifically, counter-narratives are being seeded and maintained through image memes.
There are now numerous operational and formal definitions of image memes, sometimes
referred to as “internet memes” [2,3]. The newer definitions serve to disambiguate the
image-embellished-with-text, ubiquitous on social media platforms, from the more general
“meme” originally proposed by Dawkins [4] and further developed by Blackmore [5],
Dennett [6], Heylighen [7], and others, referring to any “cultural component passed from
one individual to another by non-genetic means, or imitation” [3].
In this article, we focus on the "image meme" format (as seen in Figures 1, 3, 4, 6, and 7),
operationally defined as a shareable, digital image that contains either non-textual visual
symbols or text or a combination of both. While other media formats (e.g. GIF, audio, video)
are also ubiquitous, we focus on the image meme in particular to articulate scalable
computational systems for rhetorical annotation and analysis which could enrich current
analysis methods such as sentiment, semantic, narrative approaches.
While image memes used to be regarded as ephemeral detritus of the Internet intended
primarily to induce humor, the powerful role they can play in the formation of public belief
and sentiment is becoming increasingly evident [8,9]. Amidst the voices calling for urgent
study of the memetic construction of public belief [10], we emphasize the need to examine
image memes for their communicative (rhetorical) function as quasi-arguments in public
discourse [11].
We recommend applying the structural framework of the Toulmin model of argument
analysis to trace the public argumentation performed by image meme circulation [11]. We
advocate the application of an argument framework because image meme content is being
widely
used
to
advance claims that appear reasonable, despite minimal evidence
presented within the meme. Such memetic content has demonstrated strong potential to
shift public belief and spur public action [12]. The Toulminian model identifies three
1

## Page 3

fundamental components of an argument - claim, evidence, and warrant [13]. The claim
refers to the proposition that the audience is being asked to accept. The evidence refers to
information that supports the claim. The warrant, often not articulated, refers to each
assumption that connects evidence to claim.
Figure 1. Example Image Meme.
For example, rather than dismissing the meme above as intended purely for humor, an
audience enculturated to reject institutional narratives recognizes that the meme advances
the claim that space agencies such as NASA are engaged in long-term deceit. The meme
does this by presenting an argument that can be outlined using the Toulmin model as
below.
●
Evidence
○
IF a security camera offers only a grainy image
●
Claim
○
THEN space agencies are lying about images from space telescopes.
●
Warrants (unstated but implied claims which are already established as true for the
audience)
○
BECAUSE
2

## Page 4

i.
We should be able to see someone on a security camera much more
clearly than celestial objects in space
ii.
Space agencies have a history of presenting fiction as fact (e.g. faked
moon landing, doctored images)
The advantage to annotating memes using an argument framework is that we begin to
identify how certain ideas which are not contained within the image meme itself (i.e., NASA
is a fake organization) and sentiments (i.e., suspicion toward NASA) propagate across
publics. Argument analysis reveals the power that memes have to shape public belief by
simulating appeals to logic, even spuriously, by functioning as quasi-arguments. Memes
function as
arguments when they invite an audience to accept a claim, based on
evidentiary information that is sometimes contained in the claim and sometimes implicit.
Nevertheless, for an audience to accept the claim, the meme also relies on implicit
warrants. Interventional approaches to addressing memetic circulation of misinformation,
therefore, need to target, expose, and challenge spurious evidence and hidden warrants
that specific memes rely on, in order to induce skepticism toward the memetic form as a
mechanism for sound reasoning. Fact-checking articles designed to debunk memes have
limited success in dismantling the power of memetic argument, in part because they
assume a different rhetorical form. Counter-memes that identify and challenge the
argument components of original memes could present a more targeted strategy.
However, we advocate strong caution in the use of this approach. Counter-memes should
not be used to advance novel competing claims but instead highly purposefully to
dismantle
already
circulating
spurious
memetic
arguments,
since
the
objective
of
intervention is to challenge reliance on memes for public sense-making.
A rhetorical argument-based approach to analyzing image memes can advance our
understanding of their persuasive influence beyond the current practices of syntactic
tagging of memes, for example by text recognition [14] or classification of memes into
categories based upon visual similarity. Previously, we have argued that a rhetorical
approach fills in the gaps endemic to tagging practices by enriching analysis of image
memes with rich semantic information embedded in the parsimonious combination of the
meme components [15]. While in recent years, small-scale rhetorical analyses of image
memes have been published [16], these methods have not been implemented widely.
Much attention has been paid to specific large-scale shifts in public beliefs, such as vaccine
rejection [12], COVID denial, and rejection of various global election results. However, we
argue that what deserves more attention in academic, political, and security analyses is a
focus on the underlying common thread running through these far-ranging belief and
3

## Page 5

attitude shifts, namely the mechanism of change: the creation and maintenance of shared
beliefs, and the provocation of shared emotions through memetic persuasion [15,17–21].
Despite legitimate concerns about the degradation of public information due to the
infusion of spurious counter-institutional content (e.g., “fake news” and misinformation), we
argue that viewing the information crisis as a competition between truth and falsity
obscures the nature of the digital information crisis we are facing and, worse still,
hamstrings efforts to restore trust and rework social consensus. Framing the information
crisis as a battle between true and false information has not proven effective in regaining
the trust of those disaffected by mainstream channels of information. A simplistic
true/false dichotomy ignores the complexity inherent in counter-institutional narratives
and furthermore prevents us from studying the rhetorical conditions that enable the
subversion of mainstream narratives by competing ones. Deploying this dichotomy
through strategies such as fact-checking pop-ups that overlay memes on Facebook can
actually undermine the ability of good-faith actors to either correct or contribute to
competing narratives. Those who seek to address our information crisis will need to do
more than target and neutralize alleged sources of misinformation - they will need both a
deeper understanding of how beliefs emerge and crystallize, and a sharper situational
awareness.
The scale and rapidity with which emerging political and social events are being co-opted
into counter-narratives is possible because of the extreme parsimony and virality of
memetic
argument.
Since
the
split
between
institutional
and
counter-institutional
narratives has solidified as a social schema, emerging events create a vacuum into which
counter-institutional content can be introduced. This content, in image memetic or other
forms, has the capacity to nudge social actors into rejecting institutional narratives about
events and can reinforce the rejection of political, corporate, and social institutions.
Therefore, information systems for large scale analysis of memes as persuasive artifacts
are urgently needed. Such information systems have the potential to provide early
indications and explanations of shifts in public belief and attitude, which can then be
measured with more sensitive and reliable tools. While there are tools which offer
operational situational awareness related to sentiment in text-based artifacts [18], image
memes have proven to be a very difficult artifact to collect, classify, and analyze in
aggregate and there are no standardized practices or appropriate tools for this process.
Even outside the context of analysis, no viable and accessible methods exist for systematic
search or collection of image memes.
Accordingly, in this article, we build on previous work that proposed a computational
framework, combining rhetorical analysis with an ecosystem approach (see Figure 2), to
4

## Page 6

trace the ebb and flow of narratives through memetic circulation across digital publics [15].
We first provide background on the rhetorical form and function of the image meme. We
then offer a set of vignettes to communicate the challenges practitioners face in collection
and analysis of image memes, and to explore what tooling and related capabilities would
alleviate these challenges. Finally, we provide recommendations for developers who are
working on related technologies and those who may be interested in providing the
necessary infrastructure for an ecosystem-approach to enable situational awareness in the
(mis)information environment.
Figure 2. Digital Rhetorical Ecosystem three-tier model (DRE3), from [15].
5

## Page 7

Elaboration of the Rhetorical Form and Function
of Image Memes
We can consider image memes in terms of two complementary features - form and
function. The form of the image meme is established by the rectangular box frame which
circumscribes one or more rhetorical elements, demarcating the meme as a discrete
communication unit on platforms like Facebook, Instagram, and Twitter. While image
memes perform a variety of rhetorical functions [22,23], we restrict our attention to image
memes that play a particular rhetorical role by participating in public argumentation by
advancing claims [24].
Although image memes can be circulated to drive any narrative online, they have marked
success in the disruption of official narratives across the political spectrum [9,12]. Their
truncated or compressed form is well-suited to inject targeted challenges to mainstream
claims. The parsimonious form of the image meme provides great capacity for semantic
encoding to advance persuasive claims while diminishing burdens of proof and elaboration
that other rhetorical artifacts, like news articles, would require (or be expected to provide).
Various image meme formats exist, such as text-only, image-only (no textual elements),
screenshot, and image-text juxtaposition. These varied formats, and combinations among
them, can create polysemic affordances [25]; that is, they create the possibility of extracting
multiple and multi-layered interpretations within a range of meanings. The strategic
ambiguity inherent in memetic artifacts allows for rich semantic encoding. At the same
time, the structural features of the memetic form (i.e., the containment of its content in a
box, and the text/image syntax) strategically constrain meaning-making by setting up the
key elements of an argument and cutting off counter-arguments. Below, in Figure 3, we
illustrate the argument development contained in one sample image-text meme.
Figure 3 constructs an argument with the simple juxtaposition of two lines of text above
and below a stock photo. The choice of the photo combined with the double textual
framing relies on the contextual knowledge of discursive communities to decode the
argument. While the explicit memetic content is sparse, its signifying layers are rich, thus
allowing the meme to advance a clear and persuasive claim.
The primary claim distilled from this image-text meme is that the official masking policy to
combat the virus is not to be trusted. The rhetorical power of the meme draws from its
strategy of juxtaposing two official narratives that appear to be mutually exclusive - that is,
if the virus is virulent enough to escape the strict safety protocols of a world-class
laboratory [evidence], then ordinary masks should be ineffective [claim]. The implied
6

## Page 8

warrant in this case is that both statements cannot be true at the same time, which evokes
the broader warrant that official accounts of the virus’s origins as well as official policies to
combat the virus must be false. The meme simultaneously alleges dissonance in official
claims and expresses a disdain for those who accept the official narratives and are
oblivious to the dissonance. The meme carries both content designed to appeal to
audiences’ logical reasoning as well as to activate an emotional charge in the audience. The
logic and emotion evoked by the meme are abetted by the meme’s use of the
“Condescending Wonka” image deployed memetically since 2011 to convey patronizing
sarcasm [26].
Figure 3. “Condescending Willy Wonka” image meme.
The two lines of text interspersed with the image interpellate an audience into the persona
of Condescending Wonka, questioning not only the official COVID-19 narratives but also the
intelligence of those who have not yet figured out the contradiction. The meme positions
the audience that agrees with its claim on one side against lying officials and people that
trust official narratives on the other. The rhetorical deftness of this particular image text
meme lies in its ability to swoop an audience, in the course of a single engagement with the
meme, into both the line of reasoning set up by the meme and into an interpellated
audience identity. Even as viewers might be encountering the meme’s reasoning for the
first time, having followed the reasoning and accepted it, they come to embody the
persona of the one questioning the official narrative and distinguishing themselves from
7

## Page 9

those who don’t. The semantic decoding effort demanded by the meme works to enhance
the credibility of the meme’s claim by interpellating audiences as truth-discoverers. By
advancing claims, memes not only shape public beliefs but also constitute powerful
rhetorical audiences, knitting together discursive communities that share memes and bond
over decoding and accepting memetic claims.
The boundedness of the image meme above (i.e. its containment with the rectangular box
frame) and the parsimony of the rhetorical elements within the meme inhibit central
processing and encourage peripheral processing of the meme’s claim [27]. The particular
rhetorical form of the meme thwarts further questioning into possible reasons why the two
supposedly contradictory claims may, in fact, not contradict each other. The success of the
meme’s argument relies on the implicit warrant that the virus’s escape from a protected lab
and the possibility of a mask protecting the wearer from the virus are mutually exclusive.
The
possibility
that
initial
spread was virulent because the virus encountered an
unsuspecting maskless population is elided by the memetic structure. Likewise the claim
that masks only mitigate but do not necessarily prevent infection is also obscured by the
certainty evoked in the meme’s juxtaposition of claims. Image memes often simultaneously
function as assertive yet weak arguments. Their weakness lies in the fact that their
parsimonious form limits elaboration, specifically hiding warrants. However, the parsimony
is also responsible for obscuring the weakness of memes. The limited information, visually
bounded by the meme’s rectangular box, seals a particular conclusion while deflecting
attention from warrants that could challenge the meme’s claims.
Given the rhetorical power of meme circulation, as elaborated above, to shape public
belief, opinion, and sentiment at this time, we urge large-scale collection and analysis of
memes that circulate via social media. The ethics, legality, and implications of collecting
social media information are a complex and fraught issue, beyond the direct scope of this
article, though we discuss some related aspects. Importantly, we have argued previously
that personally-identifying information is not necessarily required in collection and analysis
of memes [15], elevating privacy protection to a key concern.
Collection and analysis of image memes at scale pose numerous unprecedented challenges
that current practices are ill-equipped to meet. This effort will require teams that are
curated and structured for efficient and optimal analytical outputs. In the next section, we
outline the obstacles that collection and analysis teams are likely to face and, accordingly,
recommend practices for structuring such teams, their processes, and outcomes.1
1
Outcomes might include situation reports, research products, actionable intelligence, or
stewardship of artifact collections for posterity (e.g., the world’s largest meme museum).
8

## Page 10

Limitations on Artifact Collection by Emergent Teams
Presently, the process for collecting image memes and similar “artifacts” [28,29] for analysis
of social narratives, sometimes referred to generally as “artifact collection”, depends almost
entirely on the community or communities involved. There are limited best practices, no
use-case specific tools, and collection is performed in a wide variety of contexts, sitting at
the intersection of myriad fields, including advertising, rhetorical analysis, information
operations, misinformation response and intervention, and cognitive security [18]. Given
both the plethora of approaches and stakeholders, and the complexity of the phenomena
they seek to address, traditional organizations find it difficult to meet objectives in the
absence of cooperation with other groups or reconfiguration [18,30]. As such, it is often the
case that interorganizational or interdepartmental teams emerge to perform collection and
analysis. Here we consider the state of the art for performing analysis of image memes at
scale by emergent, interdisciplinary, interorganizational teams seeking to understand the
patterns of public discourse around current events. Specific examples below illustrate how
memetic analysis can reveal widely-shared public beliefs and opinions on the ongoing
Russo-Ukrainian War.
Teams that intend to engage in image meme analysis may follow myriad paths in pursuit of
their goals. Below, we offer an overview of the archetypal phases encountered by three
common approaches to image meme analysis undertaken by emergent teams, based on
the experiences of the authors. The three approaches are listed in order of increasing
refinement of methodology and intensity of resource use (time and computational). The
three approaches, referred to as Haphazard Collection, Methodological Collection, and
Automated
Collection
and
Analysis,
are
each
accompanied
by a description of
recommended capabilities and affordances which would alleviate their respective pain
points (summarized in Tables 1, 2, and 3).
Haphazard Collection
An emergent team seeking to perform memetic analysis will generally begin its journey
through haphazard collection and sharing of memes over some common channel. In the
worst case, sharing occurs over email. However, even in the best case, sharing often occurs
over an asynchronous chat platform with affordances for setting specific communications
channels (i.e., Keybase, Signal, Discord, or Slack). Given that no current tools offer low-code
out-of-the-box capabilities for assisting in multi-modal media collection beyond offering
storage, the onus is on the organizers and facilitators to expend extraordinary effort to set
standards, maintain norms, and motivate members, in order to transform haphazard,
general discussion into an artifact collection pipeline. Paradoxically, the social enforcement
9

## Page 11

of such standards and rules too early can be demotivating and reduce enthusiasm for
contribution, effectively requiring a forward-thinking group to choose between losing most
information from early collection efforts by not strictly enforcing input standards or risking
demoralization and reduced engagement by doing so. It is here that many emergent teams
will experience mission drift, a slow process of disintegration, or illusory progress in the
form of discussion and collection without definable utility or outcomes.
A team that fails to formalize a methodology will rely on haphazard collection by default
and may go through the following stages and challenges (summarized in table 1):
Initial Enthusiasm. As image memes flow in, the team calibrates a
common
situational
awareness
of
the
information
environment
through discussion, links to the locations where images were found,
and informal references to the events and entities which the images
reference explicitly or implicitly. This common situational awareness
paired with the social bonding over a shared sense of purpose can
create a broad enthusiasm resulting in bursts in collection activity that
attracts new members. However, this initial enthusiasm requires
organizers and those most committed to the work to now have to focus
their attention on onboarding, administrative “housekeeping”, and
moderating discussion in order to keep the group’s focus on collection.
Internal Disruption. Relevant to discussion of this stage is a design
principle in engineering, referred to as “separation of concern,” which is
described as the adequate isolation of concerns, documentation, and
objectives of each system component, such that the component can be
error tested and distinguished from the other components with which
it interacts [31]. While at the implementation level this design principle
specifically refers to functions and blocks of code, at a conceptual level
it has been recognized for use outside of engineering domains in
guiding design granularity and modularity to improve operational
reliability, collaborative productivity, or process visibility [31–34].
Teams often attempt to implement a separation of concerns through
the use of multiple communications channels. However, the lack of
collection-specific affordances in the chat platforms in-use means there
is
often
an
intermixing
of
unstructured
discussion,
structured
comments, and collection activity, as well as non-mission focused
activity such as professional networking and sharing of unrelated
10

## Page 12

materials, all in the same spaces. This leads to miscommunications and
disruption in collation of relevant objects. Given the subjective nature
of image memes themselves and the viscerally emotional states they
may provoke, no separation of concern between collection and analysis
activities can induce political and operational schisms within the team.
These schisms do not simply re-enact broader social dynamics within
the group, instead, they introduce rhetorical divergence that can induce
cascading organizational failures. For example, difference in opinion
can lead to bifurcation (i.e., professional relationships, friendships) even
though variation in perspective on that same issue might be tolerated
by a political party or company (which may have sufficient size and
mechanisms for maintaining organizational coherence).
As a practical example, consider a team seeking to understand the
discourse around the ongoing Russo-Ukrainian war which has collected
an image meme referencing Nazi ideology in relation to Ukrainian
para-military groups (see Figure 4). The intentional or unintentional
strategic ambiguity embedded in the meme means both the quality of
the claims and the claims themselves, are in the eye of the beholder.
Thus, the team’s own diversity of perspectives, without high levels of
cognitive security and trust, becomes a complex threat surface [30].
One member might interpret the meme in Figure 4 as a reference to
Facebook’s reversal of content moderation policy regarding the Azov
Battalion, which, prior to the war, had been classified alongside other
white supremacist groups [35]. However, another might interpret the
meme as a mockery of anti-fascist movements in the US and libel
regarding
the
Ukrainian
military.
Given
the
difficulty
of
rapid
synchronization of priors, unstructured discussion will almost certainly
lead to disagreements in analysis which may expose meaningful,
underlying
sociopolitical
and
philosophical
disagreements.
Where
teams lack established affordances, roles, norms, trust, and separation
of
concerns,
there
is
an
enhanced
potential
for
such
miscommunications which degrade trust and potentially result in
disintegration of the team. Instead, a separation of concerns through
tool affordances and role-based access is more likely to ensure that the
analytic stage is focused on “trending” rather than “idiosyncratic”
interpretations of memetic claims.
11

## Page 13

Overload.
If
the
team
manages
to
maintain
momentum
and
circumvent tendency toward internal disruption and disintegration, it
will next face challenges related to the volume of its collections.
Depending
on
affordances
offered
by
the
chat platform in-use,
plain-text and links might be exported from chat for analysis or may be
searchable. There is rarely a simple method available to teams for the
search, categorization, or export of image memes. Even if exported,
brute force or manual search, as opposed to query- or attribute-based
search, is likely the only method available. This being the case, the
more successful the team is at unstructured collection, the less the
value of any particular artifact given that the time and effort costs of
brute force search increase with the size of collection. Due to this
volume-value paradox, success in collections has a direct, inverse
relationship with difficulty of analysis.
Lack of Visible Progress. If the team continues operations to collect a
relatively high number of artifacts without having resolved separation
of concern, search, and collation difficulties through standards setting
and compensating controls on inputs - it has likely already undergone
some level of “mission creep” or deviation from original goals [36]. As
such, the team will likely have no visible markers of progress, which can
result in a feedback loop of decreased activity and demoralization of
still-active
members.
If
there
is
no
clearly
defined
process
for
disintegration or removal of team members based on activity or work
requirements, this feedback loop will eventually result in the team
ceasing operations as opposed to formally closing.
12

## Page 14

Figure 4. “Fast Friends” Political Cartoon [37]
Figure 5. Balancing aspects of artifact collection
13

## Page 15

Methodological Collection
After
general,
exploratory
collection,
a
team
may
then
formalize
their
methodology for managing collections. Potential stages and key challenges are
described below and summarized in Table 2.
Tool Selection. In order to begin collections, the team must choose a
tool for artifact collection, which offers accessible input affordances
(e.g., form input) for images and text (e.g., Google Sheets, Coda.io). The
larger the team, the more likely it is for conflicts to arise during this
process of tool selection. Unfortunately, very few tools in common use
have accessible affordances for connecting data such that members
could continue using their own tools while being able to collaborate on
common digital assets.
Tool Adoption and Configuration. Technical difficulties may occur
with adoption and early use of tools or tools may run afoul of
organizational security protocols. Moreover, individuals could simply
refuse to adopt new tools due to platform or tool overload. Even where
members may be using the same platforms already, if members have
their own processes or organizational accounts for managing digital
assets, difficulties can arise in where and how to store common assets.
During this time, the team may lose members, see a decline in activity
and interest, or the team may disintegrate entirely. Poor configuration
of
the
tool’s
features
(e.g., incorrect sharing options leading to
inaccessibility) can exacerbate the impact of these challenges.
Maintaining Information Quality. Given a successful migration to a
new tool, the team must decide how to set standards for input, such as
including certain attributes, links, or other details. Here, the team has
to balance the user-experience of the person performing data input
with that of the person who will later perform analysis. The more detail
that the team requires, the more opportunities for analysis later - but
every additional required detail comes with potentially large costs.
Even a very motivated and interested team may see steep declines in
activity where input controls are too strict and details required for
collection are too voluminous or complicated. Making inputs optional
can
create
an
inconsistency
that
could
potentially
demotivate
14

## Page 16

detail-oriented and conscientious team members. Unfortunately, any
adaptation will be accompanied by tradeoffs, as the team must balance
information quality controls with artifact detail requirements and rate
of
collection
goals,
each
impacting
one
another
(see Figure 5).
Regardless of tool choice, whether or not the team achieves adoption,
and the detail of annotation of artifacts, nearly all tools will require the
user to switch back and forth between the tool and their browser
during collection. This context switching is cognitively expensive, and
can result in further declines in information quality and enthusiasm for
collection due to poor user experience.
Regardless of scope and controls the team will also have to deal with
difficulties in managing duplicates and reducing redundant collections.
As the team finds image memes during their collection activities, there
is no user-experience friendly method to ensure the item being viewed
hasn’t already been collected or if the source being accessed has
already been searched for artifacts.
In terms of avoiding redundant collections, the team runs into a
frustrating challenge. Many of the relevant sources of artifacts are not
static web pages, but instead discussion threads that can change over
time. Further, the most impactful discussion threads will change
rapidly, by merit of their impact. This means the team must risk
redundant collections in order to avoid missing new content.
One ameliorating approach is to rely on a “master spreadsheet” or an
index to maintain a “single source of truth” for what has been collected
and what sources have been searched for artifacts. This process comes
with
pitfalls
that
inevitably
increase
the
workload
and
create
inefficiencies. The work involved in duplicate-checking expands with the
size of the extant collection of artifacts, making each additional input a
contribution not only to the collection, but also to the difficulty of
further collections. This constitutes a variant of the volume-value
paradox discussed in relation to haphazard collection. Without a means
to connect the view the analyst has of the information environment
directly to existing collection data, this pain point in analysis is
effectively unresolvable.
15

## Page 17

Figure 6. A collection of image memes; (A) an image meme suggesting one would have to
be mentally ill to support Russia’s basis for engaging in conflict, (B) an image meme
suggesting Russia’s handling of protestors is over-aggressive, and (C) an image meme used
to represent the status of the relationship between Russia and Ukraine [38].
16

## Page 18

Figure 7. A collection of image memes; (A) an image meme critiquing profile picture
changes in support of trending issues, (B) a photograph of a woman who was arrested by
Russian police for holding a blank sign [39] which has been used as an image meme, (C) an
image meme suggesting basis for Russian caution in provoking the United States into
military action, (D) an image meme comparing Russian and EU negotiation strategy, (E) a
screenshot of a subreddit’s name and a recent upvoted post, used as an image meme, and
(F) an image meme conveying the relationship between Putin and Obama.
17

## Page 19

Information Integrations and Externalization. As suggested above,
maintaining information quality and the level of detail useful for
analysis will inevitably come at the cost of the rate of artifact collection.
Teams at this stage will often seek to address this challenge by
externalizing some aspects of collection, for example, opening channels
for input from other teams or the general public, or attempting to
integrate already existing collections into their own. Both methods
come with challenges. A team which externalizes its collection is now
faced not only with processing collected items, but also with managing
a crowdsourcing solution, which can be time expensive and unreliable.
Attempting to integrate existing collections can be equally challenging,
as the likelihood of finding an existing collection fit to the same scope
as the team’s is minimal, and there is no common standard for image
meme citation and collection - creating the need to do additional
processing work for relevance.
Further, it is unlikely that these collections will contain any provenance
data, which will limit analysis a great deal. The image meme found in
Figure 6-C is one such example, where the post in which it is found ties
the image to the war through its title, which is not included in the image
[38]. Similarly, the image meme found in Figure 7-A might be used in a
variety of contexts. Without relevant metadata accompanying entry of
these image memes into a shared repository, the team would have to
rely on pure speculation to identify them as relevant to the memetic
discourse around the Russo-Ukrainian War.
Once again, the team is faced with a fundamental trade-off. If they
simply accept a slower rate of collection over attempting to externalize
some aspects of collection or integrate from existing collections, the
team may quickly run into problems stemming from various forms of
bias - as no single team can possibly have all of the perspectives
necessary to prevent it.
●
Centralization bias may come in the form of collection and analysis
tending
to
have
an
implicit
or
explicit
overestimation
of
coordination, rational intent, and common direction or theme
[40,41]. For example, the team may see the image memes found in
Figure 7-C and Figure 7-D as related to the Russo-Ukrainian War or
events leading up to it, even though they were both used in
18

## Page 20

reference to past events and were picked up in collections that
could not be properly scoped by time-of-posting due to limitations
on search engines.
●
The team’s own narrative will inevitably calibrate an informational
niche which will create a feedback loop of bias in both collection
and analysis activity. This will lead them to overestimate the
importance of their own perspectives or of particular narratives
relevant to their perspectives similar to “overestimation of our own
importance”, in the context of intelligence analysis [40], or more
generally, a salience bias, leading them to prioritize that which
stands out to them as relevant given their prior experiences. For
example, consider the image meme found in Figure 7-E, which to
some may represent a simple mark of support, as opposed to an
argument-by-hypocrisy with relevant connections to other image
memes such as the one featured in Figure 4.
●
Sample bias will leave the team with blind spots. For example, the
image memes found in Figure 3 all feature different individuals and
settings. However, all of the individuals and settings featured are
from the same sitcom, Parks and Recreation. The team might not
be aware of this and therefore fail to mark the sitcom as a
referenced
entity.
These
kinds
of
details
may
seem
inconsequential, but they can be critical for certain kinds of
analyses,
such
as
those
focused
on
understanding
the
demographics
involved
in
generating
categories
of
artifacts,
understanding the audience that artifacts were created for, or
understanding and discovering coordinated activity. As another
example, if the team is unaware of recent arrests of Russian
protestors carrying blank signs [39], then the image meme in Figure
7-B (and especially its variations which do not include references to
Putin) may not be tied as relevant to the war, but instead as merely
a comical exaggeration.
●
Additionally, the scope of collections itself may result in further
blind spots. For example, the image in Figure 7-B is a photograph,
and might not be considered an image meme by an analyst, even
though it has been used as one (potentially as a protest-meme
against over-moderation). As another example, 7-E is a screenshot,
19

## Page 21

which, in conjunction with salience bias, may go uncollected
despite its previously mentioned potential value in analysis. As a
final example, the image in 7-B is now tied to a “copy-pasta” which
has spread over Twitter and Reddit:
“A man hands out leaflets on Red Square, and the KGB
arrest him. But when they get him to the station, they
find that the leaflets are all blank. And he says "Well,
everyone knows what the problem is, so why bother
writing it down?"
Given that copy-pasta is a text-based format, it would likely not
have been considered in an image meme focused collections
scope, thus limiting an analysis of 7-B’s rhetorical impact. The
inability to connect collection activity to the collection activity of
other groups with differing scope and collection requirements
deprives later analyses of a key factor related to rate of spread and
impact of content, and a key indicator of coordinated activity.
Without the ability to modularize, externalize, granularize, and connect
aspects of collection to such an extent that it limits the bias of the
team’s individuals on resulting information quality, biases will likely only
be exacerbated by further analysis.
Making Analysis Useful. The team, despite biased collections and
challenges in integration and externalization, runs into its final set of
difficulties, all related to ensuring the analysis they perform is actually
disseminated and of use to others. The problems with rate of collection
and analysis means that at this point, any analysis is likely to be a
post-mortem of events related to memetic discourse, rather than a
map of the current state of a relevant area of the information
environment. Of course all analyses are retrospective to some extent,
but
in
the
case
of memetic discourse, where the state of the
environment can shift so quickly - it is likely the case that the
information landscape has changed significantly by the time any form
of analysis is complete. As such, a team which has been successful up
to this point, might simply complete its activities by writing a report or a
paper regarding their findings, as opposed to being able to offer any
predictive value, actionable recommendations, or situational awareness
20

## Page 22

to stakeholders. Unfortunately, the value of these information products
may be limited to archival or historical purposes unless it had been
focused on fundamental research (e.g., understanding mechanism of
spread of claims or mutation of image meme format over time). Even
the archival value of the work is questionable. While some research
work allows value to be salvaged from a project in the form of re-use of
datasets, the absence of common standards and provenance data
unfortunately means that the generated meme datasets may not be
useful to other teams performing similar analyses.
Further, rhetorical analysis of the meaning of image memes is likely to
be highly subjective among disparate groups of individuals. This
significantly complicates both the process and the resulting utility of the
analysis of image memes. Attempting rhetorical analysis may result in
unintended, counterproductive outcomes that reinforce the biases of
both
the
team
and
their
stakeholders.
A
multi-user
and
multi-community process for determination of rhetorical claims could
lend a greater degree of objectivity to the analysis. However, as
discussed, these kinds of collaborations face a number of challenges.
Focusing on the sensemaking processes of content and semiotic
analysis (e.g., “What entities are referenced in this image?”, “Are any
latent objects signified by the content?”), increases the likelihood of
alignment, but this approach is only enabled by well-structured,
voluminous collection.
21

## Page 23

Automated Collection and Analysis
After performing initial exploration of structured collection (e.g. as described above), the
team may choose to refine and elaborate its approach towards collecting artifacts via
computational methods. This approach generally consists of two primary aspects: Data
Engineering and Data Analysis. The challenges of automated collection and analysis can be
generalized to those faced in Data Engineering and Data Analysis at large; therefore, a
comprehensive discussion is beyond the scope of this white paper. However, some
challenges of particular relevance to emergent teams performing automated collection and
analysis of image memes are discussed below and summarized in Table 3).
Data Engineering. Depending on the selected data source and desired
outcomes, this process could consist of several phases: acquisition,
cleaning, formatting, metadata collection, and de-identification.
●
Acquisition. Data may be directly acquired through a public API
(Application Programming Interface) provided by a digital platform,
or by scraping memes across different websites and media formats
(jpeg, png, pdf, etc). There are various types of API protocols, such
as
REST
(Representational
State
Transfer)
and SOAP (Simple
Objects
Access
Protocol),
each
of
which
requires
custom
connections that vary in terms of difficulty of implementation.
●
Formatting. Memes should be converted to a common file format
for interpretation by a computer vision package such as OpenCV.
Image memes may come in any number of sizes and shapes.
Therefore,
the
cleaning pipeline should use uniform resizing
parameters that facilitate image and text legibility, and discard
memes that do not meet the minimum threshold criteria.
●
Cleaning. This process can include correcting for errors that occur
due to font type and optical character recognition (OCR), removing
incomplete files, and potentially removing duplicates depending on
the desired outcomes of the analysis. For example, in some
circumstances, it may be helpful to understand how many times a
meme has been duplicated or how many different groups and/or
users engage with a given meme. However, in other circumstances,
duplication may bias the results. Hence, the scope of cleaning
could entail removing duplicate memes, but will certainly include
22

## Page 24

removing duplication errors that sometimes occur in the scraping
process.
●
Metadata Collection. Depending on the desired outcome of the
analysis, collecting additional data about the source of memes
might be important. For example, the team might want to record
metadata based on the source of the meme or the date that the
meme was posted. When focusing on the flow of information, the
user ID corresponding to the individual that posted might be of
interest.
The
team
may
also
want
to
determine
poster
demographics (e.g., age, location, religion, political affiliation, level
of education).
●
De-identification. While poster demographics and psychographics
can be of value to analysis, some teams may not be able to collect
these kinds of data. Moreover, many institutions require IRB
certification to use data from human subjects, which always
mandates that the subjects are de-identified before the data is
analyzed. De-identification can be done by replacing poster names
with random variables. However, it is critical to also remove
additional
demographic
information
if
these
data
could
be
leveraged to determine the poster identity (for example only two
people over age 60 work at a specific place). These kinds of
restrictions can create difficulties for collaborations among teams
and limit the applicability of datasets.
Data Analysis. Ideally, the goal of the analysis is outlined prior to data
acquisition and is not established post hoc, as it may be difficult to
perform specific analyses if the metadata are not properly collected.
While there are infinite possible analyses with respect to forum, user,
and user demographics, this section will focus on analyzing content
that
can
be
found
within
meme
text,
meme
images,
and
the
juxtaposition of images and text. Furthermore, methods that could
facilitate the automated detection of rhetorical claims in image memes,
such as functional annotation, categorization, and semiotic analysis will
be explored here.
●
Text Analysis. Meme text has to be extracted through optical
character
recognition,
which
converts
images
of
text
into
23

## Page 25

machine-readable text. The semantic content of the text can then
be analyzed through any number of natural language processing
pipelines, including pipelines for sarcasm detection in memes [42].
●
Image Analysis. Images can be analyzed for semantic content, such
as objects, people, text, scenes, and activities, through any number
of
image
analysis platforms. Gleaning semiotic content from
images
is
more
difficult.
Recent
efforts
have
attempted
to
distinguish regular images from image memes [43] and explore the
visual semantics of satire [44]. However, we are only just beginning
to unravel the complexity of semiotic and rhetorical content
embedded within image memes. While manual annotation of
memes by humans can be very helpful in creating a training set of
data and broad categories of image memes, manual annotation is a
time-consuming
task with a subjective nature that can yield
variable results [45]. Machine learning can reduce the time burden
of annotation, and can also be useful for semantic association and
classification of images [46].
●
Juxtaposition of Images and Text. The interplay between text and
images in multimodal content can be quite complex, and can have
a significant impact on the essence of a meme. The relationships
between
pictorial
and
textual
concepts
and
entities
are
characterized
by
metrics
that
include
cross-modal
mutual
information
(conceptual
overlap)
and
semantic
correlation
(meaning overlap) [47]. Moreover, content within images and text
can contribute to the rhetoric in the meme in a number of ways.
Meaning can be derived largely from the text, as in Figure 3, where
replacing the image with any number of images is not likely to
interfere with the overarching claim. In other memes the meaning
is mostly image-based, as in Figure 7-E; the caption is not necessary
within the context of the current Russo-Ukrainian war. Meaning can
also be derived equally from image and text, as in Figure 1, where
replacing either would significantly impact the underlying claim.
The relative importance of images and text has been quantified in a
metric called “status” [48].
Deep learning has been used to
successfully
categorize
image-text relationships based on the
metrics described above [48].
24

## Page 26

●
Categorization of Memes. Analysis of the entire memome is as
daunting a task as the analysis of the entire genome was at the
turn of the 21st century, and we can learn a lot from the successful
methods that emerged in the automation of genomic analysis.
Automating the ability to understand any meme, from any source,
would have to begin with coarse grained analyses (i.e. broad
categories) which are then later refined to higher levels of detail
(i.e. rhetorical claims). Machine learning methods could be used for
meme categorization, with appropriately labeled training data, and
the largest and most successful crowdsourced set of manually
annotated images and text, Wikipedia, could serve as training data
to this end. Wikipedia image and text content is conveniently
labeled with category terms that start off at broad levels and
become increasingly refined [48]. Wikipedia categories may be a
good starting point for the development of categories for a meme
ontology, a hypothetical memetic analogue to the gene ontology
that is broadly used for functional genomic annotation. Within the
memome, there is the potential to uncover memes related to
myriad stable and provisional categories. As in the genome,
studying the memome will uncover motifs, i.e. recurring patterns
with well-defined functions, that belong to categories (usually more
than one). For example, based on the images and text, the meme
in Figure 4 could belong to categories such as “comic strip”, “Nazi
Germany”, “white supremacy”, “Ukraine”, “Russo-Ukrainian war”,
“military’, and “hug”. While this level of categorization is far from a
rhetorical analysis, it can be useful in the detection of meme types
in order to understand the topics individuals or groups are
interested in discussing.
●
Detection of Rhetorical Claims. Rhetorical analysis of the claims in
image memes is difficult for both humans and computers. Each
person brings many years of their own unique prior cultural
experience into the analysis of a meme. Computational analysis
can benefit from human annotation of rhetoric in training data,
although arriving at a definitive claim for every image meme is a
daunting
task
that
may
have
unwanted
outcomes
(see
Methodological
Collection:
Making
Analysis
Useful
above).
In
understanding rhetorical claims, there are many layers of analysis
25

## Page 27

that build upon one another. At the most basic level, the concepts
and entities present in an image meme are identified. Recognizing
entity
relationships
then
facilitates
semantic
understanding.
Automated analysis of memes could be successful to this end;
however,
sensemaking beyond the semantic level requires a
human-in-the-loop to create annotated data. Uncovering latent
substance
in
memes
requires
understanding
the
potential
alternative
significance
of the meme’s concepts and entities.
Although semiotic comprehension can be deeply personal (e.g., not
everyone thinks of Grandma when they smell lilacs) there are also
signs and symbols that have been broadly adopted. An annotated
dataset that links semantics and semiotics would advance our
ability to uncover potential latent significance within memes, and
advance efforts towards the automated detection of rhetorical
claims.
Haphazard Collection
Stage
Key Challenges
Initial Enthusiasm
●
No Designated Roles
●
No Affordances for Structured Contribution
Internal Disruption
●
No Accessible Tools Designed for Collection Activity or
Implementing Related Compensating Controls
●
No Separation of Concerns
●
No Role-Based Access
●
No Affordances for Structured Contribution
Overload
●
No Affordances for Multi-Modal, Semantic Search
●
Limited Affordances for Structured Archiving
Lack of Visible
Progress
●
Mission Creep
●
Lack of Ability to Measure Progress
Table 1. Key Challenges in Haphazard Collection
26

## Page 28

Methodological Collection
Stage
Key Challenges
Tool Selection
●
Tool Overload
Tool Adoption and
Configuration
●
Tool Overload
●
Lack of Tool Interoperability
●
Lack of Accessible Connectivity Affordances Among Teams
and Platforms
Maintaining
Information Quality
●
Poor User Experience of Collection Activity
●
Manual Connections Among Tools and Datasets
●
No Accessible Tools Designed for Collection Activity or
Implementing Related Compensating Controls
●
Limited Affordances for Structured Archiving and
Annotation
●
Context Switching Between Collection Tools and the
Browsing Environment
●
Inability to Detect Exact and Near Duplicates
Information
Integrations and
Externalization
●
Lack of Accessible Connectivity Affordances Among Teams
and Platforms
●
Lack of Common Standards for Data Sharing
●
No Common Citation Method
●
Limited Affordances for Structured Archiving
●
Poor User Experience of Collection Activity
●
No Accessible Tools Designed for Collection Activity or
Implementing Related Compensating Controls
●
Bias in Analysis
Making Analysis
Useful
●
Collection and Analysis Activity Not Fast Enough to Provide
Situational Awareness in Real-time
●
Insufficient Standardization or Provenance Data to Allow
for Reusability of Datasets
Table 2. Key Challenges in Methodological Collection
27

## Page 29

Automated Collection and Analysis
Stage
Key Challenges
Data Engineering
●
Lack of Accessible Connectivity Affordances Between
Teams and Platforms
Data Analysis
●
Insufficient
Human-Annotated
Data
to
Extract
the
Significance from the Relationships Between Text and
Images
●
Lack of Common Standards and Ontology to Leverage in
Connecting Image Memes to Functional Categories or
Topics
●
Limited Human-Annotated Data Connecting Semantics
and Semiotics (which could help with the automated
extraction of latent topics within memes).
Table 3. Key Challenges in Automated Collection and Analysis
28

## Page 30

Requirements and Recommendations
Below we provide requirements for systems that might alleviate many of the key challenges
for emergent teams tasked with analysis and annotation of image memes, and offer
example use-cases if these requirements were made available.
Information Where it Matters. Analysts need access to information where it matters.
Being able to access details on-site about existing analyses, collected artifacts, and to
simply see whether or not a relevant object (e.g. an entity reference, an image meme, a
thread, or a web page) has already been the subject of collection activity would
immediately and unambiguously reduce most redundancies in collections activities. This
could be achieved through custom browsers or, to avoid impacts from requiring platform
adoption, through web and document annotation affordances. Tools that place collected
information and analysis alongside the content itself would allow analysts to see their own
shared lens on the internet without requiring content providers to adopt common
standards.
Example Use-Cases
●
Enriching images that have already been collected
with
summary
information
and
links
to extant
analysis and related artifacts.
●
Marking
discussion
threads and webpages with
summary information about when they were last
visited and what had been collected from them.
Dynamic Web Annotations. The ability to annotate and enrich content with links to and
presentation of existing information would provide numerous benefits to analysts,
including preventing redundant analysis and duplication during collection. However, the
fact that some sources of artifacts are impactful because they are expected to change often
(e.g.., discussion threads) creates numerous challenges. The ability to annotate, or attach,
‘functions' or ‘automations' (i.e., triggers to run scripts) to web pages, which dynamically
update their data and contents as opposed to presenting static content (e.g., text), could
alleviate many of these challenges and add new analysis capabilities.
29

## Page 31

Example Use-Cases
●
Updating analysts when content at a given URL has changed
substantially or when content with certain characteristics have
been detected.
●
Tracking changes to sentiment or rate of engagement, indicating
the presence of recent, potentially valuable artifacts.
●
Performing image-similarity searches to “track” already collected
images as they spread across the internet.
Proximal Collection and Tagging Affordances. The constant context-switching required
for use of most tools offering collection affordances decreases productivity, information
quality, and user experience and engagement generally. Providing collection affordances
which are proximal to the source of artifacts would greatly enhance efficiency and user
experience, thus improving information quality, rate of collection and productivity, and
general engagement. For example, Paperpile, a reference management platform for
academics, has used HTML injection and HTML template standards in order to insert
artifact collection affordances on content of both static pages and search results in order to
improve the productivity and efficiency of researchers (See Figure 8). Similar approaches
using community-related features and more general and customizable templates and
object standards could vastly improve digital artifact collection processes outside the
context of academia.
Example Use-Cases
●
Injection of collection affordances on web pages without requiring
the permission of the web page’s owners or their adoption of
common standards. This would enable users to quickly add
artifacts for processing.
●
Providing
a
computational
basis
for instituting compensating
controls on collections. If the collection affordance is proximal to
the source, a great deal of metadata about the artifact can be
collected automatically, greatly reducing the amount of time taken
per artifact while greatly increasing the information quality.
30

## Page 32

●
Providing users and communities with the opportunity to share in
common templates for computational detection and collection of
different kinds of artifacts and their metadata on different websites
(e.g., articles, posted comments, images).
●
Offering tools that allow teams to set local scope and standards for
collection
and
processing.
As
discussed, a multi-community
process for annotation of semiotic content in images could lend a
greater degree of objectivity to the analysis of rhetorical content.
Further, the ability to set clear standards for these annotations (i.e.
what and how annotation will be executed) could increase the
longevity and applicability of the resulting datasets, and help
communities choose the appropriate scope and level of detail for
collection activity. For example, while analysts are unlikely to
disagree about the semantic content within the images or text,
finding consensus on rhetorical claims and the meaning of image
memes may be impossible in some cases. Instead, comprehending
the latent representations (i.e. semiotic content) in image memes
may provide a useful common ground where human annotation
can
offer
insight
into
the
sensemaking
that
precedes
the
determination of rhetorical claims. Answering the question “What
are the hidden representations, if any, that this meme signifies?”
facilitates
multiple
answers.
An
analyst
that didn’t have the
appropriate background to uncover latent meaning could answer,
“None.” Instead of forcing the analyst to deduce a single claim,
semiotic-focused collection and analysis affords a softer approach
more amenable to multi-user and multi-community analysis. In
addition, an annotated dataset linking semantic and semiotic
content would be re-usable by future analysts, and could be
leveraged as training data to automate the process of sensemaking
that underlies more complex forms of analysis (e.g. analysis of
rhetorical claims).
31

## Page 33

Figure 8. A 4-part graphic representing how Paperpile is used for digital artifact collection.
(a) A detected artifact which has not yet been collected, (b) an artifact’s attached collection
affordance being used to search for related metadata, (c) how an artifact indicates that it
has already been collected, and (d) how an artifact is represented in aggregate with other
collected artifacts, with redactions of personal information.
32

## Page 34

Semantic Multimodal Search. Semantic multimodal search may be among the most
needed and difficult-to-fulfill requests for analyst capabilities, as it has constituted a
challenge for as long as humans have been actively attempting to refine methods for
search, sort, and summarization of information across myriad contexts, from intelligence
practice to library archiving [49–51]. While AI methods are presently the most popular
approach to semantic search and collation, human annotation and analysis is the oldest,
most auditable, and arguably the most reliable method available, despite its drawbacks
[15,49,50,52]. The most notable of these drawbacks may be speed and scalability. These
drawbacks can be addressed with crowdsourcing solutions, which adds new difficulties,
such as the potential for disagreement regarding both the standards for and the resulting
annotations.
However, given that the intent of artifact annotation in this case is to understand
underlying
claims,
referenced
entities,
and
cultural
references,
disagreement
in
well-structured and standardized annotation instead becomes valuable data for analysis
[15,53]. Further, artifact annotation allows teams which externalize their collection and
annotation activities to transcend local narrative bias and inherently limited cultural
knowledge. Human annotation, at scale, facilitated by web annotation in combination with
image-similarity algorithms, AI, and traditional ontological approaches, could yield the
necessary semantic multi-modal search for aggregate analysis of highly subjective content,
such as image memes [15]. These approaches could also provide training data to further
externalize annotation to automated systems, enable connectivity between content and
concepts, and offer the basis for identifying common hidden states, themes, claims, and
references running through disparate content [15].
Example Use-Cases
●
The PageRank algorithm is used to discover “which nodes [in a
network] are important” whereas Reverse PageRank is “often used
to determine why a particular node is important” [54]. While
PageRank, and its cousin Reverse PageRank, are known for their
use in internet search, the underlying mathematics are entirely
general, and have been used in areas such as bioinformatics,
neuroscience, literature and bibliometrics, and even sports [54].
With structured annotation, time-series metadata, and shared
catalogs of image memes, there are a number of variations of
search that become possible, from searching for which memes and
themes or relevant URLs are or were important or trending, to
33

## Page 35

collating memes which are relevant given attribute-based search
queries. Further, these forms of search may provide actionable
insights for those performing or informing interventions, as they
could point to image memes with highly specific attributes which
can
act
as
“stepping
stones”
or
defenses
for
changing
or
maintaining beliefs, respectively, given audience characteristics and
interests [18].
●
With
an
annotated
catalog of image memes and annotated
automations for detecting image memes, it may be possible to not
only search for specific memes or memes with specific attributes,
but to search for where a meme has appeared and for its potential
succeeding iterations. Similar to tagging and tracking methods in
ecology, image memes could be tracked in real-time or through
time-series projections as they move across the internet [15]. This
kind of tracking, when paired with annotated automations for
sentiment analysis over time, allows for search by sentiment
impacts
and
trajectory
potentials
-
offering
the
basis
for
dashboards, early warning systems, impact projections, and other
predictive analytics and situational awareness systems [15].
●
While search is generally associated with returned results from a
specific query, there are numerous other forms of exploratory
search, such as exploring connections in networks, interactive
visualizations, and situational awareness tools [55]. Well annotated
catalogs
of
image
memes
could
allow
for
various
exploration-facilitating
visualizations,
such
as
rhetorical
and
argument
maps
[56],
process
maps of meme lifecycles and
transitions [32], and interactive network graphs [55].
●
In addition, there could be forms of exploratory search as would be
found in bioinformatics, such as network and graph exploration via
functional annotation within the genome [57,58], allowing for
analogous “memoinformatics” driven search within the memome.
As functional significance has been imparted onto the human
genome,
our
understanding
of
organismal
biochemistry
and
physiology
has
progressed
in
turn.
A
detailed,
hierarchical
annotation
of
the
memome
that includes concepts, entities,
semantic categories, semiotics, rhetoric, and the underlying links
34

## Page 36

between
these
types
of
content,
could
vastly
increase
our
understanding
of
multimodal
human
communication,
sensemaking, and cognition.
Separate Platforms, Shared Data. The limitation of current tools and the “one platform to
rule them all” [59–61], or the “one app to replace them all” paradigm [62], has created and
exacerbated challenges for teams attempting to effectively and efficiently collect, analyze,
and implement common standards and controls for their data. As discussed, where
individuals or their organizations come to the team with tool preferences, storage
protocols and nomenclature, and rules about what tools can be used, teams are not only
disjointed by default, but can disintegrate before they even have a chance to begin work. In
addition, the team needs to be able to externalize some aspects of collection and integrate
data with other teams that allow opportunities for selective disclosure, and create
separations of concern within the team based on role and information needs. Even if there
was a tool available which offered all of the affordances and capabilities the team could
possibly need, it may still be more efficient, inclusive, and practical to “[meet] people where
they are” [63,64].
Many companies are beginning to adopt a more collaborative approach, attempting to
create new value through open standards and specifications. This open approach allows
for digital asset and signal exchange with third-parties, and contributes API (application
programming interface) connections and affordances to the “API economy” [65,66]. The API
economy is composed of integration platform as a service (iPaas) [67,68], general
automation platform (GAP) [69], cloud-based integration [70], and digital ecosystem [63]
approaches. Companies such as Zapier [71], IFTTT [72], Make (formerly Integromat) [73],
DOMO [74], MuleSoft [75], and Workato [70] provide the ability to create automations using
“trigger-action” and “if-this-then-that” frameworks [71,72] and data pipeline integration
capabilities that connect and incentivize the creation and use of APIs. These capabilities
reduce the time-to-impact and the development and personnel costs of linking and
maintaining data across multiple organizations, applications, and services [67,68]. However,
not all platforms allow for these integrations and there can be a significant amount of work
involved in enabling their use.
Incentivizing API economy participation through the use of common standards and market
mechanisms for exchange of data with third parties might increase venture capital
attention on companies using more open approaches with their data, thereby increasing
the number of interoperable platforms. It can also offer alternative revenue streams to
platforms which are currently disincentivized from sharing their data due to their reliance
35

## Page 37

on dwell time related revenues (i.e., advertising revenue). For example, while the meme
collection site “Know Your Meme” is branded as a meme research and collection platform
[76], it is also owned by a media holding company that collects advertising revenue from its
underlying brands [77]. If Know Your Meme were provided with low-cost mechanisms to
bring its collections to the API economy, the potential to offset ad revenue losses and
create new business value might incentivize making data easily available to third-parties for
other meme search, research, and curation functions. A more robust and accessible API
economy could have large impacts on interorganizational work.
Example Use-Cases
●
With accessible, low-code API integrations paired with methods for
exchanging information about data standards and controls in use,
the
same
mechanisms
which
help
teams
share
calendar
information between project management tools could be used to
share data collections between teams in real-time.
●
Role-based
access controls and privileges can be difficult to
manage and keep track of [55]. If teams have the ability to create
ad hoc real-time connections between platforms, then, by proxy,
they
can
implement
highly
complex
role-based
access
and
affordances simply by restricting membership on certain tools. This
could allow for complex intelligence pipelines in which aspects of
collection and expert analysis could be separated within teams,
externalized to other teams with varying incentive sets (e.g.,
crowdsourced collection), or done in real-time collaboration with
other teams and organizations.
●
Some teams may have information they cannot or do not want to
share, and, as discussed, some teams may have information they
cannot receive without running afoul of internal ethical or other
controls. API integration capabilities paired with clear opportunities
for bidirectional selective disclosure could allow teams that would
otherwise be unable to share information to communicate and
collaborate. Further, allowing teams to set “terms of use” and
related information on their offered or requested data would allow
for a new level of transparency for users in how their shared
information is used and governed [33].
36

## Page 38

Conclusion
As the world’s increasing complexity drives conflict into increasingly abstract spaces [78],
emergent teams involved in digital discourse analysis have a vital role to play in helping
organizations
maintain
situational
awareness and synthesize disparate perspectives
generally. Here we advanced previous work on rhetorical analysis of image memes by
presenting several archetypes of emergent teams involved in analysis, describing their
inherent challenges, and suggesting recommendations for future systems design.
In the realm of image memes, which allow for unparalleled strategic ambiguity and
plausible deniability, emergent teams may be the only viable approach to sensemaking in
the digital rhetorical ecosystem - as no single organizational configuration can capture all of
the symbols and cultural knowledge necessary to understand or estimate the significance
of the content present. As discussed, the state of the art of image meme collection by
emergent teams is not commensurate with either stakeholders’ or the team’s needs,
despite direly needed affordances being well within technological reach. In short, there is a
chasm between “how it is done today” and “how it could be done” that is not proportionate
to the gap between “what is available” and “what is possible” (see Figure 9).
Digital ecosystem and API economy approaches seem to be a viable route to addressing
many of the challenges discussed, and for enabling and contributing to the web and
document annotation approaches which address the remaining challenges. API economy
approaches have gained traction in recent years, and are now being applied across various
areas of the market including agriculture [67,73], engineering [70], research [72] and
marketing [69]. However for API economy approaches, lack of data standardization and
integration capabilities remain a problem. This problem is not specific to challenges faced
by emergent teams, and addressing it could be beneficial to a variety of sectors. Polling has
suggested that the average enterprise uses more than 1,200 applications [79], and that an
“average knowledge worker” is using up to 28 different applications [80] and is toggling
between applications up to 10 times per hour [81]. According to Deloitte’s 2021 Chief
Procurement
Officer
Survey,
among
the
top
two
barriers
to
effective
technology
implementation are data quality and poor integration capabilities across applications [82].
In addition to many other domains being able to share in the benefits of developments that
would resolve challenges for emergent teams conducting image meme analysis, other
domains can benefit from the resulting analysis. Resolving these challenges using the
approaches discussed in this white paper could result in new claims-based methods to
identify counterpublics and communities that have no formal affiliations, the capacity to
37

## Page 39

identify hidden states and themes running through public discourse, and to provide early
warning systems indicating where streams of memes might constitute the precursor for
groups to converge on (potentially violent) action.
Figure 9. Graphical overview for computational and rhetorical analysis pipelines. A) How it
is done today, and B) How it could be done.
38

## Page 40

Funding and Acknowledgements
R.J. Cordes is funded through the NSF Convergence Accelerator Trust and Authenticity in
Communication Systems Program (NSF 21-572), under award ID #49100421C0036 and is
supported in research efforts through a Nonresident Fellowship with the Atlantic Council
on appointment to the GeoTech Center.
Daniel A. Friedman is funded by the NSF program Postdoctoral Research Fellowships in
Biology (NSF 20-077), under award ID #2010290.
39

## Page 41

Works Cited
1.
Roose K. A QAnon “Digital Soldier” Marches On, Undeterred by Theory’s Unraveling.
The New York Times. 17 Jan 2021. Available:
https://www.nytimes.com/2021/01/17/technology/qanon-meme-queen.html. Accessed
14 Jul 2022.
2.
Castaño Díaz. Defining and characterizing the concept of Internet Meme. CES Psicol.
2013. Available:
http://www.scielo.org.co/scielo.php?pid=S2011-30802013000200007&script=sci_arttext
&tlng=en
3.
Molina. What makes an internet meme a meme? Six essential characteristics.
Handbook of visual communication. 2020.
doi:10.4324/9780429491115-35/makes-internet-meme-meme-six-essential-characterist
ics-maria-molina
4.
Dawkins R. The Selfish Gene. Oxford University Press; 2016.
5.
Blackmore S. The Meme Machine. OUP Oxford; 2000.
6.
Dennett DC. Darwin’s Dangerous Idea: Evolution and the Meanings of Life. Simon and
Schuster; 1996.
7.
Heylighen F. Selection criteria for the evolution of knowledge. Proc 13th Int Congress
on Cybernetics. Association Internat. de Cybernétique; 1993. pp. 524–528.
8.
Moreno-Almeida C. Memes as snapshots of participation: The role of digital amateur
activists in authoritarian regimes. New Media & Society. 2021;23: 1545–1566.
9.
de Saint Laurent C, Glăveanu VP, Literat I. Internet Memes as Partial Stories: Identifying
Political Narratives in Coronavirus Memes. Social Media + Society. 2021;7:
2056305121988932.
10. Nieubuurt JT. Internet Memes: Leaflet Propaganda of the Digital Age. Frontiers in
Communication. 2021;5. doi:10.3389/fcomm.2020.547065
11. Mascarenhas M. Memes as Quasi-Argument. Local Theories of Argument. 2021. pp.
385–390. doi:10.4324/9781003149026-65
12. Observatory, DiResta, Cryst, Masterson, Small, Miller, et al. Memes, Magnets and
Microchips: Narrative dynamics around COVID-19 vaccines. Virality Project. 2022.
Available:
https://cyber.fsi.stanford.edu/io/publication/memes-magnets-and-microchips-narrative
-dynamics-around-covid-19-vaccines
40

## Page 42

13. Toulmin SE. The Uses of Argument. Cambridge University Press; 1958.
14. Ling C, AbuHilal I, Blackburn J, De Cristofaro E, Zannettou S, Stringhini G. Dissecting the
Meme Magic: Understanding Indicators of Virality in Image Memes. Proc ACM
Hum-Comput Interact. 2021;5: 1–24.
15. Mascarenhas M, Cordes RJ, Friedman DA. Digital Rhetorical Ecosystem Analysis:
Sensemaking of Digital Memetic Discourse. Zenodo. 2021.
doi:10.5281/zenodo.5573947
16. Chagas V, Freire F, Rios D, Magalhães D. Political memes and the politics of memes: A
methodological proposal for content analysis of online political memes. First Monday.
2019. Available: https://firstmonday.org/ojs/index.php/fm/article/view/7264/7731
17. Zakem V, McBride MK, Hammerberg K. Exploring the utility of memes for US
government influence campaigns. Center for Naval Analyses Arlington United States;
2018. Available: https://apps.dtic.mil/sti/citations/AD1052398
18. Cordes RJ, David S, Maan A, Ruiz A, Sapp E, Scannell P, et al. The Narrative Campaign
Field Guide - First Edition. 1st ed. Cordes RJ, editor. Narrative Strategies Ink; 2021.
19. Cordes RJ. The Next Generation of Security: Prioritizing Information Warfare Defense. 7
Aug 2021 [cited 12 Aug 2021]. Available:
https://www.hstoday.us/subject-matter-areas/intelligence/the-next-generation-of-secu
rity-prioritizing-information-warfare-defense/
20. Cordes RJ. Games with serious consequences: Extremist movements and kayfabe -
Atlantic Council. In: atlanticcouncil.org [Internet]. 19 May 2021 [cited 3 Jun 2021].
Available:
https://www.atlanticcouncil.org/blogs/geotech-cues/games-with-serious-consequences
-extremist-movements-and-kayfabe-2/
21. Friedman DA, Cordes RJ. Infinite Games for Infinite Teams. The Great Preset: Remote
Teams and Operational Art. COGSEC; 2020.
22. Taecharungroj V, Nueangjamnong P. Humour 2.0: Styles and Types of Humour and
Virality of Memes on Facebook. Journal of Creative Communications. 2015;10: 288–302.
23. Guenther L, Ruhrmann G, Bischoff J, Penzel T, Weber A. Strategic Framing and Social
Media Engagement: Analyzing Memes Posted by the German Identitarian Movement
on Facebook. Social Media + Society. 2020;6: 2056305119898777.
24. Tindale CW. Replicating Reasons: Arguments, Memes, and the Cognitive Environment.
Philosophy & Rhetoric. 2017;50: 566–588.
25. Ceccarelli L. Polysemy: Multiple meanings in rhetorical criticism. Q J Speech. 1998;84:
41

## Page 43

395–415.
26. Wikipedia contributors. Condescending Wonka. In: Wikipedia, The Free Encyclopedia
[Internet]. 1 Jun 2021. Available:
https://en.wikipedia.org/w/index.php?title=Condescending_Wonka&oldid=1026354342
27. Petty RE, Cacioppo JT. The Elaboration Likelihood Model of Persuasion. In: Berkowitz L,
editor. Advances in Experimental Social Psychology. Academic Press; 1986. pp.
123–205.
28. Karlova N. Misinformation and disinformation in online games: an exploratory
investigation of possible cues. digital.lib.washington.edu. 2018. Available:
http://digital.lib.washington.edu/researchworks/handle/1773/42416
29. Hemsley, Snyder. Dimensions of visual misinformation in the emerging media
landscape. Misinformation and mass audiences. 2018. Available:
https://books.google.com/books?hl=en&lr=&id=jUw7DwAAQBAJ&oi=fnd&pg=PA91&dq
=misinformation+artifacts&ots=Cv4UnjEdKZ&sig=DtirCALUY0-oCZ_ROpqb8ZYBqQc
30. Cordes RJ, Friedman DA. Emergent Teams for Complex Threats. The Great Preset:
Remote Teams and Operational Art. COGSEC; 2020. pp. 1–15.
31. Hürsch WL, Lopes CV. Separation of Concerns. 1995 [cited 24 Aug 2020].
doi:10.1.1.29.5223
32. Cordes RJ, Friedman DA, Maron M. Reimagining Maps. {National GeoSpatial Intelligence
Agency Polyplexus Incubator}; 2020 Oct. doi:10.5281/zenodo.4170026
33. Karachiwalla R, Pinkow F. Understanding crowdsourcing projects: A review on the key
design elements of a crowdsourcing initiative. Creat Innov Manag. 2021;30: 563–584.
34. Friedman D, Applegate-Swanson S, Choudhury A, Cordes RJ, El Damaty S, Guénin-Carlut
A, et al. An Active Inference Ontology for Decentralized Science: from Situated
Sensemaking to the Epistemic Commons. 2022. doi:10.5281/zenodo.6320575
35. Jamal U. Facebook is reversing its ban on posts praising Ukraine’s far-right Azov
Battalion, report says. Business Insider. 25 Feb 2022. Available:
https://www.businessinsider.com/facebook-reverses-ban-praise-ukraine-far-right-force
s-2022-2. Accessed 19 May 2022.
36. Cordes RJ, Friedman DA. The Facilitator’s Catechism. In: Friedman DA, Cordes RJ,
editors. The Great Preset: Remote Teams and Operational Art. COGSEC; 2020.
37. StoneToss. Fast Friends Image Meme. In: stonetoss.com [Internet]. 24 Mar 2022 [cited
18 May 2022]. Available: https://stonetoss.com/comic/fast-friends/
42

## Page 44

38. Ukraine and Putin RN. In: 9GAG [Internet]. 1 Mar 2022 [cited 23 May 2022]. Available:
https://9gag.com/gag/a21vjMe
39. van Brugen I. Russia Arrests Multiple People for Holding Up Blank Signs. In: Newsweek
[Internet]. 14 Mar 2022 [cited 23 May 2022]. Available:
https://www.newsweek.com/russia-ukraine-war-invasion-protests-police-arrest-activist
s-holding-blank-signs-paper-1687603
40. Heuer RJ. Psychology of Intelligence Analysis. Center for the Study of Intelligence; 1999.
41. Jervis R. Perception and Misperception in International Politics. Princeton University
Press; 2017.
42. Potamias RA, Siolas G, Stafylopatis A-G. A transformer-based approach to irony and
sarcasm detection. Neural Comput Appl. 2020;32: 17309–17320.
43. Koutlis C, Schinas M, Papadopoulos S. MemeTector: Enforcing deep focus for meme
detection. arXiv [cs.CV]. 2022. Available: http://arxiv.org/abs/2205.13268
44. Sinha A, Patekar P, Mamidi R. Unsupervised Approach for Monitoring Satire on Social
Media. Proceedings of the 11th Forum for Information Retrieval Evaluation. New York,
NY, USA: Association for Computing Machinery; 2019. pp. 36–41.
45. Chen E. 30% of Google’s Emotions Dataset is Mislabeled. 14 Jul 2022 [cited 14 Jul 2022].
Available:
https://www.surgehq.ai//blog/30-percent-of-googles-reddit-emotions-dataset-is-mislab
eled
46. Wei P, He F, Zou Y. Content semantic image analysis and storage method based on
intelligent computing of machine learning annotation. Neural Comput Appl. 2020;32:
1813–1822.
47. Henning C, Ewerth R. Estimating the information gap between textual and visual
representations. International Journal of Multimedia Information Retrieval. 2018;7:
43–56.
48. Otto C, Springstein M, Anand A, Ewerth R. Characterization and classification of
semantic image-text relations. International Journal of Multimedia Information
Retrieval. 2020;9: 31–45.
49. Blair AM. Too Much to Know. Yale University Press; 2010.
50. Mangold C. A survey and classification of semantic search approaches. Int J Metadata
Semant Ontol. 2007;2: 23–34.
51. Waltz E. Knowledge Management in the Intelligence Enterprise. Artech House; 2003.
43

## Page 45

52. Kent S. Strategic Intelligence for American World Policy. Princeton University Press;
1949.
53. Tamari R, Friedman D, Fischer W, Hebert L, Shahaf D. From Users to (Sense)Makers: On
the Pivotal Role of Stigmergic Social Annotation in the Quest for Collective
Sensemaking. arXiv [cs.SI]. 2022. Available: http://arxiv.org/abs/2205.06345
54. Gleich DF. PageRank Beyond the Web. SIAM Rev. 2015;57: 321–363.
55. Cordes RJ, Applegate-Swanson S, Friedman DA, Knight VB, Mikhailova A. Narrative
Information Management. In: Cordes RJ, Friedman DA, editors. Narrative Information
Ecosystems: Conflict and Trust on the Endless Frontier. COGSEC; 2021. pp. 1–64.
56. Kashcha A. Map of Reddit. In: anvaka.github.io [Internet]. [cited 16 Jun 2022]. Available:
https://anvaka.github.io/map-of-reddit/?x=137263.0138067139&y=331978.2724596900
5&z=967321.673408511
57. Search by Functional Annotation. In: genomehubs.org [Internet]. [cited 21 Jul 2022].
Available:
https://genomehubs.org/communities/tutorials/search-by-functional-annotation/
58. Shannon P, Markiel A, Ozier O, Baliga NS, Wang JT, Ramage D, et al. Cytoscape: a
software environment for integrated models of biomolecular interaction networks.
Genome Res. 2003;13: 2498–2504.
59. Adobe. One Platform to Rule Them All: 5 reasons to choose a hybrid platform that
combines B2B and B2C capabilities. 2019 Feb.
60. Solarwinds. “One Platform to Rule Them All”: How SolarWinds provided IT visibility
across the VA’s enterprise. In: Federal News Network [Internet]. 30 Oct 2019 [cited 24
Jun 2022]. Available:
https://federalnewsnetwork.com/innovation-in-government-success-stories/2019/10/o
ne-platform-to-rule-them-all-how-solarwinds-provided-it-visibility-across-the-vas-enter
prise/
61. Dunn J. One Platform To Rule Them All: How N2N Makes Life Easier for Smaller
Institutions. In: illuminateapp.com [Internet]. 7 Jun 2019 [cited 24 Jun 2022]. Available:
https://illuminateapp.com/web/EnlightenEd/one-platform-to-rule-them-all-how-n2n-m
akes-life-easier-for-smaller-institutions/
62. Reed J. One app to replace them all? In: diginomica [Internet]. 3 Feb 2021 [cited 24 Jun
2022]. Available:
https://diginomica.com/one-app-replace-them-all-clickups-ceo-future-productivity-and-
how-clickup-addresses-proliferation
63. Mattr. Mattr - Building an Open and Interoperable Ecosystem for Digital Trust. In:
44

## Page 46

Mattr.global [Internet]. 2021 [cited 29 Jun 2022]. Available:
https://mattr.global/approach/
64. Norman D. The Design of Everyday Things: Revised and Expanded Edition. Basic Books;
2013.
65. Gamez-Diaz A, Fernandez P, Ruiz-Cortes A. An Analysis of RESTful APIs Offerings in the
Industry. Service-Oriented Computing. Springer International Publishing; 2017. pp.
589–604.
66. Soomro, Awan. Challenges and Future of Enterprise Application Integration. Int J
Comput Appl. 2012. Available:
http://www.ijcaonline.org/archives/volume42/number7/5708-7762
67. Cestari RH, Ducos S, Exposito E. iPaaS in Agriculture 4.0: An Industrial Case. 2020 IEEE
29th International Conference on Enabling Technologies: Infrastructure for
Collaborative Enterprises (WETICE). ieeexplore.ieee.org; 2020. pp. 48–53.
68. Gartner. Definition of Integration Platform as a Service (iPaaS). In: Gartner Information
Technology Glossary [Internet]. [cited 30 Jun 2022]. Available:
https://www.gartner.com/en/information-technology/glossary/information-platform-as
-a-service-ipaas
69. Ortiz A, Waldron R, Taylor H. The Beginner’s Guide to General Automation Platforms.
2019.
70. Frantz RZ, Corchuelo R, Basto-Fernandes V, Rosa-Sequeira F, Roos-Frantz F, L. Arjona J.
A cloud‐based integration platform for enterprise application integration: A
Model‐Driven Engineering approach. Softw Pract Exp. 2020. doi:10.1002/spe.2916
71. Rahmati A, Fernandes E, Jung J, Prakash A. IFTTT vs. Zapier: A Comparative Study of
Trigger-Action Programming Frameworks. arXiv [cs.CR]. 2017. Available:
http://arxiv.org/abs/1709.02788
72. Ovadia S. Automate the Internet With “If This Then That” (IFTTT). Behav Soc Sci Librar.
2014;33: 208–211.
73. Singh, Iqbal, Singh, Kumar. Smart garden with iot based plant monitoring system. IEEE
Solid-State Circuits Soc Newsl. 2020. Available:
https://www.researchgate.net/profile/Surya-Singh-2/publication/353878999_Smart_Gar
den_with_IoT_based_Plant_Monitoring_System/links/611632f00c2bfa282a3f72fb/Smart
-Garden-with-IoT-based-Plant-Monitoring-System.pdf
74. DOMO. DOMO Data Integration. In: Domo.com [Internet]. [cited 6 Jul 2022]. Available:
https://www.domo.com/solution/data-integration?utm_source=google&utm_medium=
paidsearch&campid=7015w000000vlPfAAI&gcreative=474206076414&gdevice=c&gnet
45

## Page 47

work=g&gkeyword=application%20interface&gplacement=&gmatchtype=p&gtarget=&g
adposition=&s_kwcid=AL!5964!3!474206076414!p!!g!!application%20interface&gclid=Cj
0KCQjw5ZSWBhCVARIsALERCvyz8XHgFUQc-ylxkHj8NvRiT3BoOkbAbdEp0L_-gX8k6GWCl
O5Jt3kaAghiEALw_wcB
75. MuleSoft. What is an API Economy. In: MuleSoft [Internet]. [cited 21 Jul 2022]. Available:
https://www.mulesoft.com/resources/api/what-is-an-api-economy
76. KnowYourMeme. About Know Your Meme. In: Know Your Meme [Internet]. [cited 6 Jul
2022]. Available: https://knowyourmeme.com/about
77. LiterallyMedia. Literally – Home of Legendary Brands. In: Literally Media [Internet].
[cited 6 Jul 2022]. Available: https://literally.media/
78. David S, Cordes RJ, Friedman DA. Active Inference in Modeling Conflict. 2021.
doi:10.5281/zenodo.5750935
79. Netskope Marketing. Netskope Cloud Report - August 2019. 1 Aug 2019 [cited 30 Jun
2022]. Available:
https://resources.netskope.com/cloud-security-infographics/netskope-cloud-report-au
gust-2019-2
80. Ismail K. How To Manage Digital Workplace Tool Overload. In: CMSWire.com [Internet].
1 Nov 2017 [cited 30 Jun 2022]. Available:
https://www.cmswire.com/digital-workplace/how-to-manage-digital-workplace-tool-ove
rload/
81. Eide N, Bolden-Barrett V. App overload wastes 32 days of employee productivity each
year. In: ciodive.com [Internet]. 7 Mar 2018 [cited 30 Jun 2022]. Available:
https://www.ciodive.com/news/app-overload-wastes-32-days-of-employee-productivity
-each-year/518520/
82. Kilpatrick J, Brown J, Flynn R, Addicoat A, Mitchell P. Deloitte Global 2021 Chief
Procurement Officer Survey. In: Deloitte Insights [Internet]. Deloitte; 23 Apr 2021 [cited
30 Jun 2022]. Available:
https://www2.deloitte.com/global/en/insights/topics/operations/chief-procurement-offi
cer-cpo-survey.html
46


---
*Extraction method: pymupdf*