When are copies of content appropriate, and how should you manage copies? Should content ever be repetitive?\u00a0 Is duplicative content always bad?<\/p>\n
Answers to these questions are typically provided by specialists: CMS implementers (developers skilled in PHP or another CMS programming language), SEO experts, or webmasters. Specialists tend to focus on technical effort or performance\u2014the technical penalties\u2014rather than strategic issues of how people interact with messages and information\u2014the users\u2019 goals. Discussions become overly narrow, with important issues taken off the table.\u00a0<\/p>\n
But if we only consider the technical dimensions, we can lose sight of the human factors at play. Content exists to be read. Authors and readers continually judge content according to whether it seems familiar or different. People often need to see things more than once.\u00a0They even choose to re-read some content.\u00a0<\/p>\n
Though technology is important, it\u2019s always in flux. Technology doesn\u2019t impose fixed rules and shouldn\u2019t dictate strategy.\u00a0<\/p>\n
Acknowledging the repetitiveness of content<\/h2>\n
A good amount of content repeats itself\u2014and always has. Repetition allows content to be disseminated more widely.\u00a0 Humans have copied text as long as they\u2019ve been writing. Text reuse is part of the human condition.<\/p>\n
Scholars analyze \u201cdifferent types of text reuse<\/a>, such as jokes, adverts, boilerplates, speeches, or religious texts, but also short stories and reprints of book segments. Each of them is tied to a different logic and motivation.\u201d<\/p>\n As one researcher studying the historical development of news stories notes, \u201cArticles emerge through a process of creative re-use and re-appropriation. Whole fragments, sentences and quotations are often transferred to novel contexts. In this sense, newspaper content emerges through a process of what could be called bricolage, in which content is soldered together from existing fragments and textual patterns<\/a>. In other words, newspaper content is often harvested from a wide range of available textual material.\u201d<\/p>\n Such research can help us to understand consequential issues such as:<\/p>\n Content propagation in the real world is messy. It happens organically through numerous small decisions made on a decentralized basis.\u00a0 Some decisions are opportunistic (such as plagiarism or repeating rumors), while others are motivated by a desire to spread credible information.\u00a0 No solution can be viable if it ignores the complex motivations of people conveying information.<\/p>\n Content professionals are generally wary of repeated content. They caution organizations to \u201cavoid duplication<\/a>\u201d because \u201cit\u2019s bad<\/a>.\u201d Their goal is to prevent<\/em> duplication and remediate it when it occurs.<\/p>\n The content professional\u2019s alternative to duplication<\/em> is content reuse<\/em>. Unlike duplication, content reuse is considered virtuous. Duplication and reuse are distinct approaches to repeating text, but they share similarities. They are not exact opposites. It doesn\u2019t follow that one is absolutely bad while the other is always good.\u00a0<\/p>\n Before we can consider the merits and behaviors of reuse, it\u2019s important to first understand the various manifestations of duplication, some of which overlap with content reuse.\u00a0\u00a0<\/p>\n Duplicate web pages on a website are almost always bad. A web page should live in only one place on a website. When the same page exists in several places on a website, it\u2019s fairly easy for software to locate such pages. Numerous tools can scan your website for duplicate pages using a mathematical technique called checksum.\u00a0\u00a0<\/p>\n When the same page exists across distinct web domains, the advisability of having the same content appear in multiple places gets more complicated. Sometimes, such behavior indicates a poorly governed publishing process, where a page is copied to various domains without either tracking this copying or asking if it is necessary.\u00a0 But not all situations are problems. There are legitimate use cases for publishing the same content on distinct pages on different websites.\u00a0 Content may be repeated across localized web domains or domains for subbrands of an organization.\u00a0\u00a0<\/p>\n Content syndication allows the same page to be republished on multiple domains to make it available to audiences so they can find it where they are looking for it rather than expecting they\u2019ll be hunting for it on an unfamiliar website.\u00a0 Organizations syndicate content throughout their own web properties or make it available to third parties<\/a>.<\/p>\n The audience\u2019s needs should determine whether the content should be placed on multiple websites.\u00a0<\/p>\n When identical web pages appear on multiple websites, this can be implemented in several ways.\u00a0 The pages can be shared either through RSS or an API that other websites can access. But often the original page is copied to a new website. The existence of multiple copies that are independent of one another introduces many content management inefficiencies and risks.\u00a0<\/p>\n The copying of webpages is often a consequence of the way CMSs are designed. Traditional CMSs support a single website, relying on folders and sitemaps to organize pages. Each additional website that needs the page must have the page copied into that site\u2019s page organization. While CMSs that support multiple websites have emerged recently, some still don\u2019t allow the original content to be organized independently of where on a website it will appear.\u00a0\u00a0<\/p>\n Duplicated content results from both human decisions and automated ones.\u00a0\u00a0<\/p>\n When organizations intend to duplicate content, they can do so for either good or bad faith motives.\u00a0<\/p>\n Good faith motivations reflect users\u2019 interests by making content available where they are looking for that content. Republishing of content is allowed and encouraged. The US Department of Health and Human Services encourages the syndication of its content: \u201cContent syndication allows you to place content from HHS websites onto your own site. It allows you to offer high-quality HHS content in the look and feel of your site<\/a>. The syndicated content is automatically updated in real-time, requiring no effort from your staff to keep the pages up to date.\u201d<\/p>\n Bad faith motivations include the intention to spam the user by blanketing them everywhere they might be. \u201c\u2018Copypasta\u2019<\/a> (a reference to copy-and-paste functionality to duplicate content) is an Internet slang term that refers to an attempt by multiple individuals to duplicate content from an original source and share it widely across social platforms or forums,\u201d noted a well known social media platform that subsequently changed its ownership and name. Of course, people alone aren\u2019t responsible for copypasta\u2013nowadays, bots do most of the work.<\/p>\n In other cases, duplication involves efforts to deceive who the author is or disguise the organization that is publishing the content. Bad actors can steal content and republish it through adversarial proxy mirroring<\/a> (the wholesale copying of a website that is rebranded) and web scraping (lifting published content and republishing it elsewhere without permission).\u00a0 Such copy-theft is illegal but technically easy<\/a> to perform.<\/p>\n While identical duplicate web pages are not uncommon, an even more pervasive situation is \u201cnear dupes\u201d or items that duplicate some content but also contain unique content.<\/p>\n Near duplicate content can be planned or incidental.\u00a0 Similarity in content items signals thematic repetition across multiple items. Near duplication content often represents variations on a core set of messages or information.\u00a0<\/p>\n Templates in e-commerce sites generate many pages of near duplicate content. They combine data feeds of product descriptions with boilerplate copy. Each product page has some identical wording it shares with other pages.\u00a0<\/p>\n Unlike checks for exact duplicates, auditing for near-duplicates involves noting both what\u2019s the same and what\u2019s unique. The audit needs to determine where items are dissimilar and whether that is intentional.\u00a0 Sometimes, copies of items are updated unevenly so that there are different versions of what should be identical text.\u00a0 Any variations within a copy of near-duplicates should convey distinct information or messages.<\/p>\n Also, note that near-duplicates aren\u2019t necessarily the repetition of exact prose. They may be summarizations or extensions. \u201cA near-duplicate is, in some cases, a mere paraphrasing of a previous article; in other cases, it contains corrections or added content as a follow-up<\/a>.\u201d Both publishers and readers can find value in extending what\u2019s been previously said.\u201d<\/p>\n Related content may duplicate strings or passages of text but don\u2019t replicate enough of the body of the content to appear as a near-duplicate. It emerges in various situations.\u00a0<\/p>\n Recurring phrases can signal that content items belong to a common content type.\u00a0 Content style guides may specify patterns for writing headlines, calls-to-action, and other strings.\u00a0 A recurring pattern might signify that the content item is a help topic or a hero.<\/p>\n Related content is also the product of repeating segments of content across items to support continuity in the user\u2019s content experience. Content chunks might be repeated to provide \u201csignposts,\u201d such as a preview or a takeaway.\u00a0<\/p>\n Repeating fragments of content support continuity across content items over time and through a customer journey.<\/p>\n More content management tools are focusing on repeatable content components. An example of this trend is the ubiquitous WordPress platform. WordPress\u2019 updated authoring interface, Gutenberg, manages content chunks it calls \u201cblocks.\u201d\u00a0 The interface allows authors to \u201cduplicate\u201d or \u201cshare\u201d blocks<\/a> in one item for use in another item.\u00a0 Shared blocks can be edited in any item where they are used, which will change them everywhere, though users report this behavior can be confusing and result in unanticipated changes. Because the blocks have no independent identity, their messages can be strongly influenced by the context in which they are edited.\u00a0\u00a0<\/p>\n Duplicated content can trigger a range of problems and consequences. Duplicated published<\/em> content may be bad or not. Duplicated unpublished<\/em> content is almost always problematic.<\/p>\n Let\u2019s start by looking at the internal consequences of duplicative content. Multiple versions of the same item are confusing to authors, editors, and content managers. No one can be sure which is the \u201cright\u201d version. Ironically, the latest version may not be the right one if someone creates a new copy and starts editing it without completing a full review.\u00a0 Abandoned drafts can also cloud which one is the active one. An unapproved version could be delivered to customers.\u00a0<\/p>\n The simple guideline to follow is that you shouldn\u2019t have exact copies of items in your content repository.\u00a0 Any near duplicates in your content inventory should be managed as content variants.\u00a0 (For a discussion of the distinction between versions and variants, see my post on content history<\/a>.)<\/p>\n Now, let\u2019s consider the situation of published content that\u2019s been duplicated. Is it bad for audiences?\u00a0 It can be, but won\u2019t necessarily be.\u00a0\u00a0<\/p>\n A wrong assumption often made about duplicated published content is that audiences will encounter it all at once. Many organizations rely on web crawls to simulate how audiences encounter their content.\u00a0 Web crawls often turn up duplicate pages.\u00a0 It doesn\u2019t follow that an individual will necessarily encounter these duplicates. Ironically, \u201cduplicated pages can even be introduced by the crawler itself,<\/a> when different links point to the same page.\u201d<\/p>\n An old myth in the SEO industry proclaimed that Google penalized duplicate content. But Google acknowledges that duplicate content, while potentially confusing to users, does not present a problem<\/a> for Google\u2019s search indexing: \u201cSome duplicate content on a site is normal<\/a> and it\u2019s not a violation of Google\u2019s spam policies. However, having the same content accessible through many different URLs can be a bad user experience (for example, people might wonder which is the right page and whether there\u2019s a difference between the two), and it may make it harder for you to track how your content performs in search results.\u201d<\/p>\n Duplicate content is often a symptom of other user experience issues, such as poor journey mapping or content labeling. No reader wants multiple links that all lead to the same item. When titles or links look similar, readers can\u2019t be sure whether equivalent options are identical and equally useful or are really different content items. For example, users frequently choose the wrong product support link because they are unable to understand and define distinctions between product variants.\u00a0<\/p>\n Content reuse is widely advocated but sometimes loosely defined. It\u2019s often not clear whether it refers to the internal reuse of content prior to publication or the external republication of content. Without making that distinction, it isn\u2019t clear when or whether duplication of content occurs. How does one apply the famous adage in content practice to be \u201cDRY<\/a>\u201d (Don\u2019t Repeat Yourself)? Should content not be repeated externally or only internally?<\/p>\n People may advocate reuse for a range of reasons:\u00a0<\/p>\n Content reuse implies that one copy of a content item can appear many times in various guises. The reality behind the scenes is more complicated, and it is perhaps more accurate to think about content reuse as managed<\/em> duplication.<\/p>\n Reuse implies one original content item will serve as the basis for published content that\u2019s delivered in various contexts. When implemented in publishing toolchains, there will likely be more than one copy. If you care about business continuity, your repository will likely have a mirror and backup, and it\u2019s possible an item will be cached in other systems involved in the publishing and delivery process. But while copies may exist, there\u2019ll only be one original.\u00a0<\/p>\n The original copy is sometimes referred to as the canonical one. Any changes are made only to the original; the other copies are read-only.\u00a0 Importantly, all changes are reversible since the copies are dependent on the original or are stored temporarily.\u00a0 With duplicated copies are unmanaged, by contrast, separate instances would each require updating, which often doesn\u2019t happen.<\/p>\n It\u2019s useful to distinguish delivery reuse (one item delivered to many places) from assembly reuse (one item incorporated into many other items). Most rationales for content reuse focus on internal content management requirements rather than external customer access benefits, but both are valid goals.<\/p>\n A wider perspective on reuse considers its role in contextualizing information and messages. Reused content can change the temporal and topical context.<\/p>\n Sometimes, reused content is standalone items: information or messages that need to be repeated in diverse scenarios. Such reuse allows target messages to be delivered at the right moment.<\/p>\n Other times, reused content is inserted into a larger item. But when reused content is incorporated into larger content items, content reuse can generate near-duplicates. Templated content, for example, repeats wording on multiple pages, making it hard for users to distinguish various items.\u00a0 From an external user\u2019s perspective, reused content can be indistinguishable from duplicated content.\u00a0<\/p>\n Reuse can support content customization. Organizations are expected to generate many variations of core content.\u00a0 Reuse has its roots in document management, the assembling of long-form documents that are built from both repeated text and customized text.\u00a0 But as online content moves away from long-form documents like product manuals and becomes more granular and on-demand, content customization is changing. Reuse in content assembly is still important, but more content is now reused directly by delivering standalone snippets or chunks.<\/p>\n Detecting duplicate content has become a mini-industry.\u00a0 Numerous technical approaches can identify duplicated content, and a range of vendors offer de-duplication solutions.\u00a0\u00a0<\/p>\n One vendor focuses on monitoring repetition in what\u2019s published online, asserting, \u201cThere\u2019s a wide variety of use cases for duplicate detection in the field of media monitoring<\/a>, ranging from virality analyses and content distribution tracking to plagiarism detection and web crawling.\u201d<\/p>\n Content aggregators need to filter duplicates. Another vendor sells a \u201ccontent deduplication\/travel content mapping solution<\/a>\u201d that gives customers \u201cthe opportunity to create your own hotel database and write original material.\u201d\u00a0<\/p>\n When organizations create content, they need to preclude making redundant content. One firm offers a tool to prevent writers from creating duplicate content<\/a> on intranets. The problem is not trivial: how do writers know what\u2019s already been created? They may create a new item that doesn\u2019t have the exact wording of an existing one, but with a focus that\u2019s nearly identical.\u00a0 <\/p>\n Governance based on well-defined content types (indicating a clear purpose for the content) and accurate, descriptive metadata (indicating the content\u2019s scope) is essential to preventing redundant content.\u00a0 Authors should be prompted to answer what the content is about before starting to create it.\u00a0 The inventory can check to see what existing content might be similar.<\/p>\n Since near-duplicates are more difficult to identify than exact ones, tools need to do \u201cfuzzy\u201d searches to find overlapping items.\u00a0 Techniques include \u201cMinHash\u201d and \u201cshingling\u201d that chop up strings to measure similarity thresholds.<\/p>\n While readers don\u2019t want to wade through duplicate items or have to disambiguate them, the same is true for machines \u2013 only at a larger scale. Software programs can behave oddly if the inventory of content emphasizes certain items too much.\u00a0 Duplication can introduce bias in software algorithms because programs are more inclined to select from duplicated information when performing searches or generating answers. Duplication of content has emerged as a concern in large language models<\/a>.\u00a0\u00a0<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/storyneedle.com\/wp-content\/uploads\/2024\/06\/Screenshot-2024-06-25-at-9.50.54%E2%80%AFPM.png?resize=580%2C244&ssl=1\")
<\/a>\n
Good and Bad reasons for duplicate content<\/h2>\n
\n
Near-duplicates: a pervasive phenomenon<\/h2>\n
Related content: the repetition of fragments<\/h2>\n
Looking at duplication from internal and external perspectives<\/h2>\n
Reuse: How different is it from duplication?<\/h2>\n
\n
![\"\"\/](\"https:\/\/lh7-us.googleusercontent.com\/docsz\/AD_4nXebHvJBoarZo-5isKVzx__3ebP579LaKXyRnO6JK5RCnJVRf4GsBIBQ5o8OAyrBw7Z1DV4J1qPW0S8aTIzT0svT6fee9cgT7YND1VMQKe24DqwyDGvq_e3Y2rubsJdjGwD3eegexNQqPy2_vQ-lqgcQ5o8?key=Nd2adIYV1VbyT4a88xs3HQ\")
The value of de-duplicating content<\/h2>\n