Connect with us

SEO

Are ChatGPT, Bard and Dolly 2.0 Trained On Pirated Content?

Published

on

Are ChatGPT, Bard and Dolly 2.0 Trained On Pirated Content?

Large Language Models (LLMs) like ChatGPT, Bard and even open source versions are trained on public Internet content. But there are also indications that popular AIs might also be trained on datasets created from pirated books.

Is Dolly 2.0 Trained on Pirated Content?

Dolly 2.0 is an open source AI that was recently released. The intent behind Dolly is to democratize AI by  making it available to everyone who wants to create something with it, even commercial products.

But there’s also a privacy issue with concentrating AI technology in the hands of three major corporations and trusting them with private data.

Given a choice, many businesses would prefer to not hand off private data to third parties like Google, OpenAI and Meta.

Even Mozilla, the open source browser and app company, is investing in growing the open source AI ecosystem.

Advertisement

The intent behind open source AI is unquestionably good.

But there is  an issue with the data that is used to train these large language models because some of it consists of pirated content.

Open source ChatGPT clone, Dolly 2.0, was created by a company called DataBricks  (learn more about Dolly 2.0)

Dolly 2.0 is based on an Open Source Large Language Model (LLM) called Pythia (which was created by an open source group called, EleutherAI).

EleutherAI created eight versions of LLMs of different sizes within the Pythia family of LLMs.

One version of Pythia, a 12 billion parameter version, is the one used by DataBricks to create Dolly 2.0, as well as with a dataset that DataBricks created themselves (a dataset of questions and answers that was used to train the Dolly 2.0 AI to take instructions)

Advertisement

The thing about the EleutherAI Pythia LLM is that it was trained using a dataset called the Pile.

The Pile dataset is comprised of multiple sets of English language texts, one of which is a dataset called Books3. The Books3 dataset contains the text of books that were pirated and hosted at a pirate site called, bibliotik.

This is what the DataBricks announcement says:

“Dolly 2.0 is a 12B parameter language model based on the EleutherAI pythia model family and fine-tuned exclusively on a new, high-quality human generated instruction following dataset, crowdsourced among Databricks employees.”

Pythia LLM Was Created With the Pile Dataset

The Pythia research paper by EleutherAI that mentions that Pythia was trained using the Pile dataset.

This is a quote from the Pythia research paper:

Advertisement

“We train 8 model sizes each on both the Pile …and the Pile after deduplication, providing 2 copies of the suite which can be compared.”

Deduplication means that they removed redundant data, it’s a process for creating a cleaner dataset.

So what’s in Pile? There’s a Pile research paper that explains what’s in that dataset.

Here’s a quote from the research paper for Pile where it says that they use the Books3 dataset:

“In addition we incorporate several existing highquality datasets: Books3 (Presser, 2020)…”

The Pile dataset research paper links to a tweet by Shawn Presser, that says what is in the Books3 dataset:

“Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data.

Now you do. Now everyone does.

Presenting “books3”, aka “all of bibliotik”

Advertisement

– 196,640 books
– in plain .txt
– reliable, direct download, for years: https://the-eye.eu/public/AI/pile_preliminary_components/books3.tar.gz”

So… the above quote clearly states that the Pile dataset was used to train the Pythia LLM which in turn served as the foundation for the Dolly 2.0 open source AI.

Is Google Bard Trained on Pirated Content?

The Washington Post recently published a review of Google’s Colossal Clean Crawled Corpus dataset (also known as C4 – PDF research paper here) in which they discovered that Google’s dataset also contains pirated content.

The C4 dataset is important because it’s one of the datasets used to train Google’s LaMDA LLM, a version of which is what Bard is based on.

The actual dataset is called Infiniset and the C4 dataset makes up about 12.5% of the total text used to train LaMDA. Citations to those facts about Bard can be found here.

The Washington Post news article published:

Advertisement

“The three biggest sites were patents.google.com No. 1, which contains text from patents issued around the world; wikipedia.org No. 2, the free online encyclopedia; and scribd.com No. 3, a subscription-only digital library.

Also high on the list: b-ok.org No. 190, a notorious market for pirated e-books that has since been seized by the U.S. Justice Department.

At least 27 other sites identified by the U.S. government as markets for piracy and counterfeits were present in the data set.”

The flaw in the Washington Post analysis is that they’re looking at a version of the C4 but not necessarily the one that LaMDA was trained on.

The research paper for the C4 dataset was published in July 2020. Within a year of publication another research paper was published that discovered that the C4 dataset was biased against people of color and the LGBT community.

The research paper is titled, Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (PDF research paper here).

It was discovered by the researchers that the dataset contained negative sentiment against people of Arab identies and excluded documents that were associated with Blacks, Hispanics, and documents that mention sexual orientation.

Advertisement

The researchers wrote:

“Our examination of the excluded data suggests that documents associated with Black and Hispanic authors and documents mentioning sexual orientations are significantly more likely to be excluded by C4.EN’s blocklist filtering, and that many excluded documents contained non-offensive or non-sexual content (e.g., legislative discussions of same-sex marriage, scientific and medical content).

This exclusion is a form of allocational harms …and exacerbates existing (language-based) racial inequality as well as stigmatization of LGBTQ+ identities…

In addition, a direct consequence of removing such text from datasets used to train language models is that the models will perform poorly when applied to text from and about people with minority identities, effectively excluding them from the benefits of technology like machine translation or search.”

It was concluded that the filtering of “bad words” and other attempts to “clean” the dataset was too simplistic and warranted are more nuanced approach.

Those conclusions are important because they show that it was well known that the C4 dataset was flawed.

LaMDA was developed in 2022 (two years after the C4 dataset) and the associated LaMDA research paper says that it was trained with C4.

Advertisement

But that’s just a research paper. What happens in real-life on a production model can be vastly different from what’s in the research paper.

When discussing a research paper it’s important to remember that Google consistently says that what’s in a patent or research paper isn’t necessarily what’s in use in Google’s algorithm.

Google is highly likely to be aware of those conclusions and it’s not unreasonable to assume that Google developed a new version of C4 for the production model, not just to address inequities in the dataset but to bring it up to date.

Google doesn’t say what’s in their algorithm, it’s a black box. So we can’t say with certainty that the technology underlying Google Bard was trained on pirated content.

To make it even clearer, Bard was released in 2023, using a lightweight version of LaMDA. Google has not defined what a lightweight version of LaMDA is.

So there’s no way to know what content was contained within the datasets used to train the lightweight version of LaMDA that powers Bard.

Advertisement

One can only speculate as to what content was used to train Bard.

Does GPT-4 Use Pirated Content?

OpenAI is extremely private about the datasets used to train GPT-4. The last time OpenAI mentioned datasets is in the PDF research paper for GPT-3 published in 2020 and even there it’s somewhat vague and imprecise about what’s in the datasets.

The TowardsDataScience website in 2021 published an interesting review of the available information in which they conclude that indeed some pirated content was used to train early versions of GPT.

They write:

“…we find evidence that BookCorpus directly violated copyright restrictions for hundreds of books that should not have been redistributed through a free dataset.

For example, over 200 books in BookCorpus explicitly state that they “may not be reproduced, copied and distributed for commercial or non-commercial purposes.””

It’s difficult to conclude whether GPT-4 used any pirated content.

Advertisement

Is There A Problem With Using Pirated Content?

One would think that it may be unethical to use pirated content to train a large language model and profit from the use of that content.

But the laws may actually allow this kind of use.

I asked Kenton J. Hutcherson, Internet Attorney at Hutcherson Law what he thought about the use of pirated content in the context of training large language models.

Specifically, I asked if someone uses Dolly 2.0, which may be partially created with pirated books, would commercial entities who create applications with Dolly 2.0 be exposed to copyright infringement claims?

Kenton answered:

“A claim for copyright infringement from the copyright holders of the pirated books would likely fail because of fair use.

Fair use protects transformative uses of copyrighted works.

Advertisement

Here, the pirated books are not being used as books for people to read, but as inputs to an artificial intelligence training dataset.

A similar example came into play with the use of thumbnails on search results pages. The thumbnails are not there to replace the webpages they preview. They serve a completely different function—they preview the page.

That is transformative use.”

Karen J. Bernstein of Bernstein IP offered a similar opinion.

“Is the use of the pirated content a fair use? Fair use is a commonly used defense in these instances.

The concept of the fair use defense only exists under US copyright law.

Fair use is analyzed under a multi-factor analysis that the Supreme Court set forth in a 1994 landmark case.

Advertisement

Under this scenario, there will be questions of how much of the pirated content was taken from the books and what was done to the content (was it “transformative”), and whether such content is taking the market away from the copyright creator.”

AI technology is bounding forward at an unprecedented pace, seemingly evolving on a week to week basis. Perhaps in a reflection of the competition and the financial windfall to be gained from success, Google and OpenAI are becoming increasingly private about how their AI models are trained.

Should they be more open about such information? Can they be trusted that their datasets are fair and non-biased?

The use of pirated content to create these AI models may be legally protected as fair use, but just because one can does that mean one should?

Featured image by Shutterstock/Roman Samborskyi



Source link

Advertisement
Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

SEO

How Compression Can Be Used To Detect Low Quality Pages

Published

on

By

Compression can be used by search engines to detect low-quality pages. Although not widely known, it's useful foundational knowledge for SEO.

The concept of Compressibility as a quality signal is not widely known, but SEOs should be aware of it. Search engines can use web page compressibility to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords, making it useful knowledge for SEO.

Although the following research paper demonstrates a successful use of on-page features for detecting spam, the deliberate lack of transparency by search engines makes it difficult to say with certainty if search engines are applying this or similar techniques.

What Is Compressibility?

In computing, compressibility refers to how much a file (data) can be reduced in size while retaining essential information, typically to maximize storage space or to allow more data to be transmitted over the Internet.

TL/DR Of Compression

Compression replaces repeated words and phrases with shorter references, reducing the file size by significant margins. Search engines typically compress indexed web pages to maximize storage space, reduce bandwidth, and improve retrieval speed, among other reasons.

This is a simplified explanation of how compression works:

  • Identify Patterns:
    A compression algorithm scans the text to find repeated words, patterns and phrases
  • Shorter Codes Take Up Less Space:
    The codes and symbols use less storage space then the original words and phrases, which results in a smaller file size.
  • Shorter References Use Less Bits:
    The “code” that essentially symbolizes the replaced words and phrases uses less data than the originals.

A bonus effect of using compression is that it can also be used to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords.

Research Paper About Detecting Spam

This research paper is significant because it was authored by distinguished computer scientists known for breakthroughs in AI, distributed computing, information retrieval, and other fields.

Advertisement

Marc Najork

One of the co-authors of the research paper is Marc Najork, a prominent research scientist who currently holds the title of Distinguished Research Scientist at Google DeepMind. He’s a co-author of the papers for TW-BERT, has contributed research for increasing the accuracy of using implicit user feedback like clicks, and worked on creating improved AI-based information retrieval (DSI++: Updating Transformer Memory with New Documents), among many other major breakthroughs in information retrieval.

Dennis Fetterly

Another of the co-authors is Dennis Fetterly, currently a software engineer at Google. He is listed as a co-inventor in a patent for a ranking algorithm that uses links, and is known for his research in distributed computing and information retrieval.

Those are just two of the distinguished researchers listed as co-authors of the 2006 Microsoft research paper about identifying spam through on-page content features. Among the several on-page content features the research paper analyzes is compressibility, which they discovered can be used as a classifier for indicating that a web page is spammy.

Detecting Spam Web Pages Through Content Analysis

Although the research paper was authored in 2006, its findings remain relevant to today.

Then, as now, people attempted to rank hundreds or thousands of location-based web pages that were essentially duplicate content aside from city, region, or state names. Then, as now, SEOs often created web pages for search engines by excessively repeating keywords within titles, meta descriptions, headings, internal anchor text, and within the content to improve rankings.

Section 4.6 of the research paper explains:

Advertisement

“Some search engines give higher weight to pages containing the query keywords several times. For example, for a given query term, a page that contains it ten times may be higher ranked than a page that contains it only once. To take advantage of such engines, some spam pages replicate their content several times in an attempt to rank higher.”

The research paper explains that search engines compress web pages and use the compressed version to reference the original web page. They note that excessive amounts of redundant words results in a higher level of compressibility. So they set about testing if there’s a correlation between a high level of compressibility and spam.

They write:

“Our approach in this section to locating redundant content within a page is to compress the page; to save space and disk time, search engines often compress web pages after indexing them, but before adding them to a page cache.

…We measure the redundancy of web pages by the compression ratio, the size of the uncompressed page divided by the size of the compressed page. We used GZIP …to compress pages, a fast and effective compression algorithm.”

High Compressibility Correlates To Spam

The results of the research showed that web pages with at least a compression ratio of 4.0 tended to be low quality web pages, spam. However, the highest rates of compressibility became less consistent because there were fewer data points, making it harder to interpret.

Figure 9: Prevalence of spam relative to compressibility of page.

The researchers concluded:

Advertisement

“70% of all sampled pages with a compression ratio of at least 4.0 were judged to be spam.”

But they also discovered that using the compression ratio by itself still resulted in false positives, where non-spam pages were incorrectly identified as spam:

“The compression ratio heuristic described in Section 4.6 fared best, correctly identifying 660 (27.9%) of the spam pages in our collection, while misidentifying 2, 068 (12.0%) of all judged pages.

Using all of the aforementioned features, the classification accuracy after the ten-fold cross validation process is encouraging:

95.4% of our judged pages were classified correctly, while 4.6% were classified incorrectly.

More specifically, for the spam class 1, 940 out of the 2, 364 pages, were classified correctly. For the non-spam class, 14, 440 out of the 14,804 pages were classified correctly. Consequently, 788 pages were classified incorrectly.”

The next section describes an interesting discovery about how to increase the accuracy of using on-page signals for identifying spam.

Insight Into Quality Rankings

The research paper examined multiple on-page signals, including compressibility. They discovered that each individual signal (classifier) was able to find some spam but that relying on any one signal on its own resulted in flagging non-spam pages for spam, which are commonly referred to as false positive.

Advertisement

The researchers made an important discovery that everyone interested in SEO should know, which is that using multiple classifiers increased the accuracy of detecting spam and decreased the likelihood of false positives. Just as important, the compressibility signal only identifies one kind of spam but not the full range of spam.

The takeaway is that compressibility is a good way to identify one kind of spam but there are other kinds of spam that aren’t caught with this one signal. Other kinds of spam were not caught with the compressibility signal.

This is the part that every SEO and publisher should be aware of:

“In the previous section, we presented a number of heuristics for assaying spam web pages. That is, we measured several characteristics of web pages, and found ranges of those characteristics which correlated with a page being spam. Nevertheless, when used individually, no technique uncovers most of the spam in our data set without flagging many non-spam pages as spam.

For example, considering the compression ratio heuristic described in Section 4.6, one of our most promising methods, the average probability of spam for ratios of 4.2 and higher is 72%. But only about 1.5% of all pages fall in this range. This number is far below the 13.8% of spam pages that we identified in our data set.”

So, even though compressibility was one of the better signals for identifying spam, it still was unable to uncover the full range of spam within the dataset the researchers used to test the signals.

Combining Multiple Signals

The above results indicated that individual signals of low quality are less accurate. So they tested using multiple signals. What they discovered was that combining multiple on-page signals for detecting spam resulted in a better accuracy rate with less pages misclassified as spam.

Advertisement

The researchers explained that they tested the use of multiple signals:

“One way of combining our heuristic methods is to view the spam detection problem as a classification problem. In this case, we want to create a classification model (or classifier) which, given a web page, will use the page’s features jointly in order to (correctly, we hope) classify it in one of two classes: spam and non-spam.”

These are their conclusions about using multiple signals:

“We have studied various aspects of content-based spam on the web using a real-world data set from the MSNSearch crawler. We have presented a number of heuristic methods for detecting content based spam. Some of our spam detection methods are more effective than others, however when used in isolation our methods may not identify all of the spam pages. For this reason, we combined our spam-detection methods to create a highly accurate C4.5 classifier. Our classifier can correctly identify 86.2% of all spam pages, while flagging very few legitimate pages as spam.”

Key Insight:

Misidentifying “very few legitimate pages as spam” was a significant breakthrough. The important insight that everyone involved with SEO should take away from this is that one signal by itself can result in false positives. Using multiple signals increases the accuracy.

What this means is that SEO tests of isolated ranking or quality signals will not yield reliable results that can be trusted for making strategy or business decisions.

Takeaways

We don’t know for certain if compressibility is used at the search engines but it’s an easy to use signal that combined with others could be used to catch simple kinds of spam like thousands of city name doorway pages with similar content. Yet even if the search engines don’t use this signal, it does show how easy it is to catch that kind of search engine manipulation and that it’s something search engines are well able to handle today.

Here are the key points of this article to keep in mind:

Advertisement
  • Doorway pages with duplicate content is easy to catch because they compress at a higher ratio than normal web pages.
  • Groups of web pages with a compression ratio above 4.0 were predominantly spam.
  • Negative quality signals used by themselves to catch spam can lead to false positives.
  • In this particular test, they discovered that on-page negative quality signals only catch specific types of spam.
  • When used alone, the compressibility signal only catches redundancy-type spam, fails to detect other forms of spam, and leads to false positives.
  • Combing quality signals improves spam detection accuracy and reduces false positives.
  • Search engines today have a higher accuracy of spam detection with the use of AI like Spam Brain.

Read the research paper, which is linked from the Google Scholar page of Marc Najork:

Detecting spam web pages through content analysis

Featured Image by Shutterstock/pathdoc

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

New Google Trends SEO Documentation

Published

on

By

Google publishes new documentation for how to use Google Trends for search marketing

Google Search Central published new documentation on Google Trends, explaining how to use it for search marketing. This guide serves as an easy to understand introduction for newcomers and a helpful refresher for experienced search marketers and publishers.

The new guide has six sections:

  1. About Google Trends
  2. Tutorial on monitoring trends
  3. How to do keyword research with the tool
  4. How to prioritize content with Trends data
  5. How to use Google Trends for competitor research
  6. How to use Google Trends for analyzing brand awareness and sentiment

The section about monitoring trends advises there are two kinds of rising trends, general and specific trends, which can be useful for developing content to publish on a site.

Using the Explore tool, you can leave the search box empty and view the current rising trends worldwide or use a drop down menu to focus on trends in a specific country. Users can further filter rising trends by time periods, categories and the type of search. The results show rising trends by topic and by keywords.

To search for specific trends users just need to enter the specific queries and then filter them by country, time, categories and type of search.

The section called Content Calendar describes how to use Google Trends to understand which content topics to prioritize.

Advertisement

Google explains:

“Google Trends can be helpful not only to get ideas on what to write, but also to prioritize when to publish it. To help you better prioritize which topics to focus on, try to find seasonal trends in the data. With that information, you can plan ahead to have high quality content available on your site a little before people are searching for it, so that when they do, your content is ready for them.”

Read the new Google Trends documentation:

Get started with Google Trends

Featured Image by Shutterstock/Luis Molinero

Source link

Advertisement
Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

All the best things about Ahrefs Evolve 2024

Published

on

All the best things about Ahrefs Evolve 2024

Hey all, I’m Rebekah and I am your Chosen One to “do a blog post for Ahrefs Evolve 2024”.

What does that entail exactly? I don’t know. In fact, Sam Oh asked me yesterday what the title of this post would be. “Is it like…Ahrefs Evolve 2024: Recap of day 1 and day 2…?” 

Even as I nodded, I couldn’t get over how absolutely boring that sounded. So I’m going to do THIS instead: a curation of all the best things YOU loved about Ahrefs’ first conference, lifted directly from X.

Let’s go!

OUR HUGE SCREEN

CONFERENCE VENUE ITSELF

It was recently named the best new skyscraper in the world, by the way.

 

OUR AMAZING SPEAKER LINEUP – SUPER INFORMATIVE, USEFUL TALKS!

 

Advertisement

GREAT MUSIC

 

AMAZING GOODIES

 

SELFIE BATTLE

Some background: Tim and Sam have a challenge going on to see who can take the most number of selfies with all of you. Last I heard, Sam was winning – but there is room for a comeback yet!

 

THAT BELL

Everybody’s just waiting for this one.

 

STICKER WALL

AND, OF COURSE…ALL OF YOU!

 

Advertisement

There’s a TON more content on LinkedIn – click here – but I have limited time to get this post up and can’t quite figure out how to embed LinkedIn posts so…let’s stop here for now. I’ll keep updating as we go along!



Source link

Advertisement
Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending