Connect with us


Are ChatGPT, Bard and Dolly 2.0 Trained On Pirated Content?



Are ChatGPT, Bard and Dolly 2.0 Trained On Pirated Content?

Large Language Models (LLMs) like ChatGPT, Bard and even open source versions are trained on public Internet content. But there are also indications that popular AIs might also be trained on datasets created from pirated books.

Is Dolly 2.0 Trained on Pirated Content?

Dolly 2.0 is an open source AI that was recently released. The intent behind Dolly is to democratize AI by  making it available to everyone who wants to create something with it, even commercial products.

But there’s also a privacy issue with concentrating AI technology in the hands of three major corporations and trusting them with private data.

Given a choice, many businesses would prefer to not hand off private data to third parties like Google, OpenAI and Meta.

Even Mozilla, the open source browser and app company, is investing in growing the open source AI ecosystem.

The intent behind open source AI is unquestionably good.

But there is  an issue with the data that is used to train these large language models because some of it consists of pirated content.

Open source ChatGPT clone, Dolly 2.0, was created by a company called DataBricks  (learn more about Dolly 2.0)

Dolly 2.0 is based on an Open Source Large Language Model (LLM) called Pythia (which was created by an open source group called, EleutherAI).

EleutherAI created eight versions of LLMs of different sizes within the Pythia family of LLMs.

One version of Pythia, a 12 billion parameter version, is the one used by DataBricks to create Dolly 2.0, as well as with a dataset that DataBricks created themselves (a dataset of questions and answers that was used to train the Dolly 2.0 AI to take instructions)

The thing about the EleutherAI Pythia LLM is that it was trained using a dataset called the Pile.

The Pile dataset is comprised of multiple sets of English language texts, one of which is a dataset called Books3. The Books3 dataset contains the text of books that were pirated and hosted at a pirate site called, bibliotik.

This is what the DataBricks announcement says:

“Dolly 2.0 is a 12B parameter language model based on the EleutherAI pythia model family and fine-tuned exclusively on a new, high-quality human generated instruction following dataset, crowdsourced among Databricks employees.”

Pythia LLM Was Created With the Pile Dataset

The Pythia research paper by EleutherAI that mentions that Pythia was trained using the Pile dataset.

This is a quote from the Pythia research paper:

“We train 8 model sizes each on both the Pile …and the Pile after deduplication, providing 2 copies of the suite which can be compared.”

Deduplication means that they removed redundant data, it’s a process for creating a cleaner dataset.

So what’s in Pile? There’s a Pile research paper that explains what’s in that dataset.

Here’s a quote from the research paper for Pile where it says that they use the Books3 dataset:

“In addition we incorporate several existing highquality datasets: Books3 (Presser, 2020)…”

The Pile dataset research paper links to a tweet by Shawn Presser, that says what is in the Books3 dataset:

“Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data.

Now you do. Now everyone does.

Presenting “books3”, aka “all of bibliotik”

– 196,640 books
– in plain .txt
– reliable, direct download, for years:”

So… the above quote clearly states that the Pile dataset was used to train the Pythia LLM which in turn served as the foundation for the Dolly 2.0 open source AI.

Is Google Bard Trained on Pirated Content?

The Washington Post recently published a review of Google’s Colossal Clean Crawled Corpus dataset (also known as C4 – PDF research paper here) in which they discovered that Google’s dataset also contains pirated content.

The C4 dataset is important because it’s one of the datasets used to train Google’s LaMDA LLM, a version of which is what Bard is based on.

The actual dataset is called Infiniset and the C4 dataset makes up about 12.5% of the total text used to train LaMDA. Citations to those facts about Bard can be found here.

The Washington Post news article published:

“The three biggest sites were No. 1, which contains text from patents issued around the world; No. 2, the free online encyclopedia; and No. 3, a subscription-only digital library.

Also high on the list: No. 190, a notorious market for pirated e-books that has since been seized by the U.S. Justice Department.

At least 27 other sites identified by the U.S. government as markets for piracy and counterfeits were present in the data set.”

The flaw in the Washington Post analysis is that they’re looking at a version of the C4 but not necessarily the one that LaMDA was trained on.

The research paper for the C4 dataset was published in July 2020. Within a year of publication another research paper was published that discovered that the C4 dataset was biased against people of color and the LGBT community.

The research paper is titled, Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (PDF research paper here).

It was discovered by the researchers that the dataset contained negative sentiment against people of Arab identies and excluded documents that were associated with Blacks, Hispanics, and documents that mention sexual orientation.

The researchers wrote:

“Our examination of the excluded data suggests that documents associated with Black and Hispanic authors and documents mentioning sexual orientations are significantly more likely to be excluded by C4.EN’s blocklist filtering, and that many excluded documents contained non-offensive or non-sexual content (e.g., legislative discussions of same-sex marriage, scientific and medical content).

This exclusion is a form of allocational harms …and exacerbates existing (language-based) racial inequality as well as stigmatization of LGBTQ+ identities…

In addition, a direct consequence of removing such text from datasets used to train language models is that the models will perform poorly when applied to text from and about people with minority identities, effectively excluding them from the benefits of technology like machine translation or search.”

It was concluded that the filtering of “bad words” and other attempts to “clean” the dataset was too simplistic and warranted are more nuanced approach.

Those conclusions are important because they show that it was well known that the C4 dataset was flawed.

LaMDA was developed in 2022 (two years after the C4 dataset) and the associated LaMDA research paper says that it was trained with C4.

But that’s just a research paper. What happens in real-life on a production model can be vastly different from what’s in the research paper.

When discussing a research paper it’s important to remember that Google consistently says that what’s in a patent or research paper isn’t necessarily what’s in use in Google’s algorithm.

Google is highly likely to be aware of those conclusions and it’s not unreasonable to assume that Google developed a new version of C4 for the production model, not just to address inequities in the dataset but to bring it up to date.

Google doesn’t say what’s in their algorithm, it’s a black box. So we can’t say with certainty that the technology underlying Google Bard was trained on pirated content.

To make it even clearer, Bard was released in 2023, using a lightweight version of LaMDA. Google has not defined what a lightweight version of LaMDA is.

So there’s no way to know what content was contained within the datasets used to train the lightweight version of LaMDA that powers Bard.

One can only speculate as to what content was used to train Bard.

Does GPT-4 Use Pirated Content?

OpenAI is extremely private about the datasets used to train GPT-4. The last time OpenAI mentioned datasets is in the PDF research paper for GPT-3 published in 2020 and even there it’s somewhat vague and imprecise about what’s in the datasets.

The TowardsDataScience website in 2021 published an interesting review of the available information in which they conclude that indeed some pirated content was used to train early versions of GPT.

They write:

“…we find evidence that BookCorpus directly violated copyright restrictions for hundreds of books that should not have been redistributed through a free dataset.

For example, over 200 books in BookCorpus explicitly state that they “may not be reproduced, copied and distributed for commercial or non-commercial purposes.””

It’s difficult to conclude whether GPT-4 used any pirated content.

Is There A Problem With Using Pirated Content?

One would think that it may be unethical to use pirated content to train a large language model and profit from the use of that content.

But the laws may actually allow this kind of use.

I asked Kenton J. Hutcherson, Internet Attorney at Hutcherson Law what he thought about the use of pirated content in the context of training large language models.

Specifically, I asked if someone uses Dolly 2.0, which may be partially created with pirated books, would commercial entities who create applications with Dolly 2.0 be exposed to copyright infringement claims?

Kenton answered:

“A claim for copyright infringement from the copyright holders of the pirated books would likely fail because of fair use.

Fair use protects transformative uses of copyrighted works.

Here, the pirated books are not being used as books for people to read, but as inputs to an artificial intelligence training dataset.

A similar example came into play with the use of thumbnails on search results pages. The thumbnails are not there to replace the webpages they preview. They serve a completely different function—they preview the page.

That is transformative use.”

Karen J. Bernstein of Bernstein IP offered a similar opinion.

“Is the use of the pirated content a fair use? Fair use is a commonly used defense in these instances.

The concept of the fair use defense only exists under US copyright law.

Fair use is analyzed under a multi-factor analysis that the Supreme Court set forth in a 1994 landmark case.

Under this scenario, there will be questions of how much of the pirated content was taken from the books and what was done to the content (was it “transformative”), and whether such content is taking the market away from the copyright creator.”

AI technology is bounding forward at an unprecedented pace, seemingly evolving on a week to week basis. Perhaps in a reflection of the competition and the financial windfall to be gained from success, Google and OpenAI are becoming increasingly private about how their AI models are trained.

Should they be more open about such information? Can they be trusted that their datasets are fair and non-biased?

The use of pirated content to create these AI models may be legally protected as fair use, but just because one can does that mean one should?

Featured image by Shutterstock/Roman Samborskyi

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address


GPT Store Set To Launch In 2024 After ‘Unexpected’ Delays




GPT Store Set To Launch In 2024 After 'Unexpected' Delays

OpenAI shares its plans for the GPT Store, enhancements to GPT Builder tools, privacy improvements, and updates coming to ChatGPT.

  • OpenAI has scheduled the launch of the GPT Store for early next year, aligning with its ongoing commitment to developing advanced AI technologies.
  • The GPT Builder tools have received substantial updates, including a more intuitive configuration interface and improved file handling capabilities.
  • Anticipation builds for upcoming updates to ChatGPT, highlighting OpenAI’s responsiveness to community feedback and dedication to AI innovation.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading


96.55% of Content Gets No Traffic From Google. Here’s How to Be in the Other 3.45% [New Research for 2023]



96.55% of Content Gets No Traffic From Google. Here's How to Be in the Other 3.45% [New Research for 2023]

It’s no secret that the web is growing by millions, if not billions of pages per day.

Our Content Explorer tool discovers 10 million new pages every 24 hours while being very picky about the pages that qualify for inclusion. The “main” Ahrefs web crawler crawls that number of pages every two minutes. 

But how much of this content gets organic traffic from Google?

To find out, we took the entire database from our Content Explorer tool (around 14 billion pages) and studied how many pages get traffic from organic search and why.

How many web pages get organic search traffic?

96.55% of all pages in our index get zero traffic from Google, and 1.94% get between one and ten monthly visits.

Distribution of pages by traffic from Content Explorer

Before we move on to discussing why the vast majority of pages never get any search traffic from Google (and how to avoid being one of them), it’s important to address two discrepancies with the studied data:

  1. ~14 billion pages may seem like a huge number, but it’s not the most accurate representation of the entire web. Even compared to the size of Site Explorer’s index of 340.8 billion pages, our sample size for this study is quite small and somewhat biased towards the “quality side of the web.”
  2. Our search traffic numbers are estimates. Even though our database of ~651 million keywords in Site Explorer (where our estimates come from) is arguably the largest database of its kind, it doesn’t contain every possible thing people search for in Google. There’s a chance that some of these pages get search traffic from super long-tail keywords that are not popular enough to make it into our database.

That said, these two “inaccuracies” don’t change much in the grand scheme of things: the vast majority of published pages never rank in Google and never get any search traffic. 

But why is this, and how can you be a part of the minority that gets organic search traffic from Google?

Well, there are hundreds of SEO issues that may prevent your pages from ranking well in Google. But if we focus only on the most common scenarios, assuming the page is indexed, there are only three of them.

Reason 1: The topic has no search demand

If nobody is searching for your topic, you won’t get any search traffic—even if you rank #1.

For example, I recently Googled “pull sitemap into google sheets” and clicked the top-ranking page (which solved my problem in seconds, by the way). But if you plug that URL into Ahrefs’ Site Explorer, you’ll see that it gets zero estimated organic search traffic:

The top-ranking page for this topic gets no traffic because there's no search demandThe top-ranking page for this topic gets no traffic because there's no search demand

This is because hardly anyone else is searching for this, as data from Keywords Explorer confirms:

Keyword data from Ahrefs' Keywords Explorer confirms that this topic has no search demandKeyword data from Ahrefs' Keywords Explorer confirms that this topic has no search demand

This is why it’s so important to do keyword research. You can’t just assume that people are searching for whatever you want to talk about. You need to check the data.

Our Traffic Potential (TP) metric in Keywords Explorer can help with this. It estimates how much organic search traffic the current top-ranking page for a keyword gets from all the queries it ranks for. This is a good indicator of the total search demand for a topic.

You’ll see this metric for every keyword in Keywords Explorer, and you can even filter for keywords that meet your minimum criteria (e.g., 500+ monthly traffic potential): 

Filtering for keywords with Traffic Potential (TP) in Ahrefs' Keywords ExplorerFiltering for keywords with Traffic Potential (TP) in Ahrefs' Keywords Explorer

Reason 2: The page has no backlinks

Backlinks are one of Google’s top three ranking factors, so it probably comes as no surprise that there’s a clear correlation between the number of websites linking to a page and its traffic.

Pages with more referring domains get more trafficPages with more referring domains get more traffic
Pages with more referring domains get more traffic

Same goes for the correlation between a page’s traffic and keyword rankings:

Pages with more referring domains rank for more keywordsPages with more referring domains rank for more keywords
Pages with more referring domains rank for more keywords

Does any of this data prove that backlinks help you rank higher in Google?

No, because correlation does not imply causation. However, most SEO professionals will tell you that it’s almost impossible to rank on the first page for competitive keywords without backlinks—an observation that aligns with the data above.

The key word there is “competitive.” Plenty of pages get organic traffic while having no backlinks…

Pages with more referring domains get more trafficPages with more referring domains get more traffic
How much traffic pages with no backlinks get

… but from what I can tell, almost all of them are about low-competition topics.

For example, this lyrics page for a Neil Young song gets an estimated 162 monthly visits with no backlinks: 

Example of a page with traffic but no backlinks, via Ahrefs' Content ExplorerExample of a page with traffic but no backlinks, via Ahrefs' Content Explorer

But if we check the keywords it ranks for, they almost all have Keyword Difficulty (KD) scores in the single figures:

Some of the low-difficulty keywords a page without traffic ranks forSome of the low-difficulty keywords a page without traffic ranks for

It’s the same story for this page selling upholstered headboards:

Some of the low-difficulty keywords a page without traffic ranks forSome of the low-difficulty keywords a page without traffic ranks for

You might have noticed two other things about these pages:

  • Neither of them get that much traffic. This is pretty typical. Our index contains ~20 million pages with no referring domains, yet only 2,997 of them get more than 1K search visits per month. That’s roughly 1 in every 6,671 pages with no backlinks.
  • Both of the sites they’re on have high Domain Rating (DR) scores. This metric shows the relative strength of a website’s backlink profile. Stronger sites like these have more PageRank that they can pass to pages with internal links to help them rank. 

Bottom line? If you want your pages to get search traffic, you really only have two options:

  1. Target uncompetitive topics that you can rank for with few or no backlinks.
  2. Target competitive topics and build backlinks to rank.

If you want to find uncompetitive topics, try this:

  1. Enter a topic into Keywords Explorer
  2. Go to the Matching terms report
  3. Set the Keyword Difficulty (KD) filter to max. 20
  4. Set the Lowest DR filter to your site’s DR (this will show you keywords with at least one of the same or lower DR ranking in the top 5)
Filtering for low-competition keywords in Ahrefs' Keywords ExplorerFiltering for low-competition keywords in Ahrefs' Keywords Explorer

(Remember to keep an eye on the TP column to make sure they have traffic potential.)

To rank for more competitive topics, you’ll need to earn or build high-quality backlinks to your page. If you’re not sure how to do that, start with the guides below. Keep in mind that it’ll be practically impossible to get links unless your content adds something to the conversation. 

Reason 3. The page doesn’t match search intent

Google wants to give users the most relevant results for a query. That’s why the top organic results for “best yoga mat” are blog posts with recommendations, not product pages. 

It's obviously what searchers want when they search for "best yoga mats"It's obviously what searchers want when they search for "best yoga mats"

Basically, Google knows that searchers are in research mode, not buying mode.

It’s also why this page selling yoga mats doesn’t show up, despite it having backlinks from more than six times more websites than any of the top-ranking pages:

Page selling yoga mats that has lots of backlinksPage selling yoga mats that has lots of backlinks
Number of linking websites to the top-ranking pages for "best yoga mats"Number of linking websites to the top-ranking pages for "best yoga mats"

Luckily, the page ranks for thousands of other more relevant keywords and gets tens of thousands of monthly organic visits. So it’s not such a big deal that it doesn’t rank for “best yoga mats.”

Number of keyword rankings for the page selling yoga matsNumber of keyword rankings for the page selling yoga mats

However, if you have pages with lots of backlinks but no organic traffic—and they already target a keyword with traffic potential—another quick SEO win is to re-optimize them for search intent.

We did this in 2018 with our free backlink checker.

It was originally nothing but a boring landing page explaining the benefits of our product and offering a 7-day trial: 

Original landing page for our free backlink checkerOriginal landing page for our free backlink checker

After analyzing search intent, we soon realized the issue:

People weren’t looking for a landing page, but rather a free tool they could use right away. 

So, in September 2018, we created a free tool and published it under the same URL. It ranked #1 pretty much overnight, and has remained there ever since. 

Our rankings over time for the keyword "backlink checker." You can see when we changed the pageOur rankings over time for the keyword "backlink checker." You can see when we changed the page

Organic traffic went through the roof, too. From ~14K monthly organic visits pre-optimization to almost ~200K today. 

Estimated search traffic over time to our free backlink checkerEstimated search traffic over time to our free backlink checker


96.55% of pages get no organic traffic. 

Keep your pages in the other 3.45% by building backlinks, choosing topics with organic traffic potential, and matching search intent.

Ping me on Twitter if you have any questions. 🙂

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading


Firefox URL Tracking Removal – Is This A Trend To Watch?




Firefox URL Tracking Removal - Is This A Trend To Watch?

Firefox recently announced that they are offering users a choice on whether or not to include tracking information from copied URLs, which comes on the on the heels of iOS 17 blocking user tracking via URLs. The momentum of removing tracking information from URLs appears to be gaining speed. Where is this all going and should marketers be concerned?

Is it possible that blocking URL tracking parameters in the name of privacy will become a trend industrywide?

Firefox Announcement

Firefox recently announced that beginning in the Firefox Browser version 120.0, users will be able to select whether or not they want URLs that they copied to contain tracking parameters.

When users select a link to copy and click to raise the contextual menu for it, Firefox is now giving users a choice as to whether to copy the URL with or without the URL tracking parameters that might be attached to the URL.

Screenshot Of Firefox 120 Contextual Menu

Screenshot of Firefox functionality

According to the Firefox 120 announcement:

“Firefox supports a new “Copy Link Without Site Tracking” feature in the context menu which ensures that copied links no longer contain tracking information.”

Browser Trends For Privacy

All browsers, including Google’s Chrome and Chrome variants, are adding new features that make it harder for websites to track users online through referrer information embedded in a URL when a user clicks from one site and leaves through that click to visit another site.

This trend for privacy has been ongoing for many years but it became more noticeable in 2020 when Chrome made changes to how referrer information was sent when users click links to visit other sites. Firefox and Safari followed with similar referrer behavior.

Whether the current Firefox implementation would be disruptive or if the impact is overblown is kind of besides the point.

What is the point is whether or not what Firefox and Apple did to protect privacy is a trend and if that trend will extend to more blocking of URL parameters that are stronger than what Firefox recently implemented.

I asked Kenny Hyder, CEO of online marketing agency Pixel Main, what his thoughts are about the potential disruptive aspect of what Firefox is doing and whether it’s a trend.

Kenny answered:

“It’s not disruptive from Firefox alone, which only has a 3% market share. If other popular browsers follow suit it could begin to be disruptive to a limited degree, but easily solved from a marketers prospective.

If it became more intrusive and they blocked UTM tags, it would take awhile for them all to catch on if you were to circumvent UTM tags by simply tagging things in a series of sub-directories.. ie.<tag1>/<tag2> etc.

Also, most savvy marketers are already integrating future proof workarounds for these exact scenarios.

A lot can be done with pixel based integrations rather than cookie based or UTM tracking. When set up properly they can actually provide better and more accurate tracking and attribution. Hence the name of my agency, Pixel Main.”

I think most marketers are aware that privacy is the trend. The good ones have already taken steps to keep it from becoming a problem while still respecting user privacy.”

Some URL Parameters Are Already Affected

For those who are on the periphery of what’s going on with browsers and privacy, it may come as a surprise that some tracking parameters are already affected by actions meant to protect user privacy.

Jonathan Cairo, Lead Solutions Engineer at Elevar shared that there is already a limited amount of tracking related information stripped from URLs.

But he also explained that there are limits to how much information can be stripped from URLs because the resulting negative effects would cause important web browsing functionality to fail.

Jonathan explained:

“So far, we’re seeing a selective trend where some URL parameters, like ‘fbclid’ in Safari’s private browsing, are disappearing, while others, such as TikTok’s ‘ttclid’, remain.

UTM parameters are expected to stay since they focus on user segmentation rather than individual tracking, provided they are used as intended.

The idea of completely removing all URL parameters seems improbable, as it would disrupt key functionalities on numerous websites, including banking services and search capabilities.

Such a drastic move could lead users to switch to alternative browsers.

On the other hand, if only some parameters are eliminated, there’s the possibility of marketers exploiting the remaining ones for tracking purposes.

This raises the question of whether companies like Apple will take it upon themselves to prevent such use.

Regardless, even in a scenario where all parameters are lost, there are still alternative ways to convey click IDs and UTM information to websites.”

Brad Redding of Elevar agreed about the disruptive effect from going too far with removing URL tracking information:

“There is still too much basic internet functionality that relies on query parameters, such as logging in, password resets, etc, which are effectively the same as URL parameters in a full URL path.

So we believe the privacy crackdown is going to continue on known trackers by blocking their tracking scripts, cookies generated from them, and their ability to monitor user’s activity through the browser.

As this grows, the reliance on brands to own their first party data collection and bring consent preferences down to a user-level (vs session based) will be critical so they can backfill gaps in conversion data to their advertising partners outside of the browser or device.”

The Future Of Tracking, Privacy And What Marketers Should Expect

Elevar raises good points about how far browsers can go in terms of how much blocking they can do. Their response that it’s down to brands to own their first party data collection and other strategies to accomplish analytics without compromising user privacy.

Given all the laws governing privacy and Internet tracking that have been enacted around the world it looks like privacy will continue to be a trend.

However, at this point it time, the advice is to keep monitoring how far browsers are going but there is no expectation that things will get out of hand.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading