Large Language Models (LLMs) like ChatGPT train using multiple sources of information, including web content. This data forms the basis of summaries of that content in the form of articles that are produced without attribution or benefit to those who published the original content used for training ChatGPT.

Search engines download website content (called crawling and indexing) to provide answers in the form of links to the websites.

Website publishers have the ability to opt-out of having their content crawled and indexed by search engines through the Robots Exclusion Protocol, commonly referred to as Robots.txt.

The Robots Exclusions Protocol is not an official Internet standard but it’s one that legitimate web crawlers obey.

Should web publishers be able to use the Robots.txt protocol to prevent large language models from using their website content?

Large Language Models Use Website Content Without Attribution

Some who are involved with search marketing are uncomfortable with how website data is used to train machines without giving anything back, like an acknowledgement or traffic.

Hans Petter Blindheim (LinkedIn profile), Senior Expert at Curamando shared his opinions with me.

Hans commented:

“When an author writes something after having learned something from an article on your site, they will more often than not link to your original work because it offers credibility and as a professional courtesy.

It’s called a citation.

But the scale at which ChatGPT assimilates content and does not grant anything back differentiates it from both Google and people.

A website is generally created with a business directive in mind.

Google helps people find the content, providing traffic, which has a mutual benefit to it.

But it’s not like large language models asked your permission to use your content, they just use it in a broader sense than what was expected when your content was published.

And if the AI language models do not offer value in return – why should publishers allow them to crawl and use the content?

Does their use of your content meet the standards of fair use?

When ChatGPT and Google’s own ML/AI models trains on your content without permission, spins what it learns there and uses that while keeping people away from your websites – shouldn’t the industry and also lawmakers try to take back control over the Internet by forcing them to transition to an “opt-in” model?”

The concerns that Hans expresses are reasonable.

In light of how fast technology is evolving, should laws concerning fair use be reconsidered and updated?

I asked John Rizvi, a Registered Patent Attorney (LinkedIn profile) who is board certified in Intellectual Property Law, if Internet copyright laws are outdated.

John answered:

“Yes, without a doubt.

One major bone of contention in cases like this is the fact that the law inevitably evolves far more slowly than technology does.

In the 1800s, this maybe didn’t matter so much because advances were relatively slow and so legal machinery was more or less tooled to match.

Today, however, runaway technological advances have far outstripped the ability of the law to keep up.

There are simply too many advances and too many moving parts for the law to keep up.

As it is currently constituted and administered, largely by people who are hardly experts in the areas of technology we’re discussing here, the law is poorly equipped or structured to keep pace with technology…and we must consider that this isn’t an entirely bad thing.

So, in one regard, yes, Intellectual Property law does need to evolve if it even purports, let alone hopes, to keep pace with technological advances.

The primary problem is striking a balance between keeping up with the ways various forms of tech can be used while holding back from blatant overreach or outright censorship for political gain cloaked in benevolent intentions.

The law also has to take care not to legislate against possible uses of tech so broadly as to strangle any potential benefit that may derive from them.

You could easily run afoul of the First Amendment and any number of settled cases that circumscribe how, why, and to what degree intellectual property can be used and by whom.

And attempting to envision every conceivable usage of technology years or decades before the framework exists to make it viable or even possible would be an exceedingly dangerous fool’s errand.

In situations like this, the law really cannot help but be reactive to how technology is used…not necessarily how it was intended.

That’s not likely to change anytime soon, unless we hit a massive and unanticipated tech plateau that allows the law time to catch up to current events.”

So it appears that the issue of copyright laws has many considerations to balance when it comes to how AI is trained, there is no simple answer.

OpenAI and Microsoft Sued

An interesting case that was recently filed is one in which OpenAI and Microsoft used open source code to create their CoPilot product.

The problem with using open source code is that the Creative Commons license requires attribution.

According to an article published in a scholarly journal:

“Plaintiffs allege that OpenAI and GitHub assembled and distributed a commercial product called Copilot to create generative code using publicly accessible code originally made available under various “open source”-style licenses, many of which include an attribution requirement.

As GitHub states, ‘…[t]rained on billions of lines of code, GitHub Copilot turns natural language prompts into coding suggestions across dozens of languages.’

The resulting product allegedly omitted any credit to the original creators.”

The author of that article, who is a legal expert on the subject of copyrights, wrote that many view open source Creative Commons licenses as a “free-for-all.”

Some may also consider the phrase free-for-all a fair description of the datasets comprised of Internet content are scraped and used to generate AI products like ChatGPT.

Background on LLMs and Datasets

Large language models train on multiple data sets of content. Datasets can consist of emails, books, government data, Wikipedia articles, and even datasets created of websites linked from posts on Reddit that have at least three upvotes.

Many of the datasets related to the content of the Internet have their origins in the crawl created by a non-profit organization called Common Crawl.

Their dataset, the Common Crawl dataset, is available free for download and use.

The Common Crawl dataset is the starting point for many other datasets that created from it.

For example, GPT-3 used a filtered version of Common Crawl (Language Models are Few-Shot Learners PDF).

This is how  GPT-3 researchers used the website data contained within the Common Crawl dataset:

“Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset… constituting nearly a trillion words.

This size of dataset is sufficient to train our largest models without ever updating on the same sequence twice.

However, we have found that unfiltered or lightly filtered versions of Common Crawl tend to have lower quality than more curated datasets.

Therefore, we took 3 steps to improve the average quality of our datasets:

(1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality reference corpora,

(2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy and preserve the integrity of our held-out validation set as an accurate measure of overfitting, and

(3) we also added known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.”

Google’s C4 dataset (Colossal, Cleaned Crawl Corpus), which was used to create the Text-to-Text Transfer Transformer (T5), has its roots in the Common Crawl dataset, too.

Their research paper (Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer PDF) explains:

“Before presenting the results from our large-scale empirical study, we review the necessary background topics required to understand our results, including the Transformer model architecture and the downstream tasks we evaluate on.

We also introduce our approach for treating every problem as a text-to-text task and describe our “Colossal Clean Crawled Corpus” (C4), the Common Crawl-based data set we created as a source of unlabeled text data.

We refer to our model and framework as the ‘Text-to-Text Transfer Transformer’ (T5).”

Google published an article on their AI blog that further explains how Common Crawl data (which contains content scraped from the Internet) was used to create C4.

They wrote:

“An important ingredient for transfer learning is the unlabeled dataset used for pre-training.

To accurately measure the effect of scaling up the amount of pre-training, one needs a dataset that is not only high quality and diverse, but also massive.

Existing pre-training datasets don’t meet all three of these criteria — for example, text from Wikipedia is high quality, but uniform in style and relatively small for our purposes, while the Common Crawl web scrapes are enormous and highly diverse, but fairly low quality.

To satisfy these requirements, we developed the Colossal Clean Crawled Corpus (C4), a cleaned version of Common Crawl that is two orders of magnitude larger than Wikipedia.

Our cleaning process involved deduplication, discarding incomplete sentences, and removing offensive or noisy content.

This filtering led to better results on downstream tasks, while the additional size allowed the model size to increase without overfitting during pre-training.”

Google, OpenAI, even Oracle’s Open Data are using Internet content, your content, to create datasets that are then used to create AI applications like ChatGPT.

Common Crawl Can Be Blocked

It is possible to block Common Crawl and subsequently opt-out of all the datasets that are based on Common Crawl.

But if the site has already been crawled then the website data is already in datasets. There is no way to remove your content from the Common Crawl dataset and any of the other derivative datasets like C4 and .

Using the Robots.txt protocol will only block future crawls by Common Crawl, it won’t stop researchers from using content already in the dataset.

How to Block Common Crawl From Your Data

Blocking Common Crawl is possible through the use of the Robots.txt protocol, within the above discussed limitations.

The Common Crawl bot is called, CCBot.

It is identified using the most up to date CCBot User-Agent string: CCBot/2.0

Blocking CCBot with Robots.txt is accomplished the same as with any other bot.

Here is the code for blocking CCBot with Robots.txt.

User-agent: CCBot
Disallow: /

CCBot crawls from Amazon AWS IP addresses.

CCBot also follows the nofollow Robots meta tag:

<meta name="robots" content="nofollow">

What If You’re Not Blocking Common Crawl?

Web content can be downloaded without permission, which is how browsers work, they download content.

Google or anybody else does not need permission to download and use content that is published publicly.

Website Publishers Have Limited Options

The consideration of whether it is ethical to train AI on web content doesn’t seem to be a part of any conversation about the ethics of how AI technology is developed.

It seems to be taken for granted that Internet content can be downloaded, summarized and transformed into a product called ChatGPT.

Does that seem fair? The answer is complicated.

Featured image by Shutterstock/Krakenimages.com

Google updated their Google Discover feed guidelines to emphasize the most important elements to include in the feed in order for it to be properly optimized.

Google Discover Feed

The Google Discover follow feed feature offers relevant content to Chrome Android users and represents an importance source of traffic that is matched to user interests.

The Google Discover Follow feature is a component of Google Discover, a way to capture a steady stream of traffic apart from Google News and Google Search.

Google’s Discover Follow feature works by allowing users to choose to receive updates about the latest content on a site they are interested in.

The way to do participate in Discover Follow is through an optimized RSS or Atom feed.

If the feed is properly optimized on a website, users can choose to follow a website or a specific category of a website, depending on how the publisher configures their RSS/Atom feeds.

Audiences that follow a website will see the new content populate their Discover Follow feed which in turn brings fresh waves of traffic to participating websites that are properly optimized.

According to Google:

“The Follow feature lets people follow a website and get the latest updates from that website in the Following tab within Discover in Chrome.

Currently, the Follow button is a feature that’s available to signed-in users in English in the US, New Zealand, South Africa, UK, Canada, and Australia that are using Chrome Android.”

Receiving traffic from the Discover Follow feature only happens for sites with properly optimized feeds that follow the Discover Follow feature guidelines.

Updated Guidance for Google Discover Follow Feature

Google updated their guidelines for the Discover Feed feature to emphasize the importance of the feed <title> and <link> elements, emphasizing that the feed contains these elements.

The new guidance states:

“The most important content for the Follow feature is your feed <title> element and your per item <link> elements. Make sure your feed includes these elements.”

Presumably the absence of these two elements may result in Google being unable to understand the feed and display it for users, resulting in a loss of traffic.

Site publishers who participate in the Google Discover Follow feature should verify that their RSS or Atom feeds properly display the <title> and <link> elements.

Google Discover Optimization

Publishers and SEOs are familiar with optimizing for Google Search.

But many content publishers may be unaware of how to optimize for Google Discover in order to enjoy the loads of traffic that results from properly optimizing for Google Discover and the Google Discover Follow feature.

The Follow Feed feature, a component of Google Discover, is a way to help ensure that the website obtains a steady stream of relevant traffic beyond organic search.

This is why it’s important to make sure that your RSS/Atom feeds are properly optimized.

Read Google’s announcement of the updated guidance and read the complete Follow Feature feed guidelines here.

Featured image by Shutterstock/fizkes

As of 2023, Wix provides solid options for a basic SEO setup that will cover most needs but still lacks the flexibility of more advanced and granular settings.

In this article, I go through all the nooks and crannies of Wix SEO options so you can decide if Wix is right for your needs. I’ll also share some tips on how to make your Wix website more search-friendly if you’re already a Wix user.

Wix has a bit of a bad reputation for SEO. This is because from its launch in 2006 until its first big update in 2016, it lacked many basic SEO functionalities like adding alt text and being able to change URL structures.

However, that is no longer the case. You can now do virtually every on-page SEO task using the Wix platform. It even hired expert SEOs, such as Mordy Oberstein and Crystal Carter, who have pushed for more SEO features and better communication.

Back in 2019, we ran a study comparing Wix SEO to WordPress SEO analyzing over 6.4M websites. We found that, on average, far more WordPress websites get organic traffic than Wix websites. 

However, we believe that this is due to the website owners, not the platforms themselves. On average, Wix website owners are less tech-savvy (and less educated on SEO) than WordPress users, simply because of the extra learning curve that comes with using WordPress.

Let’s also see what Googlers have to say about Wix.

Here’s a quote by John Mueller, Google’s senior search analyst, on the topic of Wix SEO:

Wix is fine for SEO. A few years back it was pretty bad in terms of SEO, but they’ve made fantastic progress, and are now a fine platform for businesses. The reputation from back then lingers on, but don’t be swayed by it.

What they’ve done in recent years is really good stuff, including making it trivial to have a really fast site (as you see in the Lighthouse scores — admittedly, speed is only a tiny part of SEO).

If Wix works for them, and they don’t need more, there’s no reason to switch.

So overall, Wix has the majority of features most website users would need to manage SEO. But there’s more to the story…

While Wix has no major SEO issues, it does have three minor issues that may stop you from wanting to use it if you’re serious about search:

1. Website builders will typically load slower than custom code – Wix inevitably has code bloat from features you will never use. This is true even if you use WordPress and install a theme builder like Elementor or Thrive Architect, so this isn’t exclusive to Wix. That said, it’s only a minor issue, and it already has great Core Web Vitals compared to other CMS types.
2. Less-than-ideal multilingual support – If you plan on publishing your blog posts in multiple languages, you may want to skip Wix. For example, you don’t have full control over the URLs for different language versions of your site. However, some of these aspects are in its feature requests and may be available soon.
3. Limited advanced SEO control – Wix lacks some advanced SEO features. For example, it’s difficult to edit the auto-generated sitemap. Additionally, Wix generates cryptic file names for images (e.g., 09a0ab7~mv2.jpg/), which is not good for ranking on Google Images. 

Ultimately, Wix’s SEO features will work for most website owners out there. 

If you are a business owner who wants to focus more time on running your business and less time on learning how to build the perfect website with the best features, Wix is an excellent choice.

To help you decide if Wix is right for you, we made this helpful table of who should and shouldn’t use Wix to build their website:

Type of website	Is Wix a great solution?	Explanation
Personal website	Yes	Wix provides all you need for small websites.
Local business	Yes	Wix provides all you need to create a quick and easy local business website and rank in local search results.
Affiliate website	Maybe	Wix can handle your needs for affiliate marketing and SEO well. But if you’re aiming to create a big, complex website, it may be worth your time to learn WordPress instead.
Content website	Maybe	Wix can handle your needs for content used to show display ads. But if you’re aiming to create a big, complex website, it may be worth your time to learn WordPress instead.
Services website	Maybe	If you offer a service such as SaaS, banking, etc., then Wix may be a good choice depending on the specific features needed. You’ll have to do your own research.
E-commerce website	Maybe	There’s no perfect out-of-the-box CMS for this. The most common choices are Shopify or WooCommerce, but Wix is a solid option for e-commerce SMBs too. It can handle even some of the more complex e-commerce SEO stuff.

I personally would never build a website on Wix over WordPress for myself. That’s because WordPress has more features and customizability. And even though it comes with a much steeper learning curve, that is something I’ve overcome. (I’ve been building WordPress websites for over a decade.)

That said, I built my dad a website for his remodeling business using Wix. I did this because it’s much easier for him to go in and edit things himself than it is with WordPress. And he’s able to rank for local keywords just fine on the Wix platform. The website is fairly new, and I will come back in a few months to update this page with the progress of his rankings.

One last thing to keep in mind is that switching your content management system (CMS) can be a massive pain. So whichever tool you choose, be ready to stick to it for a long time.

Deciding to stick with Wix? Here are five Wix-specific tips to help you make sure your website is search-optimized:

1. Complete the Wix SEO Setup Checklist

Wix has a really easy-to-use SEO Setup Checklist built in its platform. To use it, navigate to the Marketing & SEO page, then click Get Found on Google.

From there, you’ll be asked a few questions to get started, such as your business name and the top three to five keywords you want your website to rank for. If you’re not sure which keywords to target, I highly recommend reading our guide to keyword research.

Once you answer the questions, you’ll see a screen with steps you can take to optimize your website for search engines, starting with your homepage.

SEO Setup Checklist will guide you through the process of updating your pages’ meta tags, making your website mobile-friendly, and more.

Go through each of these steps, and you’ll be well on your way to a search-optimized website.

2. Set up Google Search Console and Analytics

You’ll notice one of the steps is to connect your site to Google Search Console (GSC). This is Google’s suite of tools designed for website owners like you to more easily monitor your search rankings and find issues preventing your pages from being indexed by Googlebot. 

You can set up GSC with the click of a button using the SEO Setup Checklist. If you want to learn more, check out our complete guide to Google Search Console.

Once GSC is set up, you can connect Google Analytics (GA) to your website to get more insights into where your traffic is coming from and which pages your visitors are going to.

To connect GA, navigate to the Marketing Integrations tab under Marketing & SEO. It’s the first box that appears—click Connect.

Wix will instruct you on how to create a Google Analytics Property ID and connect that ID with your Wix website. If you need more help, we also have a guide on Google Analytics 4.

Once it’s set up and your website starts getting traffic, you’ll be able to see traffic and webpage reports. This can help you identify which pages may need improvements or how many conversions you get from organic traffic.

That’s it—you’re done with step #2.

3. Create search-optimized content

If your website just has the basic homepage, as well as “about” and “contact” pages, chances are you won’t be able to rank well for much (if anything).

A crucial step in SEO is creating content that can be crawled and indexed by Googlebot. That means creating service pages if you’re a local business and possibly also creating blog content targeting relevant keywords to your industry.

Rather than making this whole article about content, I will leave you with a resource. Go check out our guide to SEO content to learn more.

4. Add internal links

Backlinks—links from another website pointing to your website—are one of the most important ranking factors in Google’s algorithm. However, they can be difficult to obtain.

Internal links from one page on your site to another on your site are almost as important as backlinks. But they are much easier to add. You just highlight some text and add the link in.

If you have pages on your website that you want to rank better, simply add more internal links to that page and you’re already on the path to higher rankings. Obviously, just adding some internal links won’t suddenly make you rank #1 for a keyword. But it’s an important—and often overlooked—step on the road to better rankings. 

To add an internal link with Wix, simply highlight the text you want to add a link to, click the “chain link” icon, then choose the page you want the link to point to.

Check out our internal linking guide to learn more about this important SEO task.

5. Schedule regular SEO audits

Once your Wix website is set up and optimized, it’s important to schedule regular SEO audits to keep tabs on your rankings and make sure nothing gets broken.

While you can do this manually, it is time consuming and easy to overlook something. 

For example, you may not realize one of your pages broke and is now a 404 page, or that a certain blog post isn’t showing up in your sitemap, or that you’re missing metadata on a certain page… the list goes on.

Instead, you can use Ahrefs Webmaster Tools to automatically run weekly or monthly audits of your website. This free tool will give you a health score from 0 to 100 on how “healthy” your website is from an SEO perspective.

You can then see specific tasks you need to do in order to fix these issues on your site. Go to the All issues report and check the issues we found while crawling your website.

You can click the error and see exactly what it means and how to fix it.

From there, you can click “View affected URLs” and go to those pages to fix the issues. Easy peasy.

Final thoughts

Overall, Wix is a perfectly capable website builder for SEO. While it isn’t as advanced and capable as more complex CMSs like WordPress, it’s plenty good for people who just want to build a website and don’t have the time for or interest in a giant learning curve.

I still use Wix for certain client sites and to build sites for friends and family who want a website where they can still make small edits themselves. It’s my favorite website builder compared to other tools like Squarespace or WordPress.com (not to be confused with WordPress.org, which I use all the time).

One more benefit to using a website builder like Wix is that it’s a complete solution and takes care of the hosting and security. In fact, John doesn’t recommend self-hosting your websites. 

That said, if you want more advanced features and to dive deeper in SEO, I suggest learning WordPress.

Ready to keep learning? Here are some other helpful guides: