Connect with us

SEO

How To Automate Ecommerce Category Page Creation With Python

Published

on

How To Automate Ecommerce Category Page Creation With Python

Clustering product inventory and automatically aligning SKUs to search demand is a great way to find opportunities to create new ecommerce categories.

Niche category pages are a proven way for ecommerce sites to align with organic search demand while simultaneously assisting users in purchasing.

If a site stocks a range of products and there is search demand, creating a dedicated landing page is an easy way to align with the demand.

But how can SEO professionals find this opportunity?

Sure, you can eyeball it, but you’ll usually leave a lot of opportunity on the table.

This problem motivated me to script something in Python, which I’m sharing today in a simple to use Streamlit application. (No coding experience required!)

The app linked above created the following output automatically using nothing more than two crawl exports!

Advertisement
Screenshot from Microsoft Excel, May 2022A csv file export showing new subcategories generated automatically using Python

Notice how the suggested categories are automatically tied back to the existing parent category?

A csv export showing that the new subcategories have been tied back to their parent category.Screenshot from Microsoft Excel, May 2022A csv export showing that the new subcategories have been tied back to their parent category.

The app even shows how many products are available to populate the category.

the number of products available to populate the new subcategories have been highlighted.Screenshot from Microsoft Excel, May 2022the number of products available to populate the new subcategories have been highlighted.

Benefits And Uses

  • Improve relevancy to high-demand, competitive queries by creating new landing pages.
  • Increase the chance of relevant site links displaying underneath the parent category.
  • Reduce CPCs to the landing page through increased relevancy.
  • Potential to inform merchandising decisions. (If there is high search demand vs. low product count – there is a potential to widen the range.0
    A mock up image displaying the new categories as sitelinks within the Google search engine.Mock-up Screenshot from Google Chrome, May 2022A mock up image displaying the new categories as sitelinks within the Google search engine.

Creating the suggested subcategories for the parent sofa category would align the site to an additional 3,500 searches per month with relatively little effort.

Features

  • Create subcategory suggestions automatically.
  • Tie subcategories back to the parent category (cuts out a lot of guesswork!).
  • Match to a minimum of X products before recommending a category.
  • Check similarity to an existing category (X % fuzzy match) before recommending a new category.
  • Set minimum search volume/CPC cut-off for category suggestions.
  • Supports search volume and CPC data from multiple countries.

Getting Started/Prepping The Files

To use this app you need two things.

At a high level, the goal is to crawl the target website with two custom extractions.

The internal_html.csv report is exported, along with an inlinks.csv export.

These exports are then uploaded to the Streamlit app, where the opportunities are processed.

Crawl And Extraction Setup

When crawling the site, you’ll need to set two extractions in Screaming Frog – one to uniquely identify product pages and another to uniquely identify category pages.

The Streamlit app understands the difference between the two types of pages when making recommendations for new pages.

The trick is to find a unique element for each page type.

(For a product page, this is usually the price or the returns policy, and for a category page, it’s usually a filter sort element.)

Extracting The Unique Page Elements

Screaming Frog allows for custom extractions of content or code from a web page when crawled.

Advertisement

This section may be daunting if you are unfamiliar with custom extractions, but it’s essential for getting the correct data into the Streamlit app.

The goal is to end up with something looking like the below image.

(A unique extraction for product and category pages with no overlap.)

A screenshot from screaming frog showing two custom extractions to unique identify product and category pagesScreenshot from Screaming Frog SEO Spider, May 2022A screenshot from screaming frog showing two custom extractions to unique identify product and category pages

The steps below walk you through manually extracting the price element for a product page.

Then, repeat for a category page afterward.

If you’re stuck or would like to read more about the web scraper tool in Screaming Frog, the official documentation is worth your time.

Manually Extracting Page Elements

Let’s start by extracting a unique element only found on a product page (usually the price).

Highlight the price element on the page with the mouse, right-click and choose Inspect.

Advertisement
A screenshot demonstrating how to use the inspect element feature of Google Chrome to extract a CSS Selector.Screenshot from Google Chrome, May 2022A screenshot demonstrating how to use the inspect element feature of Google Chrome to extract a CSS Selector.

This will open up the elements window with the correct HTML line already selected.

Right-click the pre-selected line and choose Copy > Copy selector. That’s it!

A screenshot showing how to cop the CSS selector for use in Screaming FrogScreenshot from Google Chrome, May 2022A screenshot showing how to cop the CSS selector for use in Screaming Frog

Open Screaming Frog and paste the copied selector into the custom extraction section. (Configuration > Custom > Extraction).

A screenshot from Screaming Frog showing how to use a custom extractorScreenshot from Screaming Frog SEO Spider, May 2022A screenshot from Screaming Frog showing how to use a custom extractor

Name the extractor as “product,” select the CSSPath drop down and choose Extract Text.

Repeat the process to extract a unique element from a category page. It should look like this once completed for both product and category pages.

A screenshot from Screaming Frog showing the custom extractor correctly populatedScreenshot from Screaming Frog SEO Spider, May 2022A screenshot from Screaming Frog showing the custom extractor correctly populated

Finally, start the crawl.

The crawl should look like this when viewing the Custom Extraction tab.

A screenshot showing unique extractions for product and category pagesScreenshot from Screaming Frog SEO Spider, May 2022A screenshot showing unique extractions for product and category pages

Notice how the extractions are unique to each page type? Perfect.

The script uses the extractor to identify the page type.

Internally the app will convert the extractor to tags.

(I mention this to stress that the extractors can be anything as long as they uniquely identify both page types.)

A screenshot of how the app / script interprets the custom extractions to tag each pageScreenshot from Microsoft Excel, May 2022A screenshot of how the app / script interprets the custom extractions to tag each page

Exporting The Files

Once the crawl has been completed, the last step is to export two types of CSV files.

  • internal_html.csv.
  • inlinks to product pages.

Go to the Custom Extraction tab in Screaming Frog and highlight all URLs that have an extraction for products.

(You will need to sort the column to group it.)

A screenshot showing how to select the inlinks report from Screaming Frog ready for exportingScreenshot from Screaming Frog SEO Spider, May 2022A screenshot showing how to select the inlinks report from Screaming Frog ready for exporting

Lastly, right-click the product URLs, select Export, and then Inlinks.

A screenshot showing how to right click in Screaming Frog to export the inlinks report.Screenshot from Screaming Frog SEO Spider, May 2022A screenshot showing how to right click in Screaming Frog to export the inlinks report.

You should now have a file called inlinks.csv.

Finally, we just need to export the internal_html.csv file.

Advertisement

Click the Internal tab, select HTML from the dropdown menu below and click on the adjacent Export button.

Finally, choose the option to save the file as a .csv

A screenshot in Screaming Frog showing how to export the internal_html.csv reportScreenshot from Screaming Frog SEO Spider, May 2022A screenshot in Screaming Frog showing how to export the internal_html.csv report

Congratulations! You are now ready to use the Streamlit app!

Using The Streamlit App

Using the Streamlit app is relatively simple.

The various options are set to reasonable defaults, but feel free to adjust the cut-offs to better suit your needs.

I would highly recommend using a Keywords Everywhere API key (although it is not strictly necessary as this can be looked up manually later with an existing tool if preferred.

(The script pre-qualifies opportunity by checking for search volume. If the key is missing, the final output will contain more irrelevant words.)

If you want to use a key, this is the section on the left to pay attention to.

A screenshot showing the area to paste in the option Keywords Everywhere API keyScreenshot from Streamlit.io, May 2022A screenshot showing the area to paste in the option Keywords Everywhere API key

Once you have entered the API key and adjusted the cut-offs to your links, upload the inlinks.csv crawl.

A screenshot showing how to upload the inlinks.csv report Screenshot from Streamlit.io, May 2022A screenshot showing how to upload the inlinks.csv report

Once complete, a new prompt will appear adjacent to it, prompting you to upload the internal_html.csv crawl file.

A screenshot showing how to upload the internal_html.csv reportScreenshot from Streamlit.io, May 2022A screenshot showing how to upload the internal_html.csv report

Finally, a new box will appear asking you to select the product and column names from the uploaded crawl file to be mapped correctly.

A screenshot demonstrating how to correct map the column names from the crawlScreenshot from Streamlit.io, May 2022A screenshot demonstrating how to correct map the column names from the crawl

Click submit and the script will run. Once complete, you will see the following screen and can download a handy .csv export.

A screenshot showing the Streamlit app after it has successfully run a reportScreenshot from Streamlit.io, May 2022A screenshot showing the Streamlit app after it has successfully run a report

How The Script Works

Before we dive into the script’s output, it will help to explain what’s going on under the hood at a high level.

At a glance:

Advertisement
  • Generate thousands of keywords by generating n-grams from product page H1 headings.
  • Qualify keywords by checking whether the word is in an exact or fuzzy match in a product heading.
  • Further qualify keywords by checking for search volume using the Keywords Everywhere API (optional but recommended).
  • Check whether an existing category already exists using a fuzzy match (can find words out of order, different tenses, etc.).
  • Uses the inlinks report to assign suggestions to a parent category automatically.

N-gram Generation

The script creates hundreds of thousands of n-grams from the product page H1s, most of which are completely nonsensical.

In my example for this article, n-grams generated 48,307 words – so this will need to be filtered!

An example of the script generating thousands of nonsensical n-gram combinations.Screenshot from Microsoft Excel, May 2022An example of the script generating thousands of nonsensical n-gram combinations.

The first step in the filtering process is to check whether the keywords generated via n-grams are found at least X times within the product name column.

(This can be in an exact or fuzzy match.)

Anything not found is immediately discarded, which usually removes around 90% of the generated keywords.

The second filtering stage is to check whether the remaining keywords have search demand.

Any keywords without search demand are then discarded too.

(This is why I recommend using the Keywords Everywhere API when running the script, which results in a more refined output.)

It’s worth noting you can do this manually afterward by searching Semrush/Ahrefs etc., discarding any keywords without search volume, and running a VLOOKUP in Microsoft Excel.

Cheaper if you have an existing subscription.

Advertisement

Recommendations Tied To Specific Landing Pages

Once the keyword list has been filtered the script uses the inlinks report to tie the suggested subcategory back to the landing page.

Earlier versions did not do this, but I realized that leveraging the inlinks.csv report meant it was possible.

It really helps understand the context of the suggestion at a glance during QA.

This is the reason the script requires two exports to work correctly.

Limitations

  • Not checking search volumes will result in more results for QA. (Even if you don’t use the Keywords Everywhere API, I recommend shortlisting by filtering out 0 search volume afterward.)
  • Some irrelevant keywords will have search volume and appear in the final report, even if keyword volume has been checked.
  • Words will typically appear in the singular sense for the final output (because products are singular and categories are pluralized if they sell more than a single product). It’s easy enough to add an “s” to the end of the suggestion though.

User Configurable Variables

I’ve selected what I consider to be sensible default options.

But here is a run down if you’d like to tweak and experiment.

  • Minimum products to match to (exact match) – The minimum number of products that must exist before suggesting the new category in an exact match.
  • Minimum products to match to (fuzzy match) – The minimum number of products that must exist before suggesting the new category in a fuzzy match, (words can be found in any order).
  • Minimum similarity to an existing category – This checks whether a category already exists in a fuzzy match before making the recommendation. The closer to 100 = stricter matching.
  • Minimum CPC in $ – The minimum dollar amount of the suggested category keyword. (Requires the Keywords Everywhere API.)
  • Minimum search volume – The minimum search volume of the suggested category keyword. (Requires Keywords Everywhere API.)
  • Keywords Everywhere API key – Optional, but recommended. Used to pull in CPC/search volume data. (Useful for shortlisting categories.)
  • Set the country to pull search data from – Country-specific search data is available. (Default is the USA.)
  • Set the currency for CPC data – Country-specific CPC data is available. (Default USD.)
  • Keep the longest word suggestion – With similar word suggestions, this option will keep the longest match.
  • Enable fuzzy product matching – This will search for product names in a fuzzy match. (Words can be found out of order, recommended – but slow and CPU intensive.)

Conclusion

With a small amount of preparation, it is possible to tap into a large amount of organic opportunity while improving the user experience.

Although this script was created with an ecommerce focus, according to feedback, it works well for other site types such as job listing sites.

So even if your site isn’t an ecommerce site, it’s still worth a try.

Python enthusiast?

Advertisement

I released the source code for a non-Streamlit version here.

More resources:


Featured Image: patpitchaya/Shutterstock

!function(f,b,e,v,n,t,s)
{if(f.fbq)return;n=f.fbq=function(){n.callMethod?
n.callMethod.apply(n,arguments):n.queue.push(arguments)};
if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=’2.0′;
n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];
s.parentNode.insertBefore(t,s)}(window,document,’script’,
‘https://connect.facebook.net/en_US/fbevents.js’);

if( typeof sopp !== “undefined” && sopp === ‘yes’ ){
fbq(‘dataProcessingOptions’, [‘LDU’], 1, 1000);
}else{
fbq(‘dataProcessingOptions’, []);
}

fbq(‘init’, ‘1321385257908563’);

fbq(‘track’, ‘PageView’);

fbq(‘trackSingle’, ‘1321385257908563’, ‘ViewContent’, {
content_name: ‘python-ecommerce-category-pages’,
content_category: ‘ecommerce technical-seo’
});

Advertisement

Source link

SEO

8 Pillar Page Examples to Get Inspired By

Published

on

8 Pillar Page Examples to Get Inspired By

Pillar pages are high-level introductions to a topic. They then link to other pages, which are usually more detailed guides about parts of the main topic.

Altogether, they form a content hub.

Example of a content hub

But not all pillar pages look the same. 

In this guide, we’ll look at eight examples of pillar pages to get your creative juices flowing.

Excerpt of beginner's guide to SEO by Ahrefs

Key stats

Estimated organic traffic: 1,200
Backlinks: 6,900
Referring domains: 899

Overview of Ahrefs' beginner's guide to SEO in Ahrefs' Site Explorer

This is our very own pillar page, covering the broad topic of search engine optimization (SEO)

Why I like it

Besides the fact that I’m biased, I like the custom design we created for this page, which makes it different from the articles on our blog. 

Even though the design is custom, our pillar page is still a pretty classic “hub and spoke” style pillar page. We’ve broken the topic down neatly into six different chapters and internally linked to guides we’ve created about them. There are also custom animations when you hover over each chapter:

Examples of chapters in the SEO guide

We’ve also added a glossary section that comes with a custom illustration of the SERPs. We have explanations of what each element means, with internal links to more detailed content:

Custom illustration of the SERP

Finally, it links to another “pillar page”: our SEO glossary

Takeaway

Consider creating a custom design for your pillar page so that it stands out. 

Excerpt of Doctor Diet's ketogenic diet guide

Key stats

Estimated organic traffic: 92,200
Backlinks: 21,600
Referring domains: 1,700

Overview of Diet Doctor's ketogenic diet guide in Ahrefs' Site Explorer

Diet Doctor is a health company focusing on low-carb diets. Its pillar page is a comprehensive guide on the keto diet. 

Why I like it

On the surface, it doesn’t exactly look like a pillar page; it looks like every other post on the Diet Doctor site. But that’s perfectly fine. It’s simply a different approach—you don’t have to call out the fact that it’s a pillar page. 

Advertisement

Diet Doctor’s guide is split into 10 different sections with links to its own resources. The links bring you to different types of content (not just blog posts but videos too).

Video course about keto diet for beginners

Unlike the classic pillar page, Diet Doctor’s guide goes into enough detail for anyone who is casually researching the keto diet. But it also links to further resources for anyone who’s interested in doing additional research.

Takeaway

Pillar pages need not always just be text and links. Make it multimedia: You can add videos and images and even link to your own multimedia resources (e.g., a video course).

Excerpt of Wine Folly's beginner's guide to wine

Key stats

Estimated organic traffic: 5,600
Backlinks: 2,800
Referring domains: 247

Overview of Wine Folly's beginner's guide to wine in Ahrefs' Site Explorer

Wine Folly is a content site devoted to wine knowledge and appreciation. Its pillar page, as expected, is about wine. 

Why I like it

Wine Folly’s pillar page is a classic example of a “hub and spoke” style pillar page—split into multiple sections, with some supporting text, and then internal links to other resources that support each subsection. 

Supporting text and links to other resources

This page doesn’t just serve as a pillar page for ranking purposes, though. Given that it ranks well and receives quite a significant amount of search traffic, the page also has a call to action (CTA) to Wine Folly’s book:

Short description of book; below that, CTA encouraging site visitor to purchase it

Takeaway

While most websites design pillar pages for ranking, you can also use them for other purposes: capture email addresses, sell a book, pitch your product, etc. 

Excerpt of A-Z directory of yoga poses

Key stats

Estimated organic traffic: 11,100
Backlinks: 3,400
Referring domains: 457

Overview of Yoga Journal's A-Z directory of yoga poses in Ahrefs' Site Explorer

Yoga Journal is an online and offline magazine. Its pillar page is an A-Z directory of yoga poses.

Why I like it

Yoga Journal’s pillar page is straightforward and simple. List down all possible yoga poses (in both their English and Sanskrit names) in a table form and link to them. 

List of yoga poses in table form

Since it’s listed in alphabetical order, it’s useful for anyone who knows the name of a particular pose and is interested in learning more. 

What I also like is that Yoga Journal has added an extra column on the type of pose each yoga pose belongs to. If we click on any of the pose types, we’re directed to a category page where you can find similar kinds of poses: 

Examples of standing yoga poses (in grid format)

Takeaway

The A-Z format can be a good format for your pillar page if the broad topic you’re targeting fits the style (e.g., dance moves, freestyle football tricks, etc.).

Excerpt of Atlassian's guide to agile development

Key stats

Estimated organic traffic: 115,200
Backlinks: 3,200
Referring domains: 860

Overview of Atlassian's guide to agile development in Ahrefs' Site Explorer

Atlassian is a software company. You’ve probably heard of its products: Jira, Confluence, Trello, etc. Its pillar page is on agile development.

Why I like it

Atlassian’s pillar page is split into different topics related to agile development. It then has internal links to each topic—both as a sticky table of contents and card-style widgets after the introduction: 

Sticky table of contents
Card-style widgets

I also like the “Up next” feature at the bottom of the pillar page, which makes it seem like an online book rather than a page. 

Example of "Up next" feature

Takeaway

Consider adding a table of contents to your pillar page. 

Excerpt of Muscle and Strength's workout routines database

Key stats

Estimated organic traffic: 114,400
Backlinks: 2,900
Referring domains: 592

Overview of Muscle and Strength's workout routines database in Ahrefs' Site Explorer

Muscle and Strength’s pillar page is a massive database linking to various categories of workouts. 

Why I like it

Calling it a pillar page seems to be an understatement. Muscle and Strength’s free workouts page appears to be more like a website. 

When you open the page, you’ll see that it’s neatly split into multiple categories, such as “workouts for men,” “workouts for women,” “biceps,” “abs,” etc. 

Advertisement
Workout categories (in grid format)

Clicking through to any of them leads us to a category page containing all sorts of workouts:

Types of workouts for men (in grid format)

Compared to the other pillar pages on this list, where they’re linking to other subpages, Muscle and Strength’s pillar page links to other category pages, which then link to their subpages, i.e., its massive archive of free workouts.

Takeaway

Content databases, such as the one above, are a huge undertaking for a pillar page but can be worth it if the broad topic you’re targeting fits a format like this. Ideally, the topic should be about something where the content for it is ever-growing (e.g., workout plans, recipes, email templates, etc.).

Excerpt of Tofugu's guide to learning Japanese

Key stats

Estimated organic traffic: 39,100
Backlinks: 1,100
Referring domains: 308

Overview of Tofugu's guide to learning Japanese in Ahrefs' Site Explorer

Tofugu is a site about learning Japanese. And its pillar page is about, well, learning Japanese.

Why I like it

This is an incredible (and yes, ridiculously good) guide to learning Japanese from scratch. It covers every stage you’ll go through as a complete beginner—from knowing no Japanese to having intermediate proficiency in the language. 

Unlike other pillar pages where information is usually scarce and simply links out to further resources, this page holds nothing back. Under each section, there is great detail about what that section is, why it’s important, how it works, and even an estimated time of how long that stage takes to complete. 

Another interesting aspect is how Tofugu has structured its internal links as active CTAs. Rather than “Learn more” or “Read more,” it’s all about encouraging users to do a task and completing that stage. 

CTA encouraging user to head to the next task of learning to read hiragana

Takeaway

Two takeaways here:

  • Pillar pages can be ridiculously comprehensive. It depends on the topic you’re targeting and how competitive it is.
  • CTAs can be more exciting than merely just “Read more.”
Excerpt of Zapier's guide to working remotely

Key stats

Estimated organic traffic: 890
Backlinks: 4,100
Referring domains: 1,100

Overview of Zapier's guide to working remotely in Ahrefs' Site Explorer

Zapier allows users to connect multiple software products together via “zaps.” It’s a 100% remote company, and its pillar page is about remote work. 

Why I like it

Zapier’s pillar page is basically like Wine Folly’s pillar page. Break a topic into subsections, add a couple of links of text, and then add internal links to further resources. 

In the examples above, we’ve seen all sorts of execution for pillar pages. There are those with custom designs and others that are crazily comprehensive.

But sometimes, all a pillar page needs is a simple design with links. 

Takeaway

If you already have a bunch of existing content on your website, you can create a simple pillar page like this to organize your content for your readers. 

Advertisement

Keep learning

Inspired by these examples and want to create your own pillar page? Learn how to successfully do so with these two guides:

Any questions or comments? Let me know on Twitter.  



Source link

Continue Reading

DON'T MISS ANY IMPORTANT NEWS!
Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address

Trending

en_USEnglish