Researchers discovered a way to defeat the safety guardrails in GPT4 and GPT4-Turbo, unlocking the ability to generate harmful and toxic content, essentially beating a large language model with another large language model.

The researchers discovered that the use of tree-of-thought (ToT)reasoning to repeat and refine a line of attack was useful for jailbreaking another large language model.

What they found is that the ToT approach was successful against GPT4, GPT4-Turbo, and PaLM-2, using a remarkably low number of queries to obtain a jailbreak, on average less than thirty queries.

Tree Of Thoughts Reasoning

A Google research paper from around May 2022 discovered Chain of Thought Prompting.

Chain of Thought (CoT) is a prompting strategy used on a generative AI to make it follow a sequence of steps in order to solve a problem and complete a task. The CoT method is often accompanied with examples to show the LLM how the steps work in a reasoning task.

So, rather than just ask a generative AI like Midjourney or ChatGPT to do a task, the chain of thought method instructs the AI how to follow a path of reasoning that’s composed of a series of steps.

Tree of Thoughts (ToT) reasoning, sometimes referred to as Tree of Thought (singular) is essentially a variation and improvement of CoT, but they’re two different things.

Tree of Thoughts reasoning is similar to CoT. The difference is that rather than training a generative AI to follow a single path of reasoning, ToT is built on a process that allows for multiple paths so that the AI can stop and self-assess then come up with alternate steps.

Tree of Thoughts reasoning was developed in May 2023 in a research paper titled Tree of Thoughts: Deliberate Problem Solving with Large Language Models (PDF)

The research paper describes Tree of Thought:

“…we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving.

ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.

Our experiments show that ToT significantly enhances language models’ problem-solving abilities…”

Tree Of Attacks With Pruning (TAP)

This new method of jailbreaking large language models is called Tree of Attacks with Pruning, TAP. TAP uses two LLMs, one for attacking and the other for evaluating.

TAP is able to outperform other jailbreaking methods by significant margins, only requiring black-box access to the LLM.

A black box, in computing, is where one can see what goes into an algorithm and what comes out. But what happens in the middle is unknown, thus it’s said to be in a black box.

Tree of thoughts (TAP) reasoning is used against a targeted LLM like GPT-4 to repetitively try different prompting, assess the results, then if necessary change course if that attempt is not promising.

This is called a process of iteration and pruning. Each prompting attempt is analyzed for the probability of success. If the path of attack is judged to be a dead end, the LLM will “prune” that path of attack and begin another and better series of prompting attacks.

This is why it’s called a “tree” in that rather than using a linear process of reasoning which is the hallmark of chain of thought (CoT) prompting, tree of thought prompting is non-linear because the reasoning process branches off to other areas of reasoning, much like a human might do.

The attacker issues a series of prompts, the evaluator evaluates the responses to those prompts and then makes a decision as to what the next path of attack will be by making a call as to whether the current path of attack is irrelevant or not, plus it also evaluates the results to determine the likely success of prompts that have not yet been tried.

What’s remarkable about this approach is that this process reduces the number of prompts needed to jailbreak GPT-4. Additionally, a greater number of jailbreaking prompts are discovered with TAP than with any other jailbreaking method.

The researchers observe:

“In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM.

TAP utilizes an LLM to iteratively refine candidate (attack) prompts using tree-of-thoughts reasoning until one of the generated prompts jailbreaks the target.

Crucially, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks.

Using tree-of-thought reasoning allows TAP to navigate a large search space of prompts and pruning reduces the total number of queries sent to the target.

In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4 and GPT4-Turbo) for more than 80% of the prompts using only a small number of queries. This significantly improves upon the previous state-of-the-art black-box method for generating jailbreaks.”

Tree Of Thought (ToT) Outperforms Chain Of Thought (CoT) Reasoning

Another interesting conclusion reached in the research paper is that, for this particular task, ToT reasoning outperforms CoT reasoning, even when adding pruning to the CoT method, where off topic prompting is pruned and discarded.

ToT Underperforms With GPT 3.5 Turbo

The researchers discovered that ChatGPT 3.5 Turbo didn’t perform well with CoT, revealing the limitations of GPT 3.5 Turbo. Actually, GPT 3.5 performed exceedingly poorly, dropping from 84% success rate to only a 4.2% success rate.

This is their observation about why GPT 3.5 underperforms:

“We observe that the choice of the evaluator can affect the performance of TAP: changing the attacker from GPT4 to GPT3.5-Turbo reduces the success rate from 84% to 4.2%.

The reason for the reduction in success rate is that GPT3.5-Turbo incorrectly determines that the target model is jailbroken (for the provided goal) and, hence, preemptively stops the method.

As a consequence, the variant sends significantly fewer queries than the original method…”

What This Mean For You

While it’s amusing that the researchers use the ToT method to beat an LLM with another LLM, it also highlights the usefulness of ToT for generating surprising new directions in prompting in order to achieve higher levels of output.

• TL/DR Takeaways:
• Tree of Thought prompting outperformed Chain of Thought methods
• GPT 3.5 worked significantly poorly in comparison to GPT 4 in ToT
• Pruning is a useful part of a prompting strategy
• Research showed that ToT is superior to CoT in an intensive reasoning task like jailbreaking an LLM

Read the original research paper:

Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (PDF)

Featured Image by Shutterstock/THE.STUDIO

A competitive analysis (or market competitive analysis) is a process where you collect information about competitors to gain an edge over them and get more customers.

However, the problem is that “traditional” competitive analysis is overkill for most businesses — it requires impractical data and takes too long to complete (and it’s very expensive if you choose to outsource). 

A solution to that is a lean approach to the process — and that’s what this guide is about. 

In other words, we’ll focus on the most important data you need to answer the question: “Why would people choose them over you?”. No boring theory, outtakes from marketing history, or spending hours digging up nice-to-have information.

In this guide, you will find:

• A real-life competitive analysis example.
• Templates: one for input data and one for a slide deck to present your analysis to others.
• Step-by-step instructions.

Our template consists of two documents: a slide deck and a spreadsheet. 

The Slide deck is the output document. It will help you present the analysis to your boss or your teammates.

The spreadsheet is the input document. You will find tables that act as the data source for the charts from the slide deck, as well as a prompt to use in ChatGPT to help you with user review research.

We didn’t focus on aesthetics here; every marketer likes to do slide decks their own way, so feel free to edit everything you’ll find there. 

With that out of the way, let’s talk about the process. The template consists of these six tasks: 

1. Identify your direct competitors. 
2. Compare share of voice. 
3. Compare pricing and features.
4. Find strong and weak points based on reviews.
5. Compare purchasing convenience.
6. Present conclusions.

Going forward, we’ll explain why these steps matter and show how to complete them. 

Direct competitors are businesses that offer a similar solution to the same audience. 

They matter a lot more than indirect competitors (i.e. businesses with different products but targeting the same audience as you) because you’ll be compared with them often (e.g. in product reviews and rankings). Plus, your audience is more likely to gravitate towards them when considering different options. 

You probably have a few direct competitors in mind already, but here are a few ways to find others based on organic search and paid search ads. 

Our basis for the analysis was Landingi, a SaaS for building landing pages (we chose that company randomly). So in our case, we found these 3 direct competitors. 

Look at keyword overlap

Keyword overlap uncovers sites that target the same organic keywords as you. Some sites will compete with you for traffic but not for customers (e.g. G2 may share some keywords with Landingi but they’re a different business). However, in many cases, you will find direct competitors just by looking at this marketing channel. 

• Go to Ahrefs’ Site Explorer and enter your site’s address. 
• Scroll down to Organic competitors. 
• Visit the URLs to pick 3 – 5 direct competitors.

To double-check the choice of competitors, we also looked at who was bidding for search ads on Google.

See who’s advertising 

If someone is spending money to show ads for keywords related to what you do, that’s a strong indication they are a direct competitor. 

• Go to Ahrefs’ Keywords Explorer.
• Type in a few broad keywords related to your niche, like “landing page builder” or “landing page tool”. 
• Go to the Ads history report. 
• Visit the sites that have a high presence of ads in the SERPs (Search Engine Result Pages). 

Once you’re done checking both reports, write down competitors in the deck. 

You can also take screenshots of the reports and add them to your deck to show the supporting data for your argument. 

Share of voice is a measure of your reach in any given channel compared to competitors. 

A bigger share of voice (SOV) means that your competitors are more likely to reach your audience. In other words, they may be promoting more effectively than you. 

In our example, we found that Landingi’s SOV was the lowest in both of these channels. 

Organic: 

And social media:

Here’s how we got that data using Ahrefs and Brand24.

Organic share of voice 

Before we start, make sure you have a project set up in Ahrefs’ Rank Tracker. 

Now: 

• Go to Ahrefs’ Competitive Analysis and enter your and your competitors’s sites as shown below. 

• On the next screen, set the country with the most important market for your business and set the filters like this:

• Select keywords that sound most relevant to your business (even if you don’t rank for them yet) and Add them to Rank Tracker. 

• Go to Rank Tracker, open your project, and look for Competitors/Overview. This report will uncover automatically calculated Share of Voice. 

• Add the numbers in corresponding cells inside the sheet and paste the graph inside the slide deck. 

It’s normal that the numbers don’t add up to 100%. SOV is calculated by including sites that compete with you in traffic but are not your direct competitors, e.g. blogs. 

Social share of voice 

We can also measure our share of voice across social media channels using Brand24.

• Go to Brand24.
• Start a New project for your brand and each competitor. Use the competitors’ brand name as the keyword to monitor. 
• Go to the Comparison report and compare your project with competitors. 

• Take a screenshot of the SOV charts and paste them into the slide deck. Make sure the charts are set to “social media”.

Consumers often choose solutions that offer the best value for money — simple as that. And that typically comes down to two things: 

• Whether you have the features they care about. We’ll use all features available across all plans to see how likely the product is to satisfy user needs.
• How much they will need to pay. Thing is, the topic of pricing is tricky: a) when assessing affordability, people often focus on the least expensive option available and use it as a benchmark, b) businesses in the SaaS niche offer custom plans. So to make things more practical, we’ll compare the cheapest plans, but feel free to run this analysis across all pricing tiers.

After comparing our example company to competitors, we found that it goes head-to-head with Unbounce as the most feature-rich solution on the market. 

Here’s how we got that data. 

• Note down your and your competitors’ product features. One of the best places to get this information is pricing pages. Some brands even publish their own competitor comparisons — you may find them helpful too. 
• While making the list, place a “1” in the cell corresponding to the brand that offers the solution.

• Enter the price of the cheapest plan (excluding free plans). 

• Once finished, copy the chart and paste it inside the deck. 

User reviews can show incredibly valuable insight into your competitors’ strong and weak points. Here’s why this matters:

• Improving on what your competitors’ customers appreciate could help you attract similar customers and possibly win some over.
• Dissatisfaction with competitors is a huge opportunity. Some businesses are built solely to fix what other companies can’t fix. 

Here’s a sample from our analysis: 

And here’s how we collated the data using ChatGPT. Important: repeat the process for each competitor.

• Open ChatGPT and enter the prompt from the template.

• Go to G2, Capterra, or Trustpilot and find a competitor’s reviews with ratings from 2 – 4 (i.e. one rating above the lowest and one below the highest possible). Reason:

businesses sometimes solicit five-star reviews, whereas dissatisfied customers tend to leave one-star reviews in a moment of frustration. The most actionable feedback usually comes in between.

• Copy and paste the content of the reviews into ChatGPT (don’t hit enter yet). 
• Once you’re done pasting all reviews, hit enter in ChatGPT to run the analysis.

• Paste the graphs into the deck. If you want the graphs to look different, don’t hesitate to ask the AI. 

There’s a faster alternative, but it’s a bit more advanced. 

Instead of copy-pasting, you can use a scraping tool like this one to get all reviews at once. The downside here is that not all review sources will a have scraping tool available. 

Lastly, we’ll see how easy it is to actually buy your products, and compare the experience to your competitors. 

This is a chance to simplify your checkout process, and even learn from any good habits your competitors have adopted.

For example, we found that our sample company had probably nothing to worry about in this area — they ticked almost all of the boxes. 

Here’s how to complete this step:

• Place a “1” if you or any of your competitors offer convenience features listed in the template. 
• Once done, copy the chart and paste it into the deck.

This is the part of the presentation where you sum up all of your findings and suggest a course of action. 

Here are two examples: 

• Landingi had the lowest SOV in the niche, and that is never good. So the conclusion might be to go a level deeper and do an SEO competitive analysis, and to increase social media presence by creating more share-worthy content like industry surveys, design/CRO tips, or in-house data studies.
• Although the brand had a very high purchasing convenience score, during the analysis we found that there was a $850 gap between the monthly full plan and the previous tier. The conclusion here might be to offer a custom plan (like competitors do) to fill that gap. 

We encourage you to take your time here and think about what would make the most sense for your business. 

Final thoughts 

A competitive analysis is one of the most fruitful exercises in marketing. It can show you areas for improvement, give ideas for new features, and help you discover gaps in your strategy. It wouldn’t be an exaggeration to say that it’s fundamental to running a successful business. 

Just don’t forget to balance “spying” on your competitors with innovation. After all, you probably don’t want to become an exact copy of someone else’s brand. 

In other words, use competitive analysis to keep up with your competitors, but don’t let that erase what’s unique about your brand or make you forget your big vision. 

Got comments or questions? Ping me on X. 

Security researchers at Wordfence detailed a critical security flaw in the MW WP Form plugin, affecting versions 5.0.1 and earlier. The vulnerability allows unauthenticated threat actors to exploit the plugin by uploading arbitrary files, including potentially malicious PHP backdoors, with the ability to execute these files on the server.

MW WP Form Plugin

The MW WP Form plugin helps to simplify form creation on WordPress websites using a shortcode builder.

It makes it easy for users to create and customize forms with various fields and options.

The plugin has many features, including one that allows file uploads using the [mwform_file name=”file”] shortcode for the purpose of data collection. It is this specific feature that is exploitable in this vulnerability.

Unauthenticated Arbitrary File Upload Vulnerability

An Unauthenticated Arbitrary File Upload Vulnerability is a security issue that allows hackers to upload potentially harmful files to a website. Unauthenticated means that the attacker does not need to be registered with the website or need any kind of permission level that comes with a user permission level.

These kinds of vulnerabilities can lead to remote code execution, where the uploaded files are executed on the server, with the potential to allow the attackers to exploit the website and site visitors.

The Wordfence advisory noted that the plugin has a check for unexpected filetypes but that it doesn’t function as it should.

According to the security researchers:

“Unfortunately, although the file type check function works perfectly and returns false for dangerous file types, it throws a runtime exception in the try block if a disallowed file type is uploaded, which will be caught and handled by the catch block.

…even if the dangerous file type is checked and detected, it is only logged, while the function continues to run and the file is uploaded.

This means that attackers could upload arbitrary PHP files and then access those files to trigger their execution on the server, achieving remote code execution.”

There Are Conditions For A Successful Attack

The severity of this threat depends on the requirement that the “Saving inquiry data in database” option in the form settings is required to be enabled in order for this security gap to be exploited.

The security advisory notes that the vulnerability is rated critical with a score of 9.8 out of 10.

Actions To Take

Wordfence strongly advises users of the MW WP Form plugin to update their versions of the plugin.

The vulnerability is patched in the lutes version of the plugin, version 5.0.2.

The severity of the threat is particularly critical for users who have enabled the “Saving inquiry data in database” option in the form settings and that is compounded by the fact that no permission levels are needed to execute this attack.

Read the Wordfence advisory:

Update ASAP! Critical Unauthenticated Arbitrary File Upload in MW WP Form Allows Malicious Code Execution

Featured Image by Shutterstock/Alexander_P