Google’s Gary Illyes shares an unconventional but valid method for centralizing robots.txt rules on CDNs.

• Robots.txt files can be centralized on CDNs, not just root domains.
• Websites can redirect robots.txt from main domain to CDN.
• This unorthodox approach complies with updated standards.

Microsoft is making publicly available a new technology called GraphRAG, which enables chatbots and answer engines to connect the dots across an entire dataset, outperforming standard Retrieval-Augmented Generation (RAG) by large margins.

What’s The Difference Between RAG And GraphRAG?

RAG (Retrieval-Augmented Generation) is a technology that enables an LLM to reach into a database like a search index and use that as a basis for answering a question. It can be used to bridge a large language model and a conventional search engine index.

The benefit of RAG is that it can use authoritative and trustworthy data in order to answer questions. RAG also enables generative AI chatbots to use up to date information to answer questions about topics that the LLM wasn’t trained on. This is an approach that’s used by AI search engines like Perplexity.

The upside of RAG is related to its use of embeddings. Embeddings is a way of representing the semantic relationships between words, sentences, and documents. This representation enables the retrieval part of RAG to match a search query to text in a database (like a search index).

But the downside of using embeddings is that it limits the RAG to matching text at a granular level (as opposed to a global reach across the data).

Microsoft explains:

“Since naive RAG only considers the top-k most similar chunks of input text, it fails. Even worse, it will match the question against chunks of text that are superficially similar to that question, resulting in misleading answers.”

The innovation of GraphRAG is that it enables an LLM to answer questions based on the overall dataset.

What GraphRAG does is it creates a knowledge graph out of the indexed documents, also known as unstructured data. The obvious example of unstructured data are web pages. So when GraphRAG creates a knowledge graph, it’s creating a “structured” representation of the relationships between various “entities” (like people, places, concepts, and things) which is then more easily understood by machines.

GraphRAG creates what Microsoft calls “communities” of general themes (high level) and more granular topics (low level). An LLM then creates a summarization of each of these communities, a “hierarchical summary of the data” that is then used to answer questions. This is the breakthrough because it enables a chatbot to answer questions based more on knowledge (the summarizations) than depending on embeddings.

This is how Microsoft explains it:

“Using an LLM to summarize each of these communities creates a hierarchical summary of the data, providing an overview of a dataset without needing to know which questions to ask in advance. Each community serves as the basis of a community summary that describes its entities and their relationships.

…Community summaries help answer such global questions because the graph index of entity and relationship descriptions has already considered all input texts in its construction. Therefore, we can use a map-reduce approach for question answering that retains all relevant content from the global data context…”

Examples Of RAG Versus GraphRAG

The original GraphRAG research paper illustrated the superiority of the GraphRAG approach in being able to answer questions for which there is no exact match data in the indexed documents. The example uses a limited dataset of Russian and Ukrainian news from the month of June 2023 (translated to English).

Simple Text Matching Question

The first question that was used an example was “What is Novorossiya?” and both RAG and GraphRAG answered the question, with GraphRAG offering a more detailed response.

The short answer by the way is that “Novorossiya” translates to New Russia and is a reference to Ukrainian lands that were conquered by Russia in the 18th century.

The second example question required that the machine make connections between concepts within the indexed documents, what Microsoft calls a “query-focused summarization (QFS) task” which is different than a simple text-based retrieval task. It requires what Microsoft calls, “connecting the dots.”

The question asked of the RAG and GraphRAG systems:

“What has Novorossiya done?”

This is the RAG answer:

“The text does not provide specific information on what Novorossiya has done.”

GraphRAG answered the question of “What has Novorossiya done?” with a two paragraph answer that details the results of the Novorossiya political movement.

Here’s a short excerpt from the two paragraph answer:

“Novorossiya, a political movement in Ukraine, has been involved in a series of destructive activities, particularly targeting various entities in Ukraine [Entities (6494, 912)]. The movement has been linked to plans to destroy properties of several Ukrainian entities, including Rosen, the Odessa Canning Factory, the Odessa Regional Radio Television Transmission Center, and the National Television Company of Ukraine [Relationships (15207, 15208, 15209, 15210)]…

…The Office of the General Prosecutor in Ukraine has reported on the creation of Novorossiya, indicating the government’s awareness and potential concern over the activities of this movement…”

The above is just some of the answer which was extracted from the limited one-month dataset, which illustrates how GraphRAG is able to connect the dots across all of the documents.

GraphRAG Now Publicly Available

Microsoft announced that GraphRAG is publicly available for use by anybody.

“Today, we’re pleased to announce that GraphRAG is now available on GitHub, offering more structured information retrieval and comprehensive response generation than naive RAG approaches. The GraphRAG code repository is complemented by a solution accelerator, providing an easy-to-use API experience hosted on Azure that can be deployed code-free in a few clicks.”

Microsoft released GraphRAG in order to make the solutions based on it more publicly accessible and to encourage feedback for improvements.

Read the announcement:

GraphRAG: New tool for complex data discovery now on GitHub

Featured Image by Shutterstock/Deemerwha studio

WordPress announced over the weekend that they were pausing plugin updates and initiating a force reset on plugin author passwords in order to prevent additional website compromises due to the ongoing Supply Chain Attack on WordPress plugins.

Supply Chain Attack

Hackers have been attacking plugins directly at the source using password credentials exposed in previous data breaches (unrelated to WordPress itself). The hackers are looking for compromised credentials used by plugin authors who use the same passwords across multiple websites (including passwords exposed in a previous data breach).

WordPress Takes Action To Block Attacks

Some plugins have been compromised by the WordPress community has rallied to clamp down on further plugin compromises by instituting a forced password reset and encouraging plugin authors to use 2 factor authentication.

WordPress also temporarily blocked all new plugin updates at the source unless they received team approval in order to make sure that a plugin is not being updated with malicious backdoors. By Monday WordPress updated their post to confirm that plugin releases are no longer paused.

The WordPress announcement on the forced password reset:

“We have begun to force reset passwords for all plugin authors, as well as other users whose information was found by security researchers in data breaches. This will affect some users’ ability to interact with WordPress.org or perform commits until their password is reset.

You will receive an email from the Plugin Directory when it is time for you to reset your password. There is no need to take action before you’re notified.”

A discussion in the comments section between a WordPress community member and the author of the announcement revealed that WordPress did not directly contact plugin authors who were identified as using “recycled” passwords because there was evidence that the list of users found in the data breach list whose credentials were in fact safe (false positives). WordPress also discovered that some accounts that were assumed to be safe were in fact compromised (false negatives). That is what led to to the current action of forcing password resets.

Francisco Torres of WordPress answered:

“You’re right that specifically reaching out to those individuals mentioning that their data has been found in data breaches will make them even more sensitive, but unfortunately as I’ve already mentioned that might be inaccurate for some users and there will be others that are missing. What we’ve done since the beginning of this issue is to individually notify those users that we’re certain have been compromised.”

Read the official WordPress announcement:

Password Reset Required for Plugin Authors

Featured Image by Shutterstock/Aleutie