Microsoft 365 Copilot - Lessons Learned

By Denis Molodtsov

Published in Microsoft 365

December 04, 2024

4 min read

Disclaimer

Does Copilot see document drafts?

Test: publish a major version (2.0)

Published one more major version. Result: Copilot sees it, no problem.

Copilot can't know which sections in the document have changed.

Copilot sees document contents in plain text, without formatting, colors, etc.

How does the copilot know if there are any sections in a document and where they are beginning and ending?

How does Copilot see the content?

How many search requests can Copilot perform to provide a single answer?

Does Copilot use Graph API?

Peek into the properties that Copilot sees:

Conclusion

Disclaimer

This article is based on my personal experience with Microsoft 365 Copilot and SharePoint. It was written in April 2024 and some of the information may be outdated. The article is not an official Microsoft documentation and should not be considered as such. It is intended for educational purposes only.

Does Copilot see document drafts?

No, Copilot cannot provide any answers based on the draft version. Draft document versions are entirely ignored. Therefore, you have to make sure that the documents in question are checked and published.

Test: publish a major version (2.0)

After I promoted the draft version of 1.3 to 2.0, Copilot immediately saw the latest version of the document. It seems that there is no long-lasting caching involved.

Published version is immedeately accessible to Copilot

Published one more major version. Result: Copilot sees it, no problem.

I will then repeat the test and modify it twice using two test accounts. Yet again, Copilot saw the latest version immediately

Copilot can’t know which sections in the document have changed.

Copilot does not work with Word documents using the Word object model. Instead, all document content is presented as plain text, with no structure, formatting, or colors whatsoever. For example, Tables are presented as plain text that is not well formatted. As a result, Copilot can only guess where sections should go. It guesses fairly well, but if you have a multi-level section structure, then it won’t be able to easily tell the section hierarchy.

Copilot sees document contents in plain text, without formatting, colors, etc.

How does the copilot know if there are any sections in a document and where they are beginning and ending?

Copilot determines the sections in a document based on the context and content of the text. For example, if there is a change in topic or a heading, it usually indicates a new section. In the snippet provided, the phrases “April 2024 Updates” and “May 2024 Updates” suggest the start of new sections because they introduce new topics that are different from the previous text. Similarly, “Executive Summary” typically denotes a section that provides an overview of the document’s content. These cues help me identify different sections within a document, even in plain text format.

Copilot sees documents as a wall of unformatted text.

How does Copilot see the content?

Answer: Via Search results in Snippets.

Copilot has access to the tiny document’s snippets, allowing him to see certain parts of the content. However, it does not have visibility of the entire document, which means it may not be able to see all the sections, especially if they are not included in the snippets provided by the search results.

Each snippet returned by the search service is roughly is a third of a page. It means Copilot has a “tunnel vision”.

If additional sections are not included in the snippets, Copilot would not be able to identify them. To view the full document and all its sections, you must access it directly from your enterprise system (i.e. SharePoint).

How many search requests can Copilot perform to provide a single answer?

Copilot can request multiple snippets from the search service, but they will be provided as separate results (Snippets). For example, if you want to see the “April 2024” section and the “Conclusion” section from the “Acceptable Use Policy.docx” document, I would need to perform two separate searches to retrieve each snippet. Each section would be returned as an individual snippet based on the search results. Would you like me to proceed with this request?

Does Copilot use Graph API?

I could be wrong, but based on my tests, it does not seem that Copilot uses Graph API. Instead, it likely uses a proprietary enterprise-level service that allows it to search across various data types including files, emails, and events within an organization’s Microsoft 365 environment. or need further assistance, feel free to ask.

Architecture of Copilot. I think, there is an additional service used exclusively by Copilot and is not available to anyone else

Peek into the properties that Copilot sees:

Summary

Copilot can identify sections within a document based on context and content, but it cannot determine what changes have been made to the document. It can return metadata about a document in JSON format, including properties such as type, title, author, and last modified time. Copilot disregards draft versions of documents and does not cache information.

Type: Indicates whether the item is a file, email, event, etc.
Title: The title of the document or subject of the email.
Author: The person who created or last modified the document.
LastModifiedTime: The last time the document was edited.
FileType: The format of the file, such as docx, pptx, etc.
FileName: The name of the file.
Snippet: A brief excerpt from the document content.

We can ask Copilot to return data using JSON format:

Show me all metadata about this document. list as many properties as you see. don’t modify or format them. if it’s in JSON - give me original JSON

Results

json

{
  "Enterprise": [
    {
      "Index": 1,
      "Type": "File",
      "Title": "[Acceptable Use Policy](https://contosoca.sharepoint.com/sites/it-sandbox/CopilotTestDocuments/Acceptable%20Use%20Policy.docx?web=1&EntityRepresentationId=cf44793a-e183-4c54-9ba6-59adc32623f7)",
      "Author": "[Denis Molodtsov](https://www.office.com/search?auth=2&q=Denis+Molodtsov&EntityRepresentationId=f66a5778-9e6a-4f86-9590-87c2c2a678b0);[contosormsautomate](https://www.office.com/search?auth=2&q=contosormsautomate&EntityRepresentationId=0c0349c8-3620-49c3-b15c-af3464b15415);",
      "LastModifiedTime": "about 10 minutes ago",
      "FileType": "docx",
      "FileName": "[Acceptable Use Policy.docx](https://contosoca.sharepoint.com/sites/it-sandbox/CopilotTestDocuments/Acceptable%20Use%20Policy.docx?web=1&EntityRepresentationId=8f530e1e-ddbf-4ac3-8e55-ddd4e15711ce)",
      "Snippet": "Acceptable Use Policy\r\n Acceptable Use Policy\r\n Tailspin assets, systems, and information are Tailspin property and are intended to be primarily used in the support of Tailspin business.  To protect these assets, certain security behaviors and restrictions are imposed, while leaving employees latitude to exercise judgment, in some areas, regarding their actions. Employees are permitted to make limited and reasonable personal use of Tailspin systems, where such use is not disruptive and is consistent with these guidelines.   April 2024 Updates\r\n According to the new law, Apple devices are prohibited from accessing the Tailspin infrastructure. Policy Principles\r\n Tailspin information, assets, and systems are Tailspin property and are provided to support Tailspin business. Tailspin exercises responsible care and diligence to prevent the abuse or theft of Tailspin assets and information, and of client or partner information that has been entrusted to it. Tailspin operations comply with all relevant laws, regulations, contracts, and policies."
    },
    {
      "Index": 2,
      "Type": "File",
      "Title": "[Acceptable use policy 20221103_072201_8771](https://contosoca.sharepoint.com/sites/it-sandbox/LabTestLibrary/Acceptable%20use%20policy%2020221103_072201_8771.docx?web=1&EntityRepresentationId=70c7892e-c820-4e43-b227-939203fb5179)",
      "Author": "[Denis Molodtsov](https://www.office.com/search?auth=2&q=Denis+Molodtsov&EntityRepresentationId=f66a5778-9e6a-4f86-9590-87c2c2a678b0)",
      "LastModifiedTime": "about 15 minutes ago",
      "FileType": "docx",
      "FileName": "[Acceptable use policy 20221103_072201_8771.docx](https://contosoca.sharepoint.com/sites/it-sandbox/LabTestLibrary/Acceptable%20use%20policy%2020221103_072201_8771.docx?web=1&EntityRepresentationId=ddd33969-dea6-49b3-8788-bce31c53b521)",
      "Snippet": "Acceptable use policy\r\n To ensure your employees (and visitors) don’t misuse company computers or the internet, you need an up-to-date acceptable use policy (AUP). It should outline how people can use a website, network, or internet service. For example, it might prohibit uploading or downloading certain types of files, or harassing others via company email, or leaking insider information. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Aenean nec lorem. In porttitor. Donec laoreet nonummy augue. Suspendisse dui purus, scelerisque at, vulputate vitae, pretium mattis, nunc. Mauris eget neque at sem venenatis eleifend. Ut nonummy. Fusce aliquet pede non pede."
    },
    {
      "Index": 3,
      "Type": "File",
      "Title": "[Acceptable Use Policy](https://contosoca.sharepoint.com/sites/it-sandbox/CopilotTestDocuments/Acceptable%20Use%20Policy.docx?web=1&EntityRepresentationId=cf44793a-e183-4c54-9ba6-59adc32623f7)",
      "Author": "[Denis Molodtsov](https://www.office.com/search?auth=2&q=Denis+Molodtsov&EntityRepresentationId=f66a5778-9e6a-4f86-9590-87c2c2a678b0)",
      "LastModifiedTime": "September 18th, 2022",
      "FileType": "docx",
      "FileName": "[Acceptable Use Policy.docx](https://contosoca.sharepoint.com/sites/it-sandbox/CopilotTestDocuments/Acceptable%20Use%20Policy.docx?web=1&EntityRepresentationId=8f530e1e-ddbf-4ac3-8e55-ddd4e15711ce)",
      "Snippet": "Acceptable use policy\r\n To ensure your employees (and visitors) don’t misuse company computers or the internet, you need an up-to-date acceptable use policy (AUP). It should outline how people can use a website, network, or internet service. For example, it might prohibit uploading or downloading certain types of files, or harassing others via company email, or leaking insider information. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Aenean nec lorem. In porttitor. Donec laoreet nonummy augue. Suspendisse dui purus, scelerisque at, vulputate vitae, pretium mattis, nunc. Mauris eget neque at sem venenatis eleifend. Ut nonummy. Fusce aliquet pede non pede."
    },
    // ... other entries ...
  ],
  "SearchMetadata": {
    "Total Meetings": ""
  }
}

Conclusion

Microsoft Copilot will disregard draft versions of documents regardless of whether the current user has permission to see these drafts.
As soon as the document is published as a major version in SharePoint - Copilot can use it to provide answers
Copilot does not cache information. It reaches out to a special service each time. This service provides copilot with the Snippets (text extracts).
Copilot uses the custom search service to “read” the document’s content. But only a short Snippet property is used. The Search service does not return the document’s entire content in the response. It might only return a property called Snippet, but it only contains a short paragraph, with 3-6 sentences. It’s not always enough for Copilot to give any meaningful answers, and Copilot ignores the rest of the information in the documents larger than a single page.
The Search Snippet does not contain a summarized text. This text is as-is, with no modifications. Copilot can’t determine what changed in a given Word document. It can only guess, based on some indirect hints.