Microsoft Announces Major AI Updates for Copilot, Windows 11, and Bing

Microsoft announced a series of major AI updates for its chatbot Copilot, the Windows 11 operating system, and the Bing search engine. These updates aim to create a more personalized and user-friendly experience by leveraging AI for generating search results, voice interfaces, and recommendations.

By releasing these updates, Microsoft stated that privacy and AI remain central elements of the company – after receiving negative assessments from security experts and users earlier this year following the release of the Windows 11 "Undo" feature, which continuously took screenshots and was met with negative feedback.

The Copilot application, originally launched as Bing Chat in February 2023 and renamed in November 2023, has been updated with a voice mode and the ability to observe user actions alongside their screen (with user permission and consent) and launched from the Microsoft Edge web browser and the new program called Copilot Labs.

As Microsoft reported in its blog: "Before releasing our tools to all users, we test them on a small subset to gather feedback, analyze them, and then apply these lessons to the product, making it, as we hope, immediately better and safer."

The new features include:

• Copilot Voice, which allows users to interact with AI through natural speech, offering four different voice options. It is being rolled out in English-speaking countries such as Australia, Canada, New Zealand, the United Kingdom, and the United States, and will then be available in other regions and languages.
• Copilot Daily, a feature designed to provide users with a convenient overview, including news summaries and weather forecasts tailored to their interests. Information is extracted from authorized content providers, including Reuters and The Financial Times.
• Copilot Vision, an experimental feature available to selected Copilot Pro users. Users can interact with images or web pages, and Copilot will interpret what is on the screen to help with decision-making, such as comparing product options or suggesting next steps.

Microsoft emphasized that no data from these sessions is saved or used for training.

Through Copilot Labs, the Think Deeper feature is also available, providing detailed step-by-step answers to complex queries. The goal is to offer a deeper understanding of topics such as decision-making, financial comparisons, or practical dilemmas, such as choosing a new car or a place to live.

Copilot also integrates creative tools, such as visual search, which allows users to upload images for AI analysis, be it identifying dog breeds or suggesting recipe ideas based on a photograph of a dish.

AI also helps refine text, offering recommendations for tone and style, and encouraging creativity with tools for creating stories, poems, and images.

Regarding the models behind the new Copilot experience, a Microsoft representative stated: "We continue to use the latest OpenAI models. In addition, we are starting to apply our own internal technologies. This combination gives us more flexibility to apply the best model to each client's interaction with Copilot."

With the new updates, Windows 11 continues to evolve into a more AI-centric operating system, especially for Copilot+ PC users. These AI-powered features are designed to enhance performance while maintaining a streamlined, user-friendly interface.

• Recall allows users to instantly retrieve previously viewed content on their PC. This feature is optional, protected by Windows Hello, and ensures user privacy. Information such as credit card data will remain confidential with additional security filters.
• Click to Do helps users quickly perform actions by overlaying interactive suggestions on the screen. Whether it's rewriting text, editing images, or launching web searches, this tool aims to accelerate workflows without requiring manual navigation.
• Enhanced Search in Windows 11 utilizes AI to provide a more intuitive search for files. Users can describe the content they are searching for in natural language, for example, "photos of a barbecue party," without needing to remember specific file names or their location. This feature applies to OneDrive search as well.

According to a Microsoft representative, all AI models providing enhanced search are proprietary models.

These improvements also extend to creative applications. Super Resolution in Photos allows users to upscale images with low resolution, and generative filling and erasing in Paint provide precision editing for photographs, for example, removing unwanted objects. These functions utilize built-in AI.

Bing from Microsoft, powered by AI, continues to evolve with the launch of Bing Generative Search. This feature goes beyond simple keyword-based queries, offering detailed, contextually enriched answers to more complex questions.

Launched in beta in the United States, the Bing Generative Search system allows users to try its capabilities by entering "Bing generative search" into the search bar and selecting from a carousel of demonstration queries displayed as "cards" with captions.

Bing will generate its own summary of information from multiple sources – similar to Google AI Overviews. However, Microsoft's solution is more interactive, it includes pop-up windows with links to sources and text, and an interactive table of contents.

Another new feature is the Deep Search button, which users can click to get a more in-depth and detailed explanation of a topic. The function is designed to handle informational and complex queries, such as problem-solving or exploring unfamiliar topics. The goal is to increase traffic for content creators by maintaining a balanced web ecosystem. Microsoft warns that this function is still in beta testing.

Copilot is also designed to encourage responsible use. AI provides links to sources it references, allowing users to verify information for accuracy. Microsoft has included feedback loops to allow users to report inaccurate or inappropriate content using built-in tools.

The emphasis on privacy and security emerged after Microsoft faced reactions from users in May 2024 following the announcement of the Windows 11 "Undo" feature for Copilot+ PCs, which takes screenshots stored on the user's device and allows the PC to return to its previous state upon user request. However, security researchers and engineers noted that this feature behaved in many ways like spyware, especially considering that Microsoft initially presented it as a standard feature installed and enabled by default, which users had to disable with multiple tricks. Since then, it has become available only on request.