TLDR;
This video introduces Playwright Agents, a new feature in Playwright version 1.56 designed to guide large language models in building Playwright tests. It covers three agents: Planner, Generator, and Healer. The Planner agent plans the application to be tested, the Generator agent creates tests in Playwright format based on the plan, and the Healer agent fixes failing tests. The video explains how these agents use prompts and MCP servers, and it demonstrates their use with a .NET-based application, showcasing how they can generate, run, and heal tests.
- Playwright Agents: New feature in Playwright v1.56 to guide LLMs in test creation.
- Three Agents: Planner, Generator, Healer.
- Agents use prompts and MCP servers.
- Demonstrated with a .NET-based application.
Introduction [0:00]
The video introduces Playwright Agents, a new feature in Playwright version 1.56. These agents guide large language models through the process of building Playwright tests. There are three agents: the Planner agent, the Generator agent, and the Healer agent. These agents perform planning of the application to be tested, generate tests in Playwright format, and heal failing tests, respectively. The agents use prompts created to guide the model contest protocol (MCP) server. The video aims to explain the functionality and the underlying processes, including the new npx playwright init agents command, which initializes necessary MD files and adds an MCP server.
Installing Playwright Agents [7:24]
To start from scratch, the presenter opens a new Visual Studio Code terminal with an application that doesn't have Playwright tests. The presenter removes existing GitHub chat mode and VS code settings to ensure a clean start. The presenter then executes the command npx playwright init agents to add the GitHub chat mode and VS Code folders with the MCP server.
Understanding Agents [7:53]
The video explains the purpose of the planner chat mode file, which is a markdown file that tells the agent to create a comprehensive test plan for a web application or website. The agent uses tools like edit/create file, edit/create directory, and search file. The planner agent acts as an expert web test planner with experience in quality assurance, user experience testing, and test scenario design. It navigates, explores, analyzes user flows, designs comprehensive scenarios, structures test plans, and creates documentation. The generator agent generates tests based on the test plan MD file. The healer agent debugs and fixes failing Playwright tests.
Planner Test with Playwright Agents [13:44]
The presenter starts using the agents by asking the planner agent to generate or plan a test. The presenter adds the context of the application, which includes the EA web app (front end) and the product API (back end). The presenter then instructs the agent to generate a test plan for the application, analyzing both the product API and the EA web app, specifying that the EA web app is the front end and the product API is the API and back end with a SQL database. The planner agent starts the planning operation, analyzing the application and writing the planning steps required to test it.
Generating Test with Playwright Agents [15:57]
The presenter chooses the generator agent and asks it to generate tests based on the test plan. The presenter adds the MD file to the context, which allows it to be used for the generation operation. The generator agent starts adding a folder and tests, creating files like product list spec.ts. The agent generates the tests by reading the test plans.
Running Generated Test Cases [17:58]
The generator agent completes the test case development and creates tests with 55+ test coverage. The next step is to run the tests, ensuring Docker Compose is up and Playwright is installed. The presenter asks the agent to perform these steps. The agent creates a package.json file for Playwright test execution and adds the Playwright configuration. It starts the application in Docker and performs npm install. Finally, it runs the tests using npx playwright test --project chromium, showing the test report with passing and failing tests.
Healing the Failed Test Cases with Playwright Agents [22:43]
The presenter uses the healer agent to fix the failing tests. The presenter chooses the healer agent and asks it to heal the failing tests. The healer agent runs the tests again to identify the failures and then creates to-dos to fix the issues. It addresses strict mode violations, validation message visibility, product type drop-down selector issues, test data isolation, and performance test timeouts. The agent uses the Playwright test MCP server to fix the issues.