Introducing Symfony Panther: a Browser Testing and Web Scrapping Library for PHP

Written by Symfony blog - - Aggregated on Wednesday September 12, 2018

Since the very first version of Symfony 2, the framework provides a suite of convenient tools to create functional tests. They use the BrowserKit and DomCrawler components to simulate a web browser with a developer-friendly API.

The WebTestCase helper

Let's refresh our memories by creating a tiny news website, and the corresponding functional test suite:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# create the new project
$ composer create-project symfony/skeleton news-website
$ cd news-website/

# add some dependencies
$ composer require twig annotations
$ composer require --dev maker tests

# Run the PHP built-in web server
$ php -S 127.0.0.1:8000 -t public

We're ready to code. Start by adding a class to store and retrieve the news:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// src/Repository/NewsRepository.php
namespace App\Repository;

class NewsRepository
{
    private const NEWS = [
        'week-601' => [
            'slug' => 'week-601',
            'title' => 'A week of symfony #601 (2-8 July 2018)',
            'body' => 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore.',
        ],
        'symfony-live-usa-2018' => [
            'slug' => 'symfony-live-usa-2018',
            'title' => 'Join us at SymfonyLive USA 2018!',
            'body' => 'Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur.'
        ],
    ];

    public function findAll(): iterable
    {
        return array_values(self::NEWS);
    }

    public function findOneBySlug(string $slug): ?array
    {
        return self::NEWS[$slug] ?? null;
    }
}

This implementation isn't very dynamic, but it does the job. Then, we need a controller and the corresponding Twig template to display the latest news of the community. We'll use the Maker Bundle to generate them:

1
$ ./bin/console make:controller News

Edit the generated code to fit our requirements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// src/Controller/NewsController.php
namespace App\Controller;

use App\Repository\NewsRepository;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

class NewsController extends Controller
{
    private $newsRepository;

    public function __construct(NewsRepository $newsRepository)
    {
        $this->newsRepository = $newsRepository;
    }

    /**
     * @Route("/", name="news_index")
     */
    public function index(): Response
    {
        return $this->render('news/index.html.twig', [
            'collection' => $this->newsRepository->findAll(),
        ]);
    }

    /**
     * @Route("/news/{slug}", name="news_item")
     */
    public function item(string $slug): Response
    {
        if (null === $news = $this->newsRepository->findOneBySlug($slug)) {
            throw $this->createNotFoundException();
        }

        return $this->render('news/item.html.twig', ['item' => $news]);
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{# templates/news/index.html.twig #}
{% extends 'base.html.twig' %}

{% block title %}News{% endblock %}

{% block body %}
{% for item in collection %}
    <article id="{{ item.slug }}">
        <h1><a href="{{ path('news_item', {slug: item.slug}) }}">{{ item.title }}</a></h1>
        {{ item.body }}
    </article>
{% endfor %}
{% endblock %}
1
2
3
4
5
6
7
8
{% extends 'base.html.twig' %}

{% block title %}{{ item.title }}{% endblock %}

{% block body %}
    <h1>{{ item.title }}</h1>
    {{ item.body }}
{% endblock %}

Thanks to the WebTestCase helper, adding some functional tests for this website is easy. First, generate a functional test skeleton:

1
$ ./bin/console make:functional-test NewsControllerTest

And add assertions to check that our controller works properly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
// tests/NewsControllerTest.php
namespace App\Tests;

use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;

class NewsControllerTest extends WebTestCase
{
    public function testNews()
    {
        $client = static::createClient();
        $crawler = $client->request('GET', '/');

        $this->assertCount(2, $crawler->filter('h1'));
        $this->assertSame(['week-601', 'symfony-live-usa-2018'], $crawler->filter('article')->extract('id'));

        $link = $crawler->selectLink('Join us at SymfonyLive USA 2018!')->link();
        $crawler = $client->click($link);

        $this->assertSame('Join us at SymfonyLive USA 2018!', $crawler->filter('h1')->text());
    }
}

And now, run the tests:

1
$ ./bin/phpunit

All green! Symfony provides a very convenient API to navigate the website, check that links work and assert that the expected content is displayed. It's easy to setup, and it's super fast!

Using Panther to Run the Scenario in a Browser

However, WebTestCase doesn't use a real web browser. It simulates one with pure PHP components. It doesn't even use the HTTP protocol: it creates instances of HttpFoundation's Request objects, pass them to the Symfony kernel, and allows to assert on the HttpFoundation Response instance returned by the app. Now, what if a problem preventing the webpage to load occurs in the browser? Such issues can be as diverse as a link hidden by a faulty CSS rule, a default form behavior prevented by a buggy JavaScript file, or, worst, the detection by the browser of a security vulnerability in your code.

Well, Panther allows to run this exact same scenario in real browsers! It also implements the BrowserKit and DomCrawler APIs, but under the hood it uses the Facebook PHP WebDriver library. It means that you can choose to execute the same browsing scenario in a lightning-fast pure PHP implementation (WebTestCase) or in any modern web browser, through the WebDriver browser automation protocol which became an official W3C recommendation in June.

What's even better, to use Panther, you only need a local Chrome installation. There is nothing more to install: no Selenium (but Panther supports it too), no obscure browser driver or extension... Actually, because Panther is now a dependency of the symfony/test-pack metapackage, you've already installed Panther without knowing it when you've typed composer req --dev tests earlier. You could also install Panther directly in any PHP project by running composer require symfony/panther.

Let's tweak some lines of our existing test case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// tests/NewsControllerTest.php
namespace App\Tests;

-use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
+use Symfony\Component\Panther\PantherTestCase;

-class NewsControllerTest extends WebTestCase
+class NewsControllerTest extends PantherTestCase
{
    public function testNews()
    {
-        $client = static::createClient(); // Still work, if needed
+        $client = static::createPantherClient();

Run the tests, again:

1
$ ./bin/phpunit

All green, again! But this time, we're sure that our news website works properly in Google Chrome.

Under the hood Panther has:

If you only believe what you see, try running the following:

1
$ PANTHER_NO_HEADLESS=1 ./bin/phpunit

Watch the Screencast

As you may have noticed in the recording, I've added some calls to sleep() to highlight how it works. Having access to the browser's window (and to the Dev Tools) is also very useful to debug a failing scenario.

Because both tools implement the same API, Panther can also execute web scraping scenarios written for the popular Goutte library. In test cases, Panther lets you choose if the scenario must be executed using the Symfony kernel (when available, static::createClient()), using Goutte (send real HTTP queries but no JavaScript and CSS support, static::createGoutteClient()) or using real web browsers (static::createPantherClient()).

Even if Chrome is the default choice, Panther can control any browser supporting the WebDriver protocol. It also supports remote browser testing services such as Selenium Grid (open source), SauceLabs and Browserstack.

There is also an experimental branch that uses Geckodriver to automatically start and drive a local installation of Firefox instead of Chrome.

Testing Client-side Generated HTML

Our news website looks good, and we've just proved that it works in Chrome. But now, we want to hear some feedback from the community about our frequent publications. Let's add a comment system to our website.

To do so, we'll leverage the capabilities of Symfony 4 and of the modern web platform: we'll manage the comments through a web API, and we'll render them using Web Components and Vue.js. Using JavaScript for this feature allows to improve the overall performance and user experience: each time we post a new comment, it will be displayed in the existing page without requiring a full reload.

Symfony provides an official integration with API Platform, probably the easiest way out there to create modern web APIs (hypermedia and/or GraphQL). Install it:

1
$ composer require api

Then, use the Maker Bundle again to create a Comment entity class, and expose it through a read and write API endpoint:

1
$ ./bin/console make:entity --api-resource Comment

This command is interactive, and allows to specify the fields to create. We need only two: news (the slug of the news) and body (the comment's content). news is of type string (maximum length of 255 chars), while body is of type text. Both aren't nullable.

Here is the full transcript of the interactions with the command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
New property name (press <return> to stop adding fields):
> news

Field type (enter ? to see all types) [string]:
>

Field length [255]:
>

Can this field be null in the database (nullable) (yes/no) [no]:
>

updated: src/Entity/Comment.php

Add another property? Enter the property name (or press <return> to stop adding fields):
> body

Field type (enter ? to see all types) [string]:
> text

Can this field be null in the database (nullable) (yes/no) [no]:
>

updated: src/Entity/Comment.php

Update the .env file to set the value of DATABASE_URL to the address of your RDBMS and run the following command to create the table corresponding to our entity:

1
$ ./bin/console doctrine:schema:create

If you open http://localhost:8000/api, you can see that the API is already working and documented


« Forge: Bitbucket API v2 Changes - Taylor Otwell

Laravel News - Laratables: Ajax support of DataTables … »