Using Laravel Dusk outside of Tests to upload Files

Recently I asked on Twitter, if anyone had experience with running Laravel Dusk outside of a test environment. Paul Redmond linked to the infamous Laravel News article, where Jordan Dalton automated his monthly payments using Laravel Dusk.

My public call came from a similar need. At work, we needed to automate a file upload. Not that complicated you might think. But the client uses a proprietary software to handle those files. There are no SFTP credentials or APIs to use.
Files have to be uploaded by signing in to a website, clicking through a folder hirarchy and then uploading the respective file.

At first, we did this manually. A few weeks later the cadense increased to daily uploads, and we started to think about automating this step.

In his response, Paul also shared links to a project he used for a presentation on data scraping. It was exactly what I was looking for.

The Laravel Way #

Our solution is based on a fresh new Laravel application. We've added a Browser-class similar to the one Paul shared on Twitter.
A scheduled Artisan command looks for new files to upload, boots the Browser and uploads the files through the web UI.

Here are both our Browser and Artisan command implementations.

<?php

namespace App;

use Closure;
use Exception;
use Facebook\WebDriver\Chrome\ChromeOptions;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Laravel\Dusk\Chrome\ChromeProcess;
use Laravel\Dusk\Chrome\SupportsChrome;
use Symfony\Component\Process\Process;
use Throwable;
use Laravel\Dusk\Browser as DuskBrowser;

class Browser
{
    /**
     * @var DuskBrowser|null
     */
    public $browser;

    /**
     * @var Process
     */
    public $process;

    public function startProcess()
    {
        $this->process = (new ChromeProcess)->toProcess();
        $this->process->start();
    }

    public function quit()
    {
        $this->browser->quit();
        $this->process->stop();
    }

    /**
     * @param Closure $callback
     */
    public function browse(Closure $callback)
    {
        $this->startProcess();

        if (! $this->browser) {
            $this->browser = $this->newBrowser($this->createWebDriver());
        }

        $callback($this->browser);
    }

    /**
     * @throws Exception
     */
    function __destruct()
    {
        if ($this->browser) {
            $this->closeBrowser();
        }
    }

    /**
     * @throws Exception
     */
    protected function closeBrowser(): void
    {
        if (! $this->browser) {
            throw new Exception("The browser hasn't been initialized yet");
        }

        $this->browser->quit();
        $this->browser = null;
    }

    protected function newBrowser($driver): DuskBrowser
    {
        return new DuskBrowser($driver);
    }

    protected function createWebDriver(): RemoteWebDriver
    {
        return retry(5, function () {
            return $this->driver();
        }, 50);
    }

    protected function driver(): RemoteWebDriver
    {
        $options = (new ChromeOptions)->addArguments(collect([
            '--window-size=2000,800',
            '--disable-gpu',
            '--headless',
        ])->all());

        return RemoteWebDriver::create(
            'http://localhost:9515',
            DesiredCapabilities::chrome()->setCapability(
                ChromeOptions::CAPABILITY, $options
            )
        );
    }

}

<?php

namespace App\Console\Commands;

use App\Browser;
use App\Notifications\NoFilesFoundNotification;
use App\Notifications\UploadFailedNotification;
use App\Notifications\UploadSucceededNotification;
use Illuminate\Console\Command;
use Illuminate\Support\Collection;
use Illuminate\Support\Facades\Notification;
use Illuminate\Support\Facades\Storage;
use Illuminate\Support\Str;
use Throwable;

class UploadNewFilesCommand extends Command
{
    protected $signature = 'app:upload-new-files';

    protected $description = 'Boot a Browser and Upload new Files';

    private Browser $browser;

    public function __construct(Browser $browser)
    {
        parent::__construct();

        $this->browser = $browser;
    }

    public function handle()
    {
        $files = $this->getNewAndNotUploadedFiles();
        $this->info("{$files->count()} Files found to upload");

        if ($files->count() === 0) {
            $this->notifyDevelopersAboutNoFiles();
            return 0;
        }

        $files->each(function (string $file) {
            $this->info("Start uploading {$file} …");
            $this->uploadFile($file);
        });

        $this->browser->quit();

        return 0;
    }

    private function getNewAndNotUploadedFiles(): Collection
    {
        return collect(Storage::disk('sftp')->files())
            ->reject(function (string $file) {
                return Str::startsWith($file, 'uploaded_');
            });
    }

    private function uploadFile(string $file)
    {
        try {
            $this->browser->browse(function (\Laravel\Dusk\Browser $browser) use ($file) {

                // Download file from SFTP to local disk.
                $contents = Storage::disk('sftp')->get($file);
                Storage::put($file, $contents);
                $absolutePath = Storage::path($file);

                $browser->visit('http://example.com/login')
                    ->pause(1000)
                    ->type('email', 'example')
                    ->type('password', 'secret')
                    ->pause(1000)
                    ->press('Login')
                    ->pause(1000)
                    ->click('#upload-btn')
                    ->pause(1000);


                if (app()->environment('production')) {
                    $browser->attach('#file', $absolutePath)
                        ->pause(10000)
                        ->assertSee($file)
                        ->pause(5000);
                }
            });
        } catch (Throwable $e) {
            $this->error("Upload failed: {$e->getMessage()}");
            $this->notifyDevelopersAboutFailure($file);
            Storage::delete($file);
            return;
        }

        // No Exception has been thrown. Consider Upload a Success.
        $this->notifyDevelopersAboutSuccess($file);
        $this->renameFile($file);
        $this->info("Uploaded successful. ({$file})");

        // Delete local copy of file
        Storage::delete($file);
    }

    private function notifyDevelopersAboutSuccess(string $file): void
    {
        Notification::route('slack', config('services.slack.webhook_url'))
            ->notify(new UploadSucceededNotification($file));
    }

    private function notifyDevelopersAboutFailure(string $file): void
    {
        Notification::route('slack', config('services.slack.webhook_url'))
            ->notify(new UploadFailedNotification($file));
    }

    private function notifyDevelopersAboutNoFiles(): void
    {
        Notification::route('slack', config('services.slack.webhook_url'))
            ->notify(new NoFilesFoundNotification());
    }

    private function renameFile(string $file): void
    {
        Storage::disk('sftp')->move($file, "uploaded_" . $file);
    }
}

The file upload has it's quirks though. Our system asserts that the uploaded file name appers in the UI. This doesn't happen all the time for unkown reasons.
To ensure, that all files are always uploaded, we send Slack notifiations to our team, so that a human can take a look if a file appears to not be uploaded correctly.

The Laravel Zero Way #

A couple of weeks after I've deployed our solution, I started working on a CLI for a different project. I've used Laravel Zero for it.

While reading the docs, I saw that Laravel Zero natively supports controling the Dusk browser. They call it "Web Browser Automation".

I won't rewrite our solution, as it works great for us. Next time I have to automate a process involving a web browser, I will definitely give Laravel Zero with its Dusk component a try.