Caching Laravel configuration, routes, and more

Bref provides built-in support for Laravel via a package. By default, this integration will cache the Laravel configuration on startup, i.e. on AWS Lambda cold start. That allows making subsequent requests faster.

This is the most basic step to optimizing Laravel. Laravel can "cache" many things (config, routes, etc.), and these are often done on deployment before traffic reaches the application.

We can replicate that with Bref, but the main thing to keep in mind is to do these caching operations in an environment as close to production as possible. Indeed, Laravel will sometimes hardcode absolute paths or even environment variables (if you cache the config) in the cached configuration.

To solve that, we need to run caching commands in Docker, using the Bref-provided container images.

One solution is to run Laravel commands like route:cache in a Bref container by mounting the application in /var/task (the path where the app runs in Lambda):

docker run --rm -it --entrypoint=php -v $(pwd):/var/task bref/php-84:2 artisan route:cache

Another approach would be to deploy the application as a container image instead of a zip file. In that case, we can build the image entirely the way we want to. Here’s an example:

# syntax=docker/dockerfile:1.7
FROM bref/php-84:2
# Copy the application code
COPY --link . /var/task
# Clear any cached config that might reference local paths
RUN php artisan config:clear && php artisan route:clear && php artisan view:clear
RUN php artisan optimize
# The views are compiled in /tmp/storage/framework/views, let's copy them to `storage/framework/views` so that they are persisted
RUN mkdir -p storage/framework/views \
    && cp -r /tmp/storage/framework/views/* storage/framework/views/ \
    && rm -rf /tmp/*
# Make sure PHP on Lambda will be able to read these files
RUN chmod -R 777 storage/framework/views
RUN chmod -R 777 bootstrap/cache

One important thing to note if you want to compile the configuration before deploying: all environment variables will be hardcoded in the cached configuration.

That means you must have production environment variables set during that build. One way is to create the .env file with production secrets and copy that with the application in the container image. Then, php artisan config:cache (or optimize) will be able to cache those variables. If you do this, keep in mind that the container image contains production secrets, treat it appropriately (for example by tightening permissions to ECR).

Pre-warming the opcode cache

Another option I explored was pre-compiling PHP's opcode cache before deploying. That would enable PHP-FPM, on startup, to start with a partially warm opcache (avoiding reading PHP files from disk, parsing them, and compiling them on the fly).

Usually, PHP's opcode cache is stored in memory. But it can also be stored to disk during deployment and loaded from these files on startup in Lambda.

This approach is trickier for several reasons:

1. PHP's opcode cache is 100% tied to the exact PHP version, extensions, config, etc. and not portable between systems
2. deploying PHP's opcode cache makes the deployed archive bigger, which slows down cold starts
3. if the opcode cache is deployed with the app as files, these are mounted as read-only in Lambda and PHP does not like that
4. you need a way to trigger opcache compilation to disk

Let's go through these challenges:

1. PHP's opcode cache is 100% tied to the exact PHP version, extensions, config, etc. and not portable between systems

That can be solved by using the same container image during deployment as during production, which we already did in the previous section. So this is solved if you involve Docker.

2. deploying PHP's opcode cache makes the deployed archive bigger, which slows down cold starts

That is definitely true if you deploy with zip files, which is the default with Bref. Don't try this with zip deployments because the extra MB of opcode caches will cancel all your efforts.

However, when deploying with container images, AWS Lambda performs optimizations (there's a white paper about it) and essentially streams the filesystem to Lambda as files are read. In other words, it will only load the files you actually read. This means we can add more files to the image and cold starts will not necessarily slow down.

In my experience, it was a balance between how much you compile and the resulting cold start times. For example, I tried compiling every single PHP file, but this ended up making cold starts worse. This was counter-intuitive: I expected PHP to read the .bin file instead of the .php one, so things could only be faster.

But when I compiled every PHP file and removed the original .php files from the container image, PHP crashed completely. For some reason it needed the original PHP files, and I believe that’s why cold starts would not improve 100% of the time. I know there are options to make opcache not read PHP files, and I tried to use all of them. But PHP always needed the original files no matter what. It might be something I missed, or it might be that "opcache from files" behaves differently than "opcache from memory". Feel free to continue my experiments and prove me wrong, I’d love that!

In the end, I found a good balance by compiling only the files in these directories (this was a Laravel app):

bootstrap
app
storage/framework/views
vendor/composer
vendor/bref
vendor/symfony/http-foundation
vendor/symfony/http-kernel
vendor/psr

You can use this as a starting point. I aimed at compiling the files that I knew would be used for sure for any HTTP request.

3. if the opcode cache is deployed with the app as files, these are mounted as read-only in Lambda and PHP does not like that

I explored one approach:

• set PHP's opcache directory (on disk) to /tmp/opcache, since /tmp is the only writable path in Lambda
• during deployment, compile PHP's opcache to e.g. /bref/opcache
• on cold start, copy /bref/opcache to /tmp/opcache

To clarify if you're not familiar with Lambda: /tmp will always be empty on cold starts (you can't add files there via your container image), and is not shared between Lambda invocations.

This approach wasn't working well. Copying megabytes of data on startup was slower than deploying without a pre-warmed opcache. PHP is just that fast, and it's hard to complain about it :p

I then tried something else: if PHP could read the opcache as read-only from /bref/opcache directly that would be great. After all, we don't need PHP to write to that directory after startup since it will all be kept in memory. But PHP doesn't support that: if a directory is configured for opcache, PHP expects to use it.

But it turns out I wasn't the only one wanting that: @iamacarpet opened an issue in the PHP repository and advocated that exact use case, not specifically for Lambda but also for all container-based deployments where the filesystem is read-only.

He ended up working on a pull request that was accepted and merged in… PHP 8.5!

I upgraded my test project to PHP 8.5 (in beta at the time of writing) and Bref v3 (in alpha at the time of writing), and used the new opcache.file_cache_read_only=1 option. It worked!

4. you need a way to trigger opcache compilation to disk

The last step is to actually compile some PHP files to opcache files during deployment. Here is an example script:

<?php
$dirs = ['bootstrap', 'app', 'storage/framework/views', 'vendor/composer', 'vendor/bref', 'vendor/symfony/http-foundation', 'vendor/symfony/http-kernel', 'vendor/psr'];
foreach ($dirs as $dir) {
    $it = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($dir, FilesystemIterator::SKIP_DOTS));
    foreach ($it as $f) {
        if ($f->getExtension() === 'php') {
            $filename = $f->getPathname();
            $result = opcache_compile_file($filename);
            if ($result === false) {
                echo "[opcache] Failed to compile " . $filename . "\n";
                exit(1);
            }
            // Check the file is compiled
            if (!opcache_is_script_cached($filename)) {
                echo "[opcache] File not cached " . $filename . "\n";
            }
        }
    }
}

This is a rough script (yes, it’s ugly), feel free to make it more robust and nicer.

One thing I noticed is that some files would not compile, but no error was thrown. Hence the second check with if (!opcache_is_script_cached($filename)). I could not explain why some files would not be compiled, and it even seemed a bit random. Anyway, I didn't need exactly all files to be compiled so I ignored this for my tests and moved on.

Putting all this together, here is a Dockerfile:

# syntax=docker/dockerfile:1.7
FROM bref/php-84:3
COPY --link . /var/task
RUN mkdir -p /bref/opcache
# Clear any cached config that might reference local paths
RUN php artisan config:clear && php artisan route:clear && php artisan view:clear
RUN php artisan optimize
# The views are compiled in /tmp/storage/framework/views, let's copy them to `storage/framework/views` so that they are persisted
RUN mkdir -p storage/framework/views \
    && cp -r /tmp/storage/framework/views/* storage/framework/views/ \
    && rm -rf /tmp/*
# `opcache.file_cache_read_only=0` so that opcache actually writes the opcache files
RUN php -d opcache.file_cache_read_only=0 compile-opcache.php
RUN chmod -R 777 storage/framework/views
RUN chmod -R 777 bootstrap/cache
RUN chmod -R 755 /bref/opcache

And here is the php.ini file I used (both locally and in Lambda):

opcache.file_cache=/bref/opcache
opcache.file_cache_read_only=1
opcache.file_cache_only=0
opcache.use_cwd=0
opcache.validate_timestamps=0
opcache.file_cache_consistency_checks=0

Results

All this for what? Cold starts were 40% faster. This is compared to zip deployments without any cache pre-built.

I would have loved an even more drastic improvement, but it is still significant. Overall, the improvement can be attributed to:

• switching from zip to container deployments (which has better cold starts in itself)
• compiling Laravel's config, routes, etc. before deploying
• compiling opcache before deploying

But Lambda cold starts still need to mount the application and start PHP-FPM. Unless AWS Lambda SnapStart becomes available for custom runtimes, we shouldn’t expect drastic improvements.

I enjoyed digging very deep into this problem, especially as I'm working intensely on Bref v3 and benchmarking so many things to improve performance overall. But I want to reframe the problem: I don't think it's actually one in most situations.

Cold starts are very rare. About 0.3% of all invocations.

We tend to see them as developers because when we deploy we are the first to invoke the app after deployment. Most end users don't notice them, especially compared to things like 200ms network latency when visiting a site hosted on the other side of the planet. So I think it's important to put things into perspective and not conflate "working on fun deep technical challenges" with "this must be a very important problem".

And if you do want to play with these ideas and try benchmarking on your own, a few notes for you:

• be very careful how you measure cold starts, it's easy to mess things up (see below)
• never measure averages in metrics (especially those with high variability where random extremes skew averages), use percentiles
• learn how containers start up on Lambda: AWS does a lot of optimizations on the image on the very first invocation. You may see latencies of 5 or even 8 seconds, these will mess up your metrics for no reason (end users would never see this cold start in reality). Instead "warm" the regional AWS Lambda caches for container images.
• don't measure one or two cold starts, you need at least 20 for significant results

As a bonus, here's a CloudWatch Logs Insight query you might want to use:

filter @type = "REPORT"
| stats
count(@type) as count,
min(@billedDuration) as min,
avg(@billedDuration) as avg,
pct(@billedDuration, 50) as p50,
max(@billedDuration) as max
by @log, (@initDuration > 0) as coldstart
| sort @log

However, there is one thing that I really dislike about it: how it handles server errors.

When an error occurs on the server, Inertia will show the error page in a modal. That feels very weird. Even weirded when that modal shows because an async request failed.

The official docs suggest to customize the error page, but I want to get rid of the modal entirely.

Let's replace the modal with toast notifications.

I'm using vue-toastification for toast notifications. You can use any other library, but the idea is the same.

First, let's override how Laravel turns errors into HTTP responses in bootstrap/app.php. When we are in an Inertia request, instead of returning the error page, we will return a JSON response with the error message.

return Application::configure(basePath: dirname(__DIR__))
    // ...
    ->withExceptions(function (Exceptions $exceptions) {
        $exceptions->respond(function (Response $response, Throwable $exception, Request $request) {
            $isServerError = in_array($response->getStatusCode(), [500, 503], true);
            $isInertia = $request->headers->get('X-Inertia') === 'true';
            // When in an Inertia request, we don't want to show the default error modal
            if ($isServerError && $isInertia) {
                $errorMessage = 'An internal error occurred, please try again. If the problem persists, please contact support.';
                // In local environment let's show the actual exception class & message
                if (app()->isLocal()) {
                    $errorMessage .= sprintf("\n%s: %s", get_class($exception), $exception->getMessage());
                }
                return response()->json([
                    'error_message' => $errorMessage,
                ], $response->getStatusCode());
            }
            if ($response->getStatusCode() === 419) {
                return back()->with([
                    'flash.banner' => 'The page expired, please try again.',
                ]);
            }
            return $response;
        });
    })->create();

Now that Laravel returns a custom JSON response, we need to handle it in Inertia. Let's add this to the app.js file:

import { router } from '@inertiajs/vue3'
router.on('invalid', (event) => {
    const responseBody = event.detail.response?.data;
    if (responseBody?.error_message) {
        const toast = useToast()
        toast.error(responseBody.error_message);
        event.preventDefault();
    }
});
// ...

In short, if the server returns a JSON response with an error_message key, we will show it in a toast notification. Otherwise, we will let Inertia.js handle the error as usual.

Much better!

Switching to deploying containers on AWS Lambda

The default approach to deploying PHP to AWS Lambda with Bref is to deploy via zip files, while the PHP runtime is provided by Bref via AWS Lambda layers. This is great because deployments are fast and simple. However, adding custom binaries to the Lambda environment is hard because you have to create your own "layers".

To add FFmpeg to Lambda, we can switch to deploying a container image to Lambda. This way we have all the familiar tools to install FFmpeg in the container image later on.

There is a Bref guide on how to deploy a custom container image to Lambda, but here's the short version:

1. we need a Dockerfile:

FROM bref/php-82-fpm:2
# Copy our source code in the image
COPY . /var/task
# Configure the handler file (the entrypoint that receives all HTTP requests)
CMD ["public/index.php"]

2. We change serverless.yml to deploy our container image:

service: myapp
provider:
    name: aws
    ecr:
        images:
            myimage:
                # Path to the `Dockerfile` file
                path: ./
functions:
    website:
        image:
            name: myimage
        events:
            - httpApi: '*'

Deploying is the same command as before: serverless deploy.

Installing FFmpeg in the container image

Now that we have a container image, we can install FFmpeg in it using multi-stage builds:

# This is Bref's "build" image that we can use to build custom binaries and extensions
FROM bref/build-php-82:2 as build
# Install ffmpeg
RUN set -xe; \
    mkdir -p /tmp/ffmpeg; \
    curl -Ls https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz \
    | tar xJC /tmp/ffmpeg --strip-components=1
RUN mv /tmp/ffmpeg/ffmpeg /opt/bin/ffmpeg
FROM bref/php-82-fpm:2
# Copy what we built above in our final image
COPY --from=build /opt /opt
# Copy our source code in the image
COPY . /var/task
# Configure the handler file (the entrypoint that receives all HTTP requests)
CMD ["public/index.php"]

The ffmpeg binary is now available in the $PATH of the Lambda environment, PHP can use it like any other binary:

exec('ffmpeg -i input.mp4 output.mp4');

Before deploying to AWS Lambda, you will need:

• an AWS account (to create one, go to aws.amazon.com and click Sign up),
• the serverless CLI installed on your computer.

Note: while the example of this article should stay under the AWS free tier (1 million free AWS Lambda invocations per month), be advised that building on AWS can incur costs.

You can install the serverless CLI using NPM:

npm install-gserverless

If you don't have NPM or want to learn more, read the Serverless documentation.

Now connect the serverless CLI to your AWS account via AWS access keys. Create AWS access keys by following this guide, then set them up on your computer using the following command:

serverless configcredentials--provideraws--key<key>--secret<secret>

Serverless Framework is a CLI tool that helps us create and deploy serverless applications. Its configuration is stored in a serverless.yml file, which describes what will be deployed to AWS.

To deploy a Node application, we can create a simple serverless.yml file:

service:demo# name of the application
provider:
name:aws
runtime:nodejs18.x
region:us-east-1
functions:
api:
handler:index.handler
url:true

In the configuration above, we define a single AWS Lambda function called api, running NodeJS 18, with a public URL.

Our API handler will be a handler() function returned by index.js (learn more about handlers in the AWS Lambda documentation). Let's create the index.js file:

exportasyncfunctionhandler(event) {
return {
    hello:'world'
  }
}

Note that we will be using ESM features (like export and import), so let's create a package.json file with "type": "module":

{
"type":"module"
}

Our simple Node example is ready to be deployed with serverless deploy, but let's add PlanetScale into the mix first!

In your PlanetScale account, start by creating a database in the same region as the AWS application (us-east-1 in our example). Then, click the Connect button and select "Connect with: @planetscale/database". That will let us retrieve the database username and password.

To connect to the database in our code, we will use the PlanetScale serverless driver. Let's install it with NPM:

npm install@planetscale/database

Now that the driver is installed, we can connect to our PlanetScale database with the connect() function:

import { connect } from'@planetscale/database'
constconn=connect({
// With the serverless driver, the host is always 'aws.connect.psdb.cloud'
  host:'aws.connect.psdb.cloud',
  username:'<user>',
  password:'<password>'
})
exportasyncfunctionhandler(event) {
constresult=awaitconn.execute('SELECT * FROM records')
returnresult.rows
}

Note the following details:

• We are connecting to the database outside the handler() function. This is to reuse the same connection for all HTTP requests. If we were to connect inside the handler() function, a new connection would be created for each request, which would be inefficient.
• We are querying the records table. This table doesn't exist yet, we will create it below.
• We don't want to store the database credentials in the code. We will use environment variables instead.

Let's update our code to use environment variables. For the sake of the example, we will also create the records table on the fly:

import { connect } from'@planetscale/database'
constconn=connect({
// With the serverless driver, the host is always the same
  host:'aws.connect.psdb.cloud',
  username:process.env.DATABASE_USERNAME,
  password:process.env.DATABASE_PASSWORD
})
// Create the table if it doesn't exist (just for demo purposes)
// In a real application, we would run database migrations outside the function
awaitconn.execute('CREATE TABLE IF NOT EXISTS records (id INT PRIMARY KEY auto_increment, name VARCHAR(255))')
exportasyncfunctionhandler(event) {
// Insert a new record
constqueryParameter=event.queryStringParameters?.name ??'test'
awaitconn.execute('INSERT INTO records (name) VALUES (?)', [queryParameter])
// Retrieve all records
constresult=awaitconn.execute('SELECT * FROM records')
returnresult.rows
}

We now need to set the DATABASE_USERNAME and DATABASE_PASSWORD environment variables. We can define them in serverless.yml and use AWS SSM to store the database password securely:

provider:
name:aws
runtime:nodejs18.x
region:us-east-1
environment:
DATABASE_USERNAME:<username-here>
DATABASE_PASSWORD:${ssm:/planetscale/db-password}

The database password will be stored in AWS SSM (at no extra cost) so it is not visible in the code. The ${ssm:/planetscale/db-password} variable will retrieve the value from SSM on deployment. The SSM parameter can be created with the AWS CLI via the following command:

aws ssmput-parameter--regionus-east-1--name'/planetscale/db-password'--typeSecureString--value'replace-me'
# replace the `replace-me` string with the database password!

If you don't use the AWS CLI, you can also create the parameter in the AWS Console:

Our application is now ready! Let's deploy it:

serverless deploy

When finished, the deploy command will display the URL of our Node application. The URL should look like this: https://<id>.lambda-url.us-east-1.on.aws/. We can open it in the browser or request it with curl:

curl https://<id>.lambda-url.us-east-1.on.aws/

The response should list the records in the records table. A new record will be created every time the URL is requested. We can also provide a name parameter to change the name of the record inserted in the database:

curl https://<id>.lambda-url.us-east-1.on.aws/?name=hello

Besides the incredible scalability provided by the combination of AWS Lambda and PlanetScale, another benefit we get from this setup is the ability to combine Serverless Framework stages and PlanetScale branches.

We could imagine, for example, a dev stage for development and a prod stage for production. The dev stage would use a development branch in PlanetScale, while the prod stage would use the main production branch.

Using stage parameters, we can set different credentials to use to connect to PlanetScale depending on the stage:

provider:
name:aws
runtime:nodejs18.x
region:us-east-1
environment:
DATABASE_USERNAME:${param:dbUser}
DATABASE_PASSWORD:${param:dbPassword}
params:
dev:
dbUser:<dev-username-here>
dbPassword:${ssm:/planetscale/dev/db-password}
prod:
dbUser:<prod-username-here>
dbPassword:${ssm:/planetscale/prod/db-password}

When deploying, we can specify the stage to deploy via the --stage option:

serverless deploy--stagedev
serverless deploy--stageprod

Each stage (dev and prod) will result in entirely separate infrastructures on AWS, and each one will use its own PlanetScale branch.

That setup makes it easy to test code changes and database schema changes in a development environment that is identical to and isolated from the production environment. Once approved, schema changes can be applied to the production branch with a PlanetScale deploy request, and code changes can be deployed to production via the serverless deploy command.

In this article, we learned how to integrate PlanetScale with Node applications built using the Serverless Framework on AWS. This gives us a completely serverless stack with extreme scalability yet simple to set up and maintain.

Now that we have a basic application running we can explore more complex topics, such as:

• Creating multiple HTTP routes
• Setting up a complete deployment workflow for the Node application
• Dive into the PlanetScale workflow for branching databases, non-blocking schema changes, and more

Feel free to explore the PlanetScale documentation as well as the Serverless Framework documentation to learn more.

This approach provides several benefits:

• Instant and effortless autoscaling to handle incoming traffic.
• Redundant and resilient infrastructure out of the box, without extra complexity.
• Pay-per-request billing.

PlanetScale is a great database to pair with serverless Laravel applications running on Lambda. In this article, we will create a new Laravel application, run it on AWS Lambda using Bref, connect to a PlanetScale MySQL database, and do a load test to look at the performance.

Let's start from scratch and create a new Laravel project with Composer:

composer create-projectlaravel/laravelexample-app
cdexample-app

We can run our application locally with php artisan serve, but let's run it in the cloud on AWS Lambda instead.

To deploy Laravel to AWS Lambda, we can use Bref. Bref is an open-source project that provides support for PHP on AWS Lambda. It also provides a Laravel integration that simplifies configuring Laravel for Lambda.

Prerequisites

Before deploying to AWS Lambda, you will need:

• An AWS account (to create one, go to aws.amazon.com and click "Sign up").
• The serverless CLI installed on your computer.

You can install the serverless CLI using NPM:

npm install-gserverless

If you don't have NPM or want to learn more, read the Serverless documentation.

Now, connect the serverless CLI to your AWS account via AWS access keys. Create AWS access keys by following the guide, and then set them up on your computer using the following command:

serverless configcredentials--provideraws--key<key>--secret<secret>

Now that everything is ready, let's install Bref and its Laravel integration:

composer requirebref/brefbref/laravel-bridge--update-with-dependencies

Then, let's create a serverless.yml configuration file:

php artisanvendor:publish--tag=serverless-config

This configuration file describes what will be deployed to AWS. Let's deploy now:

serverless deploy

When finished, the deploy command will display the URL of our Laravel application.

Now that Laravel is running in the cloud, let's set it up with a PlanetScale database. Start in PlanetScale by creating a new database in the same region as the AWS application (us-east-1 by default).

Click the Connect button and select "Connect with: PHP (PDO)". That will let us retrieve the host, database name, user, and password.

Edit the .env configuration file to set up the database connection:

DB_CONNECTION=mysql
DB_HOST=<host url>
DB_PORT=3306
DB_DATABASE=<database_name>
DB_USERNAME=<user>
DB_PASSWORD=<password>
MYSQL_ATTR_SSL_CA=/opt/bref/ssl/cert.pem

Don't skip the MYSQL_ATTR_SSL_CA line: SSL certificates are required to secure the connection between Laravel and PlanetScale. Note that the path in AWS Lambda (/opt/bref/ssl/cert.pem) differs from the one on your machine (likely /etc/ssl/cert.pem). If you run the application locally, you will need to change this environment variable back and forth.

Next, redeploy the application:

serverless deploy

Now that Laravel is configured, we can run database migrations to set up DB tables. To do so, we can run Laravel Artisan commands in AWS Lambda using the serverless bref:cli command:

serverless bref:cli--args="migrate --force"

That's it! Our database is ready to use.

To test the database connection, let's create sample data in the users table created out of the box by Laravel.

Edit the database/seeders/DatabaseSeeder.php class and uncomment the following line so that we can seed our database with 10 fake users:

\App\Models\User::factory(10)->create();

Now, let's create a public API route that returns all the users from the database. Add the following code to routes/api.php:

Route::get('/users',function () {
return\App\Models\User::all();
});

Let's deploy these changes:

serverless deploy

Now, let's seed the database with 10 fake users:

serverless bref:cli--args="migrate:fresh --seed --force"

We can now retrieve our 10 users via the API route we created:

curl https://<application url>/api/users

The execution model of AWS Lambda gives us instant autoscaling without any configuration. To illustrate that, I have performed a simple load test against the application we deployed above, using PlanetScale's free tier database.

The only change I made is to disable Laravel's default rate limiting for API calls (ThrottleRequests middleware) in app/Http/Kernel.php because it would get in the way of my load test.

Furthermore, I did not ramp up traffic progressively because I wanted to show Lambda's instant scalability. I used ab (Apache's benchmarking tool) to request the /api/users endpoint with 50 threads (50 HTTP requests made in parallel continuously):

ab -c50-n10000https://<my-api-url>/api/users

When looking at the AWS Lambda and API Gateway metrics, we see the following numbers:

• Laravel scaled instantly from zero to 3,800 HTTP requests/minute.
• 100% of HTTP requests were handled successfully.
• The median PHP execution time (p50) for each HTTP request is 75ms.
• 95% of requests (p95) are processed in less than 130ms.
• PlanetScale processed up to 180 queries/s.
• The median PlanetScale query execution time is 0.3ms.

The load test was performed against a freshly deployed application. That means the first requests were cold starts: New AWS Lambda instances started and scaled up to handle the incoming traffic. The cold starts usually have a much slower execution time (one second instead of 75ms). However, we do not see them in the p50 or p95 metrics because they only impacted 1% of the requests in the first minute. After the first 50 requests (cold starts), all the other requests were warm invocations.

Note that we are looking at the AWS Lambda duration instead of HTTP response time: This is to exclude any latency related to networking (and thus have reproducible and comparable results). This is not the HTTP response time real users would see as, like on any server, the network adds latency to HTTP responses.

After a few minutes, I dropped the traffic from 50 requests in parallel to one. The PHP execution time stayed identical. This illustrates that the load did not impact the response time.

Improving performance to speed up the SSL connection

For many web applications, responding in about 100ms is more than satisfactory. However, some use cases may require lower latency.

Since Laravel connects to PlanetScale over SSL, creating the SSL connection can take longer than running the SQL query itself. PlanetScale itself can easily handle unlimited connections using built-in connection pooling, which massively improves performance by keeping those database connections open between requests.

However, PHP, by design, shares nothing across requests. This means at the end of every request, PHP will close the connection to the database.

To circumvent this problem, we can use Laravel Octane to gain performance in two ways:

• Keeping the Laravel application in memory across requests using Laravel Octane.
• Reusing SQL connections across requests (instead of reconnecting every time).

Bref supports Laravel Octane natively. We need to change the serverless.yml configuration to enable it. Change the web function configuration to this:

web:
  handler: Bref\LaravelBridge\Http\OctaneHandler
  runtime: php-81
  environment:
    BREF_LOOP_MAX: 250
    OCTANE_PERSIST_DATABASE_SESSIONS: 1
  events:
    - httpApi: '*'

Let's redeploy with serverless deploy and run the load test again:

We notice the following improvements:

• The median PHP execution time (p50) went from 75ms to 14ms.
• 95% of requests (p95) are processed in less than 35ms.
• Laravel handled 1,000 more requests/minute, though this number is not important: We could simply send more requests in our load test to reach a higher number anytime.

Here are some next steps:

• Download and run the code used in this blog post.
• Learn more about using PlanetScale with Bref: MySQL compatibility, data imports, and schema changes workflow.
• Learn more about running Laravel on AWS Lambda: running Artisan commands, setting up assets, queues, and more.

You can also learn more about PlanetScale and AWS Lambda with Bref in the respective documentation.

So far, Bref has been installed more than 2 million times and powers more than 10 billion Lambda executions (aka requests) every month*.

That's 1 in every 1000 AWS Lambda executions!

Today, we celebrate these achievements, the ongoing work, and the release of Bref 2.0 ?

Let's check out what's new in v2.

Bref 2.0

Here's a summary, we'll dive in the details below:

• Simpler serverless.yml configuration for setting up PHP.
• Revamped Laravel integration.
• ARM/Graviton support (faster processors with lower Lambda costs).
• Faster deployments by default.
• vendor/bin/bref cli becomes much simpler.
• Automatically load secrets in environment variables at runtime.
• Simpler docker-compose.yml for local development.
• PHP constructs for AWS CDK support.
• The internals (the scripts that build the runtime) have been rewritten at least 4 times (just look at the number of commits on the v2 runtimes…) but they are much better now: they are now tested, we understand all the code, we optimized their size, we've made the builds as fast as possible, and contributions and maintenance as easy as possible.

What did we break? Nothing major, the upgrade should be smooth. Here are the details:

• PHP 8.0+ is now required (7.4 support is dropped).
• Serverless Framework v3 is now required (2.x is obsolete). Run serverless --version to check.
• The vendor/bin/bref commands have been moved to the serverless CLI (detailed below).
• If you have a docker-compose.yml for local development, it needs to be adjusted (detailed below).
• The separateVendor option in serverless.yml has been removed (unmaintained feature).

Bref 2.0 lets us configure the runtime and PHP version in a much simpler way in serverless.yml (#1394). Here's an example below.

Note: this new feature is optional, you can keep using the Bref v1 syntax as it still works (this is not a breaking change).

Before (Bref v1 syntax):

provider:
    name: aws
    runtime: provided.al2
functions:
    api:
        handler: public/index.php
        # ...
        layers:
            - ${bref:layer.php-81-fpm}

After (Bref v2 syntax):

provider:
    name: aws
functions:
    api:
        handler: public/index.php
        # ...
        runtime: php-81-fpm

As you can see, we no longer have to set runtime: provided.al2 and add the Bref layers. We can now directly set a PHP runtime (php-81, php-81-fpm, php-81-console) and Bref will transform this into the proper runtime + layers configuration.

This works for all the Bref runtimes (FPM, function and console) and all supported PHP versions (80, 81, and 82 at the moment). Here's a recap:

# PHP-FPM runtime (web apps)
runtime: provided.al2
layers:
    - ${bref:layer.php-81-fpm}
# becomes:
runtime: php-81-fpm
# Function runtime
runtime: provided.al2
layers:
    - ${bref:layer.php-81}
# becomes:
runtime: php-81
# Console runtime
runtime: provided.al2
layers:
    - ${bref:layer.php-81}
    - ${bref:layer.console}
# becomes:
runtime: php-81-console

The Bref documentation has been updated to reflect these changes.

New Laravel integration

Bref provides the Laravel bridge to easily deploy Laravel applications to AWS Lambda.

However, that bridge was lagging behind and was limited in some features. The community (CacheWerk, Till Kruss, and George Boot) maintained a better alternative at cachewerk/bref-laravel-bridge.

With Bref 2.0, we are joining forces and their bridge will become the new Bref Laravel bridge (#94)!

The package name will stay bref/laravel-bridge, but a new major version (2.0) has been published with these improvements:

• Laravel Octane support!
• Automatic config caching (if not already cached) on Lambda cold start.
• Maintenance mode.
• Storage directory moved entirely to /tmp.
• AWS credentials automatically set up for S3 disks, SQS queues, and DynamoDB caches.
• and more minor changes, see #94.

More importantly, it improves how Laravel Queues are supported (this is the most significant breaking change):

• Laravel bridge 1.x adapted Laravel Queues to SQS: the retry strategy and storing of failed messages was handled by SQS (configured in serverless.yml). All SQS features were supported, but only a fraction of Laravel Queues features were supported.
• Laravel bridge 2.0 follows the official behavior of Laravel Queues instead. Only a fraction of SQS features are supported, but all features of Laravel Queues are now supported (e.g. job retry, delay, rate limiting, storing failed messages…).

That should make the experience for Laravel users much simpler, as existing projects can be ported to Lambda with no changes.

Note that it is possible to stay on the 1.x version of the Laravel bridge.

Let's take the opportunity to send huge thanks to Till and George for building such an excellent integration, and for joining the Bref organization on GitHub ?

If you want to get started with Laravel on Bref, check out the documentation.

ARM/Graviton support

Since 2021, it is possible to deploy Lambda functions running on ARM processors (called Graviton) instead of Intel x86 processors. However, Bref did not support that.

These processors usually run applications faster (example here), and ARM functions cost 20% less.

With Bref v2, we can deploy on ARM by setting the architecture field to arm64:

provider:
    # ...
    architecture: arm64
functions:
    # ...

The architecture: arm64 field can also be set in each function individually.

Warning: the example above uses the new runtime: php-xx syntax introduced above. If you set layers instead, you will need to set architecture: arm64 and update layers to reference ARM layers:

provider:
    # ...
    architecture: arm64
functions:
    api:
        # ...
        layers:
            # Add the `-arm` prefix in layers ?
            - ${bref:layer.arm-php-81-fpm}

Faster deployments

There is a serverless.yml option to enable faster deployments:

provider:
    # ...
    deploymentMethod: direct

In Bref v2, this option is enabled by default (#1395). If the option was already set in your serverless.yml, you can remove it (or leave it). If it wasn't, your deployments should be about twice faster.

Simpler CLI commands

Using Bref means using 2 different CLIs:

• vendor/bin/bref
• serverless

With Bref v2, all commands (except vendor/bin/bref init) have been moved to the serverless CLI (#1303). Besides reducing confusion, integrating in the serverless CLI lets us re-use the same AWS credentials, region, stack names, function names, etc. It makes the commands simpler.

Here are the commands that have changed:

• vendor/bin/bref cli is replaced by the simpler serverless bref:cli.

  For example:

  vendor/bin/bref cli mystack-dev-artisan --region=eu-west-1 -- migrate --force
  # becomes:
  serverless bref:cli --args="migrate --force"

  No need to provide the function name or the region anymore. Read the Console documentation to learn more. You will also find alternatives if you don't use the serverless CLI.

• vendor/bin/bref local is replaced by the simpler serverless bref:local.

  For example:

  vendor/bin/bref local --handler=my-handler.php
  # becomes:
  serverless bref:local -f hello

  No need to provide the handler file name anymore, we directly use the function name. The new serverless bref:local command has similar arguments as serverless invoke.

  Read the Local Development documentation to learn more. You will also find alternatives if you don't use the serverless CLI.

• vendor/bin/bref layers is replaced by the simpler serverless layers.

  Layer versions are also available at runtimes.bref.sh if you don't use the serverless CLI.

These changes allowed us to simplify the commands (automatically use the AWS region, credentials and stage from the serverless CLI). It also allowed us to remove the biggest bref/bref Composer dependencies and make the package much lighter.

Bref v1 lets you inject secrets (API keys, DB passwords, etc.) stored in SSM into environment variables at deployment time:

provider:
    # ...
    environment:
        GITHUB_TOKEN: ${ssm:/my-app/github-token}

This relies on serverless.yml variables (${ssm:xxx}) and works well, however the drawbacks are:

• The secret value is retrieved on serverless deploy and set in plain text in the environment variable.
• The user that runs serverless deploy must have permissions to retrieve the secret value.

In Bref v2, you can have these secrets injected at runtime (when your code boots in Lambda) via a new syntax (#1376):

provider:
    # ...
    environment:
        # Different syntax that does NOT start with `$`
        GITHUB_TOKEN: bref-ssm:/my-app/github-token

In the example above, GITHUB_TOKEN will be deployed with the string bref-ssm:/my-app/github-token (i.e. it doesn't contain the secret). When Lambda starts, Bref will automatically retrieve the secret and replace the environment variable value. No changes needed in your code.

This offers a more secure solution for teams that prefer to keep secrets as tight as possible.

Read more about this new feature and secrets in general in the Secrets documentation.

Simpler docker-compose.yml for local development

Running HTTP applications locally with Bref Docker images got simpler (#38). If you used them in docker-compose.yml, you will need to update it.

Before (Bref v1):

services:
    web:
        image: bref/fpm-dev-gateway
        ports:
            - '8000:80'
        volumes:
            - .:/var/task
        depends_on:
            - php
        environment:
            HANDLER: public/index.php
            DOCUMENT_ROOT: public
    app:
        image: bref/php-80-fpm-dev
        volumes:
            - .:/var/task
    console:
        image: bref/php-80
        volumes:
            - .:/var/task
        entrypoint: php

After (Bref v2):

services:
    app:
        image: bref/php-80-fpm-dev
        ports: [ '8000:8000' ]
        volumes:
            - .:/var/task
        environment:
            HANDLER: public/index.php
            DOCUMENT_ROOT: public

The bref/php-XX-fpm-dev images can now run HTTP applications, console commands as well as event-driven functions too. Read more in web app local development.

The bref/fpm-dev-gateway image is no longer needed, and code running in bref/php-XX-fpm-dev now runs in an environment even closer to production.

PHP constructs for AWS CDK support

AWS CDK is a deployment tool that can be used as an alternative to serverless.yml.

Bref 2.0 introduces basic support for the AWS CDK (NodeJS) via PHP constructs.

In case you are not familiar with it, AWS CDK is bit more complex than serverless.yml to deploy serverless apps. These constructs will be useful to those actively looking to use the CDK with Bref.

Rewritten internals

The scripts that build the AWS Lambda runtimes and Docker images have been completely rewritten at least 4 times (just look at the number of commits on the v2 runtimes…). These scripts have also been moved to a separate repository: brefphp/aws-lambda-layers.

The internals are in a much better place now:

• We have way more test coverage.
• We optimized the runtime sizes.
• We optimized the build speed.
• We documented as much as possible.
• We now understand 99% of the build scripts (compared to ~50% before, yes I'm not joking).

Besides making maintenance and contributions much simpler, these changes allowed us to support ARM Lambda functions.

I want to extend a huge thanks to Depot for sponsoring the project and making our Docker builds 10 to 20 times faster!

I know this isn't that exciting, but I had to mention it given the incredible effort put into this over the last 1.5 years.

Thanks

A huge thanks to the 136 Bref contributors, to the community for supporting the project, and to the open-source sponsors:

and many others (shout out to @GensDeConfiance for the one-time sponsorship today).

A huge thank you to all contributors that have helped to maintain v1 and/or with this huge upcoming release, including @deleugpn who rewrote most of the Lambda runtimes, @GrahamCampbell who is helping to keep the runtimes up to date, @t-richard and @afu-dev who have been extremely helpful on the Symfony integration and in the community, @Nyholm who has been maintaining flawlessly the Bref Extra extensions, @mykiwi for keeping PHP up to date, @shouze who is helping on layers and extra extensions, @georgeboot who is contributing amazing ideas on the Laravel integrations, @tillkruss who is pushing what Bref could be for Laravel users, and so many more (@shadowhand, @kevincerro, @Guillaume-Rossignol, @nealio82…).

Thank you all!

That's it!

Hope you enjoy Bref v2!

There is a complete v2 Upgrade Guide that you can follow.

Head to the docs to get started with Bref, or check out the documentation for Laravel or Symfony.

You can also join the community in Slack, post details about your project in Built with Bref, or share your experience online and mention @brefphp on Twitter.

If you enjoy teasers, here is a preview of a redesign coming soon to Bref:

One more thing

I launched the Bref Dashboard ✨ in January. It helps you monitor and debug Bref applications:

And if you need support or help going serverless, check out the Support Plans.

What is Bref and serverless? Get started with Bref

I am checking out the AWS CDK to deploy serverless applications.

But having used Serverless Framework a lot, I find the AWS CDK very noisy and invasive in some of my projects. It doesn't have to be though.

Here are my notes on how to use the AWS CDK in a single file, without polluting your existing projects.

Disclaimer: what I am describing is only interesting is some projects, where the CDK is just "a deployment tool". I understand that some other projects can use the CDK as a framework for the whole app and the "invasiveness" is a feature.

The standard AWS CDK setup

If you have an existing project that you want to deploy using the AWS CDK, tough luck. You can't easily "introduce" the CDK in an existing project: cdk init only works on an empty directory.

But let's move past that. Let's create a new TypeScript CDK project:

cdk init app --language=typescript myapp

Here is what we have:

bin/
    cdk-myapp.ts
lib/
    cdk-myapp-stack.ts
test/
    cdk-myapp.test.ts
cdk.json
README.md
package.json
package-lock.json
tsconfig.json
jest.config.js

With that setup, we can deploy with npx cdk deploy.

Trimming the fat

If we look only at what makes the CDK actually work, we have:

bin/
    cdk-myapp.ts
lib/
    cdk-myapp-stack.ts
cdk.json
tsconfig.json

When cdk deploy runs, it executes the command listed in cdk.json:

{
    "app": "npx ts-node --prefer-ts-exts bin/cdk-myapp.ts"
}

Cool, it uses ts-node to compile TypeScript on the fly. At least we don't have to run a build step and deal with compiled files. It also means we can get rid of tsconfig.json if we don't need it for the rest of our app.

It also means we can change the bin/cdk-myapp.ts file. Let's move it to the root and rename it to cdk.ts:

lib/
    cdk-myapp-stack.ts
cdk.ts
cdk.json

Now let's look at cdk.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkMyappStack } from '../lib/cdk-myapp-stack';
const app = new cdk.App();
new CdkMyappStack(app, 'CdkMyappStack', {
  /* loads of comments here... */
});

And lib/cdk-myapp-stack.ts:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
export class CdkMyappStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // The code that defines your stack goes here
  }
}

Let's inline all the CDK code in cdk.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
class CdkMyappStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        // The code that defines your stack goes here
    }
}
const app = new cdk.App();
new CdkMyappStack(app, 'CdkMyappStack', {
  /* loads of comments here... */
});

Finally, let's inline the stack class:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
const app = new cdk.App();
new class extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        // The code that defines your stack goes here
    }
}(app, 'my-app', {
  /* stack options */
});

Final result

In the end, the AWS CDK only requires 2 files in our project:

cdk.ts
cdk.json

Those are easy to add to an existing project! We also still deploy with npx cdk deploy because cdk.json points to cdk.ts:

{
    "app": "npx ts-node --prefer-ts-exts cdk.ts"
}

And finally, all our infrastructure code is defined in cdk.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
const app = new cdk.App();
new class extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        // The code that defines your stack goes here
    }
}(app, 'my-app', {
  /* stack options */
});

Going further

We could imagine a helper function for simple apps:

#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
cdkStack('my-app', (scope: Construct, id: string, props?: cdk.StackProps) => {
    // The code that defines your stack goes here
});

That cdkStack helper would create the App and the Stack for us.

I'm tempted to it as an open-source package… Let me know what you think of the idea!

Comments

We are excited to announce Serverless Framework Compose: a new feature enabling you to deploy multiple services in one command, in parallel, or ordered by dependencies!

To use this new feature, upgrade the serverless CLI to v3.15 or greater and follow the guide below.

Composing multiple Serverless Framework services

Deploying multiple services in a monorepository is a very common pattern across larger teams. Orchestrating these deployments can become painful.

Let's take the following (simple) example project:

my-app/
  products/
    src/
    serverless.yml
  orders/
    src/
    serverless.yml

Our application contains 2 services. Each service can be deployed separately by running serverless deploy in each folder.

my-app/
  serverless-compose.yml
  products/
    src/
    serverless.yml
  orders/
    src/
    serverless.yml

The new "serverless-compose.yml" configuration file references the existing Serverless Framework projects:

# serverless-compose.yml
services:
  products:
    path: products
  orders:
    path: orders

As you can see above, each existing Serverless Framework service is referenced via a relative path. When running "serverless deploy" in the root directory, Compose will deploy both services ("products" and "orders") in parallel.

While Compose triggers the deployment, each deployment runs as usual, in the exact same conditions as a traditional Serverless Framework deployment. This was designed so that introducing Compose in an existing project would have no side-effect.

# serverless-compose.yml
services:
  products:
    path: products
  orders:
    path: orders
    params:
      productsTableName: ${products.tableName}

Setting dependencies without variables

Using variables allows us to exchange values and order deployments. It is also possible to order deployments without variables via the "dependsOn" feature:

# serverless-compose.yml
services:
  products:
    path: products
  orders:
    path: orders
    dependsOn:
      - products

In the example above, the "products" service will be deployed before "orders". This can be useful in scenarios where a service interacts with another and we want to make sure API or schema changes are deployed in a specific order.

Global commands and service commands

An interesting benefit of grouping multiple services behind a single configuration is that we can run one command in all services at once. For example:

We can also run service-specific commands without having to change directories:

What's next

Serverless Framework now lets you compose services to orchestrate their deployments.

Services can integrate together via variables and are deployed in order based on their dependencies.

There is a lot more to Serverless Framework Compose, head over to the documentation to learn more and get started.

We are also exploring exciting ideas to improve Compose in the near future, like:

• accelerating deployments for services that haven't changed
• deploying only specific services via a filter flag
• multi-region deployments

Get involved in the Serverless Framework repository to discuss these ideas!

A Lambda Function URL is a simple solution to create HTTP endpoints with AWS Lambda. Function URLs are ideal for getting started with AWS Lambda, or for single-function applications like webhooks or APIs built with web frameworks.

While not a complete replacement for API Gateway, which provides more features, a Function URL is a simpler alternative with no extra cost and a much larger timeout limit. We will explore the pros and cons in detail in this article.

Using Lambda Function URLs with Serverless Framework can be done via a single statement:

functions:
  hello:
    handler: handler.hello
    url: true

To use that new feature, upgrade Serverless Framework (v3.12.0 or greater).

Lambda URLs in details

Function URL is a new feature of AWS Lambda that exposes a function as an HTTPS endpoint.

functions:
  hello:
    handler: handler.hello
    url: true

That domain is unique to the Lambda function, and any URI path will invoke the function handler.

The request and response event payloads follow the same format as API Gateway's HTTP API "version 2" payload format. That means it is possible to switch from a Lambda Function URL to API Gateway's HTTP API without any code change.

Here is a simple NodeJS Lambda handler that responds to a Function URL:

exports.handler = async (event) => {
  return {
    statusCode: 200,
    body: 'Hello world!',
  };
};

Endpoint configuration

The endpoint is public by default. It can also be protected by IAM authentication:

functions:
  hello:
    handler: handler.hello
    # Public by default
    url: true
  world:
    handler: handler.hello
    # This URL is protected by IAM authentication
    url:
      authorizer: aws_iam

CORS can also be configured on the Function URL:

functions:
  hello:
    handler: handler.hello
    url:
      # Allow CORS for all requests from any origin
      cors: true
  world:
    handler: handler.hello
    url:
      # Configure CORS in details:
      cors:
        allowCredentials: …
        allowedHeaders: …
        allowedMethods: …
        allowedOrigins: …
        exposedResponseHeaders: …
        maxAge: ….

View the complete list of options for configuring CORS in the HTTP API event documentation.

Unlike API Gateway, Function URLs do not have a maximum execution time: the execution time is limited by the Lambda function's timeout (up to 15 minutes).

Note that Function URLs do not support custom domains. The solution would be to use CloudFront with the Function URL as the origin.

Function URLs vs. API Gateway

Lambda Function URLs let us create an HTTP endpoint for a Lambda function. Let's address the obvious question: how is that different from API Gateway?

Function URLs send all HTTP requests to one Lambda function. This pairs well with backend frameworks like Express or Apollo (Node), Flask (Python), etc. Indeed, these frameworks can handle HTTP routing, authentication, middlewares, and more. Function URLs also work great for single-purpose endpoints like webhooks.

While simple, the downside of this approach is that the whole API is contained in a single Lambda function: this is the "Mono-Lambda" pattern. The deployed package size could be big, debugging could be complex, and permissions are less granular than they could be. You can read more about that topic in The What, Why, and When of Mono-Lambda vs Single Function APIs.

API Gateway filters, processes, and routes requests to different Lambda functions. This removes the need for using backend frameworks. API Gateway can take care of transforming requests and responses, authenticating clients with different mechanisms, supporting custom domains, managing API access via API keys, and much more.

Note that the exact list of API Gateway features varies between v1 (REST API) and v2 (HTTP API): read Choosing between HTTP APIs and REST APIs to learn more.

While API Gateway provides more features, it also comes with added complexity and costs. Beyond the free tier, API Gateway v1 starts at $3.5/million requests and v2 at $1/million requests. Unlike API Gateway, Lambda Function URLs do not add any cost.

Regarding performance, one could expect Function URLs to add less latency than API Gateway. In practice, the difference is minimal. Here is the latency I measured with a quick test:

• Function URL adds 9 to 10ms to the HTTP response time
• API Gateway v2 adds 12 to 15ms to the HTTP response time

The latency mentioned above is added to the Lambda execution duration.

Let's compare all this in detail:

Use cases

Based on the comparison with API Gateway, we can conclude that Lambda Function URLs could be great for scenarios like:

• APIs built with backend frameworks, like ExpressJS/Apollo/Flask…
• Simple webhooks, where API Gateway could be overkill.
• Server-side rendered websites, built with CloudFront + Lambda (skipping API Gateway).
• APIs with a response time longer than 29 seconds (API Gateway's maximum timeout), for example to import or process data, or long-polling scenarios.

It's also worth keeping in mind that Lambda Function events are compatible between Function URLs and API Gateway's HTTP APIs. That means that Function URLs could be a great way to get started with HTTP on Lambda, and later upgrade to API Gateway to use its features.

Those are just ideas, we can't wait to see what you build with it! Check out the Serverless Framework documentation for Lambda Function URLs to get started.

‍

This new major version brings a cleaner and redesigned CLI experience as well as a brand new feature: stage parameters.

On top of that, we've worked on cleaning up the dependencies to make the serverless package 40% lighter and get rid of NPM security warnings.

Upgrading

Before we dive into the new features, let's talk about upgrading from v2 to v3.

As mentioned in the v3 beta announcement, we have revisited many deprecations and breaking changes to make the upgrade to v3 easier.

To upgrade safely, follow these 3 steps:

1. Upgrade to the latest v2 version (e.g. via "npm -g install serverless@2")
2. Fix any deprecation you encounter when deploying with v2
3. When there are no deprecations left, you are safe to upgrade to v3:

$ npm -g install serverless
...
# Check the version
$ serverless --version
Framework Core: 3.0.0

If you installed serverless as a standalone binary, read these instructions instead.

You can read the complete "Upgrading to v3" guide to read about all breaking changes and instructions for specific cases.

It is also possible to use both v2 and v3 in different projects. Read more about this in the v3 upgrade guide.

Finally, if you are looking to get started with Serverless Framework v3, check out our new Getting Started guide.

Stage parameters

Serverless Framework v3 introduces "stage parameters". It allows changing the service configuration based on the current stage.

This is particularly useful when deploying services to multiple environments, like a development/staging environment and a production environment.

Parameters can be defined under the new params key, and can be used via "${param:xxx}" variables:

# serverless.yml
service: myapp
provider:
  name: aws
  environment:
    APP_DOMAIN: ${param:domain}
params:
  prod:
    domain: myapp.com
  dev:
    domain: preview.myapp.com

In the example above, the "${param:domain}" variable will resolve to:

• myapp.com for the "prod" stage
• preview.myapp.com for the "dev" stage

It is also possible to define default parameter values via the default key. These values will apply to all the other stages:

params:
  default:
    domain: ${sls:stage}.preview.myapp.com
  prod:
    domain: myapp.com
  dev:
    domain: preview.myapp.com

Note that this new feature is born out of a common pattern: using the "custom" section with nested variables. For example:

provider:
  environment:
        APP_DOMAIN: ${self:custom.stages.${sls:stage}.domain}

If you are already using this pattern, we hope the new stage parameters can help simplify your configuration and make it more maintainable!

Finally, thanks to the optional integration with Serverless Dashboard, you can also store secret values securely and retrieve them via the "${param:my-secret}" variable syntax.

Learn everything about stage parameters in the Parameters documentation.

Redesigned CLI experience

Over the years, Serverless Framework has become the most advanced tool to create and deploy serverless applications.

This comes with a challenge: maintaining a clean and simple experience for users. As deprecations, plugins, and cloud resources multiply, so does the noisiness of the CLI.

Serverless Framework v3 is the framework you know and love, with a reimagined interface. We started from scratch and asked ourselves: "as a user, what do I need to know?" at each step of each command. The new design:

• focuses on actionable information
• removes all extra noise
• is easier on the eyes with minimalistic colors and styles

Below is a preview of the new design with the most common commands.

Deploy

The "serverless deploy" command now features a clean and minimal output. Here is a comparison of v2 (left) and v3 (right):

Verbose output

Serverless Framework v3 now supports the standard "--verbose" flag to output more details.

That option can be particularly useful in CI/CD, for example to get a detailed history of the CloudFormation deployment:

Clear errors

The error screen has been improved: any failure is now clearly signaled, secondary information is toned down and the error message is printed last, to appear right above the command prompt.

On top of that, CloudFormation errors now contain more details about resources and their statuses:

Cleaner logs

The "serverless logs" command now features a cleaner and lighter output, that brings more focus on the content of the logs.

New serverless onboarding

Serverless Framework can now interactively set up new projects: just run "serverless" in an empty directory and follow the prompt.

The interactive setup also lets you set up the Serverless Dashboard in a few steps. Run "serverless" in an existing project and get access to premium monitoring, AWS account management, parameters, and more.

Upgrading plugins

We have worked hard at helping plugins be ready for Serverless Framework v3.

That being said, given the size of the ecosystem, we have identified 3 categories of plugins:

1. Plugins that are compatible with v3 and integrate with the new CLI design.
2. Plugins that are compatible with v3.
3. Plugins that are not compatible with v3 yet.

Fortunately, most of the plugins are in categories 1 or 2. Some plugins might not integrate fully with the new design yet, but they should work fine.

We want to help developers take their plugins to the next level! This is why v3 comes with:

If you need help updating your plugin, jump in the GitHub discussion and let us know.

Architectural Guidance & Commercial Support

While Serverless Framework makes it easy to create radically efficient cloud apps, nothing beats the confidence you’ll gain from working with the team that built the Serverless Framework.

Serverless Inc's support offering includes architectural reviews to highlight improvements and standards you can leverage to scale projects and teams.  Our support offering also features a private Slack channel where you can interact directly with our team and discuss plugins, the Framework and serverless architectures on AWS.  Consider us your partner in serverless success. 

Learn more about Serverless Premium Support.

We are happy to say that Serverless Framework supports that new feature immediately via the new functionResponseType option:

functions:  
  worker:
    handler: worker.handler    
    events:      
      - sqs:          
          arn: <ARN of the SQS queue>        
          batchSize: 10
          functionResponseType: ReportBatchItemFailures

Upgrade to v2.67.0 or greater to start using it. In case you are trying out Serverless Framework v3 beta, make sure to update the v3 beta with "npm -g i serverless@pre-3".

The challenge with batch processing

Up until now, it was already possible to process SQS messages in batch with AWS Lambda, but it had limitations. Here is an example:

# serverless.yml
service: my-app
provider:  
  name: aws
functions:  
  worker:    
    handler: worker.handler    
    events:      
      - sqs:
          arn: <ARN of the SQS queue>
          batchSize: 10

Our worker.js file would contain a handler like this:

# worker.js
exports.handler = async function(event) {  
  event.Records.forEach(record => {    
    const bodyData = JSON.parse(record.body);    
    console.log(`Processing ${record.messageId}`);    
    // ...  
  });
}

The challenge is dealing with errors: if any SQS record fails to be processed (i.e. the code handling it throws an exception), then the Lambda function execution fails and the whole batch of messages is considered failed.

That also means that if one message fails, the whole batch is retried again, including messages in the batch that were successfully processed.

The SQS + Lambda integration provided no practical way to deal with those scenarios. Indeed, catching errors that occurred when processing would mean marking the whole batch of messages as "successfully processed", which was also wrong.

AWS Lambda partial batch responses

The new "partial batch response" feature lets us signal, from AWS Lambda, which SQS messages have been successfully processed, and which have failed.

Let's fix the previous example:

functions:  
  worker:    
    handler: worker.handler    
    events:      
      - sqs:          
          arn: <ARN of the SQS queue>
          batchSize: 10
          functionResponseType: ReportBatchItemFailures

In worker.js, we can now catch errors and report failed messages in the return value of our function:

# worker.js
exports.handler = async function(event) {  
  const failedMessageIds = [];  
  event.Records.forEach(record => {    
    try {      
      const bodyData = JSON.parse(record.body);      
      console.log(`Processing ${record.messageId}`);      
      // ...    
    } catch (e) {      
        failedMessageIds.push(record.messageId);    
    }  
  });
  return {    
    batchItemFailures: failedMessageIds.map(id => {      
      return {        
        itemIdentifier: id      
      }    
    })  
  }
}

One downside of this approach is that we can no longer monitor the "error rate" metric of AWS Lambda to monitor errors. Indeed, the Lambda function will execute successfully in all cases (because we catch errors).

But the upside is that only failed messages will be retried by SQS.

If you want to learn more, check out the official AWS documentation.

Today, we are very excited to announce the availability of Serverless Framework v3 beta! The 3.0 release, planned for Q1 2022, includes a cleaner and redesigned CLI experience that prioritizes actionable information.

The v3 beta lets you experience the new CLI design right now and helps prepare for the upgrade.

You can install v3 beta globally via NPM:

$ npm -g i serverless@pre-3
...
# Check the version
$ serverless --version
Framework Core: 3.0.0-pre.xxxx

Or install in a specific project only:‍

$ npm i serverless@pre-3
# Run serverless using npx to avoid conflicts with the global v2 CLI
$ npx serverless --version
Framework Core: 3.0.0-pre.xxxx

Raising the bar on developer experience

Over the years, Serverless Framework has become the most advanced tool to create and deploy serverless applications.

This comes with a challenge: maintaining a clean and simple experience for users. As deprecations, plugins, and cloud resources multiply, so does the noisiness of the CLI.

Serverless Framework v3 is the framework you know and love, with a reimagined interface. We started from scratch and asked ourselves: "as a user, what do I need to know?" at each step of each command. The new design:

• focuses on actionable information
• removes all extra noise
• is easier on the eyes with minimalistic colors and styles

Below is a preview of the new design with the most common commands.

Deploy

The "serverless deploy" command now features a clean and minimal output. Here is a comparison of v2 (left) and v3 (right):

Verbose output

Serverless Framework v3 now supports the standard "--verbose" flag to output more details.

That option can be particularly useful in CI/CD, for example to get a detailed history of the CloudFormation deployment:

Clear errors

The error screen has been improved: any failure is now clearly signaled, secondary information is toned down and the error message is printed last, to appear right above the command prompt.

On top of that, CloudFormation errors now contain more details about resources and their statuses:

Cleaner logs

The "serverless logs" command now features a cleaner and lighter output that brings more focus on the content of the logs.

New Serverless onboarding

Serverless Framework can now interactively set up new projects: just run "serverless" in an empty directory and follow the prompt.

The interactive setup also lets you set up the Serverless Dashboard in a few steps. Run serverless in an existing project and get access to premium monitoring, AWS account management, parameters, and more.

And for those of you that followed the announcement of Serverless Console last week: yes, Serverless Framework will natively integrate with Serverless Console as soon as it is available!

Join the discussion

If you have any feedback, we would love to hear it. Join the GitHub discussion and let us know what you like or how we can improve it!

Easy to upgrade

With several hundred contributors and versions, Serverless Framework v2 has accumulated improvements, but also deprecations, at a great pace.

While preparing for v3, we have revisited the list of breaking changes to make the upgrade to v3 easier.

We invite all users to upgrade to the latest v2 version: you may notice that some deprecations are gone. Most notably:

• package.exclude/package.include syntax will continue to work in v3
• provider.iamRoleStatements (and related options) will continue to work in v3
• the legacy Lambda version hashing mechanisms will continue to be supported in v3
• and more (read the complete list)

You can read more about all these features we've decided to continue supporting in v3 in the "Upgrading guide".

If the list above doesn't mean anything to you, that's OK. Here is all that you need to know:

1. Upgrade to the latest v2 version
2. If you see deprecations, take the time to fix them
3. If you don't see deprecations, your project will work with v3  ?

If you are curious, you can also read the complete "Upgrading to v3" guide.

We estimate that around 70% of serverless projects should be able to upgrade to v3 without breaking change. If you need help upgrading, feel free to open a discussion on GitHub.

Calling all plugin developers

Serverless Framework wouldn't be the same without its incredible plugin ecosystem. As you may expect, the new CLI design can also benefit plugins!

This is a call to all plugin developers: we want to help you build amazing plugins. As such, v3 comes with:

• New documentation for "Creating plugins", which is complete and up-to-date.
• Brand new plugin API for writing awesome CLI output.

The new plugin API lets plugins integrate with the new design via helpers:

• Log with sensible log levels and formatting (including --verbose and --debug)
• Create interactive progress
• Add new sections to serverless info output
• And more

You can read the documentation about these new Plugin APIs.

We have also started a community discussion with all plugin authors: join the GitHub discussion and let us know if you have feedback, questions or if you need help.

Architectural Guidance & Commercial Support

While Serverless Framework makes it easy to create radically efficient cloud apps, nothing beats the confidence you’ll gain from working with the team that built the Serverless Framework.

Talk to our experts and members of the product team to get architectural guidance as you’re designing and building your app, and get ongoing support as you go into production via email or Slack.  Learn more about serverless Enterprise Support

21 January 2021 - Matthieu Napoli

Back in January 2019, I created the Null company. My goal: help developers work better, through products, training and consulting. My secondary goal: make some of that work open-source.

Just like my financial report for 2019, here's my yearly update for 2020 on funding open-source work.

Open source funding

Of all the revenue earned by Null, here is the breakdown for 2020:

When I compare that with 2019, I am really happy to have grown the "open-source work" and "products" parts:

Development services

Most of the revenue is still related to development services, i.e. freelancing and consulting.

Serverless services

I have a specific category for serverless-related consulting and training.

This year I launched Bref Enterprise and it has been working out quite well.

Products

In 2020, I launched 2 products:

• Serverless Visually Explained, an online course for serverless
• 7777, a utility to connect easily to AWS RDS databases, built with Marco

I absolutely loved doing that, and I have more things planned for 2021 ?

Open source work

Okay, here we are:

In 2020, my company earned around 13600€ directly related to open-source work.

Last year, it was 2000€. It is still a small share of the total revenue (and consequently, the money I take home every month), but it is definitely growing!

After accounting for expenses and taxes, it represents about 430€/month net income. That is not close to a real salary, but it is no longer negligible.

Last year was a bit premature to draw any conclusion. But looking back now, I can say that even with a widely used open-source project and an active online presence, living directly and only from open-source work (sponsoring) as an individual is extremely hard.

To be clear, I am more than happy with my situation. What I am getting at is more of a "warning" for others who aspire to live of open-source: don't expect to copy Evan You or Caleb Porzio easily.

2021 goals

I want to continue the current trend and accelerate it:

• continue working on open-source and grow the sponsoring (want to help? ❤️)
• continue delivering products and services that make developers awesome at their jobs!

See you next year!

1 December 2020 - Matthieu Napoli

AWS Lambda now supports running Docker containers!

This is big news for the PHP community: while Bref runtimes provide out-of-the-box support for PHP on AWS Lambda, we can now run any container image ?

Let's check that out!

Lambda runtimes vs. containers

Here are the different ways to run code on AWS Lambda:

1. Using an official Lambda runtime:
   • runs on Amazon Linux
   • supports only a few languages, like JavaScript, Java, Python, etc. but not PHP
2. Using a custom Lambda runtime:
   • runs on Amazon Linux
   • the custom runtime is imported via a "Lambda layer" (i.e. a zip file unzipped in /opt)
   • the custom runtime is built upon the official Lambda "Runtime API"
   • can support any language we want
   • This is what Bref currently provides: custom runtimes to run PHP on Lambda
3. Using a Docker image (new)
   • runs on the Linux version you want
   • with the programs and libraries you want
   • can support any language
   • must be made compatible with the official Lambda "Runtime API" (like custom runtimes)

The last point is important: Lambda is still not made for running daemons and web servers, like Apache, Nginx, etc (which doesn't mean we can't run web applications on Lambda).

This is a good thing: Lambda's execution model is event-driven, making it extremely scalable and cost-efficient. We can use any container image, but we still need to "bridge it" with the Lambda Runtime API.

The good news is that Bref provides the "runtime client" for PHP.

Docker support in Bref

One of the perks of being an AWS Serverless Hero is that I get early access to new features :) I was able to test things beforehand and make sure that Bref was compatible.

On top of the PHP Lambda runtimes, Bref provides Docker images that mirror the Lambda environment:

• bref/php-80
• bref/php-74
• bref/php-80-fpm
• bref/php-74-fpm
• etc.

Using those images as a base for your application is the fastest way to get started: the runtime client is already included in these images. That means that they work out of the box on Lambda. Check out this Dockerfile for example:

FROM bref/php-80
# Copy our code in the container
COPY . /var/task
# Start Bref's runtime client
CMD _HANDLER=function.php /opt/bootstrap

/opt/bootstrap is a file provided by Bref that takes care of invoking your code in the "Lambda way".

In the coming weeks, we will explore the possibility of using any Docker image, including the official PHP Docker images.

What about performances/execution time/deployment size…?

The 15-minutes maximum execution time still applies with Docker containers.

According to AWS, performances should be similar to native runtimes. In my tests, warm invocations were as fast as usual. Cold starts on small Docker images were as fast as native runtimes. I've seen slower cold starts (1-2 seconds) on larger images. We'll be aggregating feedback over the next weeks because it's definitely too early to tell.

The good news is that while Lambda functions are limited to 250MB, containers can be up to 10GB. That will certainly help when deploying large monoliths to Lambda.

One limitation to keep in mind is that after 14 days of inactivity, a container-based Lambda function will switch to an "INACTIVE" state. In that state, it takes a few seconds for the Lambda to re-activate, which causes a very slow cold start. I don't expect this to be a problem on large projects, but keep that in mind.

Which one to choose?

You may have understood now that "Container support" in Lambda mostly means that you can use Docker to package your application.

It does not involve radical architecture changes for us; our code runs the same way in Docker as in native runtimes: in an event-driven way.

It is a bit early to be definitive about this, but here is my recommendation as of now:

• use Lambda runtimes by default:
  • these are simpler to use and well documented
  • no Dockerfile to create (you don't even need to know Docker)
  • with Bref, Lambda runtimes are automatically versioned in sync with the Bref version, making it much easier to keep up to date and upgrade
  • 3rd party tools support Lambda runtimes, most of them don't support containers yet
• use Docker if you have a specific reason for that:
  • you need to deploy an application larger than 250MB
  • you want control over the Linux image
  • you want to include an exotic PHP extension
  • you want to include specific system libraries or programs

Docker provides more freedom to support more use cases on Lambda. However, as far as I can tell, it is not a de-facto replacement for AWS runtimes.

Example: deploying your first PHP container using Bref

Warning: this example is advanced. If you don't know Bref, get started via its documentation. The example assumes you know about the current Bref runtimes.

Since the feature is brand new, you will need to set up everything manually. This is not representative of how containers will be supported eventually (we need to wait for container support in the Serverless framework). This example is just so that early-adopters can have a bit of fun.

Creating the Docker image

To deploy a web application using PHP-FPM (the FPM runtime), we can use a Dockerfile like this:

# Uses PHP 8.0, feel free to use php-74-fpm if you prefer
FROM bref/php-80-fpm
COPY . /var/task
CMD _HANDLER=index.php /opt/bootstrap

index.php is the front controller of the application. For the example, let's keep it simple:

<?php
echo 'Hello world!';

To deploy an event-driven function instead (the Function runtime), we can use a Dockerfile like this:

# Uses PHP 8.0, feel free to use php-74 if you prefer
FROM bref/php-80
COPY . /var/task
CMD _HANDLER=function.php /opt/bootstrap

function.php is the function to invoke:

<?php
return function () {
    return 'Hello world!';
};

As you can see, thanks to the Docker images provided by Bref it's extremely simple!

Deploying

Let's deploy that:

1. Create a Docker image on AWS:

   • open the ECR Console
   • click "Create repository"
   • set the name of your image (e.g. app) and validate

2. Push your container image to AWS:

   • in the ECR Console, open the "repository" (or image) that you created
   • click "View push commands"
   • run the commands that are displayed

3. Create a Lambda function:

   • open the Lambda Console
   • click "Create function"
   • select "Container image"
   • set a function name (e.g. app)
   • set the container image by clicking "Browse Images"
   • click "Create function"

The function is now ready to be invoked.

If you are setting up a web application (using the "FPM" runtime), you will need to set up API Gateway. The tutorial stops here: as I said it is meant to illustrate what's different with containers and guide advanced users (who know how to set up API Gateway).

In the next weeks Bref will natively support containers, so all those steps will be irrelevant.

Conclusion

That's it! If you have any question, ask me at @matthieunapoli.

Don't forget to read the official announcement from AWS to learn more.

If you want to get started, check out Bref and Serverless Visually Explained.

If you need professional support, get in touch here.

Over the years, as the Bref community grew, as AWS features landed, and as contributors worked, creating serverless PHP applications has become a reality. Bref 0.2 has seen 37 releases. Bref 0.5 has 33. In total, we've released 89 versions since the project started.

Needless to say, Bref is stable and has been for a long time.

Indeed, Bref runs more than 1 billion requests and jobs every month!

"Serverless PHP" is no longer a niche, and running scalable PHP applications has never been simpler.

To celebrate, we're finally releasing Bref 1.0!

1 billion executions per month

Thanks to Bref ping, we have an anonymous estimate of Bref's usage, i.e. the number of monthly invocations (HTTP requests, worker jobs, etc.) across all users.

It illustrates clearly that Bref is getting traction and is used in production, at scale:

Since we passed 1 billion monthly invocations, I want to celebrate with you this fantastic milestone and thank you for being a part of Bref's community.

Bref 1.0

Enough with the fun stuff, what's new?

• Lighter Lambda runtimes by moving some extensions to an extra layer instead of embedding them by default.
• Migration to Amazon Linux 2, which will be required by AWS Lambda at the end of the year.
• Huge documentation improvements: a search bar, a reorganization, documentation of typed handlers, updates all over the place, and a better onboarding experience.
• New vendor/bin/bref local command to run functions locally.
• New projects will use API Gateway v2 because they are simpler, cheaper, and faster.

What did we break? Nothing major, the upgrade should be smooth. Here are the details:

• You need to edit serverless.yml to switch to provided.al2.
• PHP 7.2 is no longer supported. PHP 7.3 still is.
• GD, Imagick, Redis, MongoDB extensions are no longer installed by default; you need to include them via Bref extra extensions.
• PHP is now compiled without ZTS (thread safety).
• Removed the deprecated lambda() function.
• Removed deprecated vendor/bin/bref commands:
  • vendor/bin/bref invoke is replaced by serverless invoke.
  • vendor/bin/bref deployment has become useless.
  • vendor/bin/bref bref.dev has become useless.

Sébastien Houzé has worked to make the Bref runtimes (AWS Lambda layers) lighter. One solution we went with was to remove built-in extensions that were not commonly used in most applications:

• GD
• Imagick
• Redis
• MongoDB

By removing them, we make your Lambda lighter, leaving more room for your code and improving cold start performances.

Of course, we don't want to simply remove those extensions. That's why they've been moved to separate layers in the Bref extra extensions repository, carefully maintained by Tobias Nyholm.

For example, to install imagick (via serverless.yml):

functions:
    myapp:
        handler: public/index.php
        layers:
            - ${bref:layer.php-74-fpm}
            - ${bref:extra.imagick-php-74}

Amazon Linux 2

AWS introduced a new base Linux image for Lambda. We need to upgrade our applications to use it.

Thanks again to Sébastien Houzé, Bref 1.0 will work with Amazon Linux 2. Upgrading is as simple as changing this line in serverless.yml:

provider:
    name: aws
-    runtime: provided
+    runtime: provided.al2

Remember to upgrade to Bref 1.0 first!

Documentation improvements

"Documentation improvements" sounds boring, but I'm extremely happy with this for many reasons:

• onboarding new users now makes much more sense,
• current users can troubleshoot and improve their knowledge of Bref and serverless,
• expert users can now dig in deeper, especially with "typed handlers".

Onboarding for new users

New users now start with the FPM runtime.

Don't worry about FaaS and functions… Start by running PHP as usual on a cheap and scalable host. Then, once you've had your first success, you can look into the "Function runtime" and its power.

Check out the "First steps" guide and see how simple it is!.

Clarity for current users

A common source of errors and confusion was the two runtimes: "FPM" and "Function".

To solve that, we clarified the wording and the structure of the documentation:

• Bref for web apps lets you run Laravel, Symfony, etc. on Lambda like on any server, using PHP-FPM (aka the "FPM" runtime),
• Bref for event-driven functions lets you handle native AWS Lambda events (aka the "Function" runtime).

Bref for web apps is the default runtime (see the "onboarding" section above), so if you're not sure: go with this one.

More for expert users

Creating event-driven functions to handle Lambda events can be done using anonymous functions:

return function ($event) {
    return 'Hello ' . $event['name'];
};

But for months now, you can also create fully typed handler classes instead. This is now documented in the Typed PHP Lambda handlers page.

You can now write asynchronous workers to process SQS queues, process uploaded files via S3 events, create decoupled microservices using EventBridge, etc. Well, you already could, but now it will be a bit easier.

Self-promotion time: I've helped enterprises refactor their microservice architectures using Lambda and SQS/EventBridge. If you are interested, get in touch to work together. You can also check out Serverless Visually Explained, it contains examples for those use cases. </self-promo>

Finally, the BREF_LOOP_MAX variable is now documented, for those ready to keep the PHP process alive between events to accelerate their workers.

The website has been slightly redesigned:

• a top menu bar has been introduced, to easily navigate between the home, documentation, news and GitHub repository,
• a search field has been introduced ?

Thanks to Algolia DocSearch for providing the search engine!

Bref provides Docker images to run web apps locally (e.g. Laravel or Symfony) and that works well.

However, when creating event-driven functions, running them locally via serverless and Docker was slow and unstable.

Bref 1.0 introduces a new command:

$ vendor/bin/bref local hello
Hello world
# With JSON event data
$ vendor/bin/bref local hello '{"name": "Jane"}'
Hello Jane

What's awesome is that you decide if you want to use Docker or not (with the Bref images). You can skip Docker for best performances and ease of use, or use Docker to run in an environment that replicates production.

Read more in the documentation: Local development for functions.

API Gateway v2

API Gateway offers two versions:

• v1's REST API
• v2's HTTP API

Bref switches from REST API to HTTP API. Why:

• HTTP API has lower latency (it's slightly faster)
• HTTP API is 70% cheaper
• HTTP API is simpler to configure, both in the AWS console and serverless.yml
• HTTP API do not have the /dev/ prefix in URLs!

The last point was a huge pain in the onboarding experience: links and redirects were broken in Symfony and Laravel. This will no longer be the case ?

HTTP API offers mostly advantages, which is why Bref's documentation and templates have changed to use the new version.

To update your existing projects, it's only 2 lines to change in serverless.yml:

functions:
    website:
        # ...
        events:
-            - http: 'ANY /'
-            - http: 'ANY /{proxy+}'
+            - httpApi: '*'

However don't do this blindly: this will delete your REST API to create a new HTTP API, which will break custom domains you may have set up. Instead, you can deploy the same Lambda with both v1 and v2, set up your domain on v2, and then delete v1:

functions:
    website:
        # ...
        events:
            - http: 'ANY /'
            - http: 'ANY /{proxy+}'
            - httpApi: '*'

Upgrading breaking changes

The deprecated lambda() function has been removed. If you were defining Lambda handlers with it, remove the call to the function. For example:

// Before
return lambda(function (array $event) {
    return 'Hello ' . $event['name'];
});
// After
return function (array $event) {
    return 'Hello ' . $event['name'];
};

The deprecated $_SERVER['LAMBDA_CONTEXT'] has been removed in favor of $_SERVER['LAMBDA_REQUEST_CONTEXT'] (same content).

Thanks

A huge thanks to the 100 Bref contributors, to the community for supporting the project, and to those sponsoring the development:

• Null
• GeckoEngage
• Laravel
• JetBrains
• Twilio

and many others. Thank you all!

That's it!

Hope you enjoy it! And stay tuned for the AWS re:Invent conference next week!

You can also join the community in Slack, post details about your project in Built with Bref or share your experience online.

What is Bref and serverless? Get started with Bref

When running asynchronous tasks on AWS, it often makes sense to send failed tasks to an SQS "Dead Letter Queue".

A dead letter queue is simply a standard SQS queue that we create to store those failed tasks.

All that is great, but what do we do with messages in that special queue? It doesn't make sense to process them again, since we know our code fails at that. What we want instead is get alerted, so that we can inspect those messages and debug the error.

It is possible to set up email alerts whenever there are messages in the queue. That is doable via a CloudWatch alarm on the "queue size" metric.

As soon as the queue is not empty, the alarm triggers and sends an email via SNS.

It's not easy to find a complete CloudFormation example online, so here it one that I wrote while working on 7777:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
    Queue:
        Type: AWS::SQS::Queue
        Properties:
            RedrivePolicy:
                # Jobs will be retried 3 times
                maxReceiveCount: 3
                # And if they still fail, they'll got to the dead letter queue
                deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
    # Failed jobs from the Queue above will end up in this queue
    # (that's the dead letter queue)
    DeadLetterQueue:
        Type: AWS::SQS::Queue
    DlqAlarm:
        Type: AWS::CloudWatch::Alarm
        Properties:
            AlarmName: My-DLQ
            AlarmDescription: 'There are failed messages in the dead letter queue.'
            Namespace: AWS/SQS
            MetricName: ApproximateNumberOfMessagesVisible
            Dimensions:
                -   Name: QueueName
                    Value: !GetAtt DeadLetterQueue.QueueName
            Statistic: Sum
            Period: 60
            EvaluationPeriods: 1
            Threshold: 0
            ComparisonOperator: GreaterThanThreshold
            AlarmActions:
                - !Ref DlqAlarmEmail
    DlqAlarmEmail:
        Type: AWS::SNS::Topic
        Properties:
            Subscription:
                -   Endpoint: me@example.com
                    Protocol: email

3 March 2020 - Matthieu Napoli

Back in January 2019, I created the Null company. My goal: help developers work better, through products, training and consulting. My secondary goal: make some of that work open-source.

Last year, I was able to work on Bref and its documentation, various open-source projects, articles in this blog, the serverless PHP newsletter, as well as share through conferences, meetups and podcasts (a first for me!).

That was awesome! Now I want to look back at 2019 and see how I am doing with funding that open-source work.

Open source funding

Of all the revenue earned by Null, here is the breakdown for 2019:

Development services

Most of the revenue was related to development services, aka freelance work and code audits.

Serverless services

Next, I have a specific category for serverless-related services. That includes:

• serverless consulting
• training (hint hint)
• serverless or Bref support

I separate this category from the first because this is a category I want to grow. I am happy with this result for the first year, especially since it is growing a lot lately. As of March 2020, I have already surpassed 2019 for this specific category.

Open source work

Okay, here we are:

In 2019, Null earned around 2000€ directly related to open-source work.

Needless to say, it is a small share of the total revenue. After taxes and company expenses, I get around 750€ as net income from that.

That year, I spent around 50 days working on open-source (conference talks not included). I also hired a friend to work on Bref for a total of 10 days. While 2000€ is probably most than most open-source developers, it would not cover 60 days of work.

That revenue comes from different sources:

• Tidelift
• more recently: GitHub sponsors

And finally, bref/symfony-messenger-sqs should have been my first open-source package whose development was paid by a company. However that bill was never paid. Ironic that my only unpaid bill is for open-source work ;)

Obviously, this is a category of revenue that I want to grow as well. In 2020, I can already see the trend improving as I am finding more sponsors. I still expect most of the open-source work to be paid for by other activities (i.e. sponsored by Null).

2020 goals

2020 started pretty strong: more open-source sponsors, brand new services finding their first clients (?), a growing interest in serverless… The future looks bright!

As for the goals:

• increase the "serverless services" category, because I love working with it: this is a technology that can transform, for the better, how we work
• work even more on open-source, which means finding more sponsors (thanks CraftCMS and JetBrains for leading the way!) and trying out new formats like sponsorware
  • as a side goal, I aim to sponsor or hire other developers to work on open-source eventually
  • want to help? your company can become a sponsor in a few clicks ❤️
• create paid products: this is new for me, and it could help a lot to fund the open-source work, stay tuned…

There is so much more to say about open-source funding. But I would rather publish that article as it is than store it somewhere and never finish it.

See you next year!

PS: you can read also Open-Source Sustainability Report (2019), a similar article by Christian Lück.

21 November 2019 - Matthieu Napoli

We are a few years in the past. I am sitting in a meeting with a very toxic colleague. As the discussion about recruitment and team organization gets heated, I can't help but lash out something like: "Our team has a culture and values that does not fit mission X."

He can't contain a snarky laugh. What kind of values?

As the discussion about recruitment and team organization gets heated, I end up bringing up some of the core values of my team: humility, kindness (as in "bienveillance" in french), tolerance.

He can't help but lash out with a snarky laugh: "Tolerance,

https://extranewsfeed.com/tolerance-is-not-a-moral-precept-1af7007d6376

21 November 2019 - Matthieu Napoli

This article is a compilation of answers to the most common questions I get about serverless PHP applications built with Bref.

How is it serverless when there are still servers?

Yes, I do get that question :) There are still servers involved, we just don't manage them. We don't even care about them. Servers are out of the equation, which is why the approach is called serverless.

How to run serverless PHP applications locally?

For HTTP applications, like websites or APIs, Bref provides Docker images to run the stack locally. The Bref documentation provides a docker-compose.yml example that you can paste into your project.

For PHP functions, you can run them locally via the serverless invoke local command.

How to handle downtime from cloud providers?

Indeed, when running applications on cloud providers like AWS, we are exposed to their downtime if they have a major incident. Those are rare, but they do happen once in a while.

However, those large cloud providers often have less downtime than if we were running our application ourselves. It is important to keep in mind that zero downtime does not exist. The question is about minimizing risk. Would you, or your team, be able to achieve better uptime than AWS, Google, or Microsoft? And if so, is it worth the cost?

How do costs scale in practice? Is Lambda still cheap with higher traffic?

Like all things, it depends. Smaller applications are very cheap on Lambda because their costs start at $0 (unlike a traditional server which has a fixed price). Larger applications often have more continuous traffic, so cost savings are less guaranteed. However, there are cases or large websites saving money thanks to AWS Lambda (even when migrating from an autoscaling infrastructure).

I encourage you to have a look at cost-calculator.bref.sh, it will help you get an idea. Be careful though: at a larger scale, using AWS ALB instead of API Gateway is much cheaper, and the calculator doesn't take that into account at the moment.

How to deal with extra costs related to DDoS attacks or traffic peaks?

AWS Lambda and related services are "pay per use". A traffic peak will result in higher costs.

There are a few ways to mitigate them. First of all, you can set billing alarms on your AWS account. These alarms monitor how much you spent this month, as well as how much you will spend by the end of the month.

Secondly, you can protect your applications from DDoS attacks using services like Cloudflare (free) or AWS Shield (expensive).

Finally, you can limit the cost impact of traffic spikes by setting limits on your application. For example, you can set the maximum number of instances your Lambda is allowed to scale up to. By default, the limit is 1000. By setting it to a lower number, like 5, you can limit the impact of a peak (at the risk of dropping requests). With a concurrency of 5, an attacker flooding a PHP website for 24 hours would cost around $7, leaving plenty of time to take action before reaching hundreds of dollars.

On top of AWS Lambda's concurrency configuration, API Gateway also has a "throttling" system. That lets you limit the number of requests per second your website or API will accept.

Everything I said above mostly applies to malicious attackers. Regular traffic spikes will usually not quadruple your AWS bill. For example, let's say you get 10 times the usual traffic on a specific day. That represents a 30% increase in your monthly bill (because all the other days in the month had the regular traffic). When this happens on my blog or on externals.io: the $0.05 monthly bill can turn into $0.08… Nothing major. Of course, on larger websites, a 30% increase can be more expensive: you have to anticipate them in your budget. You can think of it like this: instead of paying $300/month for servers, you go with a serverless architecture that costs $50/month most of the time, and $300/month on very busy months.

Conclusion

It is interesting to see many questions are related to billing. Moving to variable pricing is intimidating. However, all "going to serverless" experiences I've seen or heard about end up with cost savings. That doesn't mean that all projects will save on costs obviously.

If you are interested, check out bref.sh. I can also work with you to see if serverless can be a good fit for your project, check it out.

11 August 2019 - Matthieu Napoli

This article is part of a series of case studies of serverless PHP applications built with Bref on AWS Lambda. You can read more of those case studies here.

This case study is about migrating the externals.io website to AWS Lambda using Bref. This is the first time I write about a serverless PHP website with a MySQL database. I hope it will interest a few people ;)

Externals is a read-only interface for PHP's #internals mailing list. This mailing list is where PHP core developers discuss the future of the language.

The website (which is on GitHub) has been migrated from a classic cloud hosting to AWS Lambda in July. I have now a little bit of data to show you and draw some lessons.

Note that I will not be introducing what AWS Lambda or Bref are in this article: you can read this page to learn more.

Stack

The application is a traditional LAMP stack (Linux, Apache, MySQL, PHP) with the addition of a cron. The cron fetches the last emails of the mailing list and saves them to the database. The application used to run on Platfom.sh via a plan graciously sponsored by the company:

The new architecture is very similar. The main difference is that instead of running on one server with traditional services, the application now runs using multiple AWS services:

As you can see, while some AWS services have been added to the mix, nothing major has changed. And if you have read the Bref documentation at some point, the schema may look familiar: this architecture is documented in the "Serverless Websites" documentation on bref.sh.

To be honest, the main complexity in the stack is related to CloudFront. CloudFront is the AWS CDN, and it helps us serve assets (CSS, JS) from S3. It works well, but setting it up requires around 60 lines of YAML boilerplate. There is definitely room for improvement, and I have a few ideas in mind.

In the last section of this article, I detail everything I had to change in the code for the migration. But before that, let's talk performances and pricing.

Performances

Here is the HTTP response time from API Gateway:

As you can see, the median response time is 55ms. Pretty good for a website!

Note: this is the full HTTP response time of API Gateway, not just the Lambda execution time (PHP execution time). PHP's execution time is 15-20ms less than API Gateway's response time, i.e. 35ms on average.

What about cold starts and slow pages? Here are more numbers:

• 90% of the requests are below 100ms
• 1% of the requests are over 500ms
• 0.5% of all requests are cold starts

Having 1% of requests response in 500ms to 3s is not ideal. However it is perfectly acceptable here. Especially when you consider the full page loading time for users (with assets and JS execution), which is often more than 10 seconds on most sites.

Costs

Externals was previously hosted on Platform.sh, running on the smallest $50/month plan. Though the plan was offered by Platform.sh to support externals.io, let's keep in mind the official price for a fair comparison.

Here is the monthly cost for the serverless version on AWS:

(the free tier has been ignored, I actually pay even less than that)

As we can see, most of the cost comes from the database (which is the smallest available, and doesn't break a sweat). The rest of the website totals to $1.67/month.

The traffic received each month by the website is:

• 5200 visitors (Google Analytics)
• 100,000 HTTP requests going to PHP

And in case you wonder who pays the bill of externals.io now: null.

What about spikes?

Something I hear very often regarding the serverless billing system is:

What if there's a huge traffic spike? I'll end up paying a lot of money.

That's a real question. Let's try to give some numbers. I don't have a groundbreaking traffic spike to illustrate, but I do have a little one:

In the last few days, there has been a lot of activity on the site: internal developers are talking about major changes to the language. This has been linked over Twitter, Reddit, Facebook and Hacker News and that brought some traffic.

This caused a spike that costs me $0.01 on Lambda (went from 1c per day to 2.2c). Roughly speaking, the total extra cost of that spike should be around $0.04.

This example doesn't really answer the question of a DDoS (which can be prevented using CloudFlare or AWS WAF) and large scale traffic spikes. But I hope it gives a sense of the scale.

How to anticipate costs?

This question is a tough one. It's nice saying that the website costs only $17/month, but it's hard jumping forward and hoping for the best.

Over at Bref we have built a serverless pricing calculator. It is mostly targeted at APIs and websites, and should help you anticipate the prices a little better.

Now let's play a game and see if it would have guessed the cost correctly for externals.io:

The calculator doesn't include the database yet, but what we can note:

• AWS Lambda is off by half: this is because the calculator doesn't take into account the cron job, which actually costs about as much as the website (because fetching the emails is slow)
• CloudFront is less expensive in reality (maybe I need to account for automatic assets compression in the calculator, which reduces costs a lot)
• the rest seems on par!

I hope the calculator will be useful for you.

Migration effort

In this section, I get into the details of everything I had to change. I also link to the corresponding Bref documentation for each point.

• Move all the assets (CSS, JS, fonts) in a /assets/ subfolder. ℹ️ bref docs

With Apache or Nginx, we can configure routing to "call PHP unless a file exists, in which case serve the file". That is useful to serve assets like CSS, JS, fonts… We simply put them in a public/ directory and make that the web root.

CloudFront has no "fallback" mechanism. Requests must be routed to PHP or to S3 (assets) based on a URL prefix. I chose to move the CSS, JS and fonts into a /assets/ URL prefix so that I had 1 rule to configure instead of 3. What can I say, I'm lazy.

-<link rel="stylesheet" href="/css/main.min.css?v={{ version }}">
+<link rel="stylesheet" href="/assets/css/main.min.css?v={{ version }}">

That being said, I think getting rid of the "fallback" mechanism is a good thing: that has always been source of security issues, and it forced us to put our index.php in a public/ subdirectory. Now things are a bit more explicit and secure.

Effort: low

• Remove Platform.sh specific code for configuration and use environment variables. ℹ️ bref docs

Platform.sh populated a special environment variable containing some configuration values. I was able to remove all that and replace the configuration of the application by classic environment variables.

This was actually a win. I will spare you the diff since it's only code removal.

Effort: low

• Log to stderr instead of a file. ℹ️ bref docs

On AWS Lambda, logs should be sent to /dev/stderr to be collected by AWS. This is very simple thanks to the bref/logger PSR-3 logger where $log = new \Bref\Logger\StderrLogger() can replace Monolog.

I use PHP-DI, here is what the diff looks like:

-    LoggerInterface::class => create(Monolog\Logger::class)
-        ->constructor('app', get('logger.handlers')),
-    // more Monolog configuration...
+    LoggerInterface::class => create(Bref\Logger\StderrLogger::class),

Effort: low

• Move the Twig cache from var/cache to /tmp/cache. ℹ️ bref docs

The code is mounted as read-only in Lambda. The /tmp directory is the only writable directory, which is where I moved Twig's cache.

-    'path.cache' => __DIR__ . '/../../var/cache',
+    'path.cache' => '/tmp/cache',

Effort: low

• Move the ./console sync command into a Lambda function.

I had the ./console sync command running as a cron every 15 minutes. I could have ported exactly the same command to AWS Lambda, but I decided to make the command more "in line" with what AWS Lambda is about.

Indeed, we don't need a CLI framework and its abstractions (I mean the Symfony Console) in a cron. After all, I just want to execute a function. So I extracted the content of my Symfony command and put it in a real Lambda function:

<?php
$app = require __DIR__ . '/res/bootstrap.php';
$synchronizer = $app->getContainer()->get(Externals\EmailSynchronizer::class);
lambda(function () use ($synchronizer) {
  $synchronizer->synchronize();
});

Effort: low

• Generate a version number to bust browser cache.

The CSS is included in the page with a version number:

<link rel="stylesheet" href="/assets/css/main.min.css?v={{ version }}">

This forces browsers to clear their cache when a new version of the website is deployed. Platform.sh provided a unique version as an environment variable, but I found no such thing on AWS Lambda.

What I had to do is generate a version number when deploying. I use the timestamp in the deployment script:

export EXTERNALS_APP_VERSION=$$(date +%s)
serverless deploy

Then use that environment variable in the PHP-DI config:

-    'version' => env('PLATFORM_TREE_ID'),
+    'version' => env('EXTERNALS_APP_VERSION'),

Effort: medium

Finally, I obviously had te rewrite the configuration of the application from the Platform.sh format to serverless.yml. This is the part that required more effort. I took advantage of this to create the documentation to create websites on bref.sh, so it should definitely be much easier for you now ;)

Conclusion

Well this was a long article, I won't bore you with more text! If you want to read more of this, you can find more case studies here.

In the end, I am very happy with the result. If you have any question ask them here or ping me on Twitter.

6 June 2019 - Matthieu Napoli

Bref is an open source project that helps build serverless PHP applications. I started the project in October 2017 after discovering AWS Lambda. Over time, the project turned from an experiment into a tool used in production by multiple companies.

But since the beginning, I knew Bref was not like my other open source projects. Serverless is a movement that is changing our industry. It has good sides and its rough edges, but what I love the most about it is that it simplifies our jobs as developers. Soon everyone will be able to deploy and run applications, no matter their skills.

Because of that, Bref is a long term project. Unlike other open source projects that I maintain in my spare time, the vision I have for Bref requires time and commitment.

To give you an idea, in 2018 I tracked the equivalent of 26 full days spent working on Bref (the reality might be more around 40), i.e. 5 full-time weeks. Of course I wasn't paid for that. You can multiply that with your daily rate to get an idea of the amount of money I have invested in Bref in 2018. As of June 2019 I know I have already topped that for 2019.

This is why in January 2019 I created a company:

I hope you like the name!

Besides doing consulting work on PHP projects and application architecture, null is backing Bref to make sure it has a future for the next years.

At the moment that means that I can afford to work on Bref at least 2 days per week, as I have been doing since the beginning of the year. I am working on finding solutions to expand that and even pay more people to work on the project.

If you are a company interested to sponsor Bref, feel free to get in touch.

For those who have been following closely, one step we will take in the coming weeks is to move Bref from my personal GitHub account to the Bref organization on GitHub.

Slightly related, and I'd like to finish on that: for many months now Bref has been much more than just my contributions.

• Bubba from Signature Tech Studio has been a major contributor for the build scripts of the PHP runtimes (and he is also building a very nice Laravel integration for Bref), and his colleague Joseph has been welcoming newcomers and providing support in the Slack channel.
• Neal Brooks has contributed improvements to the Symfony integration and the documentation, and he has been spreading the word about AWS Lambda and Bref to countless meetups and conferences (the next one being Laravel Live in London).
• Alan Trope working for Sua Música has contributed a lot of improvements for high-performance applications, the latest being ALB support.
• Holger Woltersdorf is working on an open source project used by Bref: https://github.com/hollodotme/fast-cgi-client, and he has been very helpful to integrate his work and improve whatever could be improved.
• Rob Allen has been writing a lot of material to get started and learn about AWS Lambda and PHP, including Bref.
• the Enoptea team lead by Gaultier Boniface has helped Bref from the very beginning ?

There are also many more contributors, thanks to all of them.

21 March 2019 - Matthieu Napoli

This article is part of a series of case studies of serverless PHP applications built with Bref on AWS Lambda. If you are not familiar with serverless and Bref, I invite you to read Serverless and PHP: introducing Bref.

This case study is about prettyci.com, a SaaS that provides continuous integration for PHP coding standards for GitHub repositories.

The idea is that anytime you push a commit to your GitHub repository, PrettyCI will analyze that commit using PHP CodeSniffer or PHP-CS-Fixer. Since PrettyCI integrates in GitHub's checks tab you can see the build result directly in your repository without having to leave your work.

Architecture

I originally created the project using Laravel Spark. This is a Laravel project pre-built for creating SaaS applications (and it is pretty awesome by the way).

While Spark takes care of the website, I had to create:

• an API endpoint to receive GitHub webhooks whenever someone pushes new commits
• queue wokers that would analyse each commit with phpcs or php-cs-fixer

Fortunately Laravel has a queue system that is very easy to setup.

After deploying the whole thing to a small DigitalOcean server (❤️) using Laravel Forge (❤️) it was running just fine.

Serverless workers

Since I was running 4 workers on my server that meant that the system could process at most 4 commits at a time. When the 4 workers were busy, new commits to GitHub would be pending and waiting for a worker to free up.

This is not great for user experience, and while scaling up is doable (see this great article by Oh Dear for example) it is a bit more work.

Indeed, that meant adding more servers, which meant more costs and a much higher maintenance effort. This is were my lazy side kicked in.

As I was working with Lambda for returntrue.win at the time, I decided to migrate the workers to AWS Lambda using Bref.

I did the migration in July 2018 and it has been a complete success in my eyes, running great since then.

Challenges

The migration itself presented some challenges:

• the filesystem is read-only on Lambda except for /tmp so I had to change the code a bit to use this directory (to checkout repositories and run the analyses)
• while Lambda is a Linux environment, git is not available: I had to compile and upload the git binary in my lambda to be able to use it
  • Since then it is now possible to include a "git layer" like this one, which makes the whole thing much easier

Downsides

Now that the system is running, I don't see many downsides:

• the initial setup was a bit more effort than using the Laravel Queue system out of the box
• some problems are a bit harder to debug as it's impossible to SSH into a Lambda and run some tests (you have to go through the cycle of deploying + executing the lambda, or else run the tests locally)
• the /tmp folder is limited to 512MB, which made PrettyCI incompatible with projects that have a huge repository

Advantages

• jobs are never queued anymore: this is the killer thing for me here: as soon as a commit is pushed to GitHub a new environment will be launched to process it.
  • I often see "Pending build" when using Travis/Circle CI/Gitlab CI because the build is waiting for a free container. With PrettyCI that never happens, which makes the user experience awesome (most pull request statuses are updated under 5 seconds).
• infinite scaling with 0 action from my part
• pay per use, which is actually cheaper than the cheapest DigitalOcean VPS (I pay only the execution time of the workers)
• builds are isolated without me having to deal with containers or virtual machines
• high availability as all the execution environment is managed by AWS. I just have to focus on the code.

Conclusion

As explained in the Bref Maturity Matrix, running workers and jobs is an excellent use case for AWS Lambda, and this works perfectly for PrettyCI.

The migration is a bit more effort than "simply" using Laravel Queues or other similar libraries, but this is something we are trying to simplify at Bref.

If you are interested to learn more about AWS Lambda, you can check out other articles and subscribe to the serverless PHP newsletter.

10 February 2019 - Matthieu Napoli

Back in October I published a case study of a serverless website: returntrue.win. This website runs on AWS Lambda using Bref.

A new major version of Bref (v0.3) was released this month and it comes with completely new internals. I was curious, as many others were, of what it meant for performances.

I benchmarked 2 websites:

• the current returntrue.win website running with Bref 0.2
• the same website running with Bref 0.3

I made sure both lambdas were using the same configuration, which is 1024M of RAM. For those not aware, the amount of RAM available is proportional to the CPU power allocated to the lambda.

The page that we are testing is running with a custom PHP framework (not optimized, no cache), uses Twig without cache and performs 1 query to DynamoDB.

It is important to note as well that I am reporting the execution time of the lambda (i.e. the application), not the whole HTTP response time. Since we use API Gateway we need to consider that API Gateway adds about 15ms to lambda's execution times.

Cold starts

Cold starts happen when a new lambda instance is provisioned. By triggering around 20 cold starts for each function I did not see a difference.

In both cases cold starts where around 650ms.

That's a bit disappointing, hopefully we can improve this a bit. I did some tests with 2048M of RAM and cold starts where brought down to 450-500ms.

Warm results

After testing the cold starts I did performance tests on warm lambdas by running:

ab -c 1 -n 500 https://returntrue.win/

This is very good news: the website runs 6 times faster on average!

With an average response time of 39ms instead of 250ms, building website or APIs on AWS Lambda becomes a lot more interesting. You can check out below the detailed graph to see how results spread out:

• red lines: Bref 0.2 (minimum, average, maximum)
• green lines: Bref 0.3 (minimum, average, maximum)

I did some tests again with 2048M of memory and results improved even more: 35ms of average response time, with a minimum of 20ms and a maximum of 65ms.

Conclusion

These results should be taken with a grain of salt as we are testing a small application here. I intend to do more tests in the near future, and I encourage others to do some as well.

However this is very encouraging! Being able to run PHP applications under 100ms on AWS Lambda is great news as it continues to open more possibilities.

As a side note I have now deployed returntrue.win on Bref v0.3.

And if you want to give a try to Bref head over to bref.sh.

7 January 2019 - Matthieu Napoli

Serverless is a topic that is getting more and more attention lately. While this is exciting, it also means there is a lot to follow, read and test.

I have met several people who are interested about serverless and its possibilities but have a lot of trouble following everything. This is even more true in the PHP ecosystem because PHP was initially left out from AWS Lambda and other FaaS products.

In order to help I have started a monthly newsletter called Serverless PHP news.

This newsletter will contain an overview of what's new regarding serverless and how it relates to PHP. Because some news are worth knowing about only if you can relate it to the tools you are using everyday.

If you are interested, go subscribe at:

serverless-php.news

To give you an idea what to expect you can find below a copy of the first email.

Welcome to the first issue of the Serverless PHP newsletter.

First of all I want to wish you a happy new year, and thank you so much. You are more than 400 subscribed to this list and I definitely did not expect that. I hope this newsletter will live up to your expectations and I welcome any feedback (really!).

This episode is the first so I will sum up the current state of serverless PHP and talk about the main event of the last months: AWS re:Invent.

If you don't know anything about "Serverless" you can read an introduction in Martin Fowler's serverless bliki entry or Serverless and PHP: introducing Bref.

Before AWS re:Invent

AWS re:Invent is a conference held annualy by AWS. This year (November 2018) it changed a lot of things, so let's start by a recap of how PHP ran on AWS Lambda before.

Since PHP was not officially supported on AWS Lambda the only way to run PHP was to:

• create a JavaScript lambda
• compile PHP and include the binary in the lambda
• include our PHP script (or application) in the lambda as well
• write a JavaScript script that would execute the PHP binary and proxy the HTTP request to PHP

That worked fine but this had an impact on performances. Compiling PHP and writing the JS proxy script was also impractical, which is why Bref was created in the first place and this is what the current version (v0.2) is about. This is going to change!

AWS re:Invent

My selection of what was announced in November by AWS that may interest you:

• Use any programming language on AWS Lambda thanks to "lambda layers" and an "open runtime" API (explained in more details in the next section)
• Cold start improvements when using RDS: RDS is MySQL/PostgreSQL managed by AWS. Using those with Lambda works fine except it requires to put the lambda and the database in the same VPC (private virtual network). This implies a cold start of about 5 to 10 seconds, which is a huge pain for HTTP applications. This will be solved in 2019 which, I believe, is very very good news!
• RDS makes MySQL and PostgreSQL accessible via a HTTP API: this is interesting because it allows to skip managing persistent SQL connections and it is much more in line with the serverless paradigm (it is stateless). However the first version released by Amazon is definitely not ready for production right now. AWS is known for releasing alpha services and continuously improving them so let's be patient and keep an eye on it!
• Announcing Firecracker, the "micro-VM" alternative to containers and virtual machines developped by AWS to run AWS Lambda. Their goal: the security of VMs and the low costs (startup time and overhead) of containers. You don't need to learn about this as AWS takes care of the details, but it is interesting to know why they are not using containers.
• AWS SAM has received a lot of new features which makes it a very serious alternative to the serverless framework for deployments. AWS SAM is restricted to AWS but comes with very useful development tools running on Docker.
• DynamoDB goes full serverless via a new "pay per request" pricing: no need to reserve capacity, DynamoDB can now behave just like AWS Lambda: we can pay per request and it will scale on demand automatically.
• DynamoDB gets transactions support for those interested in replacing their database with DynamoDB. The new API seems limited but it may help in some scenarios.
• IntelliJ plugin to run and debug serverless applications: this will be interesting for PhpStorm users (the plugin seems to be usable only in IntelliJ for now, let's keep an eye on this)
• Websocket support in API Gateway and AWS Lambda: I love how clever this is: instead of having a complex setup with long-running processes holding the connections with the clients (I'm thinking of Laravel Echo for example) AWS makes it much simpler. API Gateway is responsible for maintaining the websocket connections for you. You then write code that reacts to websocket events (a new connection is made, a new message is received, etc.) via a Lambda function. In other words: back to stateless code, just like in a HTTP context. Check out this example application built using JavaScript. I am eager to try this in a real-world use case.
• ALB as an alternative to API Gateway for HTTP lambdas. ALB is Amazon's Application Load Balancer and it can now be used to expose Lambdas on the web without having to use API Gateway. ALB is much simpler as there is no routes to define: it proxies everything to the lambda, and it is supposedly faster as well. There is also a difference in pricing: there is a minimum of $22 per month for ALB where API Gateway starts at $0. However for bigger applications ALB is cheaper. If you are writing websites or APIs on Lambda and pay more than $22 per month on API Gateway it may be worth investigating! Remember to share your experience!

All the re:Invent talks are online on Youtube. I can recommend this talk about DynamoDB which was very enlightening for me about NoSQL in general and DynamoDB itself. I am tempted to try and rewrite externals.io using Lambda and DynamoDB to learn a bit more about it (hopefully in the coming months).

PHP on AWS Lambda

Now that we have an official way to run PHP on AWS Lambda, let's look at how it works. The idea is that lambdas can now use layers. A layer is a bunch of files that will be injected in the lambda when it starts.

We can add support for PHP in AWS Lambda by creating a layer that injects the php binary in the lambda. But that is not enough: we need a bootstrap.

The bootstrap is the file that is called when the lambda starts. It is responsible for executing the PHP script whenever there is a new event. What's great is that the bootstrap file can be written in any language, including PHP.

AWS announced that a company called Stackery worked on a PHP runtime: github.com/stackery/php-lambda-layer. However this runtime is definitely not ready for production and I found it very disappointing on several levels (I am not alone to think this).

With Bref contributors we have benchmarked several solutions for running PHP on Lambda. Those include running PHP scripts as well as HTTP applications using PHP-FPM or even PHP's built-in webserver. We have identified the best solutions and we have begun porting them in the new version of Bref.

Bref v0.3

The vision for Bref has not changed: it is not just about PHP support, it is about empowering everyone to benefit from serverless technologies. We are hard at work on the next version (v0.3) which will provide:

• stable PHP runtimes for PHP functions as well as HTTP and console applications
• much better performances thanks to these native runtimes
• a completely rewritten and extensive documentation
• a revisited deployment process that integrates with AWS tooling
• tools for local development based on Docker (thanks to AWS SAM)
• support for any PHP application or framework thanks to PHP-FPM and the console runtime

If you are interested you can follow the v0.3 pull request that summarizes all changes.

PHP outside AWS

PHP support outside of Amazon is not that great. Google Cloud Functions only supports Python and JavaScript while Microsoft Azure Functions has experimental support for PHP.

Zeit.co has basic support for PHP but I wouldn't consider it production-ready: it supports only 1 PHP file per application and runs it via go-php, a PHP 7 implementation in Go. Hopefully this will improve in the future.

IBM OpenWhisk is interesting as it can run Docker containers (so it can run PHP applications). Finally an interesting alternative that may be worth exploring is Alibaba Cloud which supports PHP functions as well as PSR-7 requests for HTTP applications.

My personal belief is that AWS is years ahead of its competitors right now, especially after the latest announcements. AWS Lambda is also very stable and reliable compared to Google Functions and Azure Functions. Let's see how things evolve in the next months and years!

Conclusion

That's it! This first episode is quite packed but a lot has been happening lately. I hope it helps you get a better overview of all this.

If you are interested in exploring the serverless world for your next big thing I hope that all of this will be useful. If you need someone to help feel free to get in touch.

29 October 2018 - Matthieu Napoli

This article is part of a series of case studies of serverless PHP applications built with Bref on AWS Lambda. If you are not familiar with serverless and Bref, I invite you to read Serverless and PHP: introducing Bref.

This case study is about the returntrue.win website. This website was my first serious experiment with serverless and Function as a Service.

Returntrue.win is a "puzzle game for PHP developers". A piece of PHP code is shown. Users can fill the missing parameters and run the code. The goal is to make the code return true, then the user can move on to another level.

Not sure how to get the best scores? Check out this complete solution.

Architecture

The website is built using the following services:

• AWS Lambda for running PHP

• API Gateway to expose the lambdas via HTTP

  This service is required if you want to make a lambda accessible on the internet as an API or website. Bref takes care of configuring API Gateway for you.

• DynamoDB to store the best scores

I chose DynamoDB because I wanted to discover that service and because it was cheaper than RDS. At first I wanted to store the data on disk in a JSON file but the distributed approach of lambdas, and the fact that the filesystem is read-only, put a stop to that. In the end I was happy with that choice and it cost me nothing. But again I stored so little in there that its usage isn't really significant.

There are 2 lambdas:

• the first lambda is the website
• the second lambda runs the code submitted by users (I like to call it "eval-as-a-service")

This separation helps ensure that code submitted by users will not affect the website, even if it crashes.

I soon discovered that building a website with AWS lambda is a bit trickier than building an API. The whole thing is not built to handle assets (CSS, JavaScript…) out of the box. A simple solution is to put assets on a CDN such as AWS S3. Since my needs were very simple I decided to include Bootstrap from a public CDN and write a few lines of inline CSS. Not the cleanest solution but that just worked.

Traffic

I released returntrue.win in February 2018 and it got a bit of attention on Twitter and Reddit. The website received a decent amount of traffic, especially the first days where I watched with attention the lambdas scale up and down.

As you would expect, I did not have anything to do except let Amazon take care of running my code for me.

Over the first month, the website served 400,000 HTTP requests which resulted in 650,000 lambda executions (remember that a user running a piece of code means 2 lambda executions: the website plus the "eval-as-a-service").

Costs

As a developer I enjoyed not worrying about whether my server would be able to handle the traffic. However I have to admit I was worried about how much this would cost me in the end :)

In total, the first month cost me $3.

Needless to say I was relieved! The website has been running since and I pay close to $0 every month.

The majority of the cost was related to API Gateway as AWS Lambda has a generous free tier. Let's break it down:

• AWS Lambda: $0 thanks to the free tier, would have paid $2.78 otherwise
• API Gateway: $2.39:
  • $2.23 for the HTTP requests
  • $1.16 for the 1.8Gb of data transfer
• DynamoDB: $0 thanks to the free tier, would probably be around $0.6 without the free tier
• AWS CloudWatch: $0
• Data Transfer: $0.04 - I still don't understand completely what this is about
• Taxes: $0.5 :)

Conclusion

While I hope this case study is useful, there are many things that could be improved:

• storing assets on a CDN
• using a real PHP framework
• caching HTTP requests (many many requests could be cached, resulting in an even lower cost)

And the website could definitely have a better design. I hope to open source the website at some point to allow users to contribute new levels.

25 May 2018 - Matthieu Napoli

Update: Since November 2018 AWS Lambda supports PHP via custom runtimes. Bref has changed to take advantage of that and the Laravel integration has changed accordingly. Read more about it in Bref's documentation for Laravel.

The article below is now obsolete.

Last week I introduced Bref as a solution to running PHP serverless.

Today let's try to deploy a Laravel application on AWS lambda using Bref. The code shown in this article is available on GitHub.

You can check out the demo Laravel application on AWS lambda here: https://k6ay4xiyld.execute-api.eu-west-3.amazonaws.com/dev. It is a simple application that uses a 3rd party HTTP API to convert between currencies.

An introduction to serverless PHP

Serverless basically means "Running apps without worrying about servers". The main difference with a traditional hosting is that you do not maintain the servers and reserve their capacity. They are scaled up or down automatically and you pay only for what you use.

One notable example is AWS S3: instead of renting a fix disk space, you pay for only the storage you are actually using. AWS S3 has no limits, you can store 10kb or 10Tb and you do not need to scale up or down the storage. AWS S3 is serverless file storage.

The idea behind serverless applications, aka Function as a Service (FaaS) is to apply the same principles:

• traditional hosting:
  • pay for fixed server resources (CPU, RAM…)
  • if you hit the limit you need to scale up the server
  • you pay for the whole server even if your application has no traffic
• serverless hosting:
  • no fixed server resources
  • the application is run whenever there is traffic
  • there are as many processes running as there are concurrent requests
  • you pay only for the execution time, an application with low traffic will have costs close to $0

Serverless hosting has the advantages of scaling very well since there are (theoretically) no limits. It can also help optimize costs by avoiding paying for unused server resources. You can read more about advantages and drawbacks here.

Making PHP work on AWS Lambda

I will take the example of AWS Lambda because it is the most popular provider for serverless applications, but there are other providers available. Unfortunately AWS Lambda does not support PHP (supported languages are for example Javascript, Go, Python…). To run PHP we must add the PHP binary in the lambda and have, for example, Javascript execute it.

The technical details can be found in this section so I will not cover them again. The conclusion is that deploying PHP on serverless providers is possible but a PITA. That is how Bref is born.

Bref's goals are:

• deploy easily on serverless providers
• make PHP frameworks work just like before

For the first part Bref builds on top of the serverless framework and brings additional tooling specific to PHP.

For the second part I have been working on bridges to make PHP frameworks work on lambdas.

Laravel on lambdas

What follows is a step-by-step explanation, if you want the short version either check out the Bref documentation or the demo on GitHub. I will also focus on AWS Lambda here because it is what I used for those tests.

Let's start by creating a new Laravel application:

$ composer create-project laravel/laravel demo
$ cd demo

Now let's install Bref (read the setup instructions) and initialize it:

$ composer require mnapoli/bref
$ vendor/bin/bref init

When applications run on lambdas they do not work like with traditional hosting providers: public/index.php is no longer the application's entry point because there is no Apache, Nginx or PHP-FPM. Bref takes care of connecting the HTTP layer (API Gateway + AWS lambda integration) to your application. It will convert the HTTP request from API Gateway into a format Laravel can understand, and will convert back the HTTP response the other way around.

Note: you don't need to know about API Gateway, Bref takes care of that part.

All we need to do is write a bref.php file: that will be the entrypoint of the application. Let's write that file at the root of the project:

<?php
define('LARAVEL_START', microtime(true));
require __DIR__.'/vendor/autoload.php';
$app = require_once __DIR__.'/bootstrap/app.php';
$kernel = $app->make(Illuminate\Contracts\Http\Kernel::class);
$app = new \Bref\Application;
$app->httpHandler(new Bref\Bridge\Laravel\LaravelAdapter($kernel));
$app->run();

As you can see it's very similar to public/index.php except that Laravel will not run itself, instead Bref will run Laravel.

That is done using the $app->httpHandler(...) line (documentation). You can use any PSR-17 framework as a HTTP handler, here we are configuring the Laravel HTTP Kernel using an adapter. The adapter is necessary because Laravel is not compliant with PSR-17 yet.

The next step is to configure which directories to deploy on the lambda. That can be configured in the serverless.yml file that was created in your project:

package:
  exclude:
    # ...
  include:
    # ...
    # Add the following directories:
    - 'app/**'
    - 'bootstrap/**'
    - 'config/**'
    - 'resources/**'
    - 'routes/**'
    - 'vendor/**'

Now all we would have to do in theory would be to deploy using:

$ vendor/bin/bref deploy

But with Laravel we have a few extra steps to take.

File storage

The filesystem on lambdas is read-only, except for the /tmp folder. That means that Laravel's storage directory will not work out of the box if we keep it inside our project.

We need to tell Laravel to use the /tmp/storage folder instead. Add this line in bootstrap/app.php after $app = new Illuminate\Foundation\Application:

/*
 * Allow overriding the storage path in production using an environment variable.
 */
$app->useStoragePath(env('APP_STORAGE', $app->storagePath()));

By using an environment variable we can keep the classic behavior when running the application locally (or on a traditional server) and we can set this variable to /tmp/storage on AWS lambda.

To set environment variables on AWS lambda we can use serverless.yml:

functions:
  main:
    ...
    environment:
      APP_STORAGE: '/tmp/storage'

Now since this directory doesn't exist by default on AWS lambda we need to create it on the fly. Add these lines to bref.php:

// ...
$app = require_once __DIR__.'/bootstrap/app.php';
// Laravel does not create that directory automatically so we have to create it
if (!is_dir(storage_path('framework/views'))) {
    if (!mkdir(storage_path('framework/views'), 0755, true)) {
        die('Cannot create directory ' . storage_path('framework/views'));
    }
}
// ...

Configuration

There are other parameters we will want to override for the production environment. What we will do is create a .env.production file in our project and configure our variables here:

APP_ENV=production
APP_DEBUG=false
# ...

To make Laravel use this configuration in production we will write a build hook. Build hooks are scripts that are executed before Bref deploys your application. Let's write those in a .bref.yml file:

hooks:
    build:
        # Rename the `.env.production` file to `.env`
        - 'rm .env && cp .env.production .env'

Note: those commands are run in a separate directory than your project, your own .env file will not be deleted.

By default Bref installs Composer dependencies and optimizes the autoloader, we do not need to do it ourselves.

In order to optimize the application we want to cache the configuration. Let's add other hooks to the list:

hooks:
    build:
        # Use the `.env.production` file as `.env`
        - 'rm .env && cp .env.production .env'
        - 'rm bootstrap/cache/*.php'
        - 'php artisan config:cache'

The php artisan config:cache dumps the optimized configuration files. The problem with that is that it will also dump all file paths as absolute paths (e.g. storage directory, views, etc.). Since Laravel is in a different directory on our computer than on AWS lambda the dumped configuration will not work in production.

We need to tell Laravel to generate relative paths. Let's do that by customizing the root path of Laravel in bootstrap/app.php as shown below. The APP_DIR variable will allow us to replace the absolute path by . (relative path) when generating the lambda.

$app = new Illuminate\Foundation\Application(
    env('APP_DIR', realpath(__DIR__.'/../'))
);

Note: we do not want to hardcode . directly here because that would mess up Laravel's behavior in other cases (for example with artisan serve or when running Laravel with Apache/Nginx).

Let's add that new environment variable to serverless.yml:

functions:
  main:
    ...
    # Laravel configuration using environment variables:
    environment:
      APP_STORAGE: '/tmp/storage'
      APP_DIR: '.'

Let's also add that variable into our .env.production file:

# ...
# This allows to generate relative file paths for the lambda
APP_DIR=.
APP_STORAGE=/tmp/storage

Everything is good except one last problem: view cache files. Laravel runs realpath() on the path containing the cached views, and we are generating a configuration with /tmp/storage on our machine -> that directory doesn't exist on our machine, and realpath() fails. That breaks Laravel's views.

We need to remove the realpath() call in config/views.php:

-    'compiled' => realpath(storage_path('framework/views')),
+    'compiled' => storage_path('framework/views'),

All is good now!

Logging

We configured Laravel to use the /tmp/storage directory, but /tmp is ephemeral: everything in there will be lost at some point. This is not a good place to store logs.

We need to change the logging driver and instead use the stderr driver: logs will be captured by Bref and sent directly to AWS Cloudwatch. That can be configured in .env.production:

# ...
LOG_CHANNEL=stderr

You can of course use any other driver you want (as long as it's not the file driver).

Sessions

Just like logging, storing sessions in /tmp/storage is not a good idea. If you are writing an API or any application that do not need sessions you can use the array driver:

# ...
SESSION_DRIVER=array

If you need sessions you can store them in database, Redis, etc.

Routing

Finally when deploying a lambda AWS creates a random URL that contains a /dev suffix. Because of that the default route (/) will not work out of the box. We can change the route for the welcome page:

// routes/web.php
Route::get('/dev', function () {
    return view('welcome');
});

If you use a custom domain however the suffix will disappear so it is only an issue when doing small tests like this.

Deploying

We are finally done for a functional serverless Laravel. Let's deploy:

$ vendor/bin/bref deploy

Our app should now be online, get the URL of the app by running vendor/bin/bref info.

Going further

There are still many things to improve with Bref and Laravel:

• simplify the setup and configuration
• build the views before deploying (this is very hard because Laravel uses absolute paths again here, I hope to submit a solution for that soon)
• integrate artisan to be able to run on lambdas
• integrate queues and scheduling to run them on lambdas
• integrate with CDN for hosting assets
• etc.

I am planning to work on those topics in the months to come, and help is always welcome.

Conclusion

If you want to learn more about Bref and Laravel:

• read the Bref documentation
• here is the serverless Laravel demo
• check out the code for the Laravel demo

To learn more about serverless in general:

• check out this article about running PHP serverless
• and this article about performances

If you want to get started I can also accompany you, drop me a line at matthieu at mnapoli dot fr.

24 May 2018 - Matthieu Napoli

Last week I introduced Bref as a solution to running PHP serverless.

Update: Since November 2018 AWS Lambda supports PHP via custom runtimes. Bref now takes advantage of that, you can read more about it at bref.sh.

Performances have changed a lot with these new versions. This article is now obsolete.

Today let's explore what performances to expect when running PHP on AWS lambda using Bref. Everything shown in this article is open source on GitHub and can be reproduced.

Everything shown here is specific to AWS Lambda. Other serverless providers (Azure, Google Cloud…) could offer different performances and costs.

Results

This is a comparison of the HTTP response time that I got by calling the lambdas from an EC2 machine in the same region, on both a NodeJS lambda and a PHP lambda. This is not the lambda's execution time but the real HTTP response time, network included. If you want to use lambdas in a non-HTTP context those results will still be useful.

I run the test more than 3 times for each combination and I keep the lowest one (because of cold starts numbers vary a lot at first).

I ran the test for several memory sizes: lambdas "sizes" are configured through the memory. More memory means more CPU, but also higher costs.

AWS Lambda allocates CPU power proportional to the memory by using the same ratio as a general purpose Amazon EC2 instance type, such as an M3 type. For example, if you allocate 256 MB memory, your Lambda function will receive twice the CPU share than if you allocated only 128 MB.

We can see that Node performances are pretty consistent, whereas PHP performances become optimal above 1024M. Performances with 512M can be acceptable depending on the use case (e.g. workers, crons, etc.). 128M performances are pretty poor.

To sum up, we should expect a 20ms penalty to using PHP over AWS Lambda (using Bref) compared to other languages.

What's interesting to note is that Node's 21ms base response time is because of the HTTP layer (API Gateway and network). Let's consider this other graph (Cloudwatch metrics):

• blue line: HTTP response time (~40ms for PHP, 13ms for Node)
• green line: PHP execution time (25ms)
• orange line: Node execution time (0ms)

This confirms that PHP adds 20ms-25ms to the lambda's execution time. The HTTP layer (API Gateway) adds ~15ms in all cases. The network between the lambdas and the EC2 machine used for the tests accounts for ~5ms.

Cold starts

The numbers above are the "cruise" numbers, when everything is warmed up and runs smoothly. But lambda cold starts should not be ignored. I will not cover that topic again since it has been covered heavily elsewhere, I can recommend reading this article.

Here are measurements made on Node and PHP lambdas, completed with cold starts from other languages found in this comprehensive article.

PHP's cold start is lower than Java and C# even though both those languages are supported natively by AWS Lambda. From 768M and up the cold starts stabilize around 230ms.

We can also see that Python has lower cold starts than Node. Bref could switch to Python as the language used to invoke PHP but the gain seems to be minimal compared to PHP's execution time.

Costs

Let's overlay the PHP performances and the cost associated. For simplicity of reading, and because the base response time is 340ms, I will be counting 1 million lambda requests that take 400ms to execute.

Note that this do not include the cost of API Gateway, data transfers, etc. (which can be much higher than that). I also do not include the free tier because the goal is to compare each row with the others.

I have removed 20ms of the execution time to go from the HTTP response time to the actual lambda's execution time.

Using the 1024M lambdas to get great performances with PHP mean a 6 times increase on costs. However this does not take into account that lambdas will also run faster, bringing costs down, which is what we can see in the last column (cost associated to the base execution time is actually decreasing). On the other hand most applications will not run 6 times faster because often the CPU is not the most limiting factor (this is rather I/O like network, databases…).

Like in all situation it is best to make your own tests to get a better idea. If your lambda is invoked through HTTP the major cost will be API Gateway anyway.

Conclusion

• use 512M memory for acceptable performances, 1024 and up for optimal performances
• expect a 20ms penalty to using PHP over AWS Lambda compared to other languages
• PHP's cold start is lower than other languages like Java and C#, from 768M and up the cold start stabilizes around 230ms

Want to get started: have a look at Bref.

Update: I realize I did not mention opcache which is an important player in PHP performances. I did not forget about it, out of the box it is not supported because PHP is running as CLI. However I am planning to work on that, opcache can be used in CLI processes.

17 May 2018 - Matthieu Napoli

Serverless basically means "Running apps without worrying about servers". Obviously there are still servers involved, the main difference is that you do not maintain the servers and reserve their capacity. They are scaled up or down automatically and you pay only for what you use.

This article intends to explain what serverless means for web applications and more specifically for PHP.

Update: Since November 2018 AWS Lambda supports PHP via custom runtimes. Bref now takes advantage of that, you can read more about it at bref.sh.

What is serverless?

Serverless file storage

AWS S3 and other file/object storage are good examples of serverless infrastructure.

If you were to store files in your application (for example file uploads, images, etc.), the traditional solution would be to store them on a drive mounted in the server. The downsides would be that you would have to monitor it and scale up the volume in case it gets full. You would also pay for the whole storage space even if you used 10% of it.

The serverless approach is rather a virtual storage with no maximum capacity and managed for you. You would also pay only for the disk space you are actually using. That is what AWS S3 offers for example.

Serverless databases

The traditional approach to using a database in an application is to run MySQL/PostgreSQL/etc. on a server. You would have to maintain it (monitoring, backups, replication…) and the database would be limited to the resources available on the server (CPU, RAM, storage…). If you hit those limits you would have to scale the server up, which means effort and downtime. In the opposite case if the application barely uses the database you would still pay for the whole server.

The serverless approach is again to abstract the server away and hide how the database service is running. You would use the database just as before, however the resources dedicated to your database would scale up or down dynamically with the load. Monitoring, backups and replication would be handled for you. You would pay only for the resources (CPU/RAM/storage) the database actually uses.

While using serverless file storage is becoming more common, serverless databases are still a bit new. This is probably because databases are a lot trickier to manage than files. Good examples of serverless databases would be AWS DynamoDB (which is a NoSQL database) and "AWS Aurora serverless" which replicates MySQL or PostgreSQL. Note however that for now AWS Aurora serverless is still in private beta.

The same could be said about other kind of databases like message queues, caches, etc.

Serverless applications, aka FaaS

The "serverless" approach for applications is a bit trickier to imagine.

The "traditional" approach to running an application means executing the application on a server. That application would be a process running on a CPU and some RAM. The application would either act immediately (like an application executed on the command line, a CRON task…) or listen and react to external events (an incoming HTTP request, a TCP connection, a new message in a message queue…), aka a daemon.

For example the code for a HTTP API in NodeJS would contain:

• the main script, starting a HTTP server on port 80, and a router handling requests, often through a framework like Express

  • a controller (JS function) that is called when a HTTP request calls GET /users

  • a controller (JS function) that is called when a HTTP request calls POST /users

The serverless approach takes things differently: instead of thinking about applications and processes, the application is considered as a collection of functions, also known as lambdas.

FaaS, which means Function as a Service, means that you only care about writing the function and that the FaaS provider will take care of running them.

Lambdas are invoked by events that can be:

• a HTTP request
• a new message in a message queue
• a new file in a file storage (like an AWS S3 bucket)
• a specific time to replicate the behavior of a cron (e.g. every day at 2am)
• etc.

If we take the same example as before, the serverless equivalent would be a NodeJS project containing 2 functions:

• a function that is set up to be called when a HTTP request calls GET /users
• a function that is set up to be called when a HTTP request calls POST /users

The serverless tooling takes care of the daemon part (e.g. the HTTP server and routing) and developers now have just to write multiple functions.

On AWS the HTTP server + routing is taken care of by API Gateway (you would basically define routes there), and the execution of the functions is done by AWS Lambda (a bit like Docker containers).

On a side note you do not necessarily have to split your application into many lambdas (one for each controller). You could also have 1 lambda that do the routing like before.

I do not know yet if one way is fundamentally better than the other, but this is definitely easier to get started with 1 lambda. This approach is also closer to current architectures and frameworks, especially in PHP where we migrated from multiple *.php files into a single index.php entrypoint a long time ago. I am not convinced that splitting everything again makes sense. I guess time will tell.

Advantages

Just like file storage and database, the main advantages are costs, scalability and the reduced amount of server maintenance.

Lambdas are executed on demand whenever they are triggered by an event (like an HTTP request). That means that if your application has no activity, nothing is running and the costs are close to 0. When your application receives a lot of requests, the FaaS provider runs as many instances as necessary to handle them. If there were to be 1000 HTTP requests at the same time, your lambda would execute 1000 times in parallel. This is useful for HTTP applications but also workers: you don't have to run a fixed number of workers at all time. Of course you could be limited by other non-scalable components of your architecture, like the database, if those are not "serverless" and scalable on-demand too.

You usually pay by the time of execution of your lambdas. Try the AWS Lambda pricing calculator to get an idea. For HTTP applications however you will also pay for API Gateway and data transfer, and those can cost more than AWS Lambda.

Drawbacks

First the technology is new, compared to traditional solutions the tooling is basic and not mature. There is less expertise and less resources out there to get started easily and build something correctly. And since we lack perspective on the subject it is harder to distinguish the use cases where serverless makes perfect sense VS the cases were serverless is not useful.

Costs can be a drawback too: if you compare the cost of a high traffic application hosted on lambdas vs a lower level solution like bare-metal or VPS hosting, it's possible that serverless will be more expensive. This is because you pay for the CPU/RAM resources and the service of managing the infrastructure. If you have an infrastructure that is already working fine as it is, the switch to serverless may not be worth it.

A drawback usually brought up is vendor lock-in. While it is a legitimate concern, it is more and more mitigated as there are open solutions that provide alternatives to AWS and Azure (e.g. serverless on Kubernetes on your own machines). There are more and more tools abstracting the provider, allowing (theoretically) to switch from one provider to another more easily. I think that given enough time there will be as much vendor lock-in as there is with any hosting provider out there, i.e. not zero but acceptable (after all, switching hosting provider is always a cost, even with tools like Ansible, Docker, etc.).

Finally, a big drawback is that the serverless approach also impacts our code and our frameworks. Those are often not ready to work out of the box with lambdas because they were not imagined for those environments and architectures. While there is often not a huge effort required, it still is an effort (PHP is a good example for that and that is detailed below).

Serverless and PHP

Update: Since November 2018 AWS Lambda supports PHP via custom runtimes. The new version of Bref takes advantage of that, you can read more about it at bref.sh.

What is described below is now obsolete.

PHP applications are different than NodeJS/Java/Go web applications because they rely on Apache/Nginx + PHP-FPM for the HTTP server layer. In the NodeJS example, the application would start and listen on a port for HTTP requests, the same process would handle all the HTTP requests.

With PHP, each HTTP request gets its own process. When the HTTP response is emitted, the PHP application terminates and everything is cleared so that on each new HTTP request the PHP application starts from scratch again (this is a simplification). This behavior is in the end very similar to lambdas: for each HTTP request a lambda is booted, handles the request and dies (this is also a simplification). That is why I think lambdas make sense for PHP: the architectural gap is not huge.

Making PHP work on AWS Lambda

I will take the example of AWS Lambda because it is the most popular provider for serverless applications. Unfortunately AWS Lambda does not support PHP (supported languages are for example Javascript, Go, Python…). To run PHP we must add the PHP binary in the lambda and have, for example, Javascript execute it.

To sum up, what we will have to do:

• compile PHP for the OS used on lambdas
• add the compiled PHP binary to the lambda
• write a Javascript handler (the code executed by the lambda) that executes the PHP binary
• write a PHP handler (the code that will be executed by the Javascript handler)
• deploy the lambda

The content of the lambda would look like this:

bin/
    php      # our compiled PHP binary
handler.js   # executes `bin/php handler.php`
handler.php  # the PHP code we want to run on the lambda

Now that we have something working, let's add the missing parts: the input and output. Lambdas take event data as input, and return a response. The event data can contain data passed by the caller, or if we are in the context of a HTTP request the event will contain the HTTP headers, request, URI, etc. The response can contain anything we want to return to the caller, and for a HTTP context it should contain the HTTP response.

Since handler.js receives the event data directly, we can pass it to handler.php through several ways: using a specific temporary file for example, or more simply through a command parameter. That means running bin/php handler.php <the-event-data>. The event data would be encoded as JSON, read by the PHP script using the $argv variable and decoded using json_decode().

For the response, handler.php could write the response in a temporary file or event output it on stdout, then handler.js would read it and return it as the response of the lambda.

Here is an example of a handler.js if you are curious.

Introducing Bref

Needless to say that the above solution is not ideal because:

• you have to compile PHP manually
• you have to write and maintain a handler.js for your PHP app
• you have to know the format of AWS lambda's event and response (for example for HTTP requests and responses)
• you cannot connect lambda's events and responses to PHP frameworks input/output or HTTP requests/responses
• you need to setup a deployment process from scratch to compile PHP, setup the project for the readonly filesystem (e.g. Composer install, prepare the caches…)

At first I tried to write a plugin for the serverless framework to support PHP but that didn't work out because of some limitations, and because that project is mostly about deployments.

I believe there are 2 problems to solve for the "serverless PHP" equation:

• deploy easily on serverless providers
• make PHP frameworks work just like before

I decided to write Bref to solve those problems.

The name bref means brief in french, in reference to the ephemeral life of lambdas. French speakers will also enjoy the double meaning of "Bref, j'ai déployé une application serverless" ;)

The deployment is implemented by a CLI tool, bref deploy, that wraps the serverless framework to add what is needed for PHP. On deployment, Bref will automatically install Composer dependencies (excluding dev dependencies) and optimize the autoloader for production. It is also possible to add additional tasks like building a cache before deployment (which is required by some frameworks like Symfony).

A simple PHP lambda can be implemented like this:

<?php
require __DIR__.'/vendor/autoload.php';
$app = new \Bref\Application;
$app->simpleHandler(function (array $event) {
    return [
        'hello' => $event['name'] ?? 'world',
    ];
});
$app->run();

While that works, that does not solve the second problem because that does not work with PHP frameworks. I decided to tackle 2 ways of integrating with PHP frameworks:

• HTTP applications
• CLI applications

HTTP applications

PHP frameworks usually read the request from global variables, and echo the HTTP response to PHP's output and headers to global functions. That does not work on lambdas.

Fortunately modern frameworks abstract the HTTP request and responses with objects. What's even better is that we have PSRs for that (PSR-7 and PSR-15).

What Bref does is turn the lambda's event data (that contains HTTP request data) into a PSR-7 request object. We now have a HTTP request that frameworks can process. We call the framework with that request and get the PSR-7 response in return. We turn that response object into a valid AWS lambda response and return that to handler.js.

By abstracting that with PSR-15's RequestHandlerInterface (and adapters for frameworks that are not directly compatible) we can support most frameworks with very little effort for the end user.

Here is an example with the Slim micro-framework:

$slim = new Slim\App;
$slim->get('/dev', function ($request, $response) {
    $response->getBody()->write('Hello world!');
    return $response;
});
// Instead of calling `$slim->run()` we use Bref
$bref = new Bref\Application;
$bref->httpHandler(new Bref\Bridge\Slim\SlimAdapter($slim));
$bref->run();

In short we use PHP frameworks just like usual but instead of running the framework directly we pass it to $bref->httpHandler(...) (for those curious here is what happens with the handler when $bref->run() is called).

CLI applications

Many frameworks let us create CLI applications (Symfony Console, Laravel Artisan, Silly…). Just like with HTTP, those frameworks read from global variables and output to stdout which doesn't work on lambdas.

But again like with HTTP, frameworks have built object-oriented abstraction over the CLI input and output.

Bref adds support to executing CLI commands in production using the bref cli command:

• run bref cli -- [arguments and options] on your machine (everything after the -- is considered as arguments and options to the target CLI command)
• the bref CLI tool will invoke the lambda and encode the provided arguments and options in the event data
• the lambda will recognize that it is called as a CLI and will decode the arguments/options from the event array into Input and Output objects
• Bref will run the CLI framework (Symfony Console, etc.) with those objects
• it will encode the output into the lambda's response
• the bref CLI tool receives the response and displays the output

Here is an example with a Silly application (which is a wrapper around the Symfony Console):

$silly = new \Silly\Application;
$silly->command('hello [name]', function (string $name = 'World!', $output) {
    $output->writeln('Hello ' . $name);
});
$app = new \Bref\Application;
$app->cliHandler($silly);
$app->run();

Running the command looks like this:

$ bref cli -- hello Bob
Hello Bob
# running the same command locally:
$ bin/console hello Bob
Hello Bob

State of the project

I want to emphasize that Bref is currently a beta project. As explained above serverless architectures is still new and many choices in Bref are not definitive. My goal in sharing this project is helping others try out serverless, and to help moving forward.

Just like with any buzzword there is hype, and there is distrust because of the hype. Bref is meant to help you make your own mind, not sell you a silver bullet.

The project is new, there are many limitations and drawbacks and there is still a lot to improve. I take the habit of filling the code with very detailed TODOs, contributors are welcome. Regarding performances (because I know that's in everyone's mind) there is obviously a penalty to executing PHP CLI like this on every request. Some benchmarks estimate it at 20ms per request, which is acceptable in many scenarios. I also think there is a lot of room for improvements, for example regarding opcache. I don't want to get too hung up on that.

I am using the project in production on a few small scale projects and it does the job. Like semver allows, I am keeping BC only between patch versions while in a 0.*.* numbering. In Composer you should use ~0.x.0 instead of ^0.x.0 to avoid BC breaks until a 1.0.0 version.

Case studies

As explained above most of this is quite new, I am collecting case studies to show what we can expect from serverless PHP applications.

• returntrue.win is a serverless application (original tweet)

  The web application is one lambda. The code evaluation is done in another lambda. The database is very small and stored in DynamoDB.

  When the project was released it got 350k HTTP requests in 2 weeks, and 700k invocations across the 2 lambdas over 1 month. Not matter the trafic the performances were always the same. The total cost for that period was $3.

• bref-symfony-demo is a Symfony application deployed as a lambda (original tweet)

• serverless workers: I have been running workers in AWS lambda in a private project for several months now and I am very happy with the processing time (no queuing time since lambdas are run on demand), the costs ($0) and the maintenance (very low)

Please share your own experience in the comments.

TL/DR

Want to try out PHP as a serverless applications? Have a look at Bref.

To clear up vocabulary confusions:

• Serverless: a type of hosting where servers are managed for you and allocated dynamically
• FaaS (Function as a Service): serverless for code
• Lambda: a function

26 November 2017 - Matthieu Napoli

Note: the french version of this article is published on Wizaplace's tech blog.

We recently discussed 2 topics seemingly unrelated with my colleagues at Wizaplace:

• how to organize code?
• how to organize teams?

Feature teams

Regarding "how to organize teams", we were discussing Spotify's Feature teams. In a "classic" organization, teams are usually formed by grouping people based on their job title:

• backend developers team
• frontend developers team
• sysadmin team
• product team

But in a "feature team" organization, teams are organized… by features:

• team e-commerce: 1 product owner, 3 developers, 1 sysadmin
• team blog: 1 product owner, 2 developers, 1 sysadmin

The pros of this kind of organization are numerous and I do not intend to go over them here. But what does this have to do with code organization?

The classic code structure

Most projects will follow this "classical" layout:

src/
    Entity/
        Product.php
        Basket.php
    Service/
        ProductService.php
        BasketService.php
    Repository/
        ProductRepository.php
        BasketRepository.php
    Resources/
        ORM/
            Product.orm.yml
            Basket.orm.yml

Code is organized by type, i.e. what each thing is (an entity, a service, a repository, …).

A developer is a developer, a sysadmin is a sysadmin, but when they both work on the same project, it makes sense to let them be in the same team. How about doing the same thing with code? Why not group code based on what it does instead of what it is?

Module-oriented code structure

Here is an alternative layout where code is grouped by "domain module", or feature:

src/
    BasketModule/
        Basket.php
        BasketService.php
        BasketRepository.php
        ORM/
            Basket.orm.yml
    ProductModule/
        Product.php
        ProductService.php
        ProductRepository.php
        ORM/
            Product.orm.yml

The first reaction one could have when seeing this is:

How do I find all the repositories (or entities) at once? I will be jumping between folders!

To which one can answer: "why would we want to find all repositories or entities?" Humans interact with code for 2 reasons:

• to read it and understand it
• to change it

In my experience it is much more common to read code to understand a feature (for example "how is stock handled in products", "when is a product published", …) rather than to understand how all repositories work or to re-read all the config files.

The same goes for writing or modifying code: we often work on fixing or improving a feature, rather that e.g. change all repositories at once. For example adding a new field on products is much easier to do when all the relevant files are in one place.

Dependencies and cohesion

When we think about it, what do ProductRepository and BasketRepository have in common? Both address different problems and should be completely decoupled, so why group them together?

On the contrary, Product, ProductService and ProductRepository are highly coupled (and for good reasons). These classes will interact together and change together over time. Let's recall the definition of cohesion:

Cohesion: degree to which the elements of a module belong together.

"Agile design"

Another great consequence of that layout is that it leaves us more freedom when designing each module.

With the "classic" code structure it becomes a no-brainer to create a repository, map the entity using Doctrine, etc. That means duplicating and perpetuating existing solutions. One could argue that it's a good thing: it brings consistency across the whole project. But it also kills creativity and does not encourage to ask the right questions, such as:

• Is Doctrine the best solution for this module? Do we even need an ORM?
• This repository contains only 2 small SQL queries, would it make more sense to get rid of it and bundle those in the service?
• Does creating an entity make sense in that particular case?

Just like Agile encourages team to find solutions that work for them (i.e. there are no one size fits all solutions), this approach to code structure encourages to think and design each module based on its use case.

The differences between a "Product" and a "Product" in e-commerce

When I joined Wizaplace I was very surprised by something in the code: the products (the basis for an e-commerce website) were modeled twice in two different ways:

• The product in the back-office can be modified by the seller, it can be in an incomplete state, without stock, etc.

  This product is editable (with all the validation rules that go with this) and it is stored in database using many tables.

• The product visible in the shop can be added to a basket and bought.

  This product is a projection of the back-office product, with all its data denormalized into a single table (no need for "joins" when retrieving it). It is read-only and only "completed" and validated products are visible in the shop. For those interested, this is a form of CQRS.

I was not impressed so much by the technical side of things, but rather by the fact that this made a lot of sense business-wise.

This "product read model" was not only a matter of caching or optimizing performances: we just do not do the same thing with those two concepts. Having two separate models in the code allows to separate the logic related to catalog management to the one related to the online shop.

What does it have to do with code organization?

With a "classical" code structure we would end up with 2 Product classes in the same folder, which would obviously be a problem. We could get around this by being creative with names but this is still confusing. Good naming does matter, but at another level. Here is an example of what we could have had:

src/
    Entity/
        Product.php
        ProductReadModel.php
    Service/
        ProductService.php
        ProductReadModelService.php

When we decided to reorganize the code into functional modules we were forced to better understand the domain and to better integrate its vocabulary and concepts into the code:

src/
    Catalog/
        Product.php
        CatalogService.php
    PIM/
        Product.php
        PIMService.php

Thanks to numerous exchanges with the product and domain experts, we managed to put words onto all of this:

• The catalog, like paper catalogs or the catalog of amazon.com, only contains validated and published products. Those cannot be edited, but they can be bought.
• The PIM (Product Information Management) is a kind of CRM for products: this is where categories, products, descriptions, prices and more can be managed. There are even software products dedicated to solving this problem.

The same concepts are being modeled into the code, but in different ways. For those familiar with Domain Driven Design you may be thinking of Bounded Contexts and you would be very right, however I do not think DDD should be a requirement to benefit from that kind of code structure.

To conclude, organizing code into domain modules is not a silver bullet but it forces to better understand the problem we are solving and better structure our code.

I follow a checklist whenever I am planning to speak at a conference. I am sharing it so that hopefully it can help you too.

The steps marked with ? are the one I failed before and don't want to fail again :) It also let me justify why something that sounds silly may be part of the list!

The checklist is quite long and might seem paranoid or more stressful than helpful, but it's actually the contrary: the more you anticipate things, the less reasons you have to stress.

Planning the talk

• I plan on writing up the structure or the story before starting the slides (just like code or books, the structure must come first)
• ? I have planned to finish the first version of my slides at least one month before the talk (because I know that the first version will suck and will be completely rewritten)
• I have planned to practice the talk from beginning to end at least 10 times: that means at least 10 days (once per day)

Writing the slides

• ? There is an underlying theme, story or structure in my slides: readers should not feel lost in an endless succession of unrelated slides
• I am using a template or standard design for the slides (unless I know what I am doing)
• ? All content is as big as possible and will be readable from the back of the room
• ? All code is as big as possible and will be readable from the back of the room (yes, code needs its own line because it's the easiest thing to mess up): ideally less than 5 lines of code per slide
• ? All useless code has been removed
• Code has syntax highlighting
• ? Unless the conference told me explicitly (with complete certainty) that the projector will be 16:9, I prepared my slides for a 4:3 format (if unsure choose 4:3)
• I did not use animated gifs, or if I did they play only once and then they stop (there is nothing more distracting than that)
• Less is more: the slides contain only the most important words that I am going to say, the rest will be said out loud (the audience cannot read and listen at the same time)
• I have a slide at the beginning to introduce myself (quickly) and I try to make it relevant to the talk (in other words: "Who am I and why am I here talking to you about this topic")

Practicing the talk

• ? I can answer thes questions "What is the message I want to convey?" (thank you @rgousi) and "Why is that message interesting to the audience?" (thank you @Louis_Remi)
• I have practiced my talk multiple times until I feel confident
• ? I have practiced my talk without my speaker notes (to simulate the case where I cannot rely on them for some reason the day of the talk)
• ? I have practiced my talk until I consistently deliver it under the allocated time (being too fast is an acceptable outcome, being too long is not because organizers will stop me)
• I can deliver my talk without reading my slides
• I have recorded myself and listened to/watched the recording (and cringed at my voice) at least once (this is super helpful)
• I have practiced my talk in front of somebody at least once (as early as possible), and I have asked for feedback (choose your person carefully, you want brutally honest feedback here)
• ? I have practiced several times with the tool I will use to switch slides (e.g. a remote)
• ? I have practiced my talk holding a fake microphone (because it will be distracting when on stage)
• ? I have practiced drinking water while holding the fake microphone and the remote at the same time (this is super hard when stressed, prepare for this); practice drinking at the right time and away from the microphone if possible
• ? I know which slides to skip if I ever have to (skipping the conclusion or the most interesting parts is the worst, anticipate that)
• ? If I need to use fake names or pronouns I try to alternate genders (or use they) so that I do not always represent developers as males
• I have practiced to pause or change rhythm on the most important parts
• ? I know the introduction by heart (this is the most stressful moment, the brain will be turned off at that point, it needs to be easy)
• ? I know the conclusion by heart (this is what people will leave with, it needs to be as polished as possible)
• ? I end the talk with "Thank you" or a sentence that starts with those words, that is the universal signal for applause; ending with "Are there any questions?" is awkward because everyone will be unsure if they should clap or if they should stay silent for questions

The day before

• I have stored the exact same version of my slides in 3 places: on my computer, online and on a USB key
• ? I have tested all 3 versions and made sure they work
• ? I have tested that both offline versions (computer and USB key) work without internet access (remember the CDN you used to load JS or fonts?)
• I have everything I need to connect my computer to: HDMI, VGA and if necessary Ethernet (tested and working)
• I have charged my laptop
• I have checked that my remote is working
• I am not planning to update my OS or any tool running on my laptop
• If possible I have checked in the room I will be speaking in
• ? I have chosen my clothes: they are comfortable, I feel good in them and they do not show sweat marks; I am also ready to be in t-shirt or sweater: the stress can make me very hot or very cold and I do not know in advance which one it will be :)

Minutes before

• I have gone to the bathroom (this is stupid but it's easy to forget)
• ? I have walked 10 minutes beforehand to evacuate the stress
• ? I have disabled the screen saver of my laptop and made sure it's not going to sleep
• I have a clean desktop and neutral background picture
• I have my remote charged and setup
• I have all my adapters close by
• I have my plan B ready: USB key
• I have my plan C ready: online slides with wifi setup or my phone ready to serve as 4G access point (but keep in mind that most of the time the wifi or 4G doesn't work in conferences, that's why it's plan C)
• ? I have put my phone on "Do not disturb" or airplane mode: never on vibrate, my pocket will vibrate because of Twitter mentions and stuff like that, and it will distract me
• ? I have enabled "Do not disturb" on my laptop to avoid popups (or disabled the wifi)
• I have closed applications I will not need
• I have muted my laptop (unless I need to use the audio)
• ? I have opened the slides and they are ready to roll
• I have a bottle of water ready
• I arrived in the room in advance (I do not attend a talk in another room right before my talk)
• ? I located the tech person or organizer, I know who to ask for help if there's a problem while setting up my laptop
• I am ready to accept that at least 1 thing will go wrong at some point but nobody will remember or even notice

During the talk

• ? I breathe calmly
• ? I drink if I need
• ? I remember not to stress because what matters is the message, not my performance

After the talk

• I linger a little bit after the talk and engage people that seem like they want to talk to me (unless I don't feel like it); But I don't linger in the room if there is another talk right after me, I take the discussion outside
• ? I do not ask people how was my talk: if people were happy with the talk they will say so, if they weren't that will either force them to lie or force them to say something they might not want (of course this may not apply to everyone)
• I share my slides on Twitter or the Joind.in event
• If the talked was recorded I watch the video (and cringe again) and take note of everything I can improve

More

If you speak french you can read the great series written by Pascal Martin on his blog. It covers the topic in more details.

Comments

19 June 2017 - Matthieu Napoli

Yes. This article is about using non-breakable spaces to name tests. And the fact that it's awesome. And why you should use them too.

public function test a user can add a product to a wishlist()
{
    // ...
}

The code above is valid PHP code and works. Non-breaking spaces (aka   in HTML) look like spaces in editors but are actually interpreted like any other character by PHP.

This is the kind of test that we usually see:

public function testAddProductToWishlist()
{
    // ...
}

This is also the kind of method names we were writing at Wizaplace a few years ago.

At that time I was very impressed by a couple of talks which got me interested in coding style. Using snake_case instead of camelCase started to make sense in test methods because it lets us explain much more clearly what the test does:

public function test_a_user_can_add_a_product_to_a_wishlist()
{
    // ...
}

(☝️ this is not PSR-2 compliant, I was dubious but yes, with time it's possible to get over it)

We ended up discussing this kind of naming in the team. Fortunately, this was also the time we were joking about writing a PHP 6 framework and playing with emojis as class or method names (which definitely works in PHP).

At some point someone trolled:

if we decide to not follow PSR-2 naming for test methods because of readability, we might as well use non-breakable spaces since it's even more readable…

That started as a joke but it just made sense. Since logic and humans don't always mix well together, we decided to try a small controlled experiment for a while and see if it was actually great in practice.

public function test a user can add a product to a wishlist()
{
    // ...
}

And it was great. So great that it has been more than a year now and we are still completely happy with it.

Test methods are clear and meaningful, here is an actual example of a diff of one of our pull request:

-     public function testProjectMultiVendorProductWithOneDetached()
+     public function test product and multivendor product projections are both updated when they are detached()
      {

Since test methods look like sentences, we think of them as sentences and it makes all tests much clearer. Here are a couple more examples:

public function test very long slugs are truncated()
{
    // ...
}
public function test there are no projects by default()
{
    self::assertEmpty($this->projectService->getProjects());
}

How to actually type a non-breaking space?

This is very easy and quickly memorized:

• MacOS: Alt+Space
• Ubuntu: Alt Gr+Space (Alt Gr is the right Alt key)

Does it work with all the tools?

In our experience yes, all the tools we use below work perfectly fine:

• git
• PhpStorm
  • update: the shortcut will invoke the "Quick definition" helper, you will need to disable (or remap) that shortcut; "Quick definition" is also available via Cmd+Y or Ctrl+Y by default so remove the shortcut should be enough
• Sublime Text
• GitHub (and formerly Gitlab)
• PHPUnit's integration in PhpStorm (right-click and "Run" still works)
• PhpStorm's analyzer and refactoring tools:

We have seen minor issues on Atom and Visual Studio Code (syntax highlighting was off), those were fixed by Florent in the following pull requests: atom/language-php#196 and Microsoft/vscode#26992.

Does it work with all the humans?

This might be the most difficult part: getting other humans on board. Every new colleague that joins the team has that "WTF" moment when reading the code. That goes against the principle of least astonishment, but we are so proud and happy with non-breaking spaces that explaining it is always a fun time :)

In our experience colleagues got on board pretty quickly, both junior and senior developers. Our team is still small though, it might be more difficult to introduce this in a large organization with multiple teams.

How can you tell if you've typed a classic space by mistake?

Don't worry, you'll see it immediately:

Again, in our experience it was never an issue.

Can it work in open source projects?

That is the only reservation I have at the moment. Using non-breaking spaces in a closed source project where the team can own the code and share the decisions is easy: if it works, keep doing it, else stop.

In open source projects it's more complex since most contributors will get their WTF moment without you by their side to explain. It may be confusing or even off-putting.

My personal stance for now is:

• use this approach for small projects that will probably get no contributors
• spread that practice as much as possible (this is what this article is about)
• hope that it catches on to be able to use it more and more without

15 January 2017 - Matthieu Napoli

Anonymous classes were added to PHP 7. This article intends to show how useful they can be for writing tests.

Mocks

Imagine the following interface:

interface UrlGenerator
{
    public function generateUrl($route);
}

If you are testing a class that depends on a UrlGenerator implementation, you can write an anonymous class to implement this on the fly:

class MyTest extends TestCase
{
    public function testSomething()
    {
        $urlGenerator = new class() implements UrlGenerator {
            public function generateUrl($route)
            {
                return '';
            }
        };
        $foo = new Foo($urlGenerator);
        // ...
    }
}

In the example above we have written a very dumb implementation. That's often useful when the mock will not be used or its behavior isn't important for the use case we are testing.

Spies

If we wanted to spy on invocations of the mock, it's trivial as well:

    public function test()
    {
        $urlGenerator = new class() implements UrlGenerator {
            public $routeToGenerate;
            public function generateUrl($route)
            {
                $this->routeToGenerate = $route;
                return '';
            }
        };
        $foo = new Foo($urlGenerator);
        // ...
        self::assertEquals('abcd', $urlGenerator->routeToGenerate);
    }

Depending on the scenario, writing mocks or spies using anonymous classes can be a bit more verbose than using a mocking framework (like PHPUnit, PhpSpec, …), so it's not a silver bullet. However it's worth keeping in mind that vanilla PHP code is often easier to understand and debug than a mocking framework.

Fixture classes

Sometimes when writing tests, you need fixture data. But in some cases you need fixture classes. That can happen when testing tools that use reflection (I've faced that a lot with PHP-DI for example).

Writing 5-10 fixture classes is verbose, boring and more importantly it makes tests very hard to read: you have to jump between separate files to retrieve and understand the fixture classes.

Here is an example of a test relying on fixture classes:

class SomeDependencyInjectionContainerTest extends TestCase
{
    public function testSomething()
    {
        $container = new Container();
        $obj = $container->get(Class1::class);
        self::assertInstanceOf(Class1::class, $obj);
        self::assertNotInstanceOf(Class2::class, $obj);
    }
}
class Class1 {
}
class Class2 {
}

Class1 and Class2 and "fixture classes", i.e. those are classes written only for tests. You can put them inside the file containing the test class (but it's not PSR-4 compliant and it leads to conflicts if you have similar classes in other files of the same namespace), or in separate files (but then it's very hard to keep a track of which test uses which fixture class).

Here is an alternative written using anonymous classes:

class SomeDependencyInjectionContainerTest extends TestCase
{
    public function testSomething()
    {
        // We create 2 fake classes on the fly
        $class1 = get_class(new class() {});
        $class2 = get_class(new class() {});
        $container = new Container();
        $obj = $container->get($class1);
        self::assertInstanceOf($class1, $obj);
        self::assertNotInstanceOf($class2, $obj);
    }
}

Don't let get_class(new class() {}); confuse you, it simply allows us to declare a new class and get its name. It's the same as doing this:

class Abcd {
}
$obj = new Abcd();
$className = get_class($obj);

With this solution fixture classes are declared inside the test, which means you only need to read the test method to understand the test entirely.

The obvious downside is that you need to understand the concept of anonymous classes, which isn't very widespread today in PHP. But like all new language feature, with time it should be more and more familiar to everyone.

12 November 2015 - Matthieu Napoli

Let's talk about how to format code and name things.

I remember when PSR-1 and PSR-2 became a thing. Jeez that "use only spaces" thing was just ridiculous, I knew that tabs where better (right?). And those crazy formatting rules were the opposite of what I was used to. Nonsense! One day I did the jump and after a few weeks, I just didn't care anymore. I was over it, just like everyone else did, and PSR-2 was my new religion.

Fast forward a few years and I'm at a talk where the speaker argues to use snake case for naming test methods. Dude, you crazy? We just finally got consistent style across all the PHP world! And we all know that camelCase looks much better right? How can I go against PSR-2? It turns out, after trying it, that this was a wonderful idea…

Habits are sometimes making us blind. We think X looks prettier than Y but that's just the habit speaking. In this article I'll try to take a rational approach at coding style. That means leaving the "it looks ugly/better" at the door.

If at any point you feel like something "just doesn't look good", breath in, breath out, and try it! Nothing beats hands-on experience, not even some random article on the internet :)

Trailing commas?

Should we use trailing commas or not? If you are not familiar with the practice, here is an array using trailing commas:

$flavors = [
   'chocolate',
   'vanilla',
];

As you can see, the last item ends with a comma even though there is no extra item after it. It's perfectly valid PHP, the trailing comma is simply ignored. Now consider this example which is not using trailing commas:

$flavors = [
   'chocolate',
   'vanilla'
];

Let's add a new value to the array:

$flavors = [
   'chocolate',
   'vanilla',
   'lemon'
];

Here is the diff we generated in the commit:

$flavors = [
   'chocolate',
-   'vanilla'
+   'vanilla',
+   'lemon'
];

With trailing commas, the diff looks like this instead:

$flavors = [
   'chocolate',
   'vanilla',
+   'lemon',
];

As you can see, trailing commas lead to simpler and cleaner commits. This is also very useful when using git blame: each line points to the real commit that added it. Conclusion: use trailing commas.

It's interesting to note that there's currently a RFC to allow trailing commas everywhere in PHP (not just arrays). Obviously, I think it would be a great addition to the language for the reasons explained above, as well as for the sake of consistency.

Value alignment?

This practice is fairly common, though not universal. PSR-2 doesn't state anything about it, but many projects enforce such rule. Here is an example with phpdoc:

/**
 * @param int             $id       ID of the thing to export.
 * @param string          $filename Name of the file in which to export.
 * @param LoggerInterface $logger   Used to log the progress of the export.
 */

And here is another one with arrays:

$formTypes = [
   'text'     => new TextField,
   'select'   => new SelectField,
   'checkbox' => new CheckboxField,
];

Or for assignments:

$firstname = 'Spongebob';
$lastName  = 'Squarepants';
$age       = 25;

Anyone who has ever modified such code knows: it's a pain. Sure it may "look good", but when modifying that you have to fill in all the extra spaces to keep the alignment. One may argue that an IDE can re-arrange that for us, but even then let's look at the diff when adding a new item to an array:

$formTypes = [
-   'text'     => new TextField,
-   'select'   => new SelectField,
-   'checkbox' => new CheckboxField,
+   'text'           => new TextField,
+   'select'         => new SelectField,
+   'checkbox'       => new CheckboxField,
+   'my_custom_type' => new CheckboxField,
];

Now all the git blame information is lost and the commit is almost unreadable.

Alternatively, if you don't bother keeping the alignment:

$formTypes = [
   'text'     => new TextField,
   'select'   => new SelectField,
   'checkbox' => new CheckboxField,
   'my_custom_type' => new CheckboxField,
];

The whole point of the alignement is lost, and there goes consistency.

Another example following an IDE refactoring (LoggerInterface was renamed to Logger):

/**
 * @param int             $id       ID of the thing to export.
 * @param string          $filename Name of the file in which to export.
 * @param Logger $logger   Used to log the progress of the export.
 */

We have all seen that!

To sum up on aligning values:

• it requires extra work to maintain
• it messes up diffs and git blame
• it leads to inconsistent alignment over time

Conclusion: do not align things with spaces.

Minimal phpdoc

Using phpdoc to document classes, functions, methods, etc. is good practice. However code considered as "well documented" usually looks like this (on GitHub):

/**
 * Constructor.
 *
 * @param HttpKernelInterface $kernel An HttpKernelInterface instance
 * @param string $cacheDir The cache directory (default used if null)
 */
public function __construct(HttpKernelInterface $kernel, $cacheDir = null)
{ ... }

There is much content in this docblock, but most of it is duplicated from what developers or tools can already get from the source:

• we get that it's a constructor, the method is __construct()
• we get that $kernel is a HttpKernelInterface, the parameter is type-hinted
• we get that a HttpKernelInterface type-hint means that the parameter must be "An HttpKernelInterface instance" (this comment has no added value)

In reality here is what the docblock provides that isn't provided by the code itself:

• $cacheDir is a string (or null)
• if $cacheDir is null, the default cache directory will be used

The docblock could be reduced to this:

/**
 * @param string|null $cacheDir The cache directory (default used if null)
 */
public function __construct(HttpKernelInterface $kernel, $cacheDir = null)
{ ... }

On top of being information overload, the duplication also becomes a problem when the code changes. Everbody has seen a docblock that doesn't match the method it describes. When information is duplicated this is much more likely to happen.

All of this is becoming even more interesting with PHP 7 (coming with scalar type-hints and return types). Here is another example to illustrate that:

interface KernelInterface
{
    ...
    /**
     * Gets the name of the kernel.
     *
     * @return string The kernel name
     */
    public function getName();
}

With PHP 7 the docblock would become entirely useless:

interface KernelInterface
{
    ...
    public function getName() : string;
}

Conclusion: use docblocks only to add information.

The "Interface" suffix

This is a topic that has already been debated (the first link is the best by the way) so let's get right to the point: in most cases, there is no need to have Interface in the name of an interface.

Let's take that example:

class Foo
{
    public function __construct(CacheInterface $cache)
    { ... }
}

One would argue here that the fact that we ask for an interface explicitly (it's visible through the name) is good: we know we ask for an interface. If the type-hint was for Cache, we wouldn't be sure whether we are asking for an interface or for an implementation.

My answer to that is that from the consumer's point of view, it doesn't matter if it's an interface or an implementation. What matters is that Foo needs a cache, end of story. There's a principle behind interfaces: either you have one implementation and you don't need an interface, either you have many implementations and you create an interface.

If for some reason there's only one implementation of the cache, then fine: just give me an instance of the class! If however there are multiples implementation of the cache, then each implementation exists for a specific reason. And that should be visible in the name. There is no reason there would be one implementation named Cache. There would be RedisCache, FileCache, ArrayCache, etc.

In the end, when we use the name Cache we either type-hint against the one implementation (no interface exists), or either against the interface. All is well!

I believe we have issues with this because :

• we are sometimes tempted to create interfaces when it's not needed, just because interfaces sound like bonus points towards clean code (thus leading to Foo implements FooInterface)
• it requires coming up with unique names for implementations, and naming things is hard

But even though this is challenging our habits, getting rid of *Interface forces us to think: better names and no unneeded interfaces.

For example if UserRepository is an interface, then you are forced to find a more specific name for your implementation. And you come up with DoctrineUserRepository and you realize that there could as well be EloquentUserRepository, PdoUserRepository or InMemoryUserRepository. Interfaces makes much more sense when they are the default. Implementations are secondary.

Let's keep in mind however that in some cases (for example in libraries/frameworks), interfaces are introduced only to allow a third party to replace the default implementation. We then have an interface with one implementation (other implementations are to be written by framework users). Given that context isn't the same as what was described above, it's harder to apply the same principles. In that case I can only advise to use critical thinking :)

The "Exception" suffix

What, again the suffixes? Yes!

this section is inspired from Kevlin's mind-blowing talk

Here is where we use exceptions in PHP:

• throw ...
• catch (...)
• @throws in phpdoc
• when creating an exception class by extending another one

That means that everywhere an exception class will appear, we will know it's an exception. Having the Exception suffix is then completely redundant.

Let's take an example: UserNotFoundException. The suffix brings absolutely no value. Even worse, it makes the (perfectly valid) sentence more obscure: UserNotFound is everything we need.

What's even more interesting is that, in some cases, removing the Exception suffix makes the name look quite bad. As an exercise, let's remove the suffix from Symfony Form's exceptions. By the way I'm taking examples from Symfony not because I believe it is bad code, but on the contrary because it is a very good code base (thus making the point stronger).

First let's start with the exceptions that would make perfect sense without the suffix:

• BadMethodCall
• InvalidArgument
• InvalidConfiguration
• OutOfBounds
• TransformationFailed
• UnexpectedType

Those sound like perfectly valid english sentences that explain an error.

Now let's look at those that sound more like an information than an exception or error:

• AlreadyBound
• AlreadySubmitted

Imagine that situation in real life: you want to get on the bus, but the driver tells you "no, somebody already got on". You would probably ask "So what?". The driver's problem is maybe that the bus is full. It's the same here: maybe a form cannot be re-bound if it has already been bound.

How about CannotRebind and CannotSubmitAgain as exception names instead?

Lastly, let's look at the exceptions left:

• ErrorMappingException (notice the combo error + exception)
• LogicException: exception/error in the logic
• RuntimeException: exception/error at runtime
• StringCastException: exception/error while casting to string?

Except RuntimeException and LogicException which are very generic exceptions (maybe they deserve to be more specific?), here are suggestions for the other two:

• InvalidMapping
• CannotBeCastToString

Not convinced? Have a look at your own code and do the same exercise, it might give you a new persepective on your errors.

Conclusion: the Exception suffix is unnecessary. Not using it also has the benefit of forcing us to come up with names that better describe an actual error.

Conclusion

In the end we only had a look at 5 actual examples, but I want to stress that the main point of this article is:

• it's possible to think about coding style logically
• sometimes doing so forces us to challenge our habits
• when unsure or dubious: just try

If you need to vent off on how some of this is stupid and ugly, there's a comment box below. I would also be happy to hear about practices you tried and you liked!

12 April 2015 - Matthieu Napoli

I have just released Silly CLI 1.1 and I think it's awesome, here's why.

What is Silly?

If you missed it, Silly is a small project I started a month ago. The idea is simple: it is a CLI micro-framework.

Much like Silex or Slim let you create a web application in a single file using closures as controllers, Silly let you create a CLI application in a single file using closures as commands:

$app = new Silly\Application();
$app->command('greet [name] [--yell]', function ($name, $yell, $output) {
    $text = $name ? 'Hello '.$name : 'Hello';
    if ($yell) {
        $text = strtoupper($text);
    }
    $output->writeln($text);
});
$app->run();

As you can see, no need for classes. Also, commands are defined using a syntax inspired from help and man pages which is short and reads nicely.

Running the application:

$ php app.php greet
Hello
$ php app.php greet john --yell
HELLO JOHN

Of course I didn't reinvent anything. All this is based on Symfony Console, which is great because:

• I'm lazy
• everybody knows the Symfony Console

This was Silly CLI 1.0, and it was good.

Scaling up

I like micro-frameworks because they allow us to start very simply and very quickly. They also generally don't force us into a specific architecture or organization which, sometimes, is good.

But the value of a micro-framework is even greater when it allows you to scale up. By that I don't mean "mongodb web-scale", I mean "my single script file is too big, I want to organize everything more properly and start doing complex stuff".

Callables

This is where a immensely useful concept of PHP comes into play: callables.

Most examples of micro-frameworks show you the "closure" example of callables because it's the most obvious one for a simple app:

$app->command('greet [name]', function ($name) {
    // ...
});

But as you probably know, closures are not the only kind of callables in PHP. Here is a list of valid PHP callables:

• a closure: function () {}
• a function name: 'strlen'
• an object method: [$object, 'theMethod']
• a class static method: ['TheClass', 'theMethod']
• invokable object: $object (an object that has an __invoke() method)

All that means is we can replace our closure with classes. Our simple example can then become:

class GreetCommand
{
    public function execute($name)
    {
        // ...
    }
}
$app->command('greet [name]', [new GreetCommand, 'execute']);

That's a nice first step to organize our growing application. Each command can be put in its own class and file.

Silly 1.1: Dependency injection

Putting our commands in separate files is a good start. But everyone knows that a growing application is not just about organizing code: it's also about managing dependencies.

Instead of turning towards singletons or global functions/static methods, Silly 1.1 helps you into using dependency injection. More specifically, it comes with dependency injection container support.

Choose your container

Silex, Slim, Cilex, … all force you into using a specific container (often Pimple). While this is practical at first, it's usually not so good for scaling up: the more classes or dependencies you have, the more configuration you will write.

Instead of forcing you into using a supposedly better DI container, Silly goes the interoperability route: use whichever container you want!

$app->useContainer($container);

The container you provide has to be compliant with the container-interop standard. If it isn't you can use Acclimate.

Callables in the container

By simply registering a container with ->useContainer(), Silly will retrieve commands from the container when they are not PHP callables:

// Valid PHP callables: doesn't use the container
$app->command('greet [name]', function () { /* ... */ });
$app->command('greet [name]', [new GreetCommand, 'execute']);
$app->command('greet [name]', ['SomeClass', 'someStaticMethod']);
$app->command('greet [name]', new InvokableClassCommand);
// Use the container
$app->command('greet [name]', ['GreetCommand', 'execute']); // execute is not a static method
$app->command('greet [name]', 'InvokableClassCommand');     // implements __invoke()

In the 2 last examples above, the container will be called to instantiate GreetCommand and InvokableClassCommand.

That means that you can configure those classes in your container and benefit from the dependency injection features of your container. For example:

class GreetCommand
{
    private $entityManager;
    public function __construct(EntityManager $entityManager)
    {
        $this->entityManager = $entityManager;
    }
    public function execute($name)
    {
        // ...
    }
}

Note that our class could also be an invokable class (i.e. one that implements __invoke()).

Dependency injection in parameters

Dependency injection as shown above only works when you write commands in classes. However you can't use it with closures!

To solve that problem, let's have a look at how dependency injection is performed in AngularJS:

angular.controller('MyController', function (myService, myOtherService) {
  // ...
});

Silly supports the same mechanism of dependency injection through the callable parameters. For example:

use Psr\Logger\LoggerInterface;
// the order of parameters doesn't matter:
$app->command('process [directory]', function (LoggerInterface $logger, $directory) {
    $logger->info('Processing directory ' . $directory);
    // ...
});

Silly can inject services and values into parameters by looking into the container using:

• the type-hint (i.e. the interface/class name): Psr\Logger\LoggerInterface
• the parameter's name: logger

Depending on how you declare your container entries you might want to enable one or the other way, or both.

$app->useContainer($container, $injectByTypeHint = true, $injectByParameterName = true);

If you set both to true, it will first look using the type-hint, then using the parameter name.

The PHP-DI and Pimple edition

Being able to use any container is nice, but sometimes you don't really care and just want to get started. You are covered!

I have created two packages that provide you Silly pre-configured with either PHP-DI or Pimple.

Those wanting the benefit of autowiring might like the former, while those familiar with Silex might feel more at ease with the latter.

Beyond Silly

I hope this example will show how a framework can be built without coupling to a dependency injection container. This was rather easy because it is a micro-framework, but I hope it will give others some ideas.

For those interested to learn more about these topics, everything explained here was actually implemented in a library called Invoker. Feel free to have a look at it and maybe use it in your projects. The same principles shown here could be applied to any framework dispatching to PHP callables. Hopefully it will catch on!

Thanks for @Darhazer and @najidev for reviewing this article.

17 December 2014 - Matthieu Napoli

Composer just got a new awesome addition thanks to Nicolas Grekas: prefer the lowest versions of your dependencies.

composer update --prefer-lowest

This amazing option will install the lowest versions possible for all your dependencies.

What for? Tests of course!

Update Composer

OK before we get started, don't forget to update Composer right now if you want to use the new option:

sudo composer self-update

Testing against lowest dependencies

It might not make a lot of sense to test an application against its lowest dependencies, because unless something is done wrong it will be installed with controlled versions.

However, for libraries (or components…) it's a very good thing.

To give you an example, I wanted to give it a try and ran composer update --prefer-lowest in PHP-DI's directory: all hell broke loose… PHPUnit wouldn't even start.

The reason for this were multiples, but they all boiled down to:

• I used a feature that didn't yet exist in version X of dependency Y
• or dependency Y had a bug in version X

and I allowed installing that problematic version X in composer.json.

In the end, I was distributing a library that could not work at all (if people had insane version constraints).

Setting up tests on Travis

Let's be proactive and use continuous integration to prevent that from happening again.

Here is an example of a .travis.yml configuration that would run the tests using the lowest dependencies with PHP 5.3.3:

language: php
php:
  - 5.3.3
  - 5.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm
matrix:
  include:
    - php: 5.3.3
      env: dependencies=lowest
before_script:
  - composer self-update
  - composer install -n
  - if [ "$dependencies" = "lowest" ]; then composer update --prefer-lowest --prefer-stable -n; fi;
script:
  - phpunit

You'll notice that we have to run composer self-update because Travis hasn't picked up the latest Composer version yet.

Here was the result on Travis:

A new job was added to the build matrix, and this job ran on PHP 5.3.3 with the lowest dependency versions.

If you are interested, have a look at the pull request.

21 September 2014 - Matthieu Napoli

Decoupling packages is a hard thing. There are not a lot of options, and this blog post is about how some options are better than others.

Let's say for example that you are writing a "package", or library, to respond to HTTP requests (that kind of package could be considered the basis for a web framework). How do you handle routing?

If you write your Router package as an independent package (which is good: small and specialized packages are more reusable and maintainable), you might not want to couple the HTTP package to the Router package: you want to leave users free to choose the router of their choice.

So, what are your options to make the HTTP package and the Router package decoupled from each other?

Events

The first option is to turn to event-driven programming. By relying on an "event manager" library/package, you can have your package raise specific events at strategic points in your code. And you can have those events affect the code flow.

This is the solution chosen by Symfony to solve the routing problem exposed earlier. Their HttpKernel component is decoupled form their Routing component through events.

Here is the simplified code flow of their HttpKernel class/component (which is the basis for an HTTP application):

$eventDispatcher->dispatch(
    KernelEvents::REQUEST,
    new GetResponseEvent($this, $request, $type)
);
$controller = $request->attributes->get('_controller');
if (! $controller) {
    throw new NotFoundHttpException();
}

So here is what's happening:

• the HttpKernel raises a "Kernel Request" event
• then it expects that a listener set the controller under the "_controller" key in the request

So is HttpKernel decoupled from the Router component? Yes. The listener that sets the controller could actually be anything.

The problem is, as emphasized above, that the HttpKernel expects something very specific from the listeners. The whole application depends on an unknown listener being actually registered for that particular event and following an exact, unspecified behavior.

I believe events should be used for hooking up in the main logic flow to extend it. But the main logic flow should be linear and not rely on the possible side effects of an event.

There are some other problems we can find with this solution:

• the package ends up coupled to the "event manager" package (you just replaced a dependency by another)
• the code is not linear: it makes it much harder for developers to put the pieces back together and contribute to the project
• behavior from decoupled packages are not specified by contracts

Regarding the pros:

• the behavior is not specified, which can also be a good thing: it leaves every possible option open for the future
• might be simpler than the other options

<disclaimer>Just to be clear, Symfony is an exceptional framework and I love it. I am sure the decision to choose this option was carefully thought through, and the important thing to remember is that it works. Symfony is probably the most popular modern PHP framework, and my blog post doesn't change that. I am just using it as an example here, I just want to expose alternative options and discuss the pros and cons.</disclaimer>

Interfaces and adapters

I was mentioning specifying behaviors with contracts. In PHP (and most OO languages), you can implement this using interfaces.

If we take the previous example, an HTTP package could contain a RouterInterface to specify how a routing component should behave (this is a very basic example):

namespace Acme\Http;
interface RouterInterface {
    /**
     * @return callable The controller to use for this request
     */
    public function route(Request $request);
}

You'll notice that not everything is specifiable, e.g. the return types. Hopefully PHP will allow that in its next major version, but until then the only solution is to use documentation.

In the HttpApplication class, we can use it in type-hints to accept any implementation (classic dependency injection here). And here is what the code logic would look like:

$controller = $router->route($request);
if (! $controller) {
    throw new NotFoundHttpException();
}

The code flow is linear and completely explicit. And the HTTP package is decoupled from any Router package!

The only problem left: Router packages would be forced to implement Acme\Http\RouterInterface if they want to be used with the HTTP package. Because of that, they end up coupled to it… So how do we decouple Router packages from the HTTP package?

A way to go around that is to use adapters:

namespace \MyApplication\Adapter;
class HttpRouterInterfaceAdapter implements \Acme\Http\RouterInterface {
    private $router;
    public function __construct(\Acme\Router\Router $router) {
        $this->router = $router;
    }
    public function route(Request $request) {
          $this->router->route($request);
    }
}

Thanks to that adapter, the Router class doesn't need to implement the RouterInterface. So it is completely decoupled from the HTTP package.

However, as you can see, it requires writing an adapter each time you want to "connect" decoupled packages.

Pros:

• linear and explicit code flow
• specified behavior (using interfaces)

Cons:

• requires to write interfaces
• requires to write adapters

This strategy of interfaces was recently used by Laravel. For the version 5.0 (IIRC), Laravel will publish a package named illuminate/contracts which contains all the interfaces used by its other packages.

That allows to have decoupled Laravel packages while not needing adapters to use them together: packages can implement the interfaces at the very small cost of being coupled to illuminate/contracts (it's a small cost because the package is very light and contains only interfaces).

Standardized interfaces

Now the last option is to go a step beyond and try to make the interfaces "standard". By that I mean that the same interface would be used by many packages, and implemented by many others.

The good example for this is obviously logging. There used to be a numerous amount of different logger libraries for PHP. Then the PHP-FIG group worked to produce PSR-3, the logger standard.

Today, many logging packages implement the Psr\Log\LoggerInterface, and most modern frameworks type-hint against that interface instead of specific implementations. That means that users can choose any PSR-3 compliant logger and have their framework use it.

Needless to say that this is an ideal situation: no coupling, no effort! But logging was kind of an easy topic. It's very hard to come up with standards for all the other components that need interfaces, mainly because implementations often differ a lot.

The PHP-FIG has been working for a few years on a Cache and an HTTP message PSR, and hopefully they will be released sometime. In the meantime, the container-interop project aims at providing interfaces to standardize the usage of dependency injection containers.

Conclusion

OK, there's not much left to add here, if you have any reaction about this I'd be happy to hear it. If I got anything wrong, I'd be happy to correct it.

I would like to finish on an idea that was suggested about a year ago on the PHP internals mailing list: "weak interfaces". Those are interfaces that define a behavior, but that do not need to be implemented by classes. It mixes the principle of static-typing with duck-typing:

If it looks like a duck and quacks like a duck, then it's a duck.

What's really good with this is that it allows packages to define their interfaces, and type-hint against it, all the while not requiring dependencies to actually implement it. As long as objects are compatible with the interface, it works. It's a sort of class X implements Y evaluated at runtime. Example:

interface FooInterface {
    public function hello();
}
// Foo does not implement FooInterface
class Foo {
    // But this method makes it compatible with FooInterface
    public function hello()
    {
        return 'Hello world';
    }
}
// That pseudo-syntax tells that this is a weak-interface type-hinting
function run(<<FooInterface>> $foo) {
    echo $foo;
}
// It works because Foo is compatible with the interface
run(new Foo);
// Error, stdClass is not compatible with FooInterface
run(new stdClass);

This example was just for fun, but I wish such a feature would land in PHP (along with static return type). It would help a lot with package interoperability and decoupling.

Update: the original RFC about what I called weak interfaces has been linked to me by its author, Anthony Ferrara. Here it is: PHP RFC: Structural Type Hinting. As you can see, it was named Structural type hinting instead of weak interface.

It has also been pointed to me in the comments that the same Anthony Ferrara wrote a userland implementation of this: ProtocolLib.

1 July 2014 - Matthieu Napoli

Today is my last day at My C-Sense, after almost 2 years working on creating and modernizing existing PHP applications. This has been an incredibly rich experience in an excellent work environment, I will surely miss this job and my colleagues.

Now is probably a good time to look back on our open source experience as a company, and as a developer. It all started when we moved all of our code to GitHub: we used a lot of open source software and libraries, and we thought we might try to contribute some day with some small internal projects too.

So that's how MyCLabs was born: let's keep the door open for open source because "why not".

jquery.confirm

The first project was born out of a simple need: a confirmation box on some buttons and links. We were using Bootstrap, but adding all the JS and HTML needed for such a simple thing was definitely worth a JS function. So instead of a quick and dirty solution, I turned to jQuery and learned how plugins worked, just for fun and because it wasn't a big investment. In no time, jquery.confirm was born (the demo is here).

The project was absolutely not ambitious, yet it's probably our most successful project yet. It has 30 forks, received 10 pull requests and has 13 releases. That's not much, but for a simple confirmation dialog, it's quite nice.

And what's interesting is that it's the perfect example of an open source "success" from the company's point of view: we receive much more contributions than we contribute to it ourselves. We get bugfixes "for free", and we even got our migration to Bootstrap 3 covered for us by a contributor. Which company wouldn't want that?

PHP Enum

The next project open sourced was a bit different. PHP Enum is an implementation of an Enum structure for PHP. That's probably not the first time you hear about something like that, so here is why we created one:

• SplEnum seems like a good solution, but it is actually a separate PHP extension that needs to be installed (which makes it useless IMO)
• the other open source implementations were not good enough: either they were too far from SplEnum (which means most of the time they were working in a weird way), either they were a direct clone of SplEnum and there was a bit of room for improvement. If you do a search today, you'll find other good libraries that were different or non-existant at the time

We ended up using it in some places in our application, and I ended up using it in PHP-DI. We received a few contributions, but obviously there's not a lot of room for bugs or new features since it's very simple.

In the end, this library has been installed more than 16 000 times, which is definitely a lot and that makes us happy (PHP-DI installs might represent a good share of that).

MyCLabs\ACL

MyCLabs\ACL is the one I'm the most proud of. We have been working on the ACL problem and access control for more than 4 years (I've been on and off at My C-Sense), changing, improving and rewriting our solution about every year. It has been a major headache, especially since it's a problem that leaks everywhere: in the model, at the database level, at the routing level, in the views, … And its impact on performance was definitely one of the biggest challenge.

We were always looking at all the existing PHP solutions, but none of them was ever even close to fit our need. We've always been amazed about this, because we are sure many other companies have the same needs as us.

So because of all this, our ACL system was something I definitely wanted to open source some day. But given its complexity, and the fact it was deeply coupled to our application, it was impossible. Then came Doctrine 2.4, and then 2.5, and it changed a lot of things. As the next big rewrite (again) was planned, I suggested we tried to make it open source. And that would not mean "just publish the code online". That would mean make the effort on making a completely decoupled library with tests and documentation.

It took a few weeks to build it, and the company paid for that effort. But it was greatly rewarded with a solution much more solid and powerful than we ever built.

Thinking and building our code on in a more generic way was definitely beneficial. We previously had an unstable, half-tested, undocumented, slow and unmaintainable ACL system. We wouldn't dare touch it in fear of breaking everything. We now have a very powerful, tested, fast and documented solution. Yes it took time, but it was definitely worth it.

And I think this library is a perfect example of how open source was beneficial, even without any external contribution.

And as a developer, making the library open source turned the most boring problem on earth (permissions!) into a fascinating one! I couldn't stop thinking about it out of work when I was working on it.

My C-Sense is now working on a v2, with a much more simpler configuration and usage (much less boilerplate code). That's so interesting that I'll probably be following the project and maybe helping out if I have the time ;)

MyCLabs\Work

The ACL project had been a success, so when we ended up moving away from Gearman and thus rewriting our worker system (coupling, coupling…), I suggested we tried building an open source solution for it. Here's MyCLabs\Work.

Of course, I had a look at everything that could fit the bill. But again, we needed a feature that wasn't provided by any open source library I could find:

That task is too long, it needs to run in background and the user will be notified by email when it finishes. But sometimes it can be very fast, so receiving an email is useless and looks stupid.

So we want to run the task in background, but if it takes less than 5 seconds, we need to show the result to the user directly (in the web page) as if it didn't run in background. And of course, the worker shouldn't send any email!

That looks like a very simple need from a user's point of view. But when you realize that it means you must have a bidirectional communication between the web page and the background worker, you start going "oh no no no".

Now we just ditched Gearman (because of too many bugs and installation problems), and we didn't want to tie ourselves again to a different work queue. So our library was built against 2 needs:

• an abstraction over different queue systems (RabbitMQ, Beanstalkd, InMemory…)
• an abstraction over the "run and wait if it finishes else …"

Of course, the run and wait wasn't possible on every queue system, so it needed to be something optional. So I ended up writing 2 interfaces:

interface WorkDispatcher
{
    public function run(Task $task);
}
interface SynchronousWorkDispatcher extends WorkDispatcher
{
    public function runAndWait(
        Task $task,
        $wait = 0, // time to wait, in seconds
        callable $completed = null,
        callable $timedout = null,
        callable $errored = null
    );
}

(the details are documented here)

For example, the Beanstalkd adapter doesn't implement SynchronousWorkDispatcher, whereas the RabbitMQ adapter does.

Now where it begins to be interesting is that there is no queue system that provides out of the box bidirectional communication with a worker (at least in the one I reviewed at that time, the one with a good PHP lib and the one that can be installed). After several days of thinking and trials, I managed to implement the runAndWait behavior in RabbitMQ with the use of a temp queues and some sort of high level "protocol". Maybe not the fastest solution on earth, but for our needs it wasn't a problem at all. If you are curious, you can checkout here how it is implemented.

In the end, this is really was this library does: it abstract 2 behaviors: simply run a task in background, and run a task and wait for its result.

MyCLabs\Work now has adapters for RabbitMQ, Beanstalkd and "In Memory" (i.e. for your development machine), you are welcome to push new adapters through pull requests. It's very simple if you don't implement the SynchronousWorkDispatcher interface, for example look at the Beanstalkd adapter.

DeepCopy

DeepCopy is a small utility that helps you create deep copies of objects. By deep copy, I mean when you want to duplicate/clone an object and it's sub-objects.

Imagine you need to provide the user a way to "duplicate a folder". The user will expect all the files and subfolders to be duplicated too. PHP's clone will do a shallow copy, which means it will only clone the root object, the properties of the cloned object will reference the same objects as the original object.

The "built-in" PHP solution would be to override __clone(), but that leads to quite complex code, and handling cycling dependencies is very hard.

DeepCopy will handle all that for you. And on top of it, it's completely configurable: you can skip properties, force them to keep their original value, or set them to null. DeepCopy also supports resolving Doctrine's collections and proxies.

ArrayComparator

ArrayComparator is a bit like DeepCopy: it's a small utility for comparing arrays containing objects. It let's you define callbacks that will be called when items are different or missing between arrays.

Contributions to other open source projects

While we use the MyCLabs organization to publish open source libraries, we also use it to host forks of other libraries. That had become quite handy when we proposed a bugfix for Doctrine: let's install our fork instead of the original repo. By the way, here is how to do it with Composer.

So up to now, we contributed mostly to Doctrine, the DoctrineExtensions and PHPExcel.

A word on documentation

This part is a bit of self-promotion :)

We ended up having 2 kinds of projects:

• small projects where the GitHub project and a single Readme would be enough
• larger projects with multi-pages documentation

For larger projects, we have been using Couscous to generate the websites from the Markdown documentation.

It is working pretty well, the local preview is useful and deploying is very simple. And with the newly implemented templates it proved to be even more useful: I have written a Couscous template for MyCLabs projects, and I use it for MyCLabs\ACL and MyCLabs\Work.

If you too want to use GitHub Pages to publish a website based on your documentation, give it a try.

Conclusion

This was a big list of most of My C-Sense's open source projects. There are even some more that I didn't mention (because the article is getting quite long), so check out MyCLabs page.

To conclude on the open source experience from the company's point of view (and this is my own opinion here), I can only find it beneficial.

Of course, don't expect to get contributions right away and have your software developed for you. Out of all the project we open sourced, we only got a dozen of pull requests. And you have to realize that it takes time to manage them, and the issues (which sometimes are just people asking for help). Remember than an open source project not maintained is hurting the community rather than helping it.

That being said, there are many advantages:

• an open source library usually mean better quality, because that code is public. I would much more restrain from committing a dirty hack if I know the code will be online for years :p But mainly it really help the developer to take a step back and really think about what the library is supposed to do in a more generic way, and also to be more open to other implementations.
• a company involved in open source projects looks sexy to developers. Recruiting is (I presume) easier, and also it allows to attract and recruit good developers: they know what kind of code and quality level to expect, just like the recruiter does when he browses GitHub profiles of candidates.
• open sourced code get fixes and improvements… sometimes. And for free… almost (if you consider the time needed to merge/handle the tickets)
• developers do a better work: publishing an open source project online, or contributing to one, is a great experience. You contribute for something more timeless than your job. The code you write might be used by your peers across the world for year! That's challenging and exciting.

The last point is also about the developer's point of view: working on open source projects is an additional source of motivation and involvement. And of course it benefits everyone.

So, companies: as long as you don't make money over it, think about open sourcing it?

Now I will finish with a very warm thank you to My C-Sense and its founders for trusting us and letting us try. Open source has had a very positive result up to now.

27 March 2014 - Matthieu Napoli

This article logically follows the previous: The Repository interface. In there I suggested a better interface for repositories based on a Collection interface so that repositories could be manipulated like collections.

This article goes further in that idea and presents an API to abstract database access behind collections of objects.

But first let's start with Doctrine Criterias.

What are Doctrine Criterias?

When you use Doctrine, you end up interacting with list of objects in two ways:

• through repositories or the entity manager
• through associations

At the entity manager or repository level, you can write DQL queries (or use the query builder that will generate them for you). This is the most powerful API Doctrine provides to interact with the database.

But inside objects you don't have access to the entity manager or the repository, because the model shouldn't know about the persistence. So what do you do when you want to filter an association without loading all the objects? And furthermore: what do you do if the association is already loaded in memory? That would be stupid to issue a DB query when you could just filter the objects in memory.

That's where Doctrine introduced the concept of Criteria. It's an API that allows you to filter an association with an abstract "query" (≠ DB query) that you apply to a collection. What's good is:

• if the collection is already loaded, the filtering happens in memory with PHP
• if the collection is not loaded, the filtering is done in a DB query

The Criteria has a limited matching language that works both on the SQL and on the PHP collection level. This means you can use collection matching interchangeably, independent of in-memory or sql-backed collections.

That's awesome!

Example of how to use it:

$criteria = Criteria::create()
    ->where(Criteria::expr()->eq('birthday', '1982-02-17'))
    ->orderBy(array('username' => Criteria::ASC))
    ->setFirstResult(0)
    ->setMaxResults(20);
$birthdayUsers = $userCollection->matching($criteria);

The Criteria API is obviously much more limited than DQL, but it's completely decoupled from persistence: you can use it in your model without polluting it with persistence problems, and you can use it against collections, loaded or not.

And the greatest thing of all: Doctrine developers didn't stop here, they also made the Repositories be compatible with those Criterias. Which means the same API whether it's a repository or not. If you read my previous article, you know how much I like that.

What's wrong then?

Well something has to be wrong else I wouldn't be writing this, and you wouldn't be wasting your time reading it.

• you cannot filter on associations of objects inside the collection, i.e. JOIN (which is pretty common)
• you cannot perform updates or deletes in bulk without loading the objects first (like set published = true for all articles in this collection or repository)
• the API uses persistence words, like "where", "order by"… It's not so much a biggy but still, it's not perfect
• you cannot chain criterias and have only 1 DB query: 1 Criteria = 1 query

The last point is a bit vague so let me show you an example:

class Blog
{
    public function getPublishedArticles()
    {
        $criteria = Criteria::create()
            ->where(Criteria::expr()->eq('published', true));
        return $this->articles->matching($criteria);
    }
}
$articles = blog->getPublishedArticles();
$criteria = Criteria::create()
    ->setFirstResult(0)
    ->setMaxResults(20);
$articlesToDisplay = $articles->matching($criteria);

Here all the published articles will be loaded in memory, and then the first 20 results will be kept.

The Collection interface

Before diving in a better alternative to criterias, let's start back with an awesome Collection interface:

interface Collection extends Countable, IteratorAggregate, ArrayAccess
{
    function add($element);
    function get($key);
    function contains($element);
    function remove($element);
    function clear();
    function toArray();
    function count();
    function filter(Expr $predicate);
    function sort($field, $direction = 'ASC');
    function slice($offset, $length = null);
    function map(Closure $function);
}

(let's also assume that the Repository interface also extends that interface :) )

Filtering!

This interface looks a lot like the Doctrine Collection interface, except that filter doesn't take a Closure anymore but an "expression", which is where the fun begin. Because it's not PHP code but an expression object (like the Criteria expression object), we can apply the same filtering in memory and in database.

Now we are get out of the scope of the interface and have a look at the implementations: what if filter, sort and slice where lazy?

What I mean is these methods would return a new "lazy" collection that would only be loaded/computed if used. That allows some pretty nice chaining!

Example:

$results = $articles->filter(Expr::eq('title', 'Hello'))
        ->filter(Expr::eq('author', $john))
        ->sort('date', 'DESC');
// Perform a COUNT() query in database
echo count($results) . ' results';
// Fetches only 10 results from the database
$pageResults = $results->slice(0, 10);
// Executes on loaded objects, no DB query here
$allAuthors = $pageResults->map(function (Article $article) {
    return $article->getAuthor();
});

This code would only issue 2 queries:

-- Count the total number of articles
SELECT COUNT(art.id) FROM Article art
INNER JOIN User ON User.id = art.author_id
WHERE art.title = 'Hello'
ORDER BY art.date DESC
-- Fetch 10 articles for the current page
SELECT ... FROM Article art
INNER JOIN User ON User.id = art.author_id
WHERE art.title = 'Hello'
ORDER BY art.date DESC
LIMIT 0, 10

What about updating and deleting?

In the same spirit, you could imagine an API to update or delete items from a collection or repository:

// Delete Bob's articles
$articles = $articles->filter(Expr::eq('author', $bob));
$articles->clear();
// Publish Alice's articles
$articles = $articles->filter(Expr::eq('author', $alice));
$articles->apply(Expr::set('published', true));

Here the delete and update queries will include the previous filtering, such that there is only 1 query executed for Bob, and 1 for Alice.

Of course, that is if the objects are not loaded in memory. If they are, then the objects are updated.

Conclusion

This article was just an idea thrown around for discussion. There is no library available, or repository containing any code.

To sum up all this quickly:

• a repository is a collection (and much more too)
• you can manipulate collections and chain filtering, sorting, etc... in a very readable way while knowing you don't trigger dozens of DB queries
• the API shown here could be a more powerful alternative to Criterias
• it can also provide a simpler alternative to DQL/Query Builder with the benefit of being persistence agnostic and usable in entities
• it could allow pagination and sorting in the controllers much more easily (you wouldn't need to add these as parameters in your repositories for example)
• it would allow for more optimized database access (counting the collection would just issue a COUNT query for example)

Drawbacks

I can't finish that article without expressing what I consider as the major drawback behind all this: you are a little tiny bit persistent-aware, because you don't filter your collections with PHP code but with an arbitrary expression language.

That doesn't mean your code is coupled to the persistence layer though because it also completely works in memory, but you deliberately don't use PHP code because you know that it's not optimized to load all the objects from the database and then filter them with PHP.

I would say that's an acceptable compromise, because let's be honest, there is no perfect solution. You can't apply PHP code to a SQL database so you have to compromise somewhere…

Penumbra

Or can you? ;)

That's the bonus note on which I will finish. There is another approach (rather controversial) taken by a new project named Penumbra. Instead of playing with "expressions" to filter collections, it allows to filter using PHP code! It then parses that code to turn it into SQL code. Pretty audacious, but apparently it works (if you keep it reasonable) :)

Example:

$MiddleAgedUsersRequest = $UserRepository->Request()
        ->Where(function (User $User) {
            return $User->GetAge() > 20 && $User->GetAge() < 50 ;
        })
        ->OrderByDescending(function (User $User) { return $User->GetLastLoginDate(); });
$SomeActiveMiddleAgedUsers = $MiddleAgedUsersRequest->AsArray();

Will result in:

SELECT Users.* FROM Users
WHERE Users.Age > 20 AND Users.Age < 50
ORDER BY Users.LastLoginDate DESC;

I will not comment on whether this is good or not, because honestly I don't know. It seems like it's heavily tested and it's a serious project, so it's not just a POC or hack. I'll let you make your own mind :)

10 March 2014 - Matthieu Napoli

Here is the definition of a repository for Eric Evans in his Domain Driven Design book:

A repository represents all objects of a certain type as a conceptual set (usually emulated). It acts like a collection, except with more elaborate querying capability. [...] For each type of object that needs global access, create an object that can provide the illusion of an in-memory collection of all objects of that type.

While I love Doctrine, I really dislike their repository interface because it doesn't look and act like a collection. And that's too bad because Doctrine even provides a very good abstraction for collections through the doctrine/collection project. It even supports filtering with criterias over collections and repositories.

I know that Doctrine is not targeted at Domain Driven Design only, but I think having a better repository interface would still benefit the project and the users.

Here is a basic repository interface I tend to use instead:

interface Repository
{
    function add($entity);
    function remove($entity);
    function count();
    /**
     * @throws NotFoundException
     * @return object
     */
    function get($id);
    /**
     * @return array
     */
    function find(Criteria $criteria);
    function toArray();
}

Of course, you should use this interface as a base and write your own repository interfaces for each aggregate root.

Collection verbs

Why use verbs likeadd and remove instead of load, save, delete, persist, …?

Because those are persistence-related terms and have nothing to do in an interface that is going to be used/extended in the domain layer.

Get versus Find

I really dislike that you can only find in Doctrine: it will return null if no entity is found.

Most of the time, that is not what you want. You actually don't want to "search and find" the entity, you just want to get it by its id and you assume it exists.

That's why you need to clearly differentiate between find and get:

• method starting with get should always return an entity, or throw an exception if not found
• method starting with find should always return a collection (that could be empty)

Going further: the collection interface

I've said it already, but a repository should behave like a collection. The interface shown above is not completely satisfying yet because it doesn't totally behave like a collection. For example you can't iterate it.

So the best solution is simple: write a sensible collection interface and have the repository extend it.

interface Collection extends Countable, IteratorAggregate, ArrayAccess
{
    function add($element);
    function remove($element);
    function clear();
    function contains($element);
    function get($key);
    function find(Criteria $criteria);
    function toArray();
    function slice($offset, $length = null);
}
interface Repository extends Collection
{
}

(this is a very simple version, not exhaustive at all)

Now this is much better. You can even type-hint against the collection and accept at the same time collections and repositories!

You can then iterate or filter the collection without having to care what the object really is. For example, you can write a ProductCriteria and use it both on the repository and collections of products.

Example: let's say you write a controller to display a product list:

class ProductListController
{
    /**
     * @var ProductCollection
     */
    private $products;
    public function __construct(ProductCollection $products)
    {
        $this->products = $products;
    }
    public function listAction()
    {
        return new View('product-list.twig.html', [$this->products]);
    }
}

Here your controller is completely reusable. If you give it the ProductRepository, it can display the whole product list. If you give it the list of favorite products of the user, it will work too. Thank you dependency injection!

And the day PHP gets generics, that will be even nicer (Collection<Product>) but that's a debate for another day!

23 September 2013 - Matthieu Napoli

If your application sends emails, you probably don't want emails to be sent when you are developing on your machine.

If you use nice libraries like SwiftMailer, it is easy to use a mock instead of sending real emails. But if you don't, you can act on PHP's configuration: instead of installing and using a real SMTP server on your machine, you can fake one using a simple script.

The fake server will be a shell script: create it as /usr/local/bin/sendmail-fake:

#!/bin/bash
{
date
echo $@
cat
} >> /var/log/sendmail-fake.log

Set up file permissions and check that it works:

$ sudo chmod +x /usr/local/bin/sendmail-fake
$ sudo chmod 777 /var/log/sendmail-fake.log
$ /usr/local/bin/sendmail-fake

Now configure PHP to use it in the php.ini:

sendmail_path = /usr/local/bin/sendmail-fake

(and restart Apache)

That's it!

You can also see the emails content in /var/log/sendmail-fake.log.

You could even go further and edit the script to open the content of the mail into a text editor, or a browser. But that would be OS dependent, so I kept with the log file here.

13 September 2013 - Matthieu Napoli

I started working a few months ago on MetaModel, a language that enables to traverse and operate your PHP model.

Today I'm going to show you how you can use MetaModel through the MetaConsole to debug your application.

Setup

To integrate the MetaConsole to your project, it's very simple using Composer:

{
    "require-dev": {
        "mnapoli/metaconsole": "dev-master"
    }
}

Now I create a bin/meta.php file (name it as you like) in my application:

#!/usr/bin/env php
<?php
use MetaModel\Bridge\Doctrine\EntityManagerBridge;
use MetaModel\Bridge\PHPDI\PHPDIBridge;
use MetaModel\MetaModel;
// Here I set up my application (bootstrap)
require_once __DIR__ . '/../application/init.php';
$metaModel = new MetaModel();
// Integrate with the Doctrine EntityManager
$entityManager = /* ... */;
$metaModel->addObjectManager(new EntityManagerBridge($entityManager));
// Integrate with PHP-DI container
$container = /* ... */;
$metaModel->addContainer(new PHPDIBridge($container));
$console = new MetaConsole\Application('MetaConsole', null, $metaModel);
$console->run();

As you can see, I can integrate the console with my ORM (Doctrine 2 here) and my DI container (PHP-DI). You can write bridges for other kinds of object managers or containers (and feel free to submit a Pull Request too).

Using

Last week, I had a bug in my application, and it was very difficult to debug because it involved data, and that was not the kind of bug you can reproduce and fix using unit tests because it involved a lot of objects. Without MetaConsole, I would have had to dig through the database using phpMyAdmin or MySQL Workbench, writing queries and joining dozens of tables.

Instead, I launched MetaConsole, and I selected an object for which I had an ID (the ID was in the URL of the webpage I wanted to debug):

AF_Model_InputSet(562)

This query will select the entity AF_Model_InputSet with ID 562, and dump it to the screen. To do this, MetaModel queries Doctrine, which roughly translate to this PHP code:

$entity = $entityManager->find('AF_Model_InputSet', 562);

Now that I had my object on screen, I could traverse the object graph (through associations, getters, arrays, …) with more precise queries:

In the end, I ended up finding my bug: one component was marked as "required" but had a number of required fields > 0 (not normal!). I fixed the code, and now I needed to rebuild some data to have everything in sync again (after my tests). That's the kind of operation that can't be done just on a single object through the website.

No problem, MetaModel let's you traverse an object graph, but also call methods on objects and on services:

Conclusion

MetaModel is pretty much stable, though there are a lot of features I want to add (arguments in method calls, …).

On the other side, MetaConsole is still in development, and I hope to provide better integration with frameworks and a more enjoyable interface. If you are interested, you can try it (it's a development tool, so there's no risk since you shouldn't use it in production), and you can improve it.

And also, ideas are welcome!

9 September 2013 - Matthieu Napoli

When working with Composer and git branches, you will end up either:

• reinstalling dependencies each time you switch branch
• or meeting weird bugs because you didn't

because composer.json may have changed between branches.

To have composer automatically re-install dependencies when you switch to a branch, simply create a .git/hooks/post-checkout file in your project repository:

#!/bin/sh
cd $GIT_DIR/..
composer install

This is a git post-checkout hook (as the name suggest)

So you chose to test your application using unit/functional tests.

How do you ensure your tests do indeed test what you expect?

Fear not! Here comes TestsTester!

Use case

<?php
class MyTest extends PHPUnit_Framework_TestCase
{
    public function testDate()
    {
        $entry = new Entry();
        $this->assertNotNull($entry->getDate());
    }
}

Woops, I forgot to test that the date is in the past:

$this->assertLessThanOrEqual(new DateTime(), $entry->getDate());

Now my sofware is filled with bugs!

Usage

With TestsTester, I would have detected that:

<?php
class MyTestTest extends TestsTester
{
    public function testTestDate()
    {
        $test = new Test('MyTest', 'testDate');
        $test->checkAssertNotNull('$entry->getDate()');
        $test->checkAssertLessThanOrEqual('new DateTime()', '$entry->getDate()');
    }
}

You can then run the test tests:

$ testtester teststests/

Result:

TestsTester 1.0.0
F
There was 1 failure:
1) MyTestTest::testTestDate
'MyTest::testDate' do not test that '$entry->getDate()' is less than or equal to 'new DateTime()'.
FAILURES!
Tests: 1, Assertions: 2, Failures: 1.

Give it a try and check out all the crazy features at TestsTester.com.

Comments

9 August 2013 - Matthieu Napoli

Using the Domain Driven Design methodology, I sometimes end up on such a situation: a behavior in a model class is complex and involving other classes. It’s time to put that behavior in a service (or a factory if the situation applies to a constructor).

The problem is that in my service, I am not in the context of the domain class. If I move Foo::something() into FooService::something($foo), then I can’t access private properties of Foo, thus limiting me to the public API of Foo.

Now I end up adding accessors, thus breaking encapsulation and complexifying everything where all I wanted was improving the code.

VB.Net has a concept of “Friend” visibility, i.e. if A is friend with B, then A can access private properties of B (or something like that it’s been a long time :p). PHP doesn’t have such a concept natively, but here is a tryout to apply it with workarounds.

Disclaimer: the code is not pretty and is not for production. This is just an idea thrown around.

<?php
class Foo {
    private $bar = 'hello world';
}
class FriendOfFoo {
    public function doSomething($foo) {
        return function() use ($foo) {
            echo $foo->bar;
        };
    }
}
$foo = new Foo();
$service = new FriendOfFoo();
$closure = $service->doSomething($foo)->bindTo(null, $foo);
$closure();

See it in action on 3v4l.org.

Here, FriendOfFoo is a service that has access to Foo’s private and protected properties.

We achieve that by writing the methods of the service into closures. We can then bind those closures to the context of Foo, and voilà!

If you see a better way of achieving this, I am interested.

1 July 2013 - Matthieu Napoli

This post is sort-of a response to the blog post of Benjamin Eberlei about Controllers in Symfony 2.

The subject is about Controllers and their dependencies:

Controllers as a service are a heated topic in the Symfony world. Developers mainly choose to extend the base class, because its much simpler to use and less to write.

With Symfony 2, you can write controllers 2 ways:

1. extend the base Controller class. This is simpler and more practical but it ties up your controller to Symfony. Also, to fetch dependencies, you have to get them from the container, which is known as the Service Locator anti-pattern (= bad).

2. create a “normal” class, and use it as a service. That means you can use dependency injection through the constructor to get your dependencies. This is clean, this looks good, but you end up with managing a lot of dependencies :/

To ease up solution n°2, Benjamin proposes to create a “ControllerUtility” class which would group the most used controller services. That way, you dramatically reduce the dependencies, and still hide the container.

I use a different solution.

Constructor injection is not the only possible injection

The idea is to keep the solution n°2, but use Property Injection instead of Constructor Injection.

Property injection is generally frowned upon, and for good reasons:

• injecting in a private property breaks encapsulation
• it is not an explicit dependency: there is no contract saying your class need the property to be set to work
• if you use annotations to mark the dependency to be injected, your class is dependent on the container

BUT

if you follow best practices, your controllers will not contain business logic (only routing calls to the models and binding returned values to view).

So:

• you will not unit-test it (that doesn’t mean you won’t write functional tests on the interface though)
• you may not need to reuse it elsewhere
• if you change the framework, you may have to rewrite it (or parts of it) anyway (because most dependencies like Request, Response, etc. will have changed)

Because of that, I chose to use Property injection.

Property injection

Here is what my controllers look like:

<?php
use DI\Annotation\Inject;
class UserController
{
    /**
     * @Inject
     * @var RouterInterface
     */
    private $router;
    /**
     * @Inject
     * @var FormFactoryInterface 
     */
    private $formFactory;
    public function createForm($type, $data, $options)
    {
        // $this->formFactory->...
    }
}

Note this is an example using PHP-DI, my alternative DI Container. It allows to mark injections using annotations.

I know many PHP devs don’t like annotations, and there are some reasons not to use it. But in this case, because of the points I explained above, I find it acceptable to use the @Inject annotation. I find it also extremely practical.

Of course, this example also applies without using annotations (using a configuration file f.e.), and it also applies to Symfony’s container.

In the end:

• controllers don’t use the container
• controllers can be reused elsewhere, given they are fetched through the container
• we have full auto-completion and refactoring support in IDEs
• controllers are easy and fast to write and read (and that’s something I value a lot)

By the way, some Java developers may find this pattern of code familiar, it’s inspired from when I was working with Spring :)

Performance note

You are injecting services that may not be used

PHP-DI and Symfony DIC both support lazy injection, i.e. injecting a proxy that will load the target service only when it is used.

9 June 2013 - Matthieu Napoli

I recently stumbled upon François Zaninotto’s blog post: You should write ugly code. While he makes several good points, I strongly disagree with him and I feel the urge to give my opinion on the matter (what did you expect, this is a blog).

The point that he makes is that he encourages developers to write “ugly code”, i.e. code that works and that doesn’t follow best practices or anything related to code quality.

[..] developers shouldn’t care about code beauty, because that’s not their job. Instead, they should focus on creating great products, which is infinitely more satisfying.

Later, he insists that by writing ugly code, we ship faster than if we had to worry and handle code quality. And shipping faster is good for the business.

Well that’s true. That’s called Technical Debt. And like any debt, you’ll have to repay later.

Technical debt diagram, borrowed from Planet Geek for the (awesome) “Clean Code cheat sheet“.

Payback is a bitch and future-you will be cursing present-you for writing ugly code. The cost of change is directly affected, and when you tell your boss that Feature B will takes 2 weeks, he will point out that Feature A took only 2 days.

I won’t get radical as well and say that you should alway write “beautiful code”. What I advocate for is to carefully choose your side.

You need to get a prototype working for a demo? Then get it done quick and dirty! Heck I even used mysql_query on an internal website once, and it worked. I know that this website will never need any real maintenance, so I can sleep without fearing for violent psychopaths.

But if you plan on writing software that will be maintained for a few years, you would definitely earn by writing beautiful code. Code quality earns you money, and that’s something difficult for managers to get (especially if they have never developed, or never seen code quality benefits before). I once took part on the rewrite of a whole application. After the rewrite, the manager was completely amazed that we needed only a few days to do what would systematically take a month or two before.

I’d like to finish on his take about “coding trends” and what he calls “trend setters” like Eric Evans with his Domain Driven Design. DDD is not a trend, it is a set of patterns and methods to help you in specific situations. Evans book is very clear about it: DDD can be unhelpful if you try to use it everywhere. That is very reductive to say that all these kind of people (Evans, Martin Fowler, Jeff Atwood) do is “to explain why pattern A is better than pattern B. Until someone else publishes a book, explaining that pattern C is much, much better.”

Thanks to these guys, and best practices in general, I make my company earn money with apps that don’t need to be rewritten every year. I can work on my colleague’s code without pulling my hair and wishing him a painful death. I can participate to Open Source projects because I can understand the code.

TL;DR

• Go quick and dirty, but expect troubles on the long run
• Care about quality, that will slow you down but get you farther

Both are valid options, but choose carefully.

8 June 2013 - Matthieu Napoli

You have a PHP project hosted on GitHub with continuous integration using Travis-CI?

How about setting up code coverage reports?

For example, here is the code coverage report of PHP-DI: (click on the link to see the details).

To do so, you will need to create an account and enable your project at Coveralls. Then add this to your composer.json:

"require-dev": {
    "satooshi/php-coveralls": "dev-master"
}

Finally, update your .travis.yml configuration:

language: php
php:
 - 5.3
 - 5.4
 - 5.5
before_script:
 - wget http://getcomposer.org/composer.phar
 - php composer.phar install --dev --no-interaction
script:
 - mkdir -p build/logs
 - phpunit --coverage-clover build/logs/clover.xml
after_script:
 - php vendor/bin/coveralls -v

Now if you commit and push, Travis will run the tests and push the code coverage results to Coverall. Check out your project page on Coverall!

16 April 2013 - Matthieu Napoli

At my company, My C-Sense, we use Doctrine amongst other PHP frameworks and libraries. When we find bugs (or need new features), we contribute to the project through our open source initiative.

The problem is when we submit a pull request on Github, several months usually happen until our fix appears in a stable release.

To be able to enjoy our bugfixes immediately, here is our workflow:

• We fork the repository of the project to our organization account
• We commit and publish the bugfix in a branch of our repository
• We submit a Pull Request
• We override the dependency to the project with our version in Composer

Overriding a dependency is quite simple: just add your git repository in your composer.json and require you branch.

But when we want to override, for example, doctrine/common which is used by doctrine/orm, then we have a problem: doctrine/orm wants a stable version of doctrine/common, it will conflict with your requirement to a dev branch.

The solution is to alias your dev branch to a stable release, and that is possible through the awesome “inline alias” functionality in Composer.

Here is an example:

{
    "require": {
        "doctrine/orm": "2.3.*",
        "doctrine/common": "dev-ChainDriverFix as 2.3.0"
    },
    "repositories": [
        {
            "type": "git",
            "url": "https://github.com/myclabs/common.git"
        }
    ]
}

Here, our branch ChainDriverFix will override the 2.3.0 version of doctrine/common, which will also be compatible with doctrine/orm!

22 March 2013 - Matthieu Napoli

The singleton is a practical design pattern, that’s the reason it is so popular amongst beginners. It is also an anti-pattern because of the problems it introduces (global state, difficult to test, …).

While I agree with that, and the fact that Singletons should be used with (a lot of) moderation, I also like an alternative pattern which comes with the advantage of the singleton and balances out its disadvantages. This can be useful if you have to work on a codebase that has singletons.

I’m calling this pattern the Optional Singleton for lack of a better name.

Simply put, this is a class which you can use as a singleton, or not (it’s optional ;):

• you can still use the handy MySingleton::getInstance()
• you can however create new instances of the class, for example for tests

There is nothing revolutionary about it, see for yourself:

Of course, this is a pattern that has to be used where it makes sense. Singletons, as cool as they can be, will never do better than dependency injection.

Update: I’ve received numerous responses (mostly “the singleton is an anti-pattern” which I agree to). Here is one of my response that I’d like to have here as well:

The entire point of the singleton pattern is that you can’t instantiate the class. That’s why the pattern is called singleton.

My answer:

Yes, but in 90% of its derived usage it’s not because we want only one instance, it’s because it’s practical.

Quote from wikipedia: “There is criticism of the use of the singleton pattern, as some consider it an anti-pattern, judging that it is overused, introduces unnecessary restrictions in situations where a sole instance of a class is not actually required, and introduces global state into an application.”

For example one may use the singleton pattern for services: accessing them is practical, you can access them anywhere with the singleton pattern. I’ve seen codebases with this pattern.

Now if I come on a codebase using the singleton for services, and if I can’t rewrite everything, I’ll turn the Singletons into “Optional Singletons” so that the existing code still work, and so that I can use Dependency Injection over those services in the new code that I’ll write.

5 March 2013 - Matthieu Napoli

This is the first open source project created at my company, so I am quite proud of it even though it is not much.

The name is jquery.confirm, which is pretty explicit. It lets you have confirmation dialogs for links or buttons. For example, "Are you sure you want to delete that comment?" kind of things.

I am going to present its basic usage and options here.

The idea is to write unobtrusive Javascript by letting the user write clean HTML:

<a href="home" class="confirm">Go to home</a>

To enable confirmation on this link, simply:

$(".confirm").confirm();

That will show a confirmation dialog each time the user clicks the link. If the user confirms, the plugin will then redirect him to the link.

You can configure the texts and labels through the options. You can also change the actions that are executed when the user confirms (follow the link) or cancels (do nothing), so you can perform AJAX requests for example.

One interesting option is to force the link to be called with a POST request instead of a GET:

$(".confirm").confirm({
    post: true
});

On your server-side code, you can check that the request is POST and refuse GET request. That can help prevent security issues like someone sending a link to delete someone else’s account for example: http://example.com/my-account/delete. If you only accept POST request, people clicking on that link won’t see their account deleted (because the request would be a GET).

If you want to learn more or try it, the website contains the official documentation and some demos.

4 February 2013 - Matthieu Napoli

Replace [client] by [boss] or anything of the kind if you prefer.

A day at work

Bug #3890 from Client

There is an application crash, it says “division by zero error in SpeedCalculator::compute()”.

Please fix ASAP!

You open SpeedCalculator.php to find:

public function compute() {
    return $this->distance / $this->time;
}

Fixing the bug

Easy! Who wrote that code anyway, how could he not think of that!

public function compute() {
    if ($this->time == 0) {
        return 0;
    }
    return $this->distance / $this->time;
}

There you go, the bug is fixed in 2 minutes.

Later, the same bug is found in RatioCalculator and MoneyCalculator, but once these are fixed too, everyone in the team is sure the problem won’t appear anywhere, it’s gone, for sure this time! The code is rock solid now!

A month later, another bug pops in. The application does not crash anymore, but the client happens to see wrong calculation results in his reports because of the return 0;.

Take a step back

What if, instead of rushing, we took a step back.

Why did this situation happen?

$this->time was set to 0.

Easy! Let’s prevent that.

public function setTime($time) {
    if ($time == 0) {
        throw new InvalidArgumentException("Invalid value");
    }
    $this->time = $time;
}

Now this is better, you guarantee the integrity of the data. But the client is not very happy! When he fills the form with a time of 0, the application shows an error page.

So you work on displaying a nice error message by catching the exception in the controller.

When you’re done, you realize you also got the same thing to do for RatioCalculator and MoneyCalculator, so you copy paste and you are good to go.

Wait a minute, the client prefers that the error message is displayed in orange rather than red. So you change the color and copy-paste the code again.

Take another step back

What if, instead of fixing a bug, you answered a need?

Why did the client put 0 in the form? Because he made a mistake.

What is needed here?

• Is it only making sure the time that the user inputs in “speedCalculationForm” is ≠ 0?
• Is it only making sure the “speedCalculationForm” contains valid data?
• Or is it validating all user inputs?

So what about a validation library for example?

Waaaaait! Don’t go and write one yourself! For the love of god, take a step back, breathe, and look at what already exists.

Needs

We, programmers, love being technical. When your client or your boss thinks out loud about what he wants, we can’t help but imagine how we could implement it.

But we need be able to take a step back. If we want to be really good in our jobs, we have to understand the needs before thinking about solutions. And that takes a lot of effort.

Does the client really need “a blinking button that moves away when you try to click on it?” or does he need something else, something that he doesn’t know about and that you could help him define? And the same goes for yourself! Do you really need to open a file and write some infos in there, or do you simply need a logging system?

Take a step back, try and see the big picture. Because one may be a very good programmer, but code’s purpose is to answer a need.

Not a "fix the bugs in the bug tracker" kind of need, but rather a "I want an application will help me calculate speed based on input data, and if I type in invalid data then for fuck’s sake just tell me don’t go and calculate some weird results".

10 December 2012 - Matthieu Napoli

Doctrine offers a command line option to validate the schema (or mapping):

./doctrine orm:validate-schema

This is very useful, when I ran it against my code, which was working by the way, I got several errors/warnings.

However, I didn’t want to have to run this tool manually once in a while. I already have tests for that. So I thought: why not integrating the schema validation to the tests!

So here is my implementation of a PHPUnit test failing when Doctrine validation find errors:

This example is an introduction to the Dependency Injection concept. It is based on the PHP library PHP-DI.

Classic implementation

Given you have:

class GoogleMapsService {
    public function getCoordinatesFromAddress($address) {
        // calls Google Maps webservice
    }
}
class OpenStreetMapService {
    public function getCoordinatesFromAddress($address) {
        // calls OpenStreetMap webservice
    }
}

The classic way of doing things is:

class StoreService {
    public function getStoreCoordinates($store) {
        $geolocationService = new GoogleMapsService();
        // or $geolocationService = GoogleMapsService::getInstance() if you use singletons
        return $geolocationService->getCoordinatesFromAddress($store->getAddress());
    }
}

Now we want to use the OpenStreetMapService instead of GoogleMapsService, how do we do? We have to change the code of StoreService, and all the other classes that use GoogleMapsService.

Without dependency injection, your classes are tightly coupled with their dependencies.

Dependency injection implementation

The StoreService now uses dependency injection:

class StoreService {
    /**
     * @Inject
     * @var GeolocationService
     */
    private $geolocationService;
    public function getStoreCoordinates($store) {
        return $this->geolocationService->getCoordinatesFromAddress($store->getAddress());
    }
}

And the services are defined using an interface:

interface GeolocationService {
    public function getCoordinatesFromAddress($address);
}
class GoogleMapsService implements GeolocationService {
    public function getCoordinatesFromAddress($address) {
        // calls Google Maps webservice
    }
}
class OpenStreetMapService implements GeolocationService {
    public function getCoordinatesFromAddress($address) {
        // calls OpenStreetMap webservice
    }
}

If you use PHP-DI (a PHP dependency injection library), you then configure which implementation will be used:

$container->set('GeolocationService')
          ->bindTo('OpenStreetMapService');

If you change your mind, there’s just one line of configuration to change.

Comments

23 November 2012 - Matthieu Napoli

Bouncing on the discussion initiated in the #52 ticket of the PHP-FIG project on Github: « Explain the scope of the PSR system », I’ll explain the case I’m trying to make.

First, PHP-FIG, which stands for Framework Interoperability Group, is a gathering of major PHP frameworks and project who try to:

talk about the commonalities between our projects and find ways we can work together.

This group has released PSR-0, PSR-1 and PSR-2, three specifications of coding standards, guide style and code organisation (for autoloading interoperability). Now the question is asked: is it the role of the PHP-FIG to define technical “code” specifications or is it out of its scope? Here is my answer.

PSR-0/1-2 are contracts between its users to ensure cohesiveness and compatibility.

Think of the PSR-0 for example, it enabled all projects to be compatible regarding class autoloading. To achieve this, no code or PHP interface was necessary because what the autoloading needed was only file names, directories and class names constraints.

Now there are other questions that need standardization for interoperability between PHP projects. And some of them need PHP interfaces.

For example, PHP (or the SPL) does not define a Collection interface (or any implementation). However, a Collection is a base object, and I bet it is used (or could be used) in many projects. Now Doctrine defined their own Collection interface (because it needed it) and I’m sure other projects did the same for the same reasons, but that situation is stupid. A Collection is a standard data structure, implementations may vary but the Collection interface should be defined once and for all.

And PHP interfaces are contracts between its users to ensure cohesiveness and compatibility.

Notice any similarity between PSR-0/1/2 and interfaces? They are the same thing, applied to different things. They are technical specifications.

I agree that the SPL was a good start and maybe would have been a good place for such things, but it is a still project, with no big changes lately, a lot of inertia and several big lacks (and who decides what’s in the SPL?). The PHP FIG is the perfect group to bring a solution to this: it is active, dynamic, open and transparent, representative of the major PHP projects, and it has the competences and the momentum to make it useful and used (that will not be “yet another PHP library”, it will be used by major frameworks).

If PHP-FIG doesn’t do it, then who will (and more importantly: who will make it a success)?

And to extend my point, have a look on the Java side (JSR), and for example JSR-107 which defines interfaces for cache API, or JSR-220 which defines JPA (specification of persistence API that Doctrine 2 has followed).

TL/DR: I think PHP-FIG should define and provide PHP interfaces. PHP-FIG defines technical specifications for interoperability between PHP projects. PHP interfaces are a form of technical specifications, they can allow PHP projects to be more compatible and work better together. PHP-FIG is the best group possible to standardize classic/mainstream API (utility classes, …). Java does it, it works, that should inspire us.

5 October 2012 - Matthieu Napoli

If you are working with Doctrine 2 and its YAML configuration files for the object mapping, you may find the documentation lacking some details.

Here is a gist of all the YAML syntax possible:

It is based on this blog article, completed, and turned into a git repository (thanks to github), so anyone can fork it and improve it.

20 September 2012 - Matthieu Napoli

I used to develop using Singletons, registries or even static classes. Those days are gone.

I decided to use Dependency Injection so that:

• my classes would be testable
• replacing an implementation by another would be not only doable, but easy (and so extending a library/module would too)
• the design of those classes wouldn’t be guided by the question of “how they will be used”
• my code would be cleaner, simpler
• and IDE auto-completion/type-hinting would always work

I gave a try to Symfony and ZF2 DI systems, but they both seem way too complicated for just a simple need (that anyone who has worked with Spring would understand):

class MyClass {
    /**
     * @Inject
     * @var MyService
     */
    private $service;
}

This short code means: Inject, using a simple annotation, an instance of another class into a property.

I started working on a framework enabling such functionality few months ago. It is now in a mature state. It is based on the Annotations library of Doctrine 2, and takes most of its ideas of Spring 3.

You can check out this framework on its official website: PHP-DI, and you are welcome to use it or contribute.