Help! This composer stuff is confusing

Notes to accompany the talk from DrupalCamp Bristol, 2017.

Introduction

I’ve encountered a lot of confusion surrounding the use of Composer with Drupal. I found it took a long time to understand what was going on, and most of the documentation and tutorials available focused on how to do things without explaining why things are the way they are.

I hope by providing some context here it will help clarify things and in doing so save you some time.

What is composer and what problem does it solve?

Composer is a tool for managing dependencies in PHP projects.

Authors of projects or libraries written in PHP can declare their requirements on other projects by means of a simple text file. Consumers of those projects, who are putting together an application, can use a command line tool to download everything in one go.

Composer is a developer focused tool. It assumes familiarity with PHP source code and the command line.

How does Composer work?

I find it helpful to think of Composer usage in three distinct steps each with their own command—require, update and install.

1. Require

Writes the composer.json file with a list of requirements.

In the top level directory we create a file called composer.json and list all of our direct dependencies. We don’t need to list any indirect dependencies here, Composer will deduce those for us. An example composer.json file looks like this:

{
    "require": {
        "data-values/geo": "^2.0"
    }
}

What we’ve done here is state that we want to use the data-values/geo library (if you’re curious, it’s for working with geospatial data). We’ve also declared we need version 2.0, or any later version that is backwards-compatible with 2.0. The caret pattern (^) will allow any version up to, but not including 3.0.

More information about Composer’s version constraints.

We could also have achieved the above using the following command, which avoids editing the composer.json file by hand:

composer require "data-values/geo:^2.0

We can even drop the version constraint in that command. If so, Composer will first determine the most appropriate version constraint to choose (usually the most recent stable release):

composer require data-values/geo

2. Update

Looks for the most up to date versions and generates the composer.lock file.

The composer.json file only states version constraints, but that isn’t enough to guarantee a predictable codebase. A statement of the form “the latest 2.x version” can vary over time, and on it’s own means we can’t share or deploy code in a reliable way.

So the next step is to work out a list of everything that needs to be installed, with exact versions. We use the composer update command to do this:

composer update data-values/geo --with-dependencies

It will determine the actual versions of each library to use, and write these exact versions to a new composer.lock file. This is situated alongside composer.json, but automatically generated by Composer.

Here’s an extract from composer.lock:

{
    "content-hash": "24dc5a680dfdd2bd93a9790b8fe75447",
    "packages": [
        {
            "name": "data-values/common",
            "version": "0.3.1"
        },
        {
            "name": "data-values/data-values",
            "version": "1.0.0"
        },
        {
            "name": "data-values/geo",
            "version": "2.0.1"
        },
        {
            "name": "data-values/interfaces",
            "version": "0.2.2"
        }
    ],
}

While composer.json specifies ^2.0, it’s the composer.lock file that determines we’re actually going to have version 2.0.1. We want to share this, so the composer.lock file should be committed to a source code repository (unless you’re producing a reusable library).

When 2.0.2 or a later version comes out, we can choose to use it by running the update update process again, after which the lock file is updated.

We can also update all the dependencies at once, but use this carefully:

composer update

3. Install

Downloads the dependencies according to what’s in composer.lock.

Once we have a composer.lock file, we can download the dependencies with the following command:

composer install

Note that conceptually, install should be thought of as happening after update.

Composer will start downloading things based on the contents of the composer.lock file. Once it has finished finished we can see a vendor directory containing the source code of all our dependencies.

Using Composer-provided code in PHP

As well as downloading the library source code, running composer.install will create an autoloader. This is some code that tells PHP where all the downloaded files are. This means we don’t really need to be concerned about the structure of the vendor folder too much.

In order to use any of the Composer-provided libraries, all we need to do is make use of the autoloader in any PHP code:

include 'vendor/autoload.php';

Other points

It’s worth pointing out a few extra things that happen:

This ensures nothing is left in a meaningless state. However, there is no harm in performing the steps laid out here explicitly, and it may be helpful to think of the distinct steps in order: require, update, install.

Why all the fuss?

Contributed modules are starting to use Composer for their own dependencies. This affects the tried and tested workflow we have already with downloads and/or drush. Here’s a very common scenario that causes quite a few problems:

  1. Download Drupal core from drupal.org (or via the drush dl command)
  2. Download some contributed modules and themes from drupal.org (or via drush)
  3. Download a module that has a third party library dependency, e.g. address.

At this point we’re stuck because we can’t enable the address module. It makes use of an external library for international address formats, and according to the project page, it must be installed via composer:

Address must be installed via Composer, in order to get the required libraries. The tarballs are provided for informative purposes only.

At this point we face a dilemma. We could edit Drupal’s composer.json file, run composer update/install, but that feels like hacking core. Surely there is a better way to do this?

The documentation on drupal.org isn’t definitive. It talks about two seemingly different projects—drupal/drupal and drupal-composer/drupal-project. At this point it’s helpful to compare those two things.


Drupal via Composer (aka drupal/drupal)

This can be thought of as: _Drupal is the project; and its dependencies are managed by Composer.__

Using this project is really just the same as downloading Drupal from drupal.org (or via drush), but by Composer. Drupal owns the composer.json and composer.lock files in this case. We have to be really careful with commands like composer update, because we will overwrite those files.

Nor can we keep using drush as before. Updates to core will overwrite our modified composer files, and we’ll have to keep applying our modifications again and again.

The Composer template (aka drupal-composer/drupal-project)

This can be thought of as: A vanilla project, with a dependency on Drupal.

What if we changed our approach to something like the following:

Start with nothing. Now give me Drupal 8.3, the address module, the bootstrap theme, and …

You would have a composer.json file that looks a bit like this, and crucially, is yours to edit:

{
  "require": {
    "drupal/core": "8.3.4",
    "drupal/address": "1.0",
    "drupal/bootstrap": "3.5"
  }
}

This is the approach taken by the Drupal composer working group, with their drupal-project template. It’s important to stress the word template— this isn’t Drupal, but a starting point for a codebase that requires Drupal and contributed modules, and possibly has some custom code in it too.

The template does a few things other than specify dependencies.

File locations

Both Drupal core and contributed modules can contain assets beyond PHP code. These assets, like HTML and CSS files, are designed to live inside the web document root so they can be served directly by the web server.

So the template defines a web directory that’s managed by Composer, and puts Drupal core and modules in there. Everything else goes in vendor as normal.

Scaffolding files

Another task of the composer template is to take care of the extra files that must live in the top directory of the document root. Because they can’t live in the core directory, they aren’t considered part of drupal/drupal-core.

These are: (amongst others)

The template provides a somewhat messy workaround for keeping these files up to date. It makes use of a drupal-scaffold utility project to download current versions of them from drupal.org upon each update/install.

Using the Composer template

The instructions for using the composer template state the following command:

composer create-project drupal-composer/drupal-project:8.x-dev some-dir --stability dev --no-interaction

Let’s examine each part of that command:

create-project makes a copy of the template, which doesn’t have a composer.lock file. It then runs composer update to generate the lock file and install the dependencies, namely Drupal core and a few other tools.

The name of the project to copy is drupal-composer/drupal-project. This is that of the template, not Drupal itself. So we get a copy of the template (which we can later edit), but Drupal core is actually provided during the install phase.

Likewise, the version number 8.x-dev refers to the version of the template, not the version of Drupal. Don’t be put off by the -dev suffix, that doesn’t mean you’ll end up using a development version of Drupal. The template itself provides the latest stable version of Drupal core by default, we can change this if we want something else.

stability --dev refers to the template version above, not to Drupal core or contributed modules. Those will default to being stable versions if available, or dev/alpha/beta/rc versions if not. The templates composer.json file has two options to control this too: minimum-stability and prefer-stable.

Extras in the composer template

The template also provides a few other useful tools: