Blue Droplet Media - Jonathan Brown

DrupalCamp Edinburgh this Saturday

jbrown — Mon, 16 May 2011 15:34:15 +0000

DrupalCamp Edinburgh is this Saturday @ the Informatics Forum: http://dcedinburgh.drupalscotland.org/

It's free and the tracks are Beginners, Community and Business.

They've lined up loads of great sessions: http://dcedinburgh.drupalscotland.org/program/session-schedule

There is free coffee and lunch, but you have to register by 5pm on 18th May.

There is also an afterparty @ Sneaky Pete's: http://www.facebook.com/event.php?eid=149619238439720

If you're down south, Drupal7Camp Leeds is the same weekend: http://2011.drupal7camp.org/

Why management needs to understand the Drupal community to fully benefit from Drupal

jbrown — Mon, 03 Jan 2011 19:52:04 +0000

The slogan of the new Drupal website is:

Come for the software, stay for the community

This is the typical process by which people learn that Drupal's ultimate strength is its community, rather than its code base.

Websites are like human beings - any two websites can be very different and do very different things, but under the hood they are 99% the same. Any website (large or small) has very small unique code base, compared to the enormous software stack that it sits upon. Other components in the stack are typically:

kernel, e.g. Linux - 13.5 million lines of code (mainly device drivers)
database, e.g. MySQL - 1.7 million
webserver, e.g. Apache - 450,000
HTTP accelerator, e.g. Varnish - 78,000

The operating system distribution itself is even larger, e.g. Debian has about 32,000 software packages, totalling almost half a billion lines of code. Of course, only some of these packages would be involved in running a given website. At a lower level, even more is going on in hardware.

Regardless of what your website is doing, all of the above components will be typical and you will probably not be reinventing any of them from scratch. Drupal core (275,000 lines) is just another component in the stack. But, as small as Drupal is compared to the rest of the software stack, it is still a very large piece of software that has taken a decade to develop. The functionality that Drupal provides is also reusable for just about any kind of website.

Drupal has the experience of having been deployed hundreds of thousands of times. It works around so many weird security problems and software incompatibilities to provide a robust plateau on which to build your site.

On top of Drupal core sits the contributed modules - these are further reusable components. There are about 6,000 of these totalling about 10 million lines of code. Of course, you should only use a few dozen modules at most on any one site.

The trick is, to have as much of the functionality provided by reusable, community or contractually supported, open source components as possible. From the operating system right up to the contributed modules. Write as little private code as possible and only do so as a last resort. It should sit at the top of the stack, like the skin of an apple. If you need some new functionality that someone else might need, write a module and contribute it back to the community.

Time and time again I have seen companies using Drupal, but having no understanding of how to utilise it correctly. It is not a technical understanding that is required, but rather a social one. Developers will download Drupal and whatever modules they like the look of, and go off on their own, making modifications to get Drupal to do exactly what they want and failing to benefit from the following advantages:

Popular modules are always getting better and better. For a company to fully benefit from improvements they make to a module, these improvements need to become part of the module on drupal.org. In order for the code to be incorporated, it will be peer reviewed and improved. Others will build upon it. This also removes the burden of maintaining a 'fork'. Updating core and modules to the latest release is considerably harder if a developer has hacked it.
If you are having a problem with a popular module, there are probably many others having the same problem - making it much easier to get to the bottom of it in the issues queue.
New engineers can be be parachuted into your site and know exactly what they are doing. They are already experts with the modules that you use. Time and money doesn't need to be wasted getting them up to speed with your bespoke effort.
Increasingly, Drupal provides functionality that is more abstract in nature. New content types used to have to be created in a custom module - now this can be done by an administrator through a point and click interface. More and more can be done without any programming. Views and Panels are examples of this.
Higher quality code - it takes a lot of time and effort to write code that is of really high quality. When code is staying private within an organisation it will be of much lower quality. Publishing forces developers to do a good job. Cutting corners with poor quality code is never a good investment.
Security - if a considerable number of organisations are using the same code, then there is a much greater collective effort to make sure it is secure. Private code simply can't compete with this.

Developers need to fully utilise the Drupal community to their benefit. The best way for this to happen is for management to understand this. This means they can instil a culture within their organisation to engage with the community, and also to permit this to happen. Developers need to be able to participate in the community without fearing that management would not understand.

How can developers and management best come to have an understanding of the community and how best utilise it? Come to Drupal events like DrupalCon.

A further advantage of management understanding the Drupal community, is that they will probably see that it is much less dysfunctional than their own organisation. The Drupal community has extremely high levels of transparency. Almost everything (except security issues) are discussed in public. This has enabled the Drupal community to be a extremely effective engineering and decision-making machine. Something to be envied by almost every other organisation in the world.

There are a couple of other blog posts on similar topics:
http://www.garfieldtech.com/blog/pay-it-forward
http://www.lullabot.com/articles/best-practices-in-open-source-developme...

[lines of code source: http://www.ohloh.net/ ]

Proposal for fixing PHP namespacing in Drupal 8

jbrown — Fri, 10 Dec 2010 01:53:28 +0000

Collisions with non-Drupal PHP code

In theory, although I have never see this happen, a PHP library that Drupal is utilising could declare a function or class that is already declared in Drupal. This would cause PHP to abort.

PHP 5.3 has a solution - namespaces (http://php.net/manual/en/language.namespaces.php).

Drupal 8 should require PHP 5.3 - it was released 2009-06-30. Drupal 7 requires 5.2 and may just get released this year. When will D8 be released? Probably 2012 or 2013 - by then PHP 5.3 will be 3 or 4 years old.

Every PHP file that is part of Drupal would be in the 'drupal' namespace. This would be declared at the top of every PHP file as follows:

<?php
namespace drupal;

Because all of Drupal is in its own namespace, there is no possibility of function or class name collisions.

We are already prefixing many functions with drupal_ - this can simply be dropped from such function declarations.

The new Database API in D7 is not really part of Drupal - it can be used without the rest of Drupal with only a few minor changes. It is a standalone library, and as such it should be in its own namespace:

namespace drupaldb;

the db_ prefix could then be dropped from all function declarations in the library.

Modules and themes

Modules and themes (and engines) would have a namespace such as

namespace drupal\taxonomy;

namespace drupal\bartik;

Function prefixes, such as 'taxonomy_' would be dropped.

However, this makes Drupal code really ugly.

Calling functions not in a module from a module would look like this:

\drupal\set_message();

Calling a db function would look like this:

\drupaldb\select('node');

Calling an API function in another module would look like:

\drupal\taxonomy\select_nodes();

The solution? Namespace importing.

Every PHP file in a module would need to declare which namespaces it needs to access, e.g.

<?php
namespace drupal\node;
use drupal, drupaldb as db;
use drupal\taxonomy;

This means the above examples would look like:

drupal\set_message();
db\select('node');
taxonomy\select_nodes();

Which is really quite pleasant!

Calling a function that is in the current namespace would not require any prefixing. For example, calling \drupal\node\access_needs_rebuild() from inside namespace 'drupal\node' would be invoked as:

access_needs_rebuild();

Namespace importing should not be confused with module dependencies - importing is specified on a per-file basis.

Magic functions (hook implementations)

There are a couple of namespacing issues that can occur with magic functions:

Ambiguity of where the module name ends and the hook name begins, e.g. views_comment_load() - is the hook 'comment_load' or 'load'?
Modules can inadvertently implement a hook for some other module.
Multiple modules can 'invent' the same hook, expecting it to do different things

Solution: use __

Magic functions would take one of the following forms:

INVENTOR__HOOK()
INVENTOR__HOOK__PARAM()
INVENTOR__HOOK__PARAM1__PARAM2() etc

INVENTOR is the name of the module that 'invented' the magic function.
HOOK is the name of the hook, e.g. form_alter.
PARAM is for additional parameters, e.g. the name of the form to be altered.

e.g.

system__menu()
block__info()
system__form_alter__search_form()

Only magic functions can have __ in their name. Modules, hook names [updated] and parameters are not allowed to have a double underscore. Drupal should not permit any machine name to have a double underscore.

A further advantage of this is that the name of a magic function for a specific hook would be the same in every module that implements the hook. This helps code portability - example hook implementations could be put in place simply by cutting and pasting code.

Theme / process / preprocess implementations

These are also magic functions.

The hook name is 'theme' / 'process' / 'preprocess'.

A module that declares a theme hook in a system__theme() is considered the 'inventor' of the magic function.

theme_user_list($variables)

becomes

user__theme__list($variables)

in both the module that invented it and in themes that are overriding it.

A module / engine / theme could implement a preprocess hook like:

system__preprocess__page()
system__preprocess__links__contextual__node()

The second example is for the 'fallback' theme items described here.

Example

The attached node.module has been modified to incorporate this proposal (and a few other code style changes).

Advanced exception handling in Drupal 7

jbrown — Tue, 21 Sep 2010 14:42:05 +0000

Earlier this year Drupal core developers were getting a strange error message while developing Drupal 7:

Fatal error: Exception thrown without a stack frame in Unknown on line 0

This was causing a lot of head scratching because it doesn't provide any information about what caused it.

Drupal 7 uses exceptions more than any previous version of Drupal and this turned out to be the problem.

PHP exceptions are handled in one of two ways:

A function further down the call stack catches the exception in a try{} catch{}.
The exception reaches the bottom of the call stack and the exception handler (_drupal_exception_handler() in Drupal 7) is invoked.

Unfortunately there are certain contexts of PHP execution where the exception handler will not be invoked - causing the above error to be output:

exception handler (already)
session save handler
registered shutdown function
class destructor method invoked during PHP shutdown

My patches to fix contexts 1 - 3 are in Drupal 7. The solution involved each of these contexts having their own try{} catch{} block so that the function had a no throw guarantee. In this situation, details about the exception are now output in a very simple manner and are not logged so nothing further can go wrong. For the case of already being in the exception handler, both the original exception and the addition one are output.

There isn't a general solution for context 4. Class destructors should not throw exceptions.

Use Git submodules to avoid storing Drupal core and contrib modules in your site's repository

jbrown — Fri, 17 Sep 2010 01:14:41 +0000

In addition to whatever custom modules / themes are required, a typical code repository for a Drupal website also contains Drupal core and stock contrib modules. This makes repositories for Drupal websites much heavier than they need to be. It shouldn't be necessary for these external projects to be checked in - they already have their own repositories on drupal.org.

I have developed a technique to avoid this using Git submodules.

Here is how it works:

Start by cloning drupal-git-base. This is a simple Git project I created that contains drupal core as a submodule and some scripts.

git clone --recursive git://github.com/bluedroplet/drupal-git-base.git mysite.com
cd mysite.com

The --recursive option means that submodules will also be cloned, so Drupal core gets cloned from git://git.drupalcode.org/project/drupal.git .

If you don't want to be using Drupal HEAD, you need to checkout a specific tag (optional):

cd drupal
git tag # see what tags are available
git checkout 7.0-alpha7
cd ..
git add drupal
git commit -m "Checkout drupal 7.0-alpha7."

We need to make a copy of the 'sites' directory outside the submodule that we can check in. The setup script creates a symlink from the drupal tree to it.

mv drupal/sites/ .
./setup

Now install Drupal in the normal way.

Next we need to add our 'sites' directory to the repository:

git add sites
git commit -m "Add sites directory."

Modules can be added as follows:

git submodule add git://git.drupalcode.org/project/devel.git sites/all/modules/devel
cd sites/all/modules/devel/
git tag # see what tags are available
git checkout 7.x-1.0-beta2
cd ../../../..
git add sites/all/modules/devel/
git commit -m "Add devel-7.x-1.0-beta2."

This is the procedure for making a clone of your repository:

git clone --recursive mysite.com test
cd test
./setup

If you want to run modified versions of core or modules then a patchset can either be maintained in your superproject, or you can maintain your own git clone of the project(s).

Unfortunately there is currently a problem with using Git submodules in this way - Drupal's update system doesn't know what contrib module and theme versions are being used. When you download a release from drupal.org it has version information in the .info file, but this is not in the repository.

Two things need to happen to fix this:

.info files need to have the release version information added in the drupalcode.org repository when they are tagged, not just in the downloadable tarballs.
The new CVS to Git migration script needs to do this retroactively.

However, Drupal core does have the correct version information in the repository.

Another problem is that you cannot use Drush to update modules. You need to enter the module's directory, perform a git pull and then checkout the desired version. This is pretty easy, but not as easy as Drush.

So what I currently recommend is that people use drupal-git-base, but do not use the Git submodule functionality for modules and themes. Check them in the old-fashioned way.

Storage API now supports Amazon CloudFront RTMP streaming

jbrown — Thu, 28 Jan 2010 15:13:26 +0000

Blue Droplet Video and Storage API now support RTMP streaming with Amazon CloudFront.

The 'Stream with CloudFront' option can be enabled when creating an S3 container, or when editing an existing one. You need CloudFront enabled in your AWS account. Streaming with CloudFront will not work immediately as it takes several minutes to setup a CloudFront distribution.

Streaming will only work with MP4 video. The old FLV format is not supported.

This functionality requires the latest development snapshots of Flowplayer, BDV and Storage API. You can try it out at SharkScopers.

Storage API now supports Amazon CloudFront

jbrown — Sat, 23 Jan 2010 04:21:01 +0000

Storage API now supports serving with Amazon CloudFront.

CloudFront is the CDN for Amazon S3. It has 14 edge locations:

United States - Ashburn, VA; Dallas/Fort Worth, TX; Los Angeles, CA; Miami, FL; Newark, NJ; Palo Alto, CA; Seattle, WA; St. Louis, MO
Europe - Amsterdam; Dublin; Frankfurt; London
Asia - Hong Kong; Tokyo

To use CloudFront, you need the latest development snapshot of Storage API. It will be in Storage API 6.x-1.2 when it is released.

The 'Serve with CloudFront' option can be enabled when creating an S3 container, or when editing an existing one. You need CloudFront enabled in your AWS account. Serving with CloudFront will not start immediately as it takes several minutes to setup a CloudFront distribution. CloudFront is polled during cron and S3 is used for serving until it is setup.

Secure serving (time limited, cryptographically signed URLs - like with S3) is supported by CloudFront, but this is not yet implemented in Storage API.

Showcase: SharkScopers

jbrown — Sun, 03 Jan 2010 21:06:03 +0000

One of the best sites that uses Blue Droplet Video and Storage API is SharkScopers.

I wrote up a showcase detailing how I helped SharkScope put it together.

Planet Drupal TV

jbrown — Tue, 28 Apr 2009 20:53:56 +0000

I have created a new site called Planet Drupal TV that aggregates Drupal video feeds in a manner similar to how Planet Drupal aggregates Drupal blog posts. Videos can be viewed in-browser or downloaded. There is a "most popular" section that utilizes the fantastic Radioactivity module.

It utilizes my Blue Droplet Video module and its Feed API integration.

I've added all the Drupal video feeds I could find, although it does not work with YouTube feeds as they do not use the enclosure tag. Miro can acquire video from YouTube, so it should be possible in future. Really, blip.tv is much better than YouTube, or you could use Blue Droplet which outputs the source video in the enclosure tag.

OpenPackage Video now supports H.264 video

jbrown — Sun, 28 Dec 2008 13:12:32 +0000

OpenPackage Video 5.x-3.15 has been released.

The major new feature of this release is MP4 encoding with H.264 video and AAC audio. These codecs have much higher performance than than the old FLV and MP3. This means you can have the same quality for less bandwidth or higher quality for the same bandwidth.

Demo

MP4 encoding is supported both with FFmpeg transcoding on your own server and with the paid transcoding service we offer at openpackage.biz .

This functionality was sponsored by SharkScopers and vertikal.dk .

You can select the MP4 format under 'transcoding parameters' in the video field configuration. You need to have run update.php if upgrading.

For MP4 videos, the user must have Flash 9 or greater to play MP4. OpenPackage Video automatically ensures that a recent enough version is installed.

If you are handling transcoding on your own server using FFmpeg, you need to make sure that your installation of FFmpeg has the required codecs. Once you have upgraded to 5.x-3.15, go to admin/settings/op_video . Important output formats and codecs are now highlighted. Make sure you can output the libfaac audio codec and the libx264 video codec. You also need qt-faststart, which is bundled with FFmpeg. See our FFmpeg documentation for more info.

Of course, if you use our remote transcoding service, you don't have to worry about configuring FFmpeg.