I installed a website on a brand install installation of Ubuntu 14.04 (opposed to an upgrade from earlier versions) and the same website would not start...

Apache2 would give me the following error:

client denied by server configuration

which did not seem to tell me anything... I search for the error and the first page that popped up was a page by Apache2 and they said to add the deny/allow trick. I tried that and it did not help.

    <Directory /my/special/directory>
        Order allow,deny
        Allow from all
    </Directory>

Search further, I found an answer on Stackoverflow asking about that problem and the answer was to add another line in there:

    <Directory /my/special/directory>
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>

Once that option was added, it worked like a charm.

The problem is that now Apache2 checks the ownership of the files and if not somehow owned by www-data, it returns a 403 Forbidden error.

Some setup may also require to add yet another option:

    <Directory /my/special/directory>
        Order allow,deny
        Allow from all
        Require all granted
        Satify any
    </Directory>

You may want to check out the documentation about Require and Satisfy to have a better idea of what these two options really do.