AssetHelper fopen and assetDownloadCurl additional HTTP headers

[jamesmacwhite]

Description

I have a feed which provides CDN image paths, we have these linked to an asset field to import but the origin server throws a 403 Forbidden error on requests for the image which are lacking a User-Agent value. I confirmed it locally by running wget/curl with and without.

Example error

thumbImage - Asset error: https://cdn.ticketsource.co.uk/images/promoter/banner/66148-1713969775145-m.png - fopen(https://cdn.ticketsource.co.uk/images/promoter/banner/66148-1713969775145-m.png): Failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden .

The setting assetDownloadCurl exists, but the issue is there is no way to set additional curl options on the asset download request

fopen can also provide HTTP headers as well, but this is currently not possible to define without forking or using FeedMe events to fetch the image with a client that provides the required headers.

Additional info

• Craft version: 5.4.9
• PHP version: 8.3
• Database driver & version: MySQL 8.0
• Plugins & versions: FeedMe 6.5.0

[brandonkelly]

Looks like it’s not documented (on it), but Feed Me supports setting a clientOptions array on your feed configs from config/feed-me.php, which will get merged into the main Guzzle config:

<?php

return [
    '*' => [
        'clientOptions' => [
            'headers' => [
                'User-Agent' => '...',
            ],
        ],
    ],
];

[jamesmacwhite]

Hi Brandon. Thank you for looking into it. Funnily enough I noted this a while back #1441.

However unfortunately in this case specifically for asset importing, this wouldn't apply I don't think. The use of clientOptions would apply on the main Feed URL request, but the AssetHelper class makes a request for the image URL from a seperate client i.e. fopen or curl if using the config setting to set the usage of curl. However, neither allow for adding or changing headers sent with this request.

I ended up writing my own asset importing logic for the CDN image paths where I can control the request headers. Providing I can get the images into asset storage before the feed runs, I can then leverage the asset find query performed on the filename to allow Feed Me to match the asset and add it to the element as needed.

It would be nice for consistency if the AssetHelper class used Guzzle which could provide extendibility, I don't know if there's a reason for using fopen/raw curl, perhaps legacy?

[i-just]

I raised a PR to support asset downloads via Guzzle too.

[jamesmacwhite]

Thank you @i-just. This will be most useful if merged!

[angrybrad]

Feed Me 5.9.0 and 6.6.0 are out with this fix!

https://github.com/craftcms/feed-me/releases/tag/5.9.0
https://github.com/craftcms/feed-me/releases/tag/6.6.0