feat[Core]: Admin and User Metadata for transactional ressources and users (#5897) #4

JustShah · 2025-01-30T18:07:01Z

Closes solidusio#5897
Concerns solidusio#5951

Breaking Changes

This feature breaks sqlite versions predating to December 2023, verify your requirements before migrating.

Introduction

Overview

This PR introduces the concept of metadata fields to solidus. The purpose of metadata fields is to allow customers and admins alike to provide additional information to transactions to either extend the informations contained or allow to insert additional data to correlate transactions with third party systems.

Examples for data can be:

any transaction itself such as orders, RMAs, payments and stock movements;
individual items contained inside an order (line items);
match users with 3rd party systems.

Possible Applications on a very high level are:

customers that want to add Purchase Order information to an order;
administrators that want to add a ticket number to an RMA.

The current scope of meta data are following ressources:

Users
Orders
Customer Returns and any from of correlated transactions
Payments
Shipments
Line Items

How is the additional information stored?

for each of the above resources two columns have been created:

customer_metadata
admin_metadata

The resources store the information as jsonb.

What is the access model?

Cancancan has been used to allow users to write data with the current scope just extended by the additional fields:

Users can create / delete / alter metadata on orders only till the payment itself is processed,
Admins have universal access R/W access to metadata at all stages of transactions.

Testing

Extensive test cases have been added.

Detailed Overview

Types of Meta Data and usage suggestions

Overview

Meta data can be broken down in three large topics visible in the table below:

	Use Case
Administrative Metadata	Information required to correlate all transactions to 3rd party systems.
Customer Metadata	Information allowing customers to customize orders (eg options to store information like a text string or link to an image) or correlate transactions (such as POs).

Administrative Metadata

Transactional resources are either by nature integrated with 3rd party systems (eg Payments, Billing in Italy with eInvoicing) or by operational requirement (ERP, CRMs like Salesforce,...). Meta data as an extension on admin site for all transactional resources to connect the transactional resources to any of such systems. Given that more and more countries will switch to a form of eInvoicing in the coming years meta data allows to fetch or store additional information in those transactions.

Customer Meta Data

Customer meta data allows users to customize a cart with either administrative (PO numbers,...) or customisation information (eg an URL to a picture to customize a shirt, text strings for incisions or in our case IMEIs to relate to a subscription) allowing to significantly extend the features without changing the underlying model. Fields should be accessible on front-end and backend.

Schema

Meta Data should net a reliable outcome:
"key1" : "value1" should be as applicable as "key2" : "value2", without restriction other than the quantity of storable key value pairs and limitations of the dimension of the value given that json as a format does not pose limits we should do that server side.

Future models to reserve or block certain key names might be possible.

Scopes & Arrays

Nature of the Values

The system is agnostic to the value contained. A decision has been made to store the data in additional columns in a jsonb which by now is relatively widely supported. Some preexisting installations might have to upgrade DBs, older sqlite versions dating before Jan 2024 are therefor not supported and need to be updated.

"key1" : "value1" should lead to the same behaviour as "key2" : "value2" without requiring modification inside the code. Reasonable provisions should be taken inside the code base to edit the quantities and content size of key value pairs and appropriate errors should be returned where rate or size limits are exceeded.

Administrative Meta Data

Allow appropriate roles to customize resources with private fields not end-user accessible.
R/W = Read and Write Access

	Admin	Use Case
Order	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Line Item	R/W	Add information to customize object such as link to an image for customization of a product or relate to a 3rd party system reference
RMA	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Payment	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Order	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Shipment	R/W	Add 3rd party system reference fields such as a delivery note
Payment	R/W	Add 3rd party system reference fields for payment conciliation
User	R/W	Add a 3rd party reference

Customer Meta Data

Allowing users to customize an incomplete order (or I'd say a cart) and call the information after a completed order. Admins should in an ideal case be allowed to edit those information after purchase, but I feel that can wait if technical limitations are currently in place.

	User	Admin	Use Case
Order	R/W	R/W	Add 3rd party system reference fields such as PO-Order,
Line Item	R/W	R/W	Add information to customize object such as link to an image for customization of a product or relate to a 3rd party system reference
User	R/W	R/W	Add additional User Fields or References

fthobe · 2025-02-03T17:28:09Z

@shahmayur001

Thomas' comment on slack:
tvdeyen
28 minutes ago
Just tried to review, but the changes in single commits are soo huge that they literaly crashed my browser.
Could you please create smaller commits that are self contained (ie. first introduce the actual metadata object, with tests but without using it and then commit by commit introduce it in the models that require it).
That way it would be much easier to review and reason about. There is a lot of good information in the PR description, but this information is missing in the commits. Writing smaller atomic commits helps also writing better commit messages, because the context is smaller and better explained that way.
This is a great resource I share often. Especially #7 is very important. We also linked some more notes about good git commit messages and why they matter in our CONTRIBUTION.md file.
https://cbea.ms/git-commit/

JustShah · 2025-02-04T07:34:48Z

permitted_<resource>_attributes Method (from base_controller.rb)

Purpose:

This method is dynamically defined for each resource in the METADATA_MODELS array (e.g., Order, CustomerReturn, Payment, etc.).

How it works:

It checks if the current user has :admin permissions for the given resource (e.g., Spree::Order, Spree::Payment, etc.).
If the user is an admin for that resource, it then appends admin_metadata_attributes.
If the user is not an admin, it simply calls super() to return the default attributes for the resource.

Key Use:

It ensures that admin users have additional metadata attributes, while non-admin users only get the default attributes.

<method_name>_attributes Method (from api_helpers.rb)

Purpose:

This method dynamically creates accessor methods for each resource in the METADATA_MODELS hash (e.g., order, payment, user, etc.).

How it works:

For each resource in the hash (e.g., order: Spree::Order, payment: Spree::Payment), it defines a method like order_attributes, payment_attributes, etc.
Each generated method calls the authorized_attributes method, which checks if the user has :admin permissions for the resource.
If the user is an admin, it adds :admin_metadata to the list of attributes defined in the Spree::Api::Config for that resource.
If the user is not an admin, it returns only the default attributes for that resource.

Key Use:

This method centralizes the logic for checking permissions and ensures that the appropriate set of attributes is returned based on user roles.

Summary:

The permitted_<resource>_attributes method from the BaseController dynamically defines which attributes are allowed for each resource, adding admin-specific metadata when appropriate.
The <method_name>_attributes method in the helper module similarly checks user permissions and adjusts the set of attributes returned for API resources, appending admin metadata when the user has admin rights.

tvdeyen

this pr is still very hard to review, because the commits lack the information on why a certain change has been necessary. also lots of changes seem to be fixups for previous commits. this is hard to follow sometimes. I would also like to keep the list of models we introduce this into to a bare minimum. we can either add them later or (better) create a generator that allows to easily integrate it in other models if necessary.

tvdeyen · 2025-02-10T11:07:51Z

core/lib/spree/app_configuration.rb

@@ -209,6 +209,18 @@ class AppConfiguration < Preferences::Configuration
    #   @return [Integer] Orders to show per-page in the admin (default: +15+)
    preference :orders_per_page, :integer, default: 15

+    # @!attribute [rw] max_keys
+    #   @return [Integer] Maximum keys that can be allocated in customer and admin metadata column(default: +6+)
+    preference :max_keys, :integer, default: 6


Can you explain why this preference could be helpful?

It limits the maximum amount of key value pairs.

As much xml expansion is not a problem here, we never the less given the low degree of auth needed to write practically anything create a stability issue.

Limiting it seemed reasonable. I will see to adapt the PR and Documentation accordingly as i believe it's important to have it there.

I'm also not convinced this is a valid addition. Can you name one real world example where this might be needed?

tvdeyen · 2025-02-10T11:10:17Z

api/app/controllers/spree/api/orders_controller.rb

+
+        params[:order].delete(:customer_metadata) if params[:order]
+      end
+


please add specs for this and explain why this is necessary in the commit message

We already have specs for this https://github.com/gms-electronics/solidus/pull/4/files#diff-2ef02110b947167ef3c10ad87001d8c36b1425578b4126fd5d39dd234e4906b7R132

Basically, This prevents users from updating customer_metadata once order it is completed

tvdeyen · 2025-02-10T11:17:23Z

api/app/controllers/spree/api/base_controller.rb

+        :StockLocation,
+        :StockMovement,
+        :Variant
+      ]


This should be configurable via the app configuration.

please amend the change to the introducing commit

tvdeyen · 2025-02-10T11:17:55Z

api/app/helpers/spree/api/api_helpers.rb

+        stock_location: Spree::StockLocation,
+        stock_movement: Spree::StockMovement,
+        user: Spree::LegacyUser
+      }


This is duplicating the constant in the api controller. Can we make this configurable?

@shahmayur001 I'd love to see this, but estimate if we can slip this in.

tvdeyen · 2025-02-10T11:19:22Z

api/app/helpers/spree/api/api_helpers.rb

+        shipment: Spree::Shipment,
+        stock_location: Spree::StockLocation,
+        stock_movement: Spree::StockMovement,
+        user: Spree::LegacyUser


Please use Spree.user_class Spree::LegacyUser is only used in the dummy app for testing.

@shahmayur001 this one makes sense

tvdeyen · 2025-02-10T11:23:32Z

api/spec/requests/spree/api/orders_spec.rb

@@ -79,6 +79,68 @@ module Spree::Api
        end
      end

+      context "when the user is not admin but has ability to create and update orders" do


these specs also belong into the same commit as the implementation

tvdeyen · 2025-02-10T11:23:45Z

api/spec/requests/spree/api/payments_spec.rb

@@ -45,6 +50,22 @@ module Spree::Api
            expect(json_response).to have_attributes(attributes)
          end

+          it "allows creating payment with customer metadata but not admin metadata" do


these specs also belong into the same commit as the implementation

tvdeyen · 2025-02-10T11:23:56Z

api/spec/requests/spree/api/users_spec.rb

+
+        expect(json_response['email']).to eq user.email
+        expect(json_response).not_to have_key('admin_metadata')
+      end


these specs also belong into the same commit as the implementation

tvdeyen · 2025-02-10T11:29:42Z

core/app/models/concerns/spree/metadata.rb

+    extend ActiveSupport::Concern
+
+    included do
+      store :customer_metadata, coder: JSON


We need to tell active record to always use json as column type in order to work in maria db

See AlchemyCMS/alchemy_cms@0817f1d

attribute :customer_metadata, :json

@shahmayur001 makes sense, please adjust

core/db/migrate/20250129061658_add_metadata_to_spree_resources.rb

fthobe · 2025-02-10T11:36:50Z

@tvdeyen can we agree on all transactional models plus users?

Orders
Shipments
Payments
Users
Line Items
RMAs and Returns

I think that would be good enough for now and we can later decide to change addresses as well.

tvdeyen · 2025-02-10T11:47:28Z

Orders

Shipments

Payments

Users

Line Items

I think that would be good enough for now and we can later decide to change addresses as well.

Yes

fthobe · 2025-02-10T14:43:45Z

@shahmayur001 to not throw the work, please squash commits that overwrite each other, keep an eye on the comments that @tvdeyen has made related to DB compatibility, we really missed that one and that was an excellent catch. Remove everything that is not transactional but users and in addition we remove stock movements as people can work (and probably will) push stock only from ERP and keep delivery notes outside.

@tvdeyen I would keep returns / rmas for the same reasons as I think that this is a needed feature at the first place (correlation with 3rd party systems). Stock movements I believe are overkill. Can you live with that?

tvdeyen · 2025-02-10T20:22:48Z

Yes, we can live with that. Stock movements table can get pretty big, pretty fast. That's why we want to keep it as simple as possible.

fthobe · 2025-02-10T20:58:41Z

@tvdeyen I can see that.

@shahmayur001 so the agreed upon version is to also keep RMA / returns.

This migration adds customer_metadata and admin_metadata to various tables

… and length Introduced validations and configurable limits for metadata attributes (`customer_metadata` and `admin_metadata`) to ensure consistency and prevent overly large or complex structures. **Implementation:** - Set default values for `customer_metadata` and `admin_metadata` to empty hashes (`{}`) for proper initialization. - Implemented validation to limit the number of keys, and the length of keys and values in metadata. - Added configuration checks to ensure metadata does not exceed the limits defined in `Spree::Config`. - Introduced preferences: `meta_data_max_keys`, `meta_data_max_key_length`, and `meta_data_max_value_length` to control metadata size: - `meta_data_max_keys`: Max keys allowed (default: 6). - `meta_data_max_key_length`: Max key length (default: 16). - `meta_data_max_value_length`: Max value length (default: 256). - Created a shared concern to handle metadata validation logic. - Added tests to ensure default values and validation logic work correctly. - Created a shared example that can be used in other specs to ensure consistent testing of metadata validation across various parts of the system.

Call the metadata concern to various models to validate and add admin and customer metadata

…tadata handling Update the API configuration to permit the `customer_metadata` attribute across various models, such as orders, payments, shipments, and return authorizations. This change allows better handling of customer metadata within API requests, ensuring consistent storage and validation of metadata attributes. Additionally, it updates the `preference` configurations for models to include `customer_metadata`. **Implementation:** - Added `customer_metadata` to the permitted attributes for several API resources. - Updated various `preference` settings, including `order_attributes`, `payment_attributes`, `shipment_attributes`, etc., to include `customer_metadata`. - Ensured that `customer_metadata` is correctly permitted in the API parameters for each relevant model, allowing admins and users to store custom metadata. - This change improves the flexibility of metadata management within the API, enabling dynamic handling and easier customizations.

User cannot update customer_metadata after the status of order is complete

Introduce the `admin_metadata` attribute to support dynamic metadata handling for admins. It enables configuration of metadata via API preferences, and ensures that metadata fields are properly permitted for admin users through new helper methods. **Implementation:** - Added `admin_metadata` to resources for admins. - Integrated `metadata_permit_parameters` and `metadata_api_parameters preference` in API configuration. - Implemented `permitted_<resource>_attributes` and `<method_name>_attributes` methods for dynamic metadata permissions. - Added `extract_metadata` in `line_items_controller` to handle metadata fields.

This configuration adds the ability to enable or disable metadata restrictions as needed. Following configurations are possible: - `meta_data_validation_enabled` : Enables or disables restrictions globally.

github-actions bot added changelog:solidus_core changelog:solidus_api changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions changelog:solidus_sample labels Jan 30, 2025

fthobe closed this Jan 30, 2025

fthobe deleted the metadata-draft-pr branch January 30, 2025 18:10

JustShah restored the metadata-draft-pr branch January 30, 2025 18:16

JustShah reopened this Jan 30, 2025

JustShah changed the base branch from store_metadata_for_solidus_resources to main January 30, 2025 18:16

github-actions bot removed changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions changelog:solidus_sample labels Jan 30, 2025

This was referenced Jan 31, 2025

MetaData support for Products, Orders and other resources solidusio/solidus#5897

Closed

Store metadata for solidus resources solidusio/solidus#5951

Closed

JustShah force-pushed the metadata-draft-pr branch from a294e67 to 32df583 Compare February 4, 2025 07:31

tvdeyen reviewed Feb 10, 2025

View reviewed changes

JustShah force-pushed the metadata-draft-pr branch 2 times, most recently from 911ba01 to f718d86 Compare February 11, 2025 10:23

github-actions bot added changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions and removed changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions labels Feb 18, 2025

JustShah force-pushed the metadata-draft-pr branch from e2d8324 to 192c365 Compare February 18, 2025 06:06

github-actions bot added changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions labels Feb 18, 2025

JustShah added 3 commits February 18, 2025 11:39

Add metadata columns for resources

cec3053

This migration adds customer_metadata and admin_metadata to various tables

Include concern to models

cc5b63c

Call the metadata concern to various models to validate and add admin and customer metadata

JustShah force-pushed the metadata-draft-pr branch from 192c365 to 3286a68 Compare February 18, 2025 06:10

github-actions bot removed changelog:solidus_backend changelog:solidus_admin changelog:solidus_promotions changelog:solidus_legacy_promotions labels Feb 18, 2025

JustShah force-pushed the metadata-draft-pr branch 2 times, most recently from b8ca440 to b08d5ee Compare February 18, 2025 06:37

JustShah added 5 commits February 18, 2025 12:12

Add restriction for metadata update after order is complete

678750d

User cannot update customer_metadata after the status of order is complete

Linting

0040a2c

Introduce Customer and Admin Metadata Restriction Preference

b954491

This configuration adds the ability to enable or disable metadata restrictions as needed. Following configurations are possible: - `meta_data_validation_enabled` : Enables or disables restrictions globally.

JustShah force-pushed the metadata-draft-pr branch from b08d5ee to b954491 Compare February 18, 2025 06:43

fthobe closed this Feb 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat[Core]: Admin and User Metadata for transactional ressources and users (#5897) #4

feat[Core]: Admin and User Metadata for transactional ressources and users (#5897) #4

JustShah commented Jan 30, 2025 •

edited by fthobe

Loading

fthobe commented Feb 3, 2025

JustShah commented Feb 4, 2025

tvdeyen left a comment

tvdeyen Feb 10, 2025

fthobe Feb 12, 2025 •

edited

Loading

kennyadsl Feb 13, 2025

tvdeyen Feb 10, 2025

JustShah Feb 13, 2025 •

edited

Loading

tvdeyen Feb 10, 2025

tvdeyen Feb 10, 2025

JustShah Feb 13, 2025

tvdeyen Feb 10, 2025

fthobe Feb 10, 2025

JustShah Feb 11, 2025

tvdeyen Feb 10, 2025

fthobe Feb 10, 2025

JustShah Feb 11, 2025

tvdeyen Feb 10, 2025

JustShah Feb 13, 2025

tvdeyen Feb 10, 2025

JustShah Feb 13, 2025

tvdeyen Feb 10, 2025

JustShah Feb 13, 2025

tvdeyen Feb 10, 2025

fthobe Feb 10, 2025

JustShah Feb 11, 2025

fthobe commented Feb 10, 2025 •

edited

Loading

tvdeyen commented Feb 10, 2025

fthobe commented Feb 10, 2025 •

edited

Loading

tvdeyen commented Feb 10, 2025 via email

fthobe commented Feb 10, 2025


		params[:order].delete(:customer_metadata) if params[:order]
		end

feat[Core]: Admin and User Metadata for transactional ressources and users (#5897) #4

feat[Core]: Admin and User Metadata for transactional ressources and users (#5897) #4

Conversation

JustShah commented Jan 30, 2025 • edited by fthobe Loading

Breaking Changes

Introduction

Overview

How is the additional information stored?

What is the access model?

Testing

Detailed Overview

Types of Meta Data and usage suggestions

Overview

Administrative Metadata

Customer Meta Data

Schema

Scopes & Arrays

Nature of the Values

Administrative Meta Data

Customer Meta Data

fthobe commented Feb 3, 2025

JustShah commented Feb 4, 2025

permitted_<resource>_attributes Method (from base_controller.rb)

Purpose:

How it works:

Key Use:

<method_name>_attributes Method (from api_helpers.rb)

Purpose:

How it works:

Key Use:

Summary:

tvdeyen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fthobe Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JustShah Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fthobe commented Feb 10, 2025 • edited Loading

tvdeyen commented Feb 10, 2025

fthobe commented Feb 10, 2025 • edited Loading

tvdeyen commented Feb 10, 2025 via email

fthobe commented Feb 10, 2025

JustShah commented Jan 30, 2025 •

edited by fthobe

Loading

fthobe Feb 12, 2025 •

edited

Loading

JustShah Feb 13, 2025 •

edited

Loading

fthobe commented Feb 10, 2025 •

edited

Loading

fthobe commented Feb 10, 2025 •

edited

Loading