No Native HTTP Retry Mechanism in Apps-Engine

[jannatkhandev]

No Native HTTP Retry Mechanism in Apps-Engine

Current Situation

Currently, the Rocket.Chat Apps-Engine lacks a native retry mechanism for HTTP requests. This means that apps need to implement their own retry logic when dealing with unreliable external services or temporary network issues, leading to:

• Duplicate code across apps
• Inconsistent retry implementations
• No standardized way to handle transient failures

Proposed Solution

Implement a new RetryHttpProvider class that can be optionally enabled by apps to provide robust retry functionality without affecting existing apps or adding overhead to the base implementation.

Implementation Details

1. Create RetryHttpProvider Class

export class RetryHttpProvider extends Http {
    private static readonly ENHANCED_RETRY_CONFIG: IHttpRetryConfig = {
        enabled: true,
        maxAttempts: 3,
        initialDelay: 1000,
        statusCodesToRetry: [
            HttpStatusCode.SERVICE_UNAVAILABLE,
            HttpStatusCode.INTERNAL_SERVER_ERROR,
            HttpStatusCode.TOO_MANY_REQUESTS,
            HttpStatusCode.GATEWAY_TIMEOUT,
            HttpStatusCode.BAD_GATEWAY
        ],
    };
    // Inherit from base Http class to maintain all existing functionality
    constructor(
        accessManager: AppAccessorManager,
        bridges: AppBridges,
        httpExtender: IHttpExtend,
        appId: string,
    ) {
        super(accessManager, bridges, httpExtender, appId);
    }

    private enhanceOptions(options?: IHttpRequest): IHttpRequest {
        return {
            ...options,
            retry: {
                ...RetryHttpProvider.ENHANCED_RETRY_CONFIG,
                ...options?.retry,
            },
        };
    }
    // Override HTTP methods to include retry functionality
    public get(url: string, options?: IHttpRequest): Promise<IHttpResponse> {
        return super.get(url, this.enhanceOptions(options));
    }
   // ... other HTTP methods
}

2. Create Configuration Manager

export class RetryHttpConfiguration {
  public async setup(configuration: IConfigurationExtend): Promise<void> {
    // Add settings for retry configuration
    await Promise.all([
 configuration.settings.provideSetting({
      id: 'enable-http-retries',
      type: SettingType.BOOLEAN,
      packageValue: false,
      required: true,
      public: true,
      i18nLabel: 'enable-http-retries',
    });
  configuration.settings.provideSetting({
          id: `${this.config.alias}-max-attempts`,
          type: SettingType.NUMBER,
          public: true,
          required: true,
          packageValue: '',
          i18nLabel: `${this.config.alias}-max-attempts`,
      }),

      configuration.settings.provideSetting({
          id: `${this.config.alias}-initial-delay`,
          type: SettingType.NUMBER,
          public: true,
          required: true,
          packageValue: '',
          i18nLabel: `${this.config.alias}-initial-delay`,
      }),

      configuration.settings.provideSetting({
        id: `${this.config.alias}-status-codes-to-retry`,
        type: SettingType.MULTI_SELECT,
        public: true,
        required: true,
        packageValue: '',
        i18nLabel: `${this.config.alias}-status-codes-to-retry`,
    }),
  }
}

3. Usage in Apps

export class YourApp implements IApp {
  private retryConfig: RetryHttpConfiguration;
  protected async extendConfiguration(configuration: IConfigurationExtend): Promise<void> {
    await this.retryConfig.setup(configuration);
}
}

Benefits

1. Zero Overhead for Existing Apps

• Existing apps continue to use the base Http class
• No performance impact on apps not using retry functionality
• No breaking changes to existing implementations

2. Opt-in Enhancement

• Apps can explicitly opt-in to retry functionality by using RetryHttpConfiguration
• Configuration is handled during app initialization
• Settings are exposed in the admin UI for easy configuration

3. Consistent Implementation

• Standardized retry behavior across all apps using the feature
• Configurable retry parameters (attempts, delay, status codes)
• Built on top of existing HTTP infrastructure

Implementation Steps

1. Core Changes

   • Add IHttpRetryConfig interface to define retry configuration
   • Implement RetryHttpProvider class extending base Http
   • Create RetryHttpConfiguration for settings management

2. Accessor Manager Modifications
   This is a critical part of the implementation that enables the override functionality:

   export class AppAccessorManager {
       private httpProviders: Map<string, IHttp>;
   
       constructor(
           private readonly bridges: AppBridges,
           private readonly httpExtender: IHttpExtend,
       ) {
           this.httpProviders = new Map();
       }
   
       public getHttp(appId: string): IHttp {
           if (!this.httpProviders.has(appId)) {
               // Check if the app has enabled retry functionality
               const settings = this.getAppSettings(appId);
               if (settings.get('enable-http-retries')) {
                   this.httpProviders.set(
                       appId,
                       new RetryHttpProvider(this, this.bridges, this.httpExtender, appId)
                   );
               } else {
                   // Use default Http implementation for apps without retry enabled
                   this.httpProviders.set(
                       appId,
                       new Http(this, this.bridges, this.httpExtender, appId)
                   );
               }
           }
   
           return this.httpProviders.get(appId);
       }
   }

   How the Override Works

   1. The AppAccessorManager is responsible for providing HTTP instances to apps
   2. When an app requests an HTTP instance via getHttp():
      • Checks if the app has enabled retry functionality in its settings
      • If enabled, provides a RetryHttpProvider instance
      • If not enabled, provides the default Http instance
   3. The instance is cached in httpProviders map for subsequent requests
   4. All HTTP calls from the app will use the provided instance

   This approach ensures:

   • Zero overhead for apps not using retry functionality
   • Automatic retry capability for apps that enable it
   • Clean separation of concerns
   • No breaking changes to existing apps

3. Integration

   • Add retry configuration settings to app configuration system
   • Implement factory pattern in AppAccessorManager for HTTP provider creation
   • Add documentation and examples

4. Testing

   • Unit tests for retry logic
   • Integration tests with various status codes
   • Performance testing to ensure no impact on non-retry apps

Migration Guide

For apps wanting to use the retry functionality:

1. Import the RetryHttpConfiguration
2. Add it to your app's configuration setup:

protected async extendConfiguration(configuration: IConfigurationExtend): Promise<void> {
const retryConfig = new RetryHttpConfiguration();
await retryConfig.setup(configuration);
}

Server Setup Information:

• Version of Rocket.Chat Server:
• License Type:
• Number of Users:
• Operating System:
• Deployment Method:
• Number of Running Instances:
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version:

Client Setup Information

• Desktop App or Browser Version:
• Operating System:

Additional context

Relevant logs:

[reetp]

Please complete the server setup info.

Thanks.

[Gustrb]

Hey @reetp, I was talking with @jannatkhandev about his idea of having a native HTTP retry mechanism so I asked @jannatkhandev to create an issue so we could discuss if we agree with his idea while having more visibility with the rest of the engineering team.
Don't worry with this one, I got it :)

[d-gubert]

Hey @jannatkhandev , thanks for raising this point.

I do think, though, that the approach you propose is way larger and more opinionated than it needs to be to solve this problem. For instance, I don't think we should automatically provide settings in the app to toggle retrying on and off - this should be a decision the app developer should make, not the user.

A suggestion I'd give in this scenario would be to export a function that would do the retrying, something along the lines of this library https://www.npmjs.com/package/exponential-backoff - it has the plus side of being agnostic to the task it's executing (not tied to HTTP) while being as flexible with the configuration as your proposal.

Something to note is that apps can already include the library above to implement retries. It would be useful to provide a function like that from the apps-engine if we were to integrate with some mechanism like logging or scheduling, for instance, by allowing the retries to be non-blocking.

Do you think that doing so would solve the problem as you see it?

[jannatkhandev]

Hey @d-gubert

Thank you for the thoughtful feedback!

I agree that my proposal is kinda overly complex and opinionated, my idea was giving admin to control whether retries should be enabled on Http calls made by the app. [this setting would be available if the app developers extends configuration to add a new Retry class implementation, similar to how settings are handled in the OAuth2Client implementation]

---

so now in order to proceed, instead of the full provider implementation, we could add a simple retry utility function to the engine that:

1. Is HTTP-agnostic (can work with any Promise-based operation)
2. Provides sensible defaults while remaining flexible
3. Integrates with the engine's existing logging system

Something like?:

export interface IRetryOptions {
    maxAttempts ? : number;
    backoff ? : (attempt: number) => number;
    shouldRetry ? : (error: any) => boolean;
}
export async function withRetry < T > (
        operation: () => Promise < T > ,
        options: IRetryOptions = {},
        logger ? : AppConsole
    ): Promise < T > {
        const {
            maxAttempts = 3,
                backoff = (attempt) => Math.min(1000 Math.pow(2, attempt), 10000),
                shouldRetry = () => true
        } = options;
        let lastError: any;
        for (let attempt = 0; attempt < maxAttempts; attempt++) {
            try {
                return await operation();
            } catch (error) {
                lastError = error;
                if (!shouldRetry(error) || attempt === maxAttempts - 1) {
                    throw error;
                }
                const delay = backoff(attempt);
                logger?.debug(Retry attempt $ {
                        attempt + 1
                    }
                    /${maxAttempts} after ${delay}ms);
                    await new Promise(resolve => setTimeout(resolve, delay));
                }
            }
            throw lastError;
        }

I'd be really grateful if you can nudge me in correct direction, will then take forward as advised.

[d-gubert]

Yup, I think that direction is the way to go