EW-1061 Course file export with Common Cartridge (#5330)

* EW-1060 testing the new export via a test endpoint

* EW-1060 moved linked task to lesson dto

* updating esbuild and plugins for esbuild

* generating files storage api client

* adding files storage client factory and module infrastructure

* working on export service

* EW-1060 implemented new cc export service via microservice

* EW-1060 resolved dependecy of the new cc mapper

* EW-1060 deleted testing dto of cc export

* EW-1060 created some classes instead of interfaces

* Regenerate lesson api client.

* EW-1060 implemented new mapping of lesson content

* EW-1060 modified mapper

* EW-1060 modified mapper of lesson dto

* EW-1060 removed items from api property

* Regenerate lesson client.

* regen lesson api.

* EW-1060 modified mapper of lesson content

* EW-1060 edited common cartridge mapper

* changing factory to adapter

* adding file resource to common cartridge

* trying file export for boards

* EW-1060 added control options of course element

* adding file export for tasks and boards

* adding modules to the learn room

* added logging

* changing parent id

* changing search id for download files

* changing download mechanics

* EW-1060 changed endpoint name of new export

* adding logging for files storage service

* EW-1060 changed export endpoint to POST

* adding debug logging

* adding more debug logging

* removing some debug logs

* changing async code execution

* EW-1060 modified mapping of some cc elements

* temp changes

* fixing imports for a module

* changing the base path

* changing logging

* changing logging

* EW-1060 deleted the old export endpoint of cc- microservice

* logging

* changing logging

* changing download mechanics

* updating open api definitions

* adding some logging

* changing factory

* changing logging

* changing logging

* adding logging

* some changes

* some changes

* some changes

* changing something

* changing something

* changes

* changes

* some changes

* some changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* changing controller

* some changes

* some changes

* updating files storage api

* some changes

* adding logging

* some changes

* EW-1060 fixed export of cards

* changing file resource manifest type

* some bug fixes, probably

* hopefully fixing encoding errors

* changing to streams

* logging

* logging

* some changes

* changes

* changes

* changes

* changes

* changes

* changes

* changes

* updating the cc micro service

* updating dependencies for c module

* naming resources

* EW-1060 changed showen title of a card

* changes

* changing mappers

* trimming file captions

* removing temp folder

* EW-1060 modified test of export uc

* removing dead code and linting

* reverting two files

* adding tests

* EW-1060 added some logs

* changes

* some changes

* EW-1060 added test for cc controller

* reverting some files and code cleanup

* reverting file and adding unit tests

* EW-1060 changed some variables name

* EW-1060 added logger module to imports

* adding unit tests

* adding unit tests and test factories

* adding unit tests

* moving method into adapter

* EW-1060 added tests for cc export service

* fixing compile errors

* adding unit tests

* code refactoring

* adding check and check:watch npm scripts

* fixing imports

* linting

* skipping one test suit

* skipping another test suit

* removing unused test

* fixing merge error

* removing linter warnings
:

* fixing merge errors

* fixing import error

* fixing imports

* skipping test

* skipping unit test

* some debugging

* test logging

* changing logging

* logging

* logging

* logging

* logging

* di update

* di update

* removing debug logs

* fixing unit tests

* fixing linter warnings

* updating sonar config

* working on coverage

* working on tests

* Update package.json

Co-authored-by: Fshmit <[email protected]>

* Update apps/server/src/modules/common-cartridge/export/interfaces/common-cartridge-resource.interface.ts

Co-authored-by: Fshmit <[email protected]>

* Update apps/server/src/modules/common-cartridge/service/common-cartridge.mapper.ts

Co-authored-by: Fshmit <[email protected]>

* working on review and renaming module

* updating module index.ts

* working on review

* fixing compile error

* fixing review comments

* fixing review comments

* working on review comments

* fixing imports

* fixing compile error

* updating package.json

* reverting lock file

* working on review comments

* fixing review comments

* removing comments

---------

Co-authored-by: Firas Shmit <[email protected]>
Co-authored-by: Maximilian Kreuzkam <[email protected]>
Co-authored-by: Fshmit <[email protected]>

apps/server/src/infra/files-storage-client/files-storage-client.adapter.spec.ts

@@ -0,0 +1,129 @@
+import { faker } from '@faker-js/faker';
+import { createMock, DeepMocked } from '@golevelup/ts-jest';
+import { HttpService } from '@nestjs/axios';
+import { ConfigService } from '@nestjs/config';
+import { REQUEST } from '@nestjs/core';
+import { Test, TestingModule } from '@nestjs/testing';
+import { ErrorLogger, Logger } from '@core/logger';
+import type { Request } from 'express';
+import { from, throwError } from 'rxjs';
+import { axiosResponseFactory } from '@testing/factory/axios-response.factory';
+import { FilesStorageClientAdapter } from './files-storage-client.adapter';
+import { FileApi } from './generated';
+
+describe(FilesStorageClientAdapter.name, () => {
+	let module: TestingModule;
+	let sut: FilesStorageClientAdapter;
+	let httpServiceMock: DeepMocked<HttpService>;
+	let errorLoggerMock: DeepMocked<ErrorLogger>;
+	let configServiceMock: DeepMocked<ConfigService>;
+
+	beforeAll(async () => {
+		module = await Test.createTestingModule({
+			providers: [
+				FilesStorageClientAdapter,
+				{
+					provide: FileApi,
+					useValue: createMock<FileApi>(),
+				},
+				{
+					provide: Logger,
+					useValue: createMock<Logger>(),
+				},
+				{
+					provide: ErrorLogger,
+					useValue: createMock<ErrorLogger>(),
+				},
+				{
+					provide: HttpService,
+					useValue: createMock<HttpService>(),
+				},
+				{
+					provide: ConfigService,
+					useValue: createMock<ConfigService>(),
+				},
+				{
+					provide: REQUEST,
+					useValue: createMock<Request>({
+						headers: {
+							authorization: `Bearer ${faker.string.alphanumeric(42)}`,
+						},
+					}),
+				},
+			],
+		}).compile();
+
+		sut = module.get(FilesStorageClientAdapter);
+		httpServiceMock = module.get(HttpService);
+		errorLoggerMock = module.get(ErrorLogger);
+		configServiceMock = module.get(ConfigService);
+	});
+
+	afterAll(async () => {
+		await module.close();
+	});
+
+	beforeEach(() => {
+		jest.clearAllMocks();
+	});
+
+	it('should be defined', () => {
+		expect(sut).toBeDefined();
+	});
+
+	describe('download', () => {
+		describe('when download succeeds', () => {
+			const setup = () => {
+				const fileRecordId = faker.string.uuid();
+				const fileName = faker.system.fileName();
+				const observable = from([axiosResponseFactory.build({ data: Buffer.from('') })]);
+
+				httpServiceMock.get.mockReturnValue(observable);
+				configServiceMock.getOrThrow.mockReturnValue(faker.internet.url());
+
+				return {
+					fileRecordId,
+					fileName,
+				};
+			};
+
+			it('should return the response buffer', async () => {
+				const { fileRecordId, fileName } = setup();
+
+				const result = await sut.download(fileRecordId, fileName);
+
+				expect(result).toEqual(Buffer.from(''));
+				expect(httpServiceMock.get).toBeCalledWith(expect.any(String), {
+					responseType: 'arraybuffer',
+					headers: {
+						Authorization: expect.any(String),
+					},
+				});
+			});
+		});
+
+		describe('when download fails', () => {
+			const setup = () => {
+				const fileRecordId = faker.string.uuid();
+				const fileName = faker.system.fileName();
+				const observable = throwError(() => new Error('error'));
+
+				httpServiceMock.get.mockReturnValue(observable);
+
+				return {
+					fileRecordId,
+					fileName,
+				};
+			};
+
+			it('should return null', async () => {
+				const { fileRecordId, fileName } = setup();
+
+				const result = await sut.download(fileRecordId, fileName);
+
+				expect(result).toBeNull();
+				expect(errorLoggerMock.error).toBeCalled();
+			});
+		});
+	});
+});

apps/server/src/infra/files-storage-client/files-storage-client.adapter.ts

@@ -0,0 +1,59 @@
+import { HttpService } from '@nestjs/axios';
+import { Inject, Injectable } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
+import { REQUEST } from '@nestjs/core';
+import { JwtExtractor } from '@shared/common/utils/jwt';
+import { AxiosErrorLoggable } from '@core/error/loggable';
+import { ErrorLogger, Logger } from '@core/logger';
+import { AxiosError } from 'axios';
+import type { Request } from 'express';
+import { lastValueFrom } from 'rxjs';
+import { FilesStorageClientConfig } from './files-storage-client.config';
+import { FileApi } from './generated';
+
+@Injectable()
+export class FilesStorageClientAdapter {
+	constructor(
+		private readonly api: FileApi,
+		private readonly logger: Logger,
+		private readonly errorLogger: ErrorLogger,
+		// these should be removed when the generated client supports downloading files as arraybuffer
+		private readonly httpService: HttpService,
+		private readonly configService: ConfigService<FilesStorageClientConfig, true>,
+		@Inject(REQUEST) private readonly req: Request
+	) {
+		this.logger.setContext(FilesStorageClientAdapter.name);
+	}
+
+	public async download(fileRecordId: string, fileName: string): Promise<Buffer | null> {
+		try {
+			// INFO: we need to download the file from the files storage service without using the generated client,
+			// because the generated client does not support downloading files as arraybuffer. Otherwise files with
+			// binary content would be corrupted like pdfs, zip files, etc. Setting the responseType to 'arraybuffer'
+			// will not work with the generated client.
+			// const response = await this.api.download(fileRecordId, fileName, undefined, {
+			// 	responseType: 'arraybuffer',
+			// });
+			const token = JwtExtractor.extractJwtFromRequest(this.req);
+			const url = new URL(
+				`${this.configService.getOrThrow<string>(
+					'FILES_STORAGE__SERVICE_BASE_URL'
+				)}/api/v3/file/download/${fileRecordId}/${fileName}`
+			);
+			const observable = this.httpService.get(url.toString(), {
+				responseType: 'arraybuffer',
+				headers: {
+					Authorization: `Bearer ${token}`,
+				},
+			});
+			const response = await lastValueFrom(observable);
+			const data = Buffer.isBuffer(response.data) ? response.data : null;
+
+			return data;
+		} catch (error: unknown) {
+			this.errorLogger.error(new AxiosErrorLoggable(error as AxiosError, 'FilesStorageClientAdapter.download'));
+
+			return null;
+		}
+	}
+}

apps/server/src/infra/files-storage-client/files-storage-client.config.ts

@@ -0,0 +1,3 @@
+export interface FilesStorageClientConfig {
+	FILES_STORAGE__SERVICE_BASE_URL: string;
+}

apps/server/src/infra/files-storage-client/files-storage-client.module.spec.ts

@@ -0,0 +1,58 @@
+import { faker } from '@faker-js/faker';
+import { createMock } from '@golevelup/ts-jest';
+import { ConfigModule, ConfigService } from '@nestjs/config';
+import { REQUEST } from '@nestjs/core';
+import { Test, TestingModule } from '@nestjs/testing';
+import { Request } from 'express';
+import { FilesStorageClientAdapter } from './files-storage-client.adapter';
+import { FilesStorageClientModule } from './files-storage-client.module';
+
+describe(FilesStorageClientModule.name, () => {
+	let module: TestingModule;
+
+	const configServiceMock = createMock<ConfigService>();
+	const requestMock = createMock<Request>({
+		headers: {
+			authorization: `Bearer ${faker.string.alphanumeric(42)}`,
+		},
+	});
+
+	beforeAll(async () => {
+		module = await Test.createTestingModule({
+			imports: [FilesStorageClientModule, ConfigModule.forRoot({ isGlobal: true })],
+		})
+			.overrideProvider(ConfigService)
+			.useValue(configServiceMock)
+			.overrideProvider(REQUEST)
+			.useValue(requestMock)
+			.compile();
+	});
+
+	afterAll(async () => {
+		await module.close();
+	});
+
+	beforeEach(() => {
+		jest.clearAllMocks();
+	});
+
+	it('should be defined', () => {
+		expect(module).toBeDefined();
+	});
+
+	describe('resolve providers', () => {
+		describe('when resolving FilesStorageRestClientAdapter', () => {
+			const setup = () => {
+				configServiceMock.getOrThrow.mockReturnValue(faker.internet.url());
+			};
+
+			it('should resolve FilesStorageRestClientAdapter', async () => {
+				setup();
+
+				const provider = await module.resolve(FilesStorageClientAdapter);
+
+				expect(provider).toBeInstanceOf(FilesStorageClientAdapter);
+			});
+		});
+	});
+});

apps/server/src/infra/files-storage-client/files-storage-client.module.ts

@@ -0,0 +1,34 @@
+import { Module, Scope } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
+import { REQUEST } from '@nestjs/core';
+import { JwtExtractor } from '@shared/common/utils/jwt';
+import { LoggerModule } from '@core/logger';
+import { Request } from 'express';
+import { HttpModule } from '@nestjs/axios';
+import { FilesStorageClientAdapter } from './files-storage-client.adapter';
+import { FilesStorageClientConfig } from './files-storage-client.config';
+import { Configuration, FileApi } from './generated';
+
+@Module({
+	imports: [LoggerModule, HttpModule],
+	providers: [
+		FilesStorageClientAdapter,
+		{
+			provide: FileApi,
+			scope: Scope.REQUEST,
+			useFactory: (configService: ConfigService<FilesStorageClientConfig, true>, request: Request): FileApi => {
+				const basePath = configService.getOrThrow<string>('FILES_STORAGE__SERVICE_BASE_URL');
+
+				const config = new Configuration({
+					accessToken: JwtExtractor.extractJwtFromRequest(request),
+					basePath: `${basePath}/api/v3`,
+				});
+
+				return new FileApi(config);
+			},
+			inject: [ConfigService, REQUEST],
+		},
+	],
+	exports: [FilesStorageClientAdapter],
+})
+export class FilesStorageClientModule {}

apps/server/src/infra/files-storage-client/generated/.gitignore

@@ -0,0 +1,4 @@
+wwwroot/*.js
+node_modules
+typings
+dist

apps/server/src/infra/files-storage-client/generated/.npmignore

@@ -0,0 +1 @@
+# empty npmignore to ensure all required files (e.g., in the dist folder) are published by npm

apps/server/src/infra/files-storage-client/generated/.openapi-generator-ignore

@@ -0,0 +1,23 @@
+# OpenAPI Generator Ignore
+# Generated by openapi-generator https://github.com/openapitools/openapi-generator
+
+# Use this file to prevent files from being overwritten by the generator.
+# The patterns follow closely to .gitignore or .dockerignore.
+
+# As an example, the C# client generator defines ApiClient.cs.
+# You can make changes and tell OpenAPI Generator to ignore just this file by uncommenting the following line:
+#ApiClient.cs
+
+# You can match any string of characters against a directory, file or extension with a single asterisk (*):
+#foo/*/qux
+# The above matches foo/bar/qux and foo/baz/qux, but not foo/bar/baz/qux
+
+# You can recursively match patterns against a directory, file or extension with a double asterisk (**):
+#foo/**/qux
+# This matches foo/bar/qux, foo/baz/qux, and foo/bar/baz/qux
+
+# You can also negate patterns with an exclamation (!).
+# For example, you can ignore all files in a docs folder with the file extension .md:
+#docs/*.md
+# Then explicitly reverse the ignore rule for a single file:
+#!docs/README.md

apps/server/src/infra/files-storage-client/generated/.openapi-generator/FILES

@@ -0,0 +1,12 @@
+.gitignore
+.npmignore
+.openapi-generator-ignore
+api.ts
+api/file-api.ts
+base.ts
+common.ts
+configuration.ts
+git_push.sh
+index.ts
+models/file-record-response.ts
+models/index.ts

apps/server/src/infra/files-storage-client/generated/api.ts

@@ -0,0 +1,18 @@
+/* tslint:disable */
+/* eslint-disable */
+/**
+ * Schulcloud-Verbund-Software Server API
+ * This is v3 of Schulcloud-Verbund-Software Server. Checkout /docs for v1.
+ *
+ * The version of the OpenAPI document: 3.0
+ * 
+ *
+ * NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
+ * https://openapi-generator.tech
+ * Do not edit the class manually.
+ */
+
+
+
+export * from './api/file-api';
+