[Relayminer] add forward capabilities to send request to suppliers

[eddyzags]

Summary

This PR aims to add capabilities to forward requests to HTTP and a WebSocket server in the relayminer.

It sets up a configurable authenticated endpoint (/services/{service_id}/forward) whose responsibility would be to forward raw requests from the Relay Server to the service ID (more precisely, service node or data node).

Issue

• Issue: [RelayMiner] Add an ability to test RelayMiner configuration without application #447

Type of change

Select one or more from the following:

• New feature, functionality or library
• Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Sanity Checklist

• I have updated the GitHub Issue assignees, reviewers, labels, project, iteration and milestone
• For docs, I have run make docusaurus_start
• For code, I have run make go_develop_and_test and make test_e2e
• For code, I have added the devnet-test-e2e label to run E2E tests in CI
• For configurations, I have update the documentation
• I added TODOs where applicable

[github-actions]

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks.

You may need to run make trigger_ci to submit an empty commit that'll trigger the tests.

GCP workloads (requires changing the namespace to 1147)
Grafana network dashboard for devnet-issue-1147

[eddyzags]

Here is the new PR @Olshansk !
I'm not sure what label I am supposed to set for myself and the reviewers. Is there any documentation about it please?

[Olshansk]

@eddyzags I've added the relayminer label. Please hold off on adding devnet-e2e until we've gone through a few rounds of review. I want to avoid wasting infrastructure resources.

[Olshansk]

@eddyzags Thanks for the change!

1. A few minor nits/comments/edits
2. Please merge with main
3. In case you missed it, do check out make send_relay_path_WEBSOCKET: it's websockets leveraging the PATH gateway
4. Can you provide some more detail/rationale on the:

• Token
• Use case
• Maybe an E2E mermaid diagram of how this works

I'm still struggling to understand what use case this serves that was possible before, but am pretty sure it's just an oversight on my part.

[Olshansk]

Can you #PUC why we need two websockeet connections and one is not enough?

[eddyzags]

It is written in the comment (103 & 104).

I need two connections to proxy the websocket traffic from the client to the backend websocket service. One connection (clientConn) between your proxy and the client, and one connection (serviceConn) between your proxy and the backend service.

[jorgecuesta]

@Olshansk @eddyzags is this a kind of --simRelays of more? if this is that, it should be on next iter, non-blocking release because you already have ping feature which will verify the communication to the backend, right?

[Olshansk]

Delegating to @eddyzags to answer this question.

[eddyzags]

is this a kind of --simRelays of more?

Yes. As stated in this issue #447 , the objective is to:
add a functionality to RelayMiner: an ability to send relay through a supplier without providing meta information (session, signature, etc.) so operators can verify their RelayMiner configuration.

if this is that, it should be on next iter, non-blocking release because you already have ping feature which will verify the communication to the backend, right?

I don't have any opinions on whether this should be in the next iteration or a non-blocking release. But, the /ping and /forward endpoints serve two purposes (or assert two different things).

From an OSI standpoint, the /ping endpoint ensures that the Physical, Data Link, and network layers are working correctly in the intercommunication between the client (relayminer) and the server (data node).
On the other hand, the /forward endpoint asserts a bit more. It also gives operators confidence about the Session, Presentation, and Application layers.

I think we should give both capabilities to the Relayminer's users.

cc @jorgecuesta @Olshansk

[Olshansk]

@eddyzags Your explanation above makes sense to me!

Love the OSI analogy you put together as well.

For reference, I'm working on a pocketd relayminer relay utility here: #1298

Please re-request a review once this is ready.

[eddyzags]

Friendly reminder @bryanchriswhite @Olshansk 👍🏾

[Olshansk]

@eddyzags Do you mind resolving the conflicts with main so we can re-review and get this in?

[eddyzags]

@Olshansk, there were quite a few changes to this part in the main branch. I'll have to understand it first to resolve the conflicts while keeping the code working. I am working on it. I try to push the changes as soon as I can.

[eddyzags]

@Olshansk I got mixed up with the rebase. I was better and faster for me to start from a clean slate. See follow-up here: #1681

@bryanchriswhite I am still waiting for some answers on some discussions in this PR. You can either answer here or in the new PR, I'll backport the modifications.

Sorry for the inconvenience. In the meantime, I am closing this PR.

Follow-up available here: #1681