Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE #63] Function Manager and Health Service update #98

Merged
merged 44 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
75713a4
cache: Use cache to store cluster and runtime in the database because…
Lambert-Rao Apr 12, 2024
93aab64
remove HealthCheckConfig, it's useless
Lambert-Rao Apr 12, 2024
088a5e4
entities modified
Lambert-Rao Apr 12, 2024
36c7ca3
mappers modified
Lambert-Rao Apr 12, 2024
61e07af
data services(dao) modified
Lambert-Rao Apr 12, 2024
fd1af97
models: use @SuperBuilder to support builder from father
Lambert-Rao Apr 12, 2024
45c1bf3
feat: add Nacos Runtime Remoting service and its implementation. this…
Lambert-Rao Apr 12, 2024
cb77197
remoting service modify
Lambert-Rao Apr 12, 2024
9e7b956
refactor: let redis check use SDK manager
Lambert-Rao Apr 12, 2024
73cced6
refactor: let rocketmq check use SDK manager
Lambert-Rao Apr 12, 2024
376de0b
fix & docs: health check module; Optimize operation logic
Lambert-Rao Apr 12, 2024
8f22f12
sql small modifies, remove redundant fields, add needed fields, and r…
Lambert-Rao Apr 12, 2024
234fa78
nacos get runtime check
Lambert-Rao Apr 14, 2024
e625fc1
test resources, they insert test data before test
Lambert-Rao Apr 14, 2024
afb0515
feat: add metadata manager who take control of the process of data sy…
Lambert-Rao Apr 14, 2024
6791d8b
test resources, they insert test data before test
Lambert-Rao Apr 14, 2024
eb90db1
feat: add Sync Service. They get data from source(mostly cluster).
Lambert-Rao Apr 14, 2024
56f0fb5
feat: add converter, which is used to convert from metadata type into…
Lambert-Rao Apr 14, 2024
0111cf4
feat: add handler, they insert new data into database
Lambert-Rao Apr 14, 2024
f733d11
feat: add FunctionManager, it manages all sub functions, like health …
Lambert-Rao Apr 14, 2024
2cc5607
test: metadata
Lambert-Rao Apr 14, 2024
1fde7fa
feat: add meta controller, the entrance of a new cluster
Lambert-Rao Apr 14, 2024
6332dcf
application property
Lambert-Rao Apr 14, 2024
ffbbd66
test: cache and application
Lambert-Rao Apr 14, 2024
5b063c5
feat: data service wrapper. A collection of DAOs.
Lambert-Rao Apr 14, 2024
e43f4e6
feat: data runtime interface model
Lambert-Rao Apr 14, 2024
ea83a25
feat: newMetaDTO
Lambert-Rao Apr 14, 2024
8fb932c
feat: topic interface to insert data into cluster
Lambert-Rao Apr 14, 2024
da15ab4
fix: fix a typo in sql
Lambert-Rao Apr 14, 2024
22bf6d9
fix: get runtimes by 3 protocols from meta
Lambert-Rao Apr 15, 2024
e757a07
fix: move functions from FunctionManager into HealthService and Metad…
Lambert-Rao Apr 16, 2024
2e66653
refactor: remove converter
Lambert-Rao Apr 17, 2024
093b3e5
chore: use log level info to get less log
Lambert-Rao Apr 17, 2024
fc416e4
fix: remove a test that leads to the fail of github test
Lambert-Rao Apr 17, 2024
bb7e082
Merge remote-tracking branch 'apache/main' into feat_function_manager_63
Lambert-Rao Apr 18, 2024
1f05117
fix: get runtime by cluster from database
Lambert-Rao Apr 18, 2024
0a2a03c
fix: remove meta controller
Lambert-Rao Apr 18, 2024
32e4170
fix: let null password not fail health check
Lambert-Rao Apr 18, 2024
2f8366d
fix: add time limit to avoid block
Lambert-Rao Apr 18, 2024
a2d4fe5
Merge branch 'main' into feat_function_manager_63
Pil0tXia Apr 18, 2024
aff1519
Merge branch 'main' into feat_function_manager_63
Lambert-Rao Apr 19, 2024
512fc42
Merge remote-tracking branch 'origin/feat_function_manager_63' into f…
Lambert-Rao Apr 19, 2024
985c710
fix: add time limit to SDK related tests
Lambert-Rao Apr 19, 2024
4ec7f6f
fix: add try catch block on health check unit tests
Lambert-Rao Apr 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,9 @@ public class AclMetadata extends MetadataConfig {
private String resourceType;
private String resourceName;
private Integer patternType;

@Override
public String getUnique() {
return null;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,9 @@ public class ClientMetadata extends MetadataConfig {
* protocol used to connect to runtime.
*/
private String protocol;

@Override
public String getUnique() {
return host + ":" + port;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,9 @@ public class ClusterMetadata extends MetadataConfig {
private StoreType storeType;

private String description;

@Override
public String getUnique() {
return clusterName + "/" + registryAddress;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,9 @@ public class ConfigMetadata extends MetadataConfig {
private Integer instanceType;

private Long instanceId;

@Override
public String getUnique() {
return configKey;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,9 @@ public class ConnectionMetadata extends MetadataConfig {
private Long groupId;

private String description;

@Override
public String getUnique() {
return getClusterId() + "/" + sourceId + "/" + sinkId + "/" + topic;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,9 @@ public class ConnectorMetadata extends MetadataConfig {
* @see KubernetesPodStatus
*/
private Integer podState;

@Override
public String getUnique() {
return host + ":" + port;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,9 @@ public class GroupMetadata extends MetadataConfig {
private Integer type;

private String state;

@Override
public String getUnique() {
return name;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,29 @@

package org.apache.eventmesh.dashboard.common.model.metadata;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;

/**
* This class is used to represent a piece of metadata, which can be used in create, update or delete operations to metadata service(eventmesh meta
* center, eventmesh runtime cluster)<p> follow method should be called in init block which is used to indicate the type of metadata:
* {@code this.setServiceTypeEnums(MetadataServiceTypeEnums.RUNTIME);}
*/
@Data
public class MetadataConfig {
@AllArgsConstructor
@NoArgsConstructor
@SuperBuilder
public abstract class MetadataConfig {

//eventmesh registry url
private String registryAddress;
//cluster id in database
private Long clusterId;

/**
* @return A string that is unique to the source, usually a url
*/
public abstract String getUnique();
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@
import org.apache.eventmesh.dashboard.common.enums.RecordStatus;

import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;

@Data
@SuperBuilder
@NoArgsConstructor
public class RegistryMetadata extends MetadataConfig {


Expand Down Expand Up @@ -49,4 +53,9 @@ public class RegistryMetadata extends MetadataConfig {
* @see RecordStatus
*/
private Integer status;

@Override
public String getUnique() {
return host + ":" + port;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,16 @@
package org.apache.eventmesh.dashboard.common.model.metadata;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
@SuperBuilder
@EqualsAndHashCode(callSuper = true)
public class RuntimeMetadata extends MetadataConfig {

private String host;
Expand All @@ -40,7 +42,12 @@ public class RuntimeMetadata extends MetadataConfig {

private Long storageClusterId;

private Long startTimeStamp;
private Long startTimestamp;

private String clusterName;

@Override
public String getUnique() {
return host + ":" + port;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@
import org.apache.eventmesh.dashboard.common.enums.StoreType;

import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;

@Data
@SuperBuilder
@NoArgsConstructor
public class TopicMetadata extends MetadataConfig {

private StoreType storeType;
Expand All @@ -42,4 +46,9 @@ public class TopicMetadata extends MetadataConfig {
private Integer type;

private String description;

@Override
public String getUnique() {
return topicName;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,13 @@

import java.util.List;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@AllArgsConstructor
public class GetRuntimeResponse {

private List<RuntimeMetadata> runtimeMetadataList;
Expand Down
1 change: 1 addition & 0 deletions eventmesh-dashboard-console/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@
<configuration>
<excludes>
<exclude>**/org/apache/eventmesh/dashboard/console/integration/**/*.java</exclude>
<exclude>**/org/apache/eventmesh/dashboard/console/EventMeshDashboardApplicationTest.java</exclude>
</excludes>
</configuration>
</plugin>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.eventmesh.dashboard.console.cache;


import org.apache.eventmesh.dashboard.console.entity.cluster.ClusterEntity;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import lombok.Getter;

public class ClusterCache {

@Getter
private static final ClusterCache INSTANCE = new ClusterCache();

//cluster name
private HashMap<String, ClusterEntity> clusterNameMap = new HashMap<>();

private HashMap<Long, ClusterEntity> clusterIdMap = new HashMap<>();

private static final Object lock = new Object();

public ClusterEntity getClusterById(Long id) {
return INSTANCE.clusterIdMap.get(id);
}

public ClusterEntity getClusterByName(String name) {
return INSTANCE.clusterNameMap.get(name);
}

public ClusterEntity getClusterByRegistryAddress(String registryAddress) {
for (ClusterEntity clusterEntity : INSTANCE.clusterIdMap.values()) {
if (clusterEntity.getRegistryAddress().equals(registryAddress)) {
return clusterEntity;
}
}
return null;
}

public List<ClusterEntity> getClusters() {
return new ArrayList<>(INSTANCE.clusterIdMap.values());
}

public void addCluster(ClusterEntity clusterEntity) {
synchronized (lock) {
if (INSTANCE.clusterIdMap.containsKey(clusterEntity.getId())
|| INSTANCE.clusterNameMap.containsKey(clusterEntity.getName())) {
return;
}
INSTANCE.clusterIdMap.put(clusterEntity.getId(), clusterEntity);
INSTANCE.clusterNameMap.put(clusterEntity.getName(), clusterEntity);
}
}

public void deleteClusterById(Long id) {
synchronized (lock) {
ClusterEntity clusterEntity = INSTANCE.clusterIdMap.get(id);
INSTANCE.clusterIdMap.remove(id);
INSTANCE.clusterNameMap.remove(clusterEntity.getName());
}
}

public void deleteClusterByName(String name) {
synchronized (lock) {
ClusterEntity clusterEntity = INSTANCE.clusterNameMap.get(name);
INSTANCE.clusterNameMap.remove(name);
INSTANCE.clusterIdMap.remove(clusterEntity.getId());
}
}

public void syncClusters(List<ClusterEntity> clusters) {
synchronized (lock) {
INSTANCE.clusterIdMap.clear();
INSTANCE.clusterNameMap.clear();
for (ClusterEntity clusterEntity : clusters) {
INSTANCE.clusterIdMap.put(clusterEntity.getId(), clusterEntity);
INSTANCE.clusterNameMap.put(clusterEntity.getName(), clusterEntity);
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.eventmesh.dashboard.console.cache;

import org.apache.eventmesh.dashboard.console.entity.runtime.RuntimeEntity;

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class RuntimeCache {

private static final RuntimeCache INSTANCE = new RuntimeCache();

public static final RuntimeCache getInstance() {
return INSTANCE;
}

// ip:port -> runtime
private Map<String, RuntimeEntity> runtimeMap = new ConcurrentHashMap<>();

private RuntimeCache() {
}

public void addRuntime(RuntimeEntity runtimeEntity) {
runtimeMap.put(runtimeEntity.getHost() + ":" + runtimeEntity.getPort(), runtimeEntity);
}

public Collection<RuntimeEntity> getRuntimeList() {
return runtimeMap.values();
}

public void deleteRuntime(RuntimeEntity runtimeEntity) {
runtimeMap.remove(runtimeEntity.getHost() + ":" + runtimeEntity.getPort());
}

public void replaceAllRuntime(List<RuntimeEntity> runtimeEntities) {
Map<String, RuntimeEntity> newRuntimeList = new ConcurrentHashMap<>();
runtimeEntities.forEach(runtimeEntity -> {
newRuntimeList.put(runtimeEntity.getHost() + ":" + runtimeEntity.getPort(), runtimeEntity);
});
runtimeMap = newRuntimeList;
}
}
Loading
Loading