Skip to content

Concepts (Hub)

Jump to bottom
Koushik Ramachandra edited this page Apr 28, 2022 · 2 revisions

The simple ranger client comes with its set of limitations.

Imagine a scenario where a service S needs to discover tens of other services. The traditional way to do this with ranger would be to create as many service finders. Causing...

Having to write a tonne of boilerplate code to create said service finders Having to create as many connections to the data source from each of the finders. A strict binding to ZK, where ZK is the only data repository that could be used to contain node information. To help solve the above, a hub is introduced, which is a collection of service finders that can work atop any data source

Service Finder Hub

To create a service finder hub

ServiceFinderHub hub = new ServiceFinderHub(serviceDataSource, serviceFinderFactory);

The hub needs to be started for the finders to get initialized atop the specified service data source. To start the hub

hub.start();

To stop the hub

hub.stop();

The service data source and finder factory are explained below.

Service Data Source

A service data source is any data source that could be plugged to act as a repository to contain node information. Ranger comes with two pre-bundled node data sources

  • Zk data source
  • HTTP data source.

Zookeeper data source

A zookeeper data source needs the following to be initialized.

  • Zookeeper details - Can be any one of the following:
    • Connection String - Zookeeper connections string to connect to ZK cluster.
    • CuratorFramework object - A prebuilt CuratorFramework object.
  • Namespace - A namespace for the service. For example, the team name this service belongs to.

To create the data source

new ZkServiceDataSource(namespace, connectionString, curatorFramework);

Http Data Source

An HTTP data source needs the following to be initialized.

  • HttpClientConfig - Contains the following
    • host - Http host where the data repository is housed
    • port - Http port where the data repository is housed
    • secure - If the connection is on https on http.
    • connectionTimeoutMs - Connection timeout in ms
    • operationTimeoutMs - Operation timeout (read/write) in ms
  • Objectmapper - To help for serialization/deserialization

To create the data source

return new HttpDataSource(httpClientConfig, objectMapper)

Service Finder Factory

A service finder factory is the repository of service names to independent finders, A service finder factory along with the Service data source forms a hub.

There are two kinds of finder factories available, zookeeper and HTTP. Each of them is available in both sharded and unsharded modes.

Assuming the following shardInfo

private static final class TestShardInfo {
        private int shardId;

        public TestShardInfo(int shardId) {
            this.shardId = shardId;
        }

        public TestShardInfo() {
        }

        public int getShardId() {
            return shardId;
        }

        public void setShardId(int shardId) {
            this.shardId = shardId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (o == null || getClass() != o.getClass()) return false;

            TestShardInfo that = (TestShardInfo) o;

            if (shardId != that.shardId) return false;

            return true;
        }

        @Override
        public int hashCode() {
            return shardId;
        }
    }

zookeeper

To initialize the sharded zookeeper finder factory we'll need,

  • ShardInfo type - This is a type parameter to the ServiceFinderFactory class. This can be any class that can be serialized and deserialized by the Serializer and Deserializer provided (see below). The hashCode() for this class is used to match and find the matching shard for a query. So be sure to implement this properly.
  • Zookeeper details - Can be any one of the following:
    • Connection String - Zookeeper connections string to connect to ZK cluster.
    • CuratorFramework object - A prebuilt CuratorFramework object.
  • shardselector - A shard selector interface that would be used to select the shards from the repository. A shard selector helps get the matching nodes given a predicate. There are already utility shardSelectors that have been provided in ListShardSelector, MatchingShardSelector, NooPShardSelector
  • nodeselector - A node selector helps pick up one node amongst the list of nodes that a shard selector may return. There are prebuild nodeselectors in RoundRobinNodeSelecotr, RandomNodeSelector available.
  • deserializer - A deserializer implementation that will be used to deserialize the retrieved shard information from the service data source
  • nodeRefreshIntervalMs - Time to refresh the service data source to fetch the new node information

To create a sharded ZK finder factory

ZkShardedServiceFinderFactory.<TestShardInfo>builder()
    .curatorFramework(curatorFramework)
    .connectionString(connectionString)
    .nodeRefreshIntervalMs(nodeRefreshIntervalMs)
    .disablePushUpdaters(isDisablePushUpdaters())
    .deserializer(getDeserializer())
    .shardSelector(shardSelector)
    .nodeSelector(nodeSelector)
    .build();

To create an unsharded ZK finder factory

ZKUnshardedServiceFinderFactory.<TestShardInfo>builder()
    .curatorFramework(curatorFramework)
    .connectionString(connectionString)
    .nodeRefreshIntervalMs(nodeRefreshIntervalMs)
    .disablePushUpdaters(isDisablePushUpdaters())
    .deserializer(getDeserializer())
    .shardSelector(shardSelector)
    .nodeSelector(nodeSelector)
    .build();

http

  • ShardInfo type - This is a type parameter to the ServiceFinderFactory class. This can be any class that can be serialized and deserialized by the Serializer and Deserializer provided (see below). The hashCode() for this class is used to match and find the matching shard for a query. So be sure to implement this properly.
  • HttpClientConfig - Contains the following
    • host - Http host where the data repository is housed
    • port - Http port where the data repository is housed
    • secure - If the connection is on HTTPS on HTTP.
    • connectionTimeoutMs - Connection timeout in ms
    • operationTimeoutMs - Operation timeout (read/write) in ms
  • Objectmapper - To help for serialization/deserialization
  • shardselector - A shard selector interface that would be used to select the shards from the repository. A shard selector helps get the matching nodes given a predicate. There are already utility shardSelectors that have been provided in ListShardSelector, MatchingShardSelector, NooPShardSelector
  • nodeselector - A node selector helps pick up one node amongst the list of nodes that a shard selector may return. There are prebuild nodeselectors in RoundRobinNodeSelecotr, RandomNodeSelector available.
  • deserializer - A deserializer implementation that will be used to deserialize the retrieved shard information from the service data source
  • nodeRefreshIntervalMs - Time to refresh the service data source to fetch the new node information

To create a sharded HTTP finder factory

HttpShardedServiceFinderFactory.<TestShardInfo>builder()
    .httpClientConfig(clientConfig)
    .nodeRefreshIntervalMs(nodeRefreshIntervalMs)
    .deserializer(deserializer)
    .shardSelector(shardSelector)
    .nodeSelector(nodeSelector)
    .mapper(mapper)
    .build();

To create an unsharded HTTP finder factory

HttpUnShardedServiceFinderFactory.<TestShardInfo>builder()
    .httpClientConfig(clientConfig)
    .nodeRefreshIntervalMs(nodeRefreshIntervalMs)
    .deserializer(deserializer)
    .shardSelector(shardSelector)
    .nodeSelector(nodeSelector)
    .mapper(mapper)
    .build();
Clone this wiki locally