GSoC Proposal: Support for Kubernetes Service Discovery(en)

2023-04-08 约 2710 字预计阅读 6 分钟

Abstract

While using ShenYu, I found errors in the documentation of the registry component and submitted an Issue. Through communication with the community developers and reading the source code, I confirmed the issue and submitted a PR to fix it. This gave me great confidence in applying for this project.

This application mainly involves the architecture of the Apache ShenYu registry center, how to register microservices deployed in Kubernetes to ShenYu (overall solution design, code details), achievements, time plan, commitments, etc.

In GSoC 2023, I only submitted a proposal to ShenYu.

1 Background

1.1 Apache ShenYu

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios.

However, ShenYu’s support for Kubernetes is still relatively weak.

1.2 Task Description

To make ShenYu a fully “Cloud-Native” gateway, it is not something that can be achieved overnight.

The topic of the current study is the first step towards the cloud-native transformation of ShenYu: Enabling microservices deployed in Kubernetes to be registered in ShenYu Admin and using Kubernetes as the registration center.

2 Analyze the service registration mechanism of Apache ShenYu

After studying the official documents “Client Registry Design” and “Register Center Source Code Analysis of Http Register” on the Apache ShenYu blog, I created a core flowchart of the ShenYu service registration mechanism. I removed caching components such as Disruptor because these details are not relevant to this topic.

As shown in the diagram, Microservice represents the microservice that needs to be integrated with ShenYu. ShenYu Admin is a ShenYu component used to handle service registration and plugin rule configuration, while ShenYu Gateway is the actual ShenYu gateway component that handles and forwards network requests.

The yellow lines represent the service registration process, while the green lines represent the flow path when a user initiates a network request.

Based on this diagram, let me introduce the service registration process and features of Apache ShenYu.

2.1 Service registration process

After startup, the microservice sends its service information (metadata, URI, port) to the Register Center via the Register Client.
ShenYu Admin monitors data changes in the Register Center in real-time, and performs data parsing when changes occur.
ShenYu Admin synchronizes the microservice’s detailed information to ShenYu Gateway via the Data Sync module. Gateway then caches this data as a basis for subsequent processing/forwarding of network traffic.

Some noteworthy details:

The Register Client does not need to be manually implemented by the microservice. ShenYu provides an SDK. For example, for gRPC protocol service registration, simply import the following dependencies and configure the connection information for the register center:
```
<dependency>
    <groupId>org.apache.shenyu</groupId>
    <artifactId>shenyu-spring-boot-starter-plugin-grpc</artifactId>
    <version>${project.version}</version>
</dependency>
```
There is a Disruptor queue between the Register Client and Register Center, and between the Register Center and ShenYu Admin, which provides decoupling and buffering.
The “Data Sync” process between ShenYu Admin and ShenYu Gateway actually uses another Register Center for implementation, with technical details hidden in the diagram.

2.2 Service Registration Features of Apache ShenYu

The biggest feature of service registration in Apache ShenYu is that the registration action is initiated by the microservice side.

This brings several characteristics:

More microservice information can be carried during registration. Compared with Kubernetes’ native service discovery, Kubernetes cannot directly perceive the details of each backend interface running inside the Pod, and must be declared in YAML. ShenYu provides Java annotations, allowing microservices to register each backend interface directly to ShenYu Admin during code writing without additional configuration.
Decoupling of Register Client and Register Server is achieved using the registration center, making extension easier. However, there is a new cost: each microservice must have the permission to read and write the registration center. Because microservices do not communicate directly with ShenYu Admin (except for the HTTP registration method), but instead write microservice data into the registration center.

3 Solution Design - Using Kubernetes as Register Center

Register Center is essentially a middleware for storing microservice metadata, where Register Client writes and Register Server reads. So what we need to do is to store the metadata and uri (including port) information of microservices in Kubernetes and listen to it on the Register Server side.

3.1 Choosing the Storage Method

There are currently three main ways to store information in K8s:

Store information in Kubernetes Pod Annotations/Labels. As mentioned earlier, ShenYu’s service registration is independent and actively registered by each microservice, and a microservice corresponds to a process, which is a Kubernetes Pod. It seems better to use Pod Annotations/Labels (such as dubbo-go), but since ShenYu’s metadata exists independently of a specific microservice instance (i.e., Pod), it is not used.
Store information in ConfigMap. This is consistent with the original intention of ConfigMap to store configuration, so I decided to adopt this approach. At the same time, use Labels to filter specific ConfigMap.
Store information in Secert. This is similar to ConfigMap, with data encryption added, but it is not very meaningful and therefore not used.

3.2 Storage Details

3.2.1 Single ConfigMap or Multiple ConfigMaps

We can choose to use a single ConfigMap to store all the key-value data of the microservices, but this is clearly not reasonable:

Every time a microservice changes, the same ConfigMap needs to be modified.
There is a limit to the size of Kubernetes object resources.

Therefore, we use multiple ConfigMaps to store microservice data, using Labels for filtering.

3.2.2 Detailed Design of ConfigMap

Let’s first take a look at how data is stored when using etcd as the registry center:

shenyu
   ├──regsiter
   ├    ├──metadata
   ├    ├     ├──${rpcType}
   ├    ├     ├      ├────${contextPath}
   ├    ├     ├               ├──${ruleName} : save metadata data of MetaDataRegisterDTO
   ├    ├──uri
   ├    ├     ├──${rpcType}
   ├    ├     ├      ├────${contextPath}
   ├    ├     ├               ├──${ip:prot} : save uri data of URIRegisterDTO
   ├    ├     ├               ├──${ip:prot}

We can see that the data is mainly divided into the following layers:

register->metadata/uri->${rpcType}->${contextPath}->details

Now we will design a new storage method for ConfigMap:

Top-level design

First, I store all ConfigMaps in the shenyu namespace, and their names all start with register-. I also add a label shenyu.apache.org/register: true to distinguish them from other ConfigMaps that are not related to the registration center.

Handling metadata/uri

Next, the metadata and uri are stored in two separate ConfigMaps, with the prefixes register-metadata- and register-uri-, respectively. They are also labeled with shenyu.apache.org/register-type: metadata and shenyu.apache.org/register-type: uri, respectively. (The different prefixes are for human readability, while the different labels are for programmatic filtering.)

Handling rpcType

The next level is rpcType. Similarly, the prefix format is register-metadata-${rpcType}- or register-uri-${rpcType}-, such as register-metadata-http. The label shenyu.apache.org/register-rpc-type: ${rpcType} is also used, such as shenyu.apache.org/register-rpc-type: http.

Handling contextPath

For filtering, the label shenyu.apache.org/register-context-path: ${contextPath} is added.

For naming, since contextPath may not conform to the naming rules of ConfigMaps (not a DNS subdomain), it is concatenated with the ruleName or ip:port and then hashed. A mechanism is used to ensure that ConfigMap names are not duplicated, such as adding a timestamp.

Handling ruleName and ip:port

ruleName records metadata, while ip:port records uri information, and both are stored as JSON.

Considering that a microservice may be particularly large and cause individual contextPath data to exceed the capacity limit, I plan to store only one ruleName or ip:port record per ConfigMap, using the labels shenyu.apache.org/register-rule-name: ${ruleName} and shenyu.apache.org/register-ip-port: ${ipPort}. The content format of the ConfigMap is data: {...}.

3.2.3 Example

In the following example, a microservice named app1 is created with a contextPath of /app1. Two interfaces are registered and two instances (Pods) are running, with IP addresses 10.42.0.34:8189 and 10.42.0.36:8189.

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: shenyu
  name: register-metadata-http-asdfio-1679973837912
  labels:
    "shenyu.apache.org/register": "true"
    "shenyu.apache.org/register-type": "metadata"
    "shenyu.apache.org/register-context-path": "app1"
    "shenyu.apache.org/register-rule-name": "app1--app1-hello"
data:
  data: |
    {"appName":"app1","contextPath":"/app1","path":"/app1/hello","pathDesc":"spring annotation register","rpcType":"http","serviceName":"org.apache.shenyu.examples.http.controller.SpringMvcMappingPathController","methodName":"hello","ruleName":"/app1/hello","enabled":true,"pluginNames":[],"registerMetaData":true,"timeMillis":1679973837912,"addPrefixed":false}    

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: shenyu
  name: register-metadata-http-dfgjkl-1679973837913
  labels:
    "shenyu.apache.org/register": "true"
    "shenyu.apache.org/register-type": "metadata"
    "shenyu.apache.org/register-context-path": "app1"
    "shenyu.apache.org/register-rule-name": "app1--app1-request-**"
data:
  data: |
    {"appName":"app1","contextPath":"/app1","path":"/app1/request/**","rpcType":"http","serviceName":"org.apache.shenyu.examples.http.controller.RequestController","ruleName":"/app1/request/**","enabled":true,"pluginNames":[],"registerMetaData":true,"timeMillis":1679973837911,"addPrefixed":false}    

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: shenyu
  name: register-metadata-http-oujijk-1679973837914
  labels:
    "shenyu.apache.org/register": "true"
    "shenyu.apache.org/register-type": "uri"
    "shenyu.apache.org/register-context-path": "app1"
    "shenyu.apache.org/register-ip-port": "10.42.0.34:8189"
data:
  data: |
    {"protocol":"http://","appName":"app1","contextPath":"/app1","rpcType":"http","host":"10.42.0.34","port":8189}    
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: shenyu
  name: register-metadata-http-cxvhyu-1679973837915
  labels:
    "shenyu.apache.org/register": "true"
    "shenyu.apache.org/register-type": "uri"
    "shenyu.apache.org/register-context-path": "app1"
    "shenyu.apache.org/register-ip-port": "10.42.0.36:8189"
data:
  data: |
    {"protocol":"http://","appName":"app1","contextPath":"/app1","rpcType":"http","host":"10.42.0.36","port":8189}

3.3 Register Client and Register Server Code Implementation

The core of implementing the Register Client and Register Server lies in implementing the ShenyuClientRegisterRepository and ShenyuServerRegisterRepository SPIs.

3.3.1 ShenyuClientRegisterRepository

Here is the definition of ShenyuClientRegisterRepository:

/**
 * Shenyu client register repository.
 */
@SPI
public interface ShenyuClientRegisterRepository {
    default void init(ShenyuRegisterCenterConfig config) {}
    void persistInterface(MetaDataRegisterDTO metadata);
    default void persistURI(URIRegisterDTO registerDTO) {}
    default void persistApiDoc(ApiDocRegisterDTO apiDocRegisterDTO) {}
    default void close() {}
}

Init(): I will generate a Java Client that operates on Kubernetes based on user configuration, referring to the client construction and ConfigMap operation code in spring-cloud-kubernetes.
persistInterface(): Store metadata in ConfigMap according to the storage method mentioned above, using Kubernetes Java Client.
persistURI(): Store metadata in ConfigMap using Kubernetes Java Client.
persistApiDoc(): This is a recently added interface that is still under development and has not yet been implemented by other registry centers.
close(): Close the Kubernetes Java Client.

3.3.2 ShenyuServerRegisterRepository

Here is the definition of ShenyuServerRegisterRepository :

/**
 * Shenyu client server register repository.
 */
@SPI
public interface ShenyuClientServerRegisterRepository {
    
    default void init(ShenyuRegisterCenterConfig config) {}
    
    default void init(ShenyuClientServerRegisterPublisher publisher, ShenyuRegisterCenterConfig config) {}

    default void close() {}
}

The main work to be done is mainly on the second method:

Initialize the Kubernetes Java Client.
Use Kubernetes List-Watch mechanism to subscribe to ConfigMap updates.
Handle ConfigMap changes in a timely manner when detected.

4 Details of registering microservices deployed in Kubernetes with ShenYu.

In addition to the core design of the registration center described in the previous section, there are some details that need to be addressed in this project.

4.1 Network Connectivity Issues

When a microservice actively registers with ShenYu Admin, it can only obtain the Pod IP of the microservice (which is virtual and can only be used within Kubernetes). However, all user network requests are forwarded through ShenYu Gateway, so if ShenYu Gateway is not inside Kubernetes, the network will be interrupted.

The solution is to deploy both ShenYu Admin and ShenYu Gateway within Kubernetes. After reviewing the documentation and conducting experiments, the various CNI plugins of Kubernetes have implemented cross-node access to Pod IP, so we do not need to worry.

So how do we deploy the various components of ShenYu inside Kubernetes? Fortunately, the community has already developed an initial version of the ShenYu Helm Chart, which can quickly deploy ShenYu in Kubernetes.

4.2 Kubernetes Authorization Issues - Service Account

Since each microservice and ShenYu Admin need to read and write ConfigMap, and both microservices and ShenYu Admin are in Pods, a Service Account needs to be configured for each Pod to grant it access to Kubernetes. For more information, please refer to Configure Service Accounts for Pods.

The configuration looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: shenyu-configmap-full-access
  namespace: shenyu
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: shenyu-kubernetes-register
  namespace: shenyu
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shenyu-configmap-full-access-binding
  namespace: shenyu
subjects:
- kind: ServiceAccount
  name: shenyu-register
  namespace: shenyu
roleRef:
  kind: Role
  name: shenyu-configmap-full-access
  apiGroup: rbac.authorization.k8s.io

Then we need to set the spec.serviceAccountName of ShenYu Admin and all microservice Pods to shenyu-kubernetes-register.

Manually creating ServiceAccount, Role, and RoleBinding can be cumbersome. Is there a simpler way?

Of course! This project will attempt to update the ShenYu Helm Chart to automatically generate the correct ServiceAccount and other objects. Users only need to modify the spec.serviceAccountName of the Pod during deployment.

5 Results for the Shenyu community

Upon completion of this project, the following contributions will be made to the Apache Shenyu community:

Take a big step towards supporting cloud-native by allowing microservices deployed in Kubernetes to register with ShenYu
Contribute a new registration center to ShenYu, based on Kubernetes Configmap
Optimize ShenYu’s Kubernetes ecosystem, such as strengthening the ShenYu Helm Chart
Provide experience for other developers’ “cloud” development journeys, such as how to quickly develop and debug ShenYu’s Kubernetes-related components locally.

6 Project Deliverables

Support deploying microservices deployed in Kubernetes to ShenYu.
Design the storage details of metadata and URI of ShenYu microservice registration using ConfigMap.
Implement a Kubernetes registry center (mainly ShenyuClientRegisterRepository and ShenyuServerRegisterRepository SPI).
Improve other details, such as automatically creating Service Account in ShenYu Helm Chart.
Improve development and user documentation, perform necessary manual testing, and add unit tests, integration tests, etc.

7 Proposed Schedule

Timeline	Work content
Preparing(Before Accepted GSoC contributor projects announced)	Maintain communication with the community, continue to contribute, discover deployment and cloud related bugs, submit issues and pull requests.
May 4 - 28(Community Bonding Period)	Familiarize oneself with the Kubernetes Java Client; Explore the best development process for integrating ShenYu and Kubernetes.
May 29 - July 09(Before Midterm Evaluations)	Complete the demo, implement support for registering microservices within Kubernetes, and submit the midterm evaluation.
July 10 - August 21(Work Period)	Optimize code details, write tests, and improve documentation.
August 21 - 28(Final week)	Submit the final work product and prepare the final mentor evaluation.

8 About Me

I am a second-year master’s student in computer science who loves programming, especially interested in the field of cloud-native technology.

I have made contributions to DevStream, mainly in DevOps-related work. There, I experienced the joy of open source and fell in love with the collaborative coding, asynchronous communication, and collaborative atmosphere unique to open source.

At ShenYu, as a newcomer in the gateway field, I am eager to contribute to the cloud-native transformation of ShenYu.

Related Skill

Kubernetes：Medium
Java：Medium
Spring-framework：Elementary（Keep Learning）

9 Community engagement

I joined the Apache ShenYu mailing list and attended community meetings to actively communicate with community developers.
When writing the proposal, I found some problems in the documentation and submitted an issue: [BUG] Inconsistent of register-center-access.
After communicating with the community, it was confirmed that the documentation was incorrect. I submitted a pull request to fix the documentation: [ISSUE #876] Inconsistent of register-center-access.
While debugging the code, I found another bug in the project’s main repository and submitted a pull request to fix it: [Fix] missing actuator dependency and port error in examples http.
I also contributed code to the ShenYu Helm Chart.
- feat: set replicas separately
- feat: support the latest version of shenyu 2.5.1

10 Other commitments

About GSoC

Time: The period for coding in GSoC happens to coincide with my summer vacation. Therefore, I have the entire summer to participate in GSoC projects and contribute to the community. I plan to dedicate 40 hours per week to this project in GSoC.
Ability: I have some knowledge of Kubernetes and have validated the core technologies of the project (such as observing the data storage format of etcd as a registry center and testing the network of Kubernetes). I believe that through my efforts and active communication with the community, I can complete this project.

Continued Involvement

After the end of GSoC, I will continue to contribute to the ShenYu community.

Continue to promote the cloud-native transformation of ShenYu
Participate in the maintenance of ShenYu Client Golang (I am also interested in the Go language)
Optimize the user experience for ShenYu, such as the documentation issues mentioned earlier.

目录