Connecting to Multiple Clusters Using kubectl
Background
kubectl relies on the kubeconfig files to locate the necessary authentication information to select a cluster and communicate with its API server. By default, kubectl uses the $HOME/.kube/config file as the credential for accessing the cluster.
When working with CCE clusters on a daily basis, it is common to manage multiple clusters simultaneously. However, this can make using kubectl to connect to clusters cumbersome, as it requires frequent switching of the kubeconfig file during routine O&M. This section introduces how to connect to multiple clusters using the same kubectl client.
The file used to configure cluster access is called kubeconfig, but it does not mean that the file name is kubeconfig.
Solution
When performing O&M on Kubernetes clusters, it is often necessary to switch between multiple clusters. The following shows some typical solutions for cluster switchover:
- Solution 1: Specify --kubeconfig of kubectl to select the kubeconfig file used by each cluster and use aliases to simplify commands.
- Solution 2: Combine clusters, users, and credentials in multiple kubeconfig files into one configuration file and run kubectl config use-context to switch between clusters.
Compared with solution 1, this solution requires manual configuration of the kubeconfig file, which is relatively complex.
Prerequisites
- You have a Linux VM with the kubectl command line tool installed. The kubectl version must match the cluster version. For details, see Install Tools.
- The VM where kubectl is installed must be able to access each cluster.
kubeconfig File Structure
kubeconfig is the configuration file of kubectl. You can download it on the cluster details page.
The content of the kubeconfig file is as follows:
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [{
"name": "internalCluster",
"cluster": {
"server": "https://192.168.0.85:5443",
"certificate-authority-data": "LS0tLS1CRUULIE..."
}
}, {
"name": "externalCluster",
"cluster": {
"server": "https://xxx.xxx.xxx.xxx:5443",
"insecure-skip-tls-verify": true
}
}],
"users": [{
"name": "user",
"user": {
"client-certificate-data": "LS0tLS1CRUdJTiBDRVJ...",
"client-key-data": "LS0tLS1CRUdJTiBS..."
}
}],
"contexts": [{
"name": "internal",
"context": {
"cluster": "internalCluster",
"user": "user"
}
}, {
"name": "external",
"context": {
"cluster": "externalCluster",
"user": "user"
}
}],
"current-context": "external"
} It mainly consists of the following sections:
- clusters: describes the cluster information, mainly the access address of a cluster.
- users: describes information about the users who can access the cluster. It includes the client-certificate-data and client-key-data certificate files.
- contexts: describes the configuration contexts. You can switch between contexts to access different clusters. A context is associated with user and cluster, that is, it defines which user accesses which cluster.
This kubeconfig file defines the private network address and public network address of the cluster as two clusters with different contexts. You can switch the context to use different addresses to access the cluster.
Solution 1: Specify Different kubeconfig Files in Commands
- Log in to the VM where kubectl is installed.
- Download the kubeconfig files of the two clusters to the /home directory on the kubectl client. The names listed below are taken as examples.
Cluster Name
kubeconfig File Name
Cluster A
kubeconfig-a.json
Cluster B
kubeconfig-b.json
- Make cluster A the default cluster for kubectl access and move the kubeconfig-a.json file to $HOME/.kube/config.
cd /home mkdir -p $HOME/.kube mv -f kubeconfig-a.json $HOME/.kube/config - Move the kubeconfig-b.json file of cluster B to $HOME/.kube/config-test.
mv -f kubeconfig-b.json $HOME/.kube/config-testYou can change config-test as needed.
- Add --kubeconfig to specify the credential used by kubectl when accessing cluster B. There is no need to add --kubeconfig when using kubectl to access cluster A because kubectl can access cluster A by default. For example, run the following command to check the nodes in cluster B:
kubectl --kubeconfig=$HOME/.kube/config-test get node
This command is long and inconvenient in case of frequent use. To simplify the command, use aliases. For example:alias ka='kubectl --kubeconfig=$HOME/.kube/config' alias kb='kubectl --kubeconfig=$HOME/.kube/config-test'
ka and kb can be custom aliases. When using kubectl, directly enter ka or kb to replace kubectl. The --kubeconfig parameter is added automatically. For example, the command for checking nodes in cluster B can be simplified as follows:
kb get node
Solution 2: Combine the kubeconfig Files of the Two Clusters Together
Two clusters are used as an example to describe how to combine the kubeconfig files to access multiple clusters.
This example configures only the Internet access to the clusters. To access multiple clusters over a private network, retain the clusters field and ensure that the clusters can be accessed over the private network. Its configuration is similar to that described in this example.
- Download the kubeconfig files of the two clusters and delete the lines related to private network access.
- Cluster A:
{ "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "externalCluster", "cluster": { "server": "https://119.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }], "users": [{ "name": "user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0tLS1CRUdJTiB...." } }], "contexts": [{ "name": "external", "context": { "cluster": "externalCluster", "user": "user" } }], "current-context": "external" } - Cluster B:
{ "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "externalCluster", "cluster": { "server": "https://124.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }], "users": [{ "name": "user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0rTUideUdJTiB...." } }], "contexts": [{ "name": "external", "context": { "cluster": "externalCluster", "user": "user" } }], "current-context": "external" }These files have the same structure except the client-certificate-data and client-key-data fields of user and the clusters.cluster.server field.
- Cluster A:
- Modify the name field as follows:
- Cluster A:
{ "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "Cluster-A", "cluster": { "server": "https://119.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }], "users": [{ "name": "Cluster-A-user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0tLS1CRUdJTiB...." } }], "contexts": [{ "name": "Cluster-A-Context", "context": { "cluster": "Cluster-A", "user": "Cluster-A-user" } }], "current-context": "Cluster-A-Context" } - Cluster B:
{ "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "Cluster-B", "cluster": { "server": "https://124.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }], "users": [{ "name": "Cluster-B-user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0rTUideUdJTiB...." } }], "contexts": [{ "name": "Cluster-B-Context", "context": { "cluster": "Cluster-B", "user": "Cluster-B-user" } }], "current-context": "Cluster-B-Context" }
- Cluster A:
- Combine these files.
The file structure remains unchanged. Combine the contents of clusters, users, and contexts as follows:
{ "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "Cluster-A", "cluster": { "server": "https://119.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }, { "name": "Cluster-B", "cluster": { "server": "https://124.xxx.xxx.xxx:5443", "insecure-skip-tls-verify": true } }], "users": [{ "name": "Cluster-A-user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0tLS1CRUdJTiB...." } }, { "name": "Cluster-B-user", "user": { "client-certificate-data": "LS0tLS1CRUdJTxM...", "client-key-data": "LS0rTUideUdJTiB...." } }], "contexts": [{ "name": "Cluster-A-Context", "context": { "cluster": "Cluster-A", "user": "Cluster-A-user" } }, { "name": "Cluster-B-Context", "context": { "cluster": "Cluster-B", "user": "Cluster-B-user" } }], "current-context": "Cluster-A-Context" } - Copy the combined file to the kubectl configuration path.
mkdir -p $HOME/.kube
mv -f kubeconfig.json $HOME/.kube/config
- Check whether the two clusters can be accessed using kubectl.
# kubectl config use-context Cluster-A-Context Switched to context "Cluster-A-Context". # kubectl cluster-info Kubernetes control plane is running at https://119.xxx.xxx.xxx:5443 CoreDNS is running at https://119.xxx.xxx.xxx:5443/api/v1/namespaces/kube-system/services/coredns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. # kubectl config use-context Cluster-B-Context Switched to context "Cluster-B-Context". # kubectl cluster-info Kubernetes control plane is running at https://124.xxx.xxx.xxx:5443 CoreDNS is running at https://124.xxx.xxx.xxx:5443/api/v1/namespaces/kube-system/services/coredns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
These commands are long and inconvenient in case of frequent switchover. To simplify the commands, use aliases. For example:
alias ka='kubectl config use-context Cluster-A-Context;kubectl' alias kb='kubectl config use-context Cluster-B-Context;kubectl'
ka and kb can be custom aliases. When using kubectl, directly enter ka or kb to replace kubectl. You need to switch the context and then run the kubectl commands. For example:
# ka cluster-info Switched to context "Cluster-A-Context". Kubernetes control plane is running at https://119.xxx.xxx.xxx:5443 CoreDNS is running at https://119.xxx.xxx.xxx:5443/api/v1/namespaces/kube-system/services/coredns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
