- What's New
- Product Bulletin
- Service Overview
- Billing
- Getting Started
-
User Guide
-
UCS Clusters
- Overview
- Huawei Cloud Clusters
-
On-Premises Clusters
- Overview
- Service Planning for On-Premises Cluster Installation
- Registering an On-Premises Cluster
- Installing an On-Premises Cluster
- Managing an On-Premises Cluster
- Attached Clusters
- Multi-Cloud Clusters
- Single-Cluster Management
- Fleets
-
Cluster Federation
- Overview
- Enabling Cluster Federation
- Using kubectl to Connect to a Federation
- Upgrading a Federation
-
Workloads
- Workload Creation
-
Container Settings
- Setting Basic Container Information
- Setting Container Specifications
- Setting Container Lifecycle Parameters
- Setting Health Check for a Container
- Setting Environment Variables
- Configuring a Workload Upgrade Policy
- Configuring a Scheduling Policy (Affinity/Anti-affinity)
- Configuring Scheduling and Differentiation
- Managing a Workload
- ConfigMaps and Secrets
- Services and Ingresses
- MCI
- MCS
- DNS Policies
- Storage
- Namespaces
- Multi-Cluster Workload Scaling
- Adding Labels and Taints to a Cluster
- RBAC Authorization for Cluster Federations
- Image Repositories
- Permissions
-
Policy Center
- Overview
- Basic Concepts
- Enabling Policy Center
- Creating and Managing Policy Instances
- Example: Using Policy Center for Kubernetes Resource Compliance Governance
-
Policy Definition Library
- Overview
- k8spspvolumetypes
- k8spspallowedusers
- k8spspselinuxv2
- k8spspseccomp
- k8spspreadonlyrootfilesystem
- k8spspprocmount
- k8spspprivilegedcontainer
- k8spsphostnetworkingports
- k8spsphostnamespace
- k8spsphostfilesystem
- k8spspfsgroup
- k8spspforbiddensysctls
- k8spspflexvolumes
- k8spspcapabilities
- k8spspapparmor
- k8spspallowprivilegeescalationcontainer
- k8srequiredprobes
- k8srequiredlabels
- k8srequiredannotations
- k8sreplicalimits
- noupdateserviceaccount
- k8simagedigests
- k8sexternalips
- k8sdisallowedtags
- k8sdisallowanonymous
- k8srequiredresources
- k8scontainerratios
- k8scontainerrequests
- k8scontainerlimits
- k8sblockwildcardingress
- k8sblocknodeport
- k8sblockloadbalancer
- k8sblockendpointeditdefaultrole
- k8spspautomountserviceaccounttokenpod
- k8sallowedrepos
- Configuration Management
- Traffic Distribution
- Observability
- Container Migration
- Pipeline
- Error Codes
-
UCS Clusters
- Best Practices
-
API Reference
- Before You Start
- Calling APIs
-
API
- UCS Cluster
-
Fleet
- Adding a Cluster to a Fleet
- Removing a Cluster from a Fleet
- Registering a Fleet
- Deleting a Fleet
- Querying a Fleet
- Adding Clusters to a Fleet
- Updating Fleet Description
- Updating Permission Policies Associated with a Fleet
- Updating the Zone Associated with the Federation of a Fleet
- Obtaining the Fleet List
- Enabling Fleet Federation
- Disabling Cluster Federation
- Querying Federation Enabling Progress
- Creating a Federation Connection and Downloading kubeconfig
- Creating a Federation Connection
- Downloading Federation kubeconfig
- Permissions Management
- Using the Karmada API
- Appendix
-
FAQs
- About UCS
-
Billing
- How Is UCS Billed?
- What Status of a Cluster Will Incur UCS Charges?
- Why Am I Still Being Billed After I Purchase a Resource Package?
- How Do I Change the Billing Mode of a Cluster from Pay-per-Use to Yearly/Monthly?
- What Types of Invoices Are There?
- Can I Unsubscribe from or Modify a Resource Package?
-
Permissions
- How Do I Configure Access Permissions for Each Function of the UCS Console?
- What Can I Do If an IAM User Cannot Obtain Cluster or Fleet Information After Logging In to UCS?
- How Do I Restore ucs_admin_trust I Deleted or Modified?
- What Can I Do If I Cannot Associate the Permission Policy with a Fleet or Cluster?
- How Do I Clear RBAC Resources After a Cluster Is Unregistered?
- Policy Center
-
Fleets
- What Can I Do If Cluster Federation Verification Fails to Be Enabled for a Fleet?
- What Can I Do If an Abnormal, Federated Cluster Fails to Be Removed from the Fleet?
- What Can I Do If an Nginx Ingress Is in the Unready State After Being Deployed?
- What Can I Do If "Error from server (Forbidden)" Is Displayed When I Run the kubectl Command?
- Huawei Cloud Clusters
- Attached Clusters
-
On-Premises Clusters
- What Can I Do If an On-Premises Cluster Fails to Be Connected?
- How Do I Manually Clear Nodes of an On-Premises Cluster?
- How Do I Downgrade a cgroup?
- What Can I Do If the VM SSH Connection Times Out?
- How Do I Expand the Disk Capacity of the CIA Add-on in an On-Premises Cluster?
- What Can I Do If the Cluster Console Is Unavailable After the Master Node Is Shut Down?
- What Can I Do If a Node Is Not Ready After Its Scale-Out?
- How Do I Update the CA/TLS Certificate of an On-Premises Cluster?
- What Can I Do If an On-Premises Cluster Fails to Be Installed?
- Multi-Cloud Clusters
-
Cluster Federation
- What Can I Do If the Pre-upgrade Check of the Cluster Federation Fails?
- What Can I Do If a Cluster Fails to Be Added to a Federation?
- What Can I Do If Status Verification Fails When Clusters Are Added to a Federation?
- What Can I Do If an HPA Created on the Cluster Federation Management Plane Fails to Be Distributed to Member Clusters?
- What Can I Do If an MCI Object Fails to Be Created?
- What Can I Do If I Fail to Access a Service Through MCI?
- What Can I Do If an MCS Object Fails to Be Created?
- What Can I Do If an MCS or MCI Instance Fails to Be Deleted?
- Traffic Distribution
- Container Intelligent Analysis
- General Reference
Copied.
What Can I Do If I Cannot Associate the Permission Policy with a Fleet or Cluster?
Symptom
When associating the permission policy with a fleet or a cluster not in the fleet, the association may fail due to cluster connection exceptions. In this case, detailed exception events will be displayed on the Set Permissions page of the fleet or cluster. Check and rectify the fault in the cluster, and then click Retry to associate the permission policy again.
Troubleshooting
If an exception occurs when the permission policy is being associated with a fleet or cluster, locate the fault based on the error message, as shown in Table 1.
Error Message |
Description |
Check Item |
---|---|---|
ClusterRole failed reason:Get \"https://kubernetes.default.svc.cluster.local/apis/rbac.authorization.k8s.io/v1/clusterroles/XXXXXXX?timeout=30s\": Precondition Required" Or Get ClusterRole failed reason:an error on the server (\"unknown\") has prevented the request from succeeding (get clusterroles.rbac.authorization.k8s.io |
The cluster has not been connected, proxy-agent in the connected cluster is abnormal, or the network is abnormal. |
|
Unauthorized |
Rectify the fault based on the returned status code. For example, status code 401 indicates that the user does not have the access permission. A possible cause is that the cluster authentication information has expired. |
|
Get cluster namesapce[x] failed. Or Reason:namespace "x" not found. |
There is no corresponding namespace in the cluster. |
Create a namespace in the cluster and try again. Example: kubectl create namespace ns_name If the namespace is not required, ignore this exception event. |
Check Item 1: proxy-agent
After the cluster is unregistered from UCS, the authentication information contained in the original proxy-agent configuration file becomes invalid. You need to delete the proxy-agent pods deployed in the cluster. To connect the cluster to UCS again, download the proxy-agent configuration file from the UCS console again and use it for re-deployment.
- Log in to the master node of the destination cluster.
- Check the deployment of the cluster agent.
kubectl -n kube-system get pod | grep proxy-agent
Expected output for successful deployment:
proxy-agent-*** 1/1 Running 0 9s
If proxy-agent is not in the Running state, run the kubectl -n kube-system describe pod proxy-agent-*** command to view the pod alarms. For details, see What Can I Do If proxy-agent Fails to Be Deployed?.
NOTE:
By default, proxy-agent is deployed with two pods, and can provide services as long as one pod is running properly. However, one pod cannot ensure high availability.
- Print the pod logs of proxy-agent and check whether the agent program can connect to UCS.
kubectl -n kube-system logs proxy-agent-*** | grep "Start serving"
If no "Start serving" log is printed but the proxy-agent pods are working, check other check items.
Check Item 2: Network Connection Between the Cluster and UCS
Public network access
- Check whether a public IP is bound to the cluster or a public NAT gateway is configured.
- Check whether the outbound traffic of the cluster security group is allowed. To perform access control on the outbound traffic, contact technical support to obtain the destination IP and port number.
- After rectifying network faults, delete the existing proxy-agent pods to rebuild pods. Check whether the logs of the new pods contain "Start serving".
kubectl -n kube-system logs proxy-agent-*** | grep "Start serving"
- If desired logs are printed, refresh the UCS console page and check whether the cluster is properly connected.
Private network access
- Check whether the outbound traffic of the cluster security group is allowed. To perform access control on the outbound traffic, contact technical support to obtain the destination IP and port number.
- Rectify the network connection faults between the cluster and UCS or IDC.
Refer to the following guides according to your network connection type:
- Direct Connect: Troubleshooting
- Virtual Private Network (VPN): Troubleshooting
- Rectify the VPC endpoint fault. The VPC endpoint status must be Accepted. If the VPC endpoint is deleted by mistake, create one again. For details, see How Do I Restore a Deleted VPC Endpoint for a Cluster Connected Over a Private Network?.
Figure 1 Checking the VPC endpoint status
- After rectifying network faults, delete the existing proxy-agent pods to rebuild pods. Check whether the logs of the new pods contain "Start serving".
kubectl -n kube-system logs proxy-agent-*** | grep "Start serving"
- If desired logs are printed, refresh the UCS console page and check whether the cluster is properly connected.
Check Item 3: Cluster Authentication Information Changes
If the error message "cluster responded with non-successful status: [401][Unauthorized]" is displayed, the IAM network connection may be faulty, according to the /var/paas/sys/log/kubernetes/auth-server.log of the three master nodes in the cluster. Ensure that the IAM domain name resolution and the IAM service connectivity are normal.
The common issue logs are as follows:
- Failed to authenticate token: *******: dial tcp: lookup iam.myhuaweicloud.com on *.*.*.*:53: no such host
This log indicates that the node is not capable of resolving iam.myhuaweicloud.com. Configure the corresponding domain name resolution by referring to Preparing for Installation.
- Failed to authenticate token: Get *******: dial tcp *.*.*.*:443: i/o timeout
This log indicates that the node's access to IAM times out. Ensure that the node can communicate with IAM properly.
- currently only supports Agency token
This log indicates that the request is not initiated by UCS. Currently, on-premises clusters can only be connected to UCS using IAM tokens.
- IAM assumed user has no authorization/iam assumed user should allowed by TEAdmin
This log indicates that the connection between UCS and the cluster is abnormal. Contact Huawei technical support for troubleshooting.
- Failed to authenticate token: token expired, please acquire a new token
This log indicates that the token has expired. Run the date command to check whether the time difference is too large. If yes, synchronize the time and check whether the cluster is working. If the fault persists for a long time, you may need to reinstall the cluster. In this case, contact Huawei technical support.
After the preceding problem is resolved, run the crictl ps | grep auth | awk '{print $1}' | xargs crictl stop command to restart the auth-server container.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot