對使用 PersistentVolumeClaims (PVC) 的任務進行故障診斷 - Amazon EMR

本文為英文版的機器翻譯版本,如內容有任何歧義或不一致之處,概以英文版為準。

對使用 PersistentVolumeClaims (PVC) 的任務進行故障診斷

如果您需要為任務建立、列出或刪除 PersistentVolumeClaims (PVC),但未將PVC許可新增至預設 Kubernetes 角色 emr-containers,則提交任務時,任務會失敗。如果沒有這些許可,emr-containers 角色就無法為 Spark 驅動程式或 Spark 用戶端建立必要的角色。如錯誤訊息所建議,將許可新增至 Spark 驅動程式或用戶端角色是不夠的。emr-containers 主要角色也必須包含必要的許可。本章節說明如何將必要的許可新增至 emr-containers 主要角色。

驗證

若要驗證您的 emr-containers 角色是否具有必要的許可,請使用您自己的值設定NAMESPACE變數,然後執行下列命令:

export NAMESPACE=YOUR_VALUE kubectl describe role emr-containers -n ${NAMESPACE}

此外,若要驗證 Spark 和用戶端角色是否具有必要的許可,請執行下列命令:

kubectl describe role emr-containers-role-spark-driver -n ${NAMESPACE} kubectl describe role emr-containers-role-spark-client -n ${NAMESPACE}

如果許可不存在,請繼續執行修補程式,如下所示。

修補程式

  1. 如果沒有許可的作業目前正在執行中,則請停止這些作業。

  2. 建立名為 RBAC_Patch.py 的檔案,如下所示:

    import os import subprocess as sp import tempfile as temp import json import argparse import uuid def delete_if_exists(dictionary: dict, key: str): if dictionary.get(key, None) is not None: del dictionary[key] def doTerminalCmd(cmd): with temp.TemporaryFile() as f: process = sp.Popen(cmd, stdout=f, stderr=f) process.wait() f.seek(0) msg = f.read().decode() return msg def patchRole(roleName, namespace, extraRules, skipConfirmation=False): cmd = f"kubectl get role {roleName} -n {namespace} --output json".split(" ") msg = doTerminalCmd(cmd) if "(NotFound)" in msg and "Error" in msg: print(msg) return False role = json.loads(msg) rules = role["rules"] rulesToAssign = extraRules[::] passedRules = [] for rule in rules: apiGroups = set(rule["apiGroups"]) resources = set(rule["resources"]) verbs = set(rule["verbs"]) for extraRule in extraRules: passes = 0 apiGroupsExtra = set(extraRule["apiGroups"]) resourcesExtra = set(extraRule["resources"]) verbsExtra = set(extraRule["verbs"]) passes += len(apiGroupsExtra.intersection(apiGroups)) >= len(apiGroupsExtra) passes += len(resourcesExtra.intersection(resources)) >= len(resourcesExtra) passes += len(verbsExtra.intersection(verbs)) >= len(verbsExtra) if passes >= 3: if extraRule not in passedRules: passedRules.append(extraRule) if extraRule in rulesToAssign: rulesToAssign.remove(extraRule) break prompt_text = "Apply Changes?" if len(rulesToAssign) == 0: print(f"The role {roleName} seems to already have the necessary permissions!") prompt_text = "Proceed anyways?" for ruleToAssign in rulesToAssign: role["rules"].append(ruleToAssign) delete_if_exists(role, "creationTimestamp") delete_if_exists(role, "resourceVersion") delete_if_exists(role, "uid") new_role = json.dumps(role, indent=3) uid = uuid.uuid4() filename = f"Role-{roleName}-New_Permissions-{uid}-TemporaryFile.json" try: with open(filename, "w+") as f: f.write(new_role) f.flush() prompt = "y" if not skipConfirmation: prompt = input( doTerminalCmd(f"kubectl diff -f {filename}".split(" ")) + f"\n{prompt_text} y/n: " ).lower().strip() while prompt != "y" and prompt != "n": prompt = input("Please make a valid selection. y/n: ").lower().strip() if prompt == "y": print(doTerminalCmd(f"kubectl apply -f {filename}".split(" "))) except Exception as e: print(e) os.remove(f"./{filename}") if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument("-n", "--namespace", help="Namespace of the Role. By default its the VirtualCluster's namespace", required=True, dest="namespace" ) parser.add_argument("-p", "--no-prompt", help="Applies the patches without asking first", dest="no_prompt", default=False, action="store_true" ) args = parser.parse_args() emrRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete", "patch"] } ] driverRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete", "patch"] }, { "apiGroups": [""], "resources": ["services"], "verbs": ["get", "list", "describe", "create", "delete", "watch"] } ] clientRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete", "patch"] } ] patchRole("emr-containers", args.namespace, emrRoleRules, args.no_prompt) patchRole("emr-containers-role-spark-driver", args.namespace, driverRoleRules, args.no_prompt) patchRole("emr-containers-role-spark-client", args.namespace, clientRoleRules, args.no_prompt)
  3. 執行 Python 指令碼:

    python3 RBAC_Patch.py -n ${NAMESPACE}
  4. 新許可與舊許可之間的 kubectl 差異會出現。按 y 來修補角色。

  5. 驗證具有其他許可的三個角色,如下所示:

    kubectl describe role -n ${NAMESPACE}
  6. 執行 Python 指令碼:

    python3 RBAC_Patch.py -n ${NAMESPACE}
  7. 執行命令後,它將顯示新許可和舊許可之間的 kubectl 差異。按 y 來修補角色。

  8. 驗證具有其他許可的三個角色:

    kubectl describe role -n ${NAMESPACE}
  9. 再次提交作業。

手動修補

如果您的應用程式需要的許可適用於PVC規則以外的其他項目,您可以視需要手動新增 Amazon EMR虛擬叢集的 Kubernetes 許可。

注意

emr-containers 角色是主要角色。這意味著它必須提供所有必要的許可,然後才能變更基礎驅動程式或用戶端角色。

  1. 透過執行以下命令將當前許可下載到 yaml 檔案中:

    kubectl get role -n ${NAMESPACE} emr-containers -o yaml >> emr-containers-role-patch.yaml kubectl get role -n ${NAMESPACE} emr-containers-role-spark-driver -o yaml >> driver-role-patch.yaml kubectl get role -n ${NAMESPACE} emr-containers-role-spark-client -o yaml >> client-role-patch.yaml
  2. 根據應用程式所需的許可,編輯每個檔案並新增其他規則,例如:

    • emr-containers-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete - patch
    • driver-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete - patch - apiGroups: - "" resources: - services verbs: - get - list - describe - create - delete - watch
    • client-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete - patch
  3. 移除下列屬性及其值。這對於套用更新是必要的。

    • creationTimestamp

    • resourceVersion

    • uid

  4. 最後,執行修補程式:

    kubectl apply -f emr-containers-role-patch.yaml kubectl apply -f driver-role-patch.yaml kubectl apply -f client-role-patch.yaml