Deploying a model with Ray Serve

You deploy a model with a RayService resource. The serveConfigV2 field holds your Serve applications and their deployments, and KubeRay creates the Ray cluster that runs them. HyperPod does not modify KubeRay, so a RayService you already run continues to work.

A RayService manifest

The following manifest runs one Serve application with a single GPU-backed deployment. Replace import_path with the module and application object in your working directory.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: my-service
  namespace: my-namespace
spec:
  serveConfigV2: |
    applications:
      - name: my-app
        import_path: my_module:app
        route_prefix: /
        deployments:
          - name: Model
            num_replicas: 2
            ray_actor_options:
              num_gpus: 1
  rayClusterConfig:
    rayVersion: "2.56.1"
    headGroupSpec:
      rayStartParams:
        dashboard-host: "0.0.0.0"
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.56.1
              ports:
                - { containerPort: 8265, name: dashboard }
                - { containerPort: 8000, name: serve }
    workerGroupSpecs:
      - groupName: gpu-workers
        replicas: 1
        rayStartParams: {}
        template:
          spec:
            nodeSelector:
              node.kubernetes.io/instance-type: ml.g5.xlarge
            containers:
              - name: ray-worker
                image: rayproject/ray:2.56.1-gpu
                resources:
                  limits: { nvidia.com/gpu: "1" }
                  requests: { cpu: "4", memory: "16Gi", nvidia.com/gpu: "1" }

Apply and confirm the service is ready:

kubectl apply -f my-service.yaml -n my-namespace
kubectl get rayservice my-service -n my-namespace

Reaching the endpoint

Ray Serve listens on port 8000 on the head pod. From inside the cluster, send requests to the service that KubeRay creates for the RayService. For quick testing, you can use kubectl port-forward:

kubectl port-forward svc/ray-service-head-svc 8000:8000
curl http://localhost:8000/

For the Serve deployment API and request handling, see Ray Serve API in the Ray documentation.