🔐GPU TEE

The implementation for running LLMs in GPU TEE is available in the private-ml-sdk GitHub repository. This project is built by Phala Network and was made possible through a grant from NEARAI. The SDK provides the necessary tools and infrastructure to deploy and run LLMs securely within GPU TEE.

Introduction

A TEE as a general hardware-based confidential computation infrastructure can offer a practical solution compared to other cryptographic methods like ZK and FHE in AI Inference:

• The computational overhead is significantly lower, with nearly native speed of execution

• Verification using TEEs is also more economical compared to ZKPs. An ECDSA signature suffices for on-chain verification, reducing the complexity and cost of ensuring computation integrity.

• NVIDIA's series of GPUs such as H100 and H200 natively support TEEs, providing hardware-accelerated secure environments for AI workloads. This native support ensures seamless integration and optimized performance for AI fine-tuning and inference.

Our TEE-based solution can provide following features for AI Inference:

1. Tamper-Proof Data: Ensuring that user request/response data cannot be altered by a middleman is fundamental. This necessitates secure communication channels and robust encryption mechanisms.

2. Secure Execution Environment: Both hardware and software must be protected against attacks. This involves leveraging TEE that provides isolated environments for secure computation.

3. Open Source and Reproducible Builds: The entire software stack, from the operating system to the application code must be reproducible. This allows auditors to verify the integrity of the system.

4. Verifiable Execution Results: The results of AI computations must be verifiable, ensuring that the outputs are trustworthy and have not been tampered with.

Study Cases

Host DeepSeek-R1 in GPU TEE

What's Next?

1. Host LLM in GPU TEE.

2. GPU TEE Inference API

3. Benchmark of running LLM in TEE.