Getting Started
Last updated
Was this helpful?
Last updated
Was this helpful?
WARNING
This tutorial assumes your have access to a compatible Intel TDX Server ready to be configured with Dstack. Check the for more information or skip setting everything up from scratch and .
Dstack is a developer friendly and security first SDK to simplify the deployment of arbitrary Docker-based apps into TEE.
Main features:
🔒 Deploy Docker apps securely in TEE in minutes
🛠️ Use familiar tools - just write a docker-compose.yaml
🔑 Safely manage secrets and sensitive data
📡 Expose services via built-in TLS termination
Dstack is community driven. Open sourced and built by and many others from , inspired by (Flashbots & Teleport), and contributed by and many others.
Components in Dstack:
teepod: A service running in bare TDX host to manage CVMs
tproxy: A reverse proxy to forward TLS connections to CVMs
kms: A KMS server to generate keys for CVMs
tappd: A service running in CVM to serve containers' key derivation and attestation requests
meta-dstack: A Yocto meta layer to build CVM guest images
The overall architecture is shown below:
Public IPv4 address assigned to the machine
A domain name you can modify DNS records
At least 16GB RAM, 100GB free disk space.
Now edit the config file. The following configurations values must be changed properly according to your environment:
Run build.sh again to build the artifacts.
Now you can open 3 terminals to start the components:
Run ./kms
Run sudo ./tproxy
Run ./teepod
After the container deployed, it should need some time to start the CVM and the containers. Time would be vary depending on your workload.
Click the [Logs] button to see the logs of the CVM, you can see if the container is finished starting there.
Once the container is running, you can click the [Dashboard] button to see some information of the container. And the logs of the containers can be seen in the [Dashboard] page.
When deploying a new App, you can pass private data via Encrypted Environment Variables. These variables can be referenced in the docker-compose.yaml file as shown below:
The environment variables will be encrypted in the client-side and decrypted in the CVM before being passed to the containers.
Once the app is deployed and listening on an HTTP port, you can access the HTTP service via tproxy's public domain. The ingress mapping rules are:
<id>[s].<base_domain> maps to port 80 or 443 if with s in the CVM.
<id>-<port>[s].<base_domain> maps to port <port> in the CVM.
For example, 3327603e03f5bd1f830812ca4a789277fc31f577-8080.app.kvin.wang maps to port 8080 in the CVM.
Where the <id> can be either the app id or the instance id. If the app id is used, one of the instances will be selected by the load balancer.
If the id-port part ends with s, it means the TLS connection will be passthrough to the app rather than terminating at tproxy.
You can also ssh into the CVM to inspect more information, if your deployment uses the image dstack-x.x.x-dev:
To get a TDX quote within app containers:
Mount /var/run/tappd.sock to the target container in docker-compose.yaml
Execute the quote request command in the container.
Container logs can be obtained from the CVM's dashboard page or by curl:
Replace <appid> and <container name> with actual values. Available parameters:
since=0: Starting Unix timestamp for log retrieval
until=0: Ending Unix timestamp for log retrieval
follow: Enables continuous log streaming
text: Returns human-readable text instead of base64 encoding
timestamps: Adds timestamps to each log line
bare: Returns the raw log lines without json format
The response of the RPC looks like:
The build configuration for TLS Passthrough is:
With this configuration, tproxy listens port 9008 for incoming TLS connections and forwards them to the appropriate Tapp based on SNI, where SNI represents your custom domain and the forwarding destination is determined by your DNS records.
For example, assuming I've deployed an app at 3327603e03f5bd1f830812ca4a789277fc31f577, as shown below:
Now, I want to use my custom domain tapp-nginx.kvin.wang to access the Tapp. I need to set up two DNS records with my DNS provider (Cloudflare in my case):
A or CNAME record to point the domain to the tdx machine:
TXT record to instruct the Tproxy to direct the request to the specified Tapp:
Where
_tapp-address.tapp-nginx.kvin.wang means configuring the tapp destination address of domain tapp-nginx.kvin.wang.
The TXT record value 3327603e03f5bd1f830812ca4a789277fc31f577:8043 means that requests sent to tapp-nginx.kvin.wang will be processed by Tapp 3327603e03f5bd1f830812ca4a789277fc31f577 on port 8043
Got to the teepod webpage, click the [Upgrade] button, select or paste the compose file you want to upgrade to, and click the [Upgrade] button again. Upon successful initiation of the upgrade, you'll see a message prompting you to run the following command in your terminal to authorize the upgrade through KMS:
The app id does not change after the upgrade. Stop and start the app to apply the upgrade.
In the tutorial above, we used a TLS certificate with a private key external to the TEE (Tproxy-CVM here). To establish trust, we need to generate and maintain the certificate's private key within the TEE and provide evidence that all TLS certificates for the domain were originate solely from Tproxy-CVM.
By combining Certificate Transparency Logs and CAA DNS records, we can make best effort to minimize security risks. Here's our approach:
Set CAA records to allow only the account created in Tproxy-CVM to request Certificates.
Launch a program to monitor Certificate Transparency Log and give alarm once any certificate issued to a pubkey that isn’t generated by Tproxy.
To launch Certbot, you need to own a domain hosted on Cloudflare. Obtain an API token with DNS operation permissions from the Cloudflare dashboard. Configure it in the build-config.sh:
Then re-run the ../build.sh:
Then run the certbot in the build/ and you will see the following log:
Where the command did are:
Registered to letsencrypt and got a new account https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853
Auto set CAA records for the domain on cloudflare, you can open the CF dashboard to see the record:
Auto requested a new certificate from Let's Encrypt. Automatically renews the certificate to maintain its validity
Execute tproxy with sudo ./tproxy, then access the web portal to check the Tproxy-CVM managed Let's Encrypt account. The account's private key remains securely sealed within the TEE.
To enhance security, we've limited TLS certificate issuance to Tproxy via CAA records. However, since these records can be modified through Cloudflare's domain management, we need to implement global CA certificate monitoring to maintain security oversight.
Then choose a new range of the CID not conflicting with the CID in use. You can change build/teepod.toml file and restart teepod. This error should disappear. For example, you may find 33000-34000 free to use:
When building the dstack from scratch, you should change the CID configs in build-config.sh instead, because teepod.toml file is generated by build.sh. Its content is derived from build-config.sh.
You may encounter this problem when upgrading from an older version of dstack, because CID was introduced in build-config.sh in later versions. In such case, please follow the docs to add the missing entries in build-config.sh and rebuild dstack.
When running ../build.sh guest, you might encounter this error:
This error occurs because Ubuntu 23.10 and later versions restrict unprivileged user namespaces by default. To fix this, run:
Dstack is proudly built by open source and Pi-rateship contributors:
And many more...
This project cannot be built without standing on the shoulders of giants:
Together, we’re shaping the future of TEE in Web3, paving the way for more secure and developer-accessible confidential computing!
A TDX host machine setup following
Open the teepod webpage (change the port according to your configuration) on your local machine to deploy a docker-compose.yaml file:
You can open tproxy's dashboard at to see the CVM's wireguard ip address, as shown below:
Given the config TPROXY_LISTEN_PORT_PASSTHROUGH=9008, now we can go to and the request will be handled by the service listening on 8043 in Tapp 3327603e03f5bd1f830812ca4a789277fc31f577.
ct_monitor tracks Certificate Transparency logs via , comparing their public key with the ones got from Tproxy RPC. It immediately alerts when detecting unauthorized certificates not issued through Tproxy:
teepod may throw this error when creating a new VM if the is occupied. To solve the problem, first, you should list the occupied CID:
Then try building again. For more information about this restriction, see the .
Phala Network: , ,
Teleport: ,
Flashbots: , , ,
Ithaca:
Fabric:
The inspiration for this work stems from ’s pioneering concept of a .
Special acknowledgment to for building a community around TEE. The TEE Hacker House initiative, organized by and led by , has brought together TEE builders to develop tools for TEE-Web3 integration. This collaborative journey has generated invaluable insights for advancing secure, confidential environments within Web3.
For a full list of the direct contributors to this repo, see on GitHub.