Temporal is a powerful workflow orchestration platform, but: by default, it doesn’t provide at-rest encryption for your workloads. While this might be acceptable in a self-hosted Temporal OSS environment, it’s a more pressing concern in a multi-tenant setup like Temporal Cloud and In fin-tech or when dealing with customer financial/KYC data, it will come heavily under audit scope if encryption isn’t considered. But either ways whether you use self hosted or cloud you should consider encryption based on what kind of data you use in the payload of your workflows.
To protect ourselves from potential customer data exposure, we can leverage Temporal’s DataConverter codecs. This approach allows us to encrypt payloads before they reach Temporal and decrypt them when our Temporal workers receive them.
Understanding Temporal’s Data Flow
Before we dive into the encryption process, let’s review how data flows through Temporal:
1. Client sends a workflow or activity task to Temporal
2. Temporal stores this task in its persistence layer (usually Cassandra or MySQL/Postgres)
3. Workers poll Temporal for tasks
4. Temporal sends the task to a worker
5. Worker processes the task and sends results back to Temporal
Without encryption, sensitive data is exposed at steps 2 and 4.
Implementing Encryption with DataConverter
Temporal’s DataConverter is the key to solving this security issue. It’s an interface that defines how data is serialized and deserialized when it’s sent to or received from Temporal.
Here’s a basic implementation of a DataConverter that uses AES encryption:
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"io"
commonpb "go.temporal.io/api/common/v1"
"go.temporal.io/sdk/converter"
)
type EncryptionDataConverter struct {
converter.DataConverter
key []byte
}
func NewEncryptionDataConverter(key []byte) *EncryptionDataConverter {
return &EncryptionDataConverter{
DataConverter: converter.GetDefaultDataConverter(),
key: key,
}
}
func (c *EncryptionDataConverter) ToPayload(value interface{}) (*commonpb.Payload, error) {
payload, err := c.DataConverter.ToPayload(value)
if err != nil {
return nil, err
}
block, err := aes.NewCipher(c.key)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return nil, err
}
nonce := make([]byte, gcm.NonceSize())
if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
return nil, err
}
ciphertext := gcm.Seal(nonce, nonce, payload.Data, nil)
payload.Data = []byte(base64.StdEncoding.EncodeToString(ciphertext))
return payload, nil
}
func (c *EncryptionDataConverter) FromPayload(payload *commonpb.Payload, valuePtr interface{}) error {
block, err := aes.NewCipher(c.key)
if err != nil {
return err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return err
}
ciphertext, err := base64.StdEncoding.DecodeString(string(payload.Data))
if err != nil {
return err
}
nonceSize := gcm.NonceSize()
if len(ciphertext) < nonceSize {
return err
}
nonce, ciphertext := ciphertext[:nonceSize], ciphertext[nonceSize:]
plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
if err != nil {
return err
}
payload.Data = plaintext
return c.DataConverter.FromPayload(payload, valuePtr)
}
This DataConverter encrypts data before it’s sent to Temporal and decrypts it when it’s received. The encryption key is stored securely on our side, never shared with Temporal Cloud.
Setting Up the Encrypted DataConverter
To use this DataConverter, we need to set it up in our Temporal client:
key := []byte("32-byte-long-secret-AES-key-here")
dataConverter := NewEncryptionDataConverter(key)
c, err := client.Dial(client.Options{
HostPort: temporalHostPort,
Namespace: namespace,
DataConverter: dataConverter,
})
if err != nil {
log.Fatalln("Unable to create client", err)
}
defer c.Close()The Codec Server: Decrypting Data for UI Access
Note: If you don’t want to configure this, it will be a little manual. You can copy the payload from the UI and run a local script with the encryption key to decrypt the data, but access to the key has to be sensitive.While our data is now secure in Temporal, we’ve introduced a new problem: we can’t view the encrypted data in the Temporal UI. This is where the Codec Server comes in.
The Codec Server is an HTTP server that implements the same encryption/decryption logic as our DataConverter. When configured in the Temporal UI, it allows the UI to decrypt and display our data.
Here’s a basic implementation of a Codec Server:
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/decode", DecodeHandler).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", r))
}
func DecodeHandler(w http.ResponseWriter, r *http.Request) {
var request struct {
EncodedValue string `json:"encodedValue"`
}
err := json.NewDecoder(r.Body).Decode(&request)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Decrypt the data here using the same logic as in the DataConverter
decryptedValue := decrypt(request.EncodedValue)
json.NewEncoder(w).Encode(map[string]string{"decodedValue": decryptedValue})
}
func decrypt(encodedValue string) string {
// Implement decryption logic here
// This should match the decryption in the DataConverter
return "decrypted value"
}To use this Codec Server, we need to configure it in the Temporal UI under “Data Encoder Configuration”.
Security Considerations
there are a few points to keep in mind:
1. Key Management: The encryption key needs to be securely stored and rotated periodically, use services like secrets manager.
2. Performance: Encryption and decryption add some overhead to each operation, load test it as well.
3. Codec Server Security: The Codec Server needs to be secured, as it has access to decrypted data.
4. Search Limitations: Encrypted data can’t be searched directly in Temporal.
Remember, security is an ongoing process. Regularly review and update your encryption strategies to stay ahead of potential threats.
Some reading:
- https://docs.temporal.io/production-readiness/develop?lang=python#securing-your-data
- https://keithtenzer.com/temporal/demystify_the_temporal_dataconverter_and_codec_server/
- https://github.com/temporalio/samples-python/tree/main/encryption
- https://github.com/ktenzer/samples-go/blob/main/encryption/data_converter.go
