How to migrate data to a new Redis service instance

In this tutorial you will learn how to migrate your data from existing Redis service instance to a new created.

Requirements

redis-cli: (Command line client for Redis)
gunzip: (Command-line utility used to decompress files)
openssl: (Toolkit for the SSL and TLS protocols, offering secure network communication and cryptography)
cf CLI v. 7: (Command Line Interface for Cloud Foundry, simplifying app and space management)

Before Migration

Install tools

To ensure a smooth migration, please proceed with installing all the tools listed in the requirements.

For installation, we recommend consulting the official documentation of each tool, which provides comprehensive guides and instructions. Depending on your operating system, you may also utilize package managers such as apt for Linux, homebrew for macOS, or choco (Chocolatey) for Windows to facilitate the installation process. These tools are essential for managing software packages and can simplify the setup.

After installation, it's important to check that each tool is operational, typically by invoking a version command or executing a basic operation specific to the tool.

Isolate source database

It is highly recommended to isolate the Redis service instance before starting the migration. This avoids potential data inconsistencies that could be caused if the Redis service instance is receiving new data while processing the migration at the same time.

Overall the goal is to avoid client connections. That way, we ensure that we are not writing data into the Redis service instance during the migration process. This can be achieved by unbinding/disconnecting all applications from your Redis service instance.

Create backup of the source database

We recommend saving your service instance's RDB file as a failsafe in the event of unexpected data loss. This file must be stored securely until the consistency of the database has been confirmed post-migration.

Open the Service Dashboard to manually trigger a backup of your source service instance. Ensure you set an encryption password and store it safely.
Download the latest backup from the Redis Service Dashboard and store it securely.

For more details, please refer to the following documentation:

Creation of RDB dump files for Redis v7 Data Services using `redis-cli` is not supported.

Create target Redis database

Create a new and empty Redis service instance that will be the target for the data migration from an existing Redis source service instance.

Do not insert any data inside of the machine before the migration and checking are done.
Create a service instance with the same capabilities or greater.

Create credentials

Create credentials using STACKIT Portal or via cf CLI (for Cloud Foundry Runtime Redis Data Services ordered in the Cloud Foundry Marketplace) for source and target Redis databases.

Database connection strings

Database connection strings can be directly obtained in the STACKIT Portal within Data Service Dashboard for exposed Data Services.

Please use corresponding cf CLI commands for Cloud Foundry Runtime Data Services to retrieve service keys of your instances.

# List your organizations
cf orgs

# Target to specific organization
cf target -o <YOUR_ORGANIZATION_NAME> -s <YOUR_SPACE_NAME>

# List your services of targeted organization
cf services

# Create service key for your Data Service
cf create-service-key <YOUR_SERVICE_INSTANCE_NAME> <YOUR_KEY_NAME>

# Show created service key
cf service-key <YOUR_SERVICE_INSTANCE_NAME> <YOUR_KEY_NAME>

# List all available service keys for a specific instance
cf service-keys <YOUR_SERVICE_INSTANCE_NAME>

Re-format obtained service keys as database connection strings:

Redis v6 db connection string "rediss://<PASSWORD>@<FQDN_HOST>:<PORT>"

Redis v7 db connection string "rediss://<USER><PASSWORD>@<FQDN_HOST>:<PORT>"

Please always use the value of "\host": for the source and target databases!

<FQDN_HOST>: Fully qualified domain name

Retrieve backend host for migration

This step is required only when migrating to an exposed STACKIT Redis Data Service (not for Cloud Foundry Runtime Redis Data Services).

To perform the migration, you need to retrieve the backend hostname for the target Redis instance. The MIGRATE command requires a direct TCP connection from the source to the target database. Therefore, using exposed frontend hosts like red297348-master-3.data.eu01.onstackit.cloud as the target will cause the migration to fail. Instead, use the backend host for the target database.

It is not necessary to order Cloud Foundry Runtime Service to execute this step!

Login with cf CLI: Please see Interacting with the Cloud Foundry.
Select Your cf Organization:
- Use the provided menu or the following commands:
  - cf orgs: List all available organizations of all STACKIT Projects you have access to.
  - cf target -o <ORGANIZATION_NAME>: Switch to your organization.
    - Typical org has the following naming schema: stackit_portal_prod_<portal_project_name>_id.
    - Example: If your STACKIT Project is called My Cloud, the organization name would be like stackit_portal_prod_my_cloud_4eFgBq5P.
List Your Data Service Instances:
- Use cf services to identify your Redis target database by name (first column of the output). The correct backend instance contains -internal as a name postfix.
Create a Service Key:
- Use the command cf create-service-key <TARGET_INSTANCE_NAME> <YOUR_KEY_NAME>.
  - Example: cf create-service-key my-redis-target-db-internal mykey.
Show the Service Key:
- Use cf service-key <TARGET_INSTANCE_NAME> <YOUR_KEY_NAME> and store the whole JSON output. This will be required in a later step.

Get CA certificates

Depending on your STACKIT project type (Public or Internal) and the Redis Data Service configuration (exposed or within the Cloud Foundry Runtime), you may need different CA certificates to establish an encrypted database connection.

STACKIT Project Type	Exposed Redis Data Services (Frontend Hosts)	Cloud Foundry Runtime Redis Data Services (Backend Hosts)
Public (Web)	CA certificates (Root CA and Intermediate CAs) are optional and required only if your environment does not trust the Public SwissSign Authority by default.	CA certificate is mandatory and can be retrieved from the service key.
Internal (Schwarz)	CA certificates (Root CA and Intermediate CAs) are not required and trusted by default.	CA certificate is mandatory and can be retrieved from the service key.

Exposed STACKIT Redis Data Services in Public Projects

Please visit the SwissSign support page for an overview of all certificates: SwissSign certificate overview

Required certificate to build up the chain are:

SwissSign RSA TLS OV ICA 2022 - 1

Key-ID: 7C:6F:0A:6F:13:0F:D9:8C:24:6F:26:34:F3:5C:6B:43:6D:B7:23:B6
Serial: 6A:EC:7C:44:41:7B:9B:44:1F:B9:76:34:CB:C6:A7:80:B0:04:1E:01
Fingerprint (SHA2): 33:2F:9E:AE:36:50:C7:74:54:AF:14:FE:1A:62:1A:24:98:FD:12:87:73:66:28:90:A0:D1:28:35:B3:43:6E:23
Download Cert PEM

SwissSign RSA TLS Root CA 2022 - 1 (Cross Signed By Gold 2)

Key-ID: 6F:8E:62:8B:93:43:B0:E1:40:F6:A7:C3:FD:F1:0F:B8:0F:15:38:A5
Serial: 68:6f:43:b4:dc:40:4c:06:7e:23:0e:3f:af:c3:2b
Fingerprint (SHA2): 28:8B:4A:9F:60:5B:09:B9:99:B2:15:85:08:25:C8:1F:9B:53:7D:BA:F2:36:64:AC:A9:8B:F6:BA:98:ED:C3:79
Download Cert PEM

Gold G2

Key-ID: 5B:25:7B:96:A4:65:51:7E:B8:39:F3:C0:78:66:5E:E8:3A:E7:F0:EE
Serial: BB:40:1C:43:F5:5E:4F:B0
Fingerprint (SHA2): 62:DD:0B:E9:B9:F5:0A:16:3E:A0:F8:E7:5C:05:3B:1E:CA:57:EA:55:C8:68:8F:64:7C:68:81:F2:C8:35:7B:95
Download Cert PEM

Migration process

Main strategy: Migration by Redis MIGRATE command

The main strategy to migrate the whole dataset from one Redis service instance to another is based on the Redis MIGRATE feature.

This feature comes in the form of a simple command that can be run through redis-cli. Using the CLI, we shall connect to the existing instance, and execute the MIGRATE command. The command accepts the hostname and credentials for the new instance, which implies that there will be a direct connection between the two instances during the migration.

Based on internal and isolated tests, migrating 1GB takes about 50 seconds on average. However, it will depend on each environment and its particularities. Be aware that it might take some time, so it is necessary to plan wisely.

Use the redis-cli to retrieve the data base keys and migrate between the Redis service instances. Replace all placeholder values in the command below and execute it afterwards. These commands may change depending on the version of Redis, make sure to run the appropriate command for your case.

If you are using a logical database, and you have more than one, this command must be executed as many times as the amount of the logical databases. It is necessary to change the `-n <DB_NUM>` value accordingly.

If the origin Redis Data Service instance contains keys with names containing special characters this migration process will not work. For example, if one key contains a breakline character this can be a problem.

Alternative strategy: Migration by script

If the MIGRATE feature is not working or failing, you can use a Python script called copy_redis.py to copy and verify all keys from the source to the target database.

Please follow the instructions provided in the README file for using the copy_redis.py script.

Migration

Command to migrate Redis Keys

# Redis v6 target database
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*' | \
xargs redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> \
MIGRATE <FQDN_TARGET_BACKEND_HOST> <TARGET_BACKEND_PORT> "" <DB_NUM> <TIMEOUT> \
COPY AUTH <BACKEND_TARGET_PASSWORD> KEYS > migration-output-checker.txt

# Redis v7 target database
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*' | \
xargs redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> \
MIGRATE <FQDN_TARGET_BACKEND_HOST> <TARGET_BACKEND_PORT> "" <DB_NUM> <TIMEOUT> \
COPY AUTH2 <BACKEND_TARGET_USER> <BACKEND_TARGET_PASSWORD> KEYS > migration-output-checker.txt

Explanation

Source Redis CLI Command: redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> --tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*'
- -u "<SOURCE_DB_CONNECTION_STRING>": The connection string for the source Redis database.
- -n <DB_NUM>: Logical database number to use.
- --tls: Enable TLS encryption.
- --sni <FQDN_SOURCE_FRONTEND_HOST>: The fully qualified domain name of the source frontend host.
- --cacert </PATH/TO/CA_CERT.PEM>: Path to the CA certificate
- --raw KEYS '*': Retrieve all keys from the source database
Piping Keys to xargs: | xargs redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> --tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> MIGRATE \
- - xargs: Pass the keys retrieved from the source database to the next redis-cli command.
Target Redis CLI Command: redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> --tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> MIGRATE \ <FQDN_TARGET_BACKEND_HOST> <TARGET_BACKEND_PORT> "" <DB_NUM> <TIMEOUT> \ COPY AUTH <BACKEND_TARGET_PASSWORD> KEYS > migration-output-checker.txt
- MIGRATE <FQDN_TARGET_BACKEND_HOST> <TARGET_BACKEND_PORT> "" <DB_NUM> <TIMEOUT>: Migrate keys to the target Redis database.
- <FQDN_TARGET_BACKEND_HOST>: The fully qualified domain name of the target backend host.
- <TARGET_BACKEND_PORT>: The port of the target backend.
- "": The key to migrate (provided by xargs).
- <DB_NUM>: The database number to use.
- <TIMEOUT>: The timeout for the migration.
- COPY: Copy the key instead of moving it.
- AUTH <BACKEND_TARGET_PASSWORD>: The password for the Redis v6 target backend
- AUTH2 <BACKEND_TARGET_USER> <BACKEND_TARGET_PASSWORD>: The username and password for the Redis v7 target backend.
- KEYS > migration-output-checker.txt: Output the result to a file for checking.

Example

# Redis v6 target database
redis-cli -u "rediss://a9******@red140220-master-1.data.eu01.onstackit.cloud:49620" -n 0 \
--tls --sni red140220-master-1.data.eu01.onstackit.cloud --cacert ca_certs.pem --raw KEYS '*' | \
xargs redis-cli -u "rediss://a9******@red140220-master-1.data.eu01.onstackit.cloud:49620" -n 0 \
--tls --sni red140220-master-1.data.eu01.onstackit.cloud --cacert ca_certs.pem \
MIGRATE red297348-master.service.dc1.a9ssvc 6379 "" 0 30 \
COPY AUTH a9****** KEYS > migration-output-checker.txt

# Redis v7 target database
redis-cli -u "rediss://a9******@red140220-master-1.data.eu01.onstackit.cloud:49620" -n 0 \
--tls --sni red140220-master-1.data.eu01.onstackit.cloud --cacert ca_certs.pem --raw KEYS '*' | \
xargs redis-cli -u "rediss://a9******@red140220-master-1.data.eu01.onstackit.cloud:49620" -n 0 \
--tls --sni red140220-master-1.data.eu01.onstackit.cloud --cacert ca_certs.pem \
MIGRATE red297348-master.service.dc1.a9ssvc 6379 "" 0 30 \
COPY AUTH2 a9****** k85***** KEYS > migration-output-checker.txt

Important considerations for Redis migration

When migrating data using the MIGRATE command, it is crucial to understand the difference between frontend (exposed) hosts and backend hosts (Cloud Foundry Runtime Data Services).

Frontend (exposed) hosts vs. backend hosts

Frontend (Exposed) Hosts: These are the publicly accessible endpoints provided for external access to the Redis Data Service. They are typically used for connecting to the Redis Data Service from outside the Cloud Foundry environment. Examples include:
- *.data.eu01.onstackit.cloud
- *.data.eu01.onstackit.schwarz
Backend Hosts: These are the internal endpoints used within the Cloud Foundry environment. They are not exposed to the public and are used for internal communication between services. Examples include:
- *.service.dc1.a9ssvc

Understanding the Redis MIGRATE command

The MIGRATE command requires a direct connection between the source and target databases. This means that the target database must be accessible from the source database's network.

Exposed Redis Data Services: If you are migrating data to an exposed Redis service, using the frontend host as the target will cause the migration to fail. This is because the MIGRATE command cannot establish a direct connection to the frontend host. Please use backend host instead.
Cloud Foundry Runtime Redis Data Services: Use the backend hosts provided by the service keys. These hosts are accessible within the Cloud Foundry environment, allowing the MIGRATE command to establish a direct connection between databases.

Steps for successful migration

Identify the Correct Hosts:
- For exposed Redis Data Services as a target, please ensure you have the backend host information.
- For Cloud Foundry Runtime Redis services, use the backend hosts provided by the service.
Use the Backend Host for Migration:
- When performing the migration, specify the backend host as the target in the MIGRATE command.
Example Command:

redis-cli -u "<SOURCE_CONNECTION_STRING_WITH_FQDN_FRONTEND_HOST>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*' | \
xargs redis-cli -u "<SOURCE_CONNECTION_STRING_WITH_FQDN_FRONTEND_HOST>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> --cacert </PATH/TO/CA_CERT.PEM> \
MIGRATE <FQDN_TARGET_BACKEND_HOST> <TARGET_BACKEND_PORT> "" <DB_NUM> <TIMEOUT> \
COPY AUTH <PASSWORD> KEYS > migration-output-checker.txt

Authentication changes in Redis v7

When migrating data to Redis 7 using the MIGRATE command, please note that Redis v7 requires a username in addition to a password for authentication. This is a change from Redis v6, which only required a password.

Additionally, the AUTH command has been replaced with AUTH2 to accommodate the new authentication requirements. Ensure that you use AUTH2 with both the username and password when performing migrations to Redis 7.

Please adapt your migration command accordingly:

# Redix v6
.. COPY AUTH <PASSWORD> KEYS > migration-output-checker.txt

# Redis v7
.. COPY AUTH2 <USERNAME> <PASSWORD> KEYS > migration-output-checker.txt

Migration of Cloud Foundry Runtime Redis Data Services

In order to get a connection to Cloud Foundry Runtime Redis Data Service database a SSH forward tunnel is required. Local port forwarding allows you to forward a port from your local machine to a remote server. This is useful when you want to access a service running on a remote server from your local machine.

To connect to the Redis Data Service database in the Cloud Foundry Runtime (ordered via Cloud Foundry Marketplace or API), an SSH forward tunnel is required. This technique, known as local port forwarding, allows you to securely forward a port from your local machine to a remote server. This is particularly useful when you need to access a service running on a remote server from your local environment.

What is local port forwarding?

Local port forwarding is a method used to redirect network traffic from a local port on your machine to a specified port on a remote server. This is achieved through an SSH tunnel, which encrypts the data being transmitted, ensuring secure communication between your local machine and the remote service.

Why Use local port forwarding?

Local port forwarding is beneficial in scenarios where direct access to a remote service is restricted due to network security policies, firewalls, or NAT (Network Address Translation). By creating an SSH tunnel, you can bypass these restrictions and securely access the remote service as if it were running locally on your machine.

Steps to establish an SSH Tunnel

Prerequisites:
- Ensure you have SSH access to the Cloud Foundry application instance.
- Verify that the SSH feature is enabled for your Cloud Foundry application.
- Ensure that your application is bound to the Redis Data Service instance.
Bind the Application to the Redis Data Service: Use the following command to bind your Cloud Foundry application to the Redis Data Service instance:

cf bind-service <YOUR_CF_APP_NAME> <YOUR_REDIS_SERVICE_INSTANCE>

Replace the placeholders with the appropriate values:

<YOUR_CF_APP_NAME>: The name of your Cloud Foundry application.
<YOUR_REDIS_SERVICE_INSTANCE>: The name of your Redis Data Service instance.

Restage the Application: After binding the service, restage your application to apply the changes:

cf restage <YOUR_CF_APP_NAME>
Create the SSH Tunnel: Use the following command to create an SSH tunnel from your local machine to the Redis Data Service instance:

cf ssh <YOUR_CF_APP_NAME> -L <LOCAL_PORT>:<REDIS_SERVICE_HOST>:<REDIS_SERVICE_PORT>

Replace the placeholders with the appropriate values:

<YOUR_CF_APP_NAME>: The name of your Cloud Foundry application.
<LOCAL_PORT>: The local port on your machine (e.g., 6379)
<REDIS_SERVICE_HOST>: The hostname of the Redis Data Service instance.
<REDIS_SERVICE_PORT>: The port on which the Redis Data Service is running (typically 6379).

Example:

cf ssh my-redis-app -L 6379:red198b63-master.service.dc1.a9ssvc:6379

Access the Redis Data Service: Once the SSH tunnel is established, you can connect to the Redis service using a Redis client on your local machine. For example, you can use the redis-cli command-line tool:

redis-cli -u "rediss://a9********:********@localhost:6379" --tls --cacert /path/to/ca_cert.pem

This command connects to the Redis service running on red198b63-master.service.dc1.a9ssvc through the SSH tunnel, using the local port 6379.

Check for migration errors

The migration-output-checker.txt contains the output of the MIGRATE command, and it is used in a further step to make sure the migration was executed with success. Check the content:

cat migration-output-checker.txt | grep --invert-match "OK"

The output of this command must be empty, otherwise it will return an error message with details on what went wrong during the migration. Below we list some of the potential error messages:

Error Message	Possible Cause	Troubleshooting
`NOKEY`	The key does not exist in the source Redis service instance.	Double check if the first part of the command (`redis-cli --raw KEYS ''`) is correct. The output of this command must have all* keys present in the Redis instance.
`IOERR error or timeout connecting to the client`	It is likely a communication problem between the two Redis instances.	Double check if the source Redis service instance has access to the destination Redis service instance. Please also check for correct target backend host for migration of exposed Redis Data Services.
`ERR Target instance replied with error: ERR invalid password`	Authentication does not work.	Double check if the password is correct.

Check database consistency

Check the database consistency between the original Redis service instance and the new Redis service instance databases.

If by any chance the source Redis service instance was not isolated **and** kept receiving new data during the migration the following check will not work.

Compare data between the Redis Service Instances

Execute the info keyspace redis-cli command under each host. The keys and expires values must be equal.

# Redis source db
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> INFO KEYSPACE

# Redis target db
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" \
--tls --sni <FQDN_TARGET_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> INFO KEYSPACE

Execute the info memory redis-cli command under each host. The values must be similar with only a few kilobytes of difference, if any.

# Redis source db
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> INFO MEMORY | grep "used_memory:\|used_memory_human"

# Redis target db
redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" \
--tls --sni <FQDN_TARGET_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> INFO MEMORY | grep "used_memory:\|used_memory_human"

Compare keys between the Redis Service Instances

This step will compare all keys between both Redis instances.

Get all source Redis service instance keys and sort them

redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*' > redis-source.txt && \  
sort --parallel=2 -uo redis-source-sorted.txt redis-source.txt

Get all target Redis service instance keys and sort them

redis-cli -u "<SOURCE_DB_CONNECTION_STRING>" -n <DB_NUM> \
--tls --sni <FQDN_SOURCE_FRONTEND_HOST> \
--cacert </PATH/TO/CA_CERT.PEM> --raw KEYS '*' > redis-target.txt && \  
sort --parallel=2 -uo redis-target-sorted.txt redis-target.txt

Compare keys from both Redis Service instances
- Please ensure that both files are not empty and contain keys before comparing.

diff redis-source-sorted.txt redis-target-sorted.txt

The output MUST NOT have differences.