Use Redis-shake To Migrate Self-built Redis To Alibaba Cloud

1. Experiment

1.1 Knowledge points

The experiment uses ApsaraDB for Redis. It introduces the process of migrating the user’s local Redis inventory and incremental data to Alibaba Cloud Redis with Redis-shake migration tool. ApsaraDB for Redis is a database service that is compatible with the open source Redis protocol standard and provides hybrid storage. It is designed base on the dual-system hot standby architecture and cluster architecture, which enables it to meet requirements of high throughput, low latency, and flexible configuration.

1.2 Experiment process

• Self-built Redis environment preparation
• Alibaba Cloud Redis environment preparation
• Use Redis-shake to migrate data

1.3 Scene architecture diagram

1.4 Cloud resources required

• ECS
• Redis

1.5 Prerequisites

• During the experiment, you are allowed to choose your own Alibaba Cloud account over the account provided by this lab. However, to make sure the experiment goes smoothly, you need to choose the same Ubuntu 16.04 operation system for your ECS.
• Before starting the experiment, please confirm that the previous experiment has been closed normally and has been exited.

2. Start the experiment environment

Click Start Lab in the upper right corner of the page to start the experiment.

.

After the experiment environment is started and the system has deployed resources needed for the experiment in the background. For example, the ECS instance, RDS instance, Server Load Balancer instance, and OSS bucket. An account consisting of the username and password for logging on to the Web console of Alibaba Cloud is also provided.

After the experiment environment is started and related resources are properly deployed, the experiment starts a countdown. You have an hour to perform experimental operations. After the countdown ends, the experiment stops, and related resources are released. During the experiment, pay attention to the remaining time and arrange your time wisely. Next, use the username and password provided by the system to log on to the Web console of Alibaba Cloud and view related resources:

Go to the logon page of Alibaba Cloud console.

Fill in the sub-user account and click on Next.

Fill in the sub-user password and click on Login.

After successfully log on to the console, you will see the page showing as the following figure.

3. Prepare self-built Redis environment

3.1 Log on to ECS

Click on Elastic Compute Service, as shown in the following figure.

We can see one running ECS instance in Silicon Valley region.

Copy this ECS instance’s Internet IP address and remotely log on to this ECS (Ubuntu system) instance. For details of remote login, refer to login。

The default account name and password of the ECS instance:

Account name: root

Password: nkYHG890..

3.2 Install Redis

Enter the following command to download the Redis installation package.

wget https://labex-ali-data.oss-us-west-1.aliyuncs.com/redis/redis-5.0.12.tar.gz

Enter the following command to decompress the installation package.

tar -xzf redis-5.0.12.tar.gz

Enter the following command to compile Redis.

cd redis-5.0.12 && make

Enter the command vim /etc/profile, copy the following content to the file, save and exit.

export PATH=$PATH:/root/redis-5.0.12/src

Enter the following command to for the modifications to take effect.

source /etc/profile

Enter the following command to start the Redis server.

nohup redis-server &

Enter the following command to start the Redis client.

redis-cli

dbsize

Enter the following command to set the key in redis.

set name labex

get name

3.3 Prepare the data

Enter exit to exit the Redis client.

Enter the following command to install the pip3 tool.

cd && apt update && apt -y install python3-pip

Enter the following command,

export LC_ALL=C

Enter the following command to install the Redis dependency package of python.

pip3 install redis

Enter the command vim data.py, copy the following content to the file, save and exit.

#!/usr/bin/python3

import redis
import random, string, os, time
from multiprocessing import Process

def getRedis():
    r = redis.Redis(host = "127.0.0.1", port = 6379, db = 0)
    return r

def createKey():
    print ("Process :" + str(os.getpid()) + " is running")
    r = getRedis()
    for i in range(10000):
        r.set('labex-' + str(os.getpid()) + '-' + str(i), 'data' + str(i))

if __name__ == "__main__":
    list_process = []
    for i in range(10):
        p = Process(target = createKey)
        list_process.append(p)
    for p in list_process:
        p.daemon = True
        p.start()
    for p in list_process:
        p.join()
    print ("success")

Enter the following command to execute the script and insert 100,000 pieces of data into Redis. The process takes about 1~2 minutes.

python3 data.py

The data insert is complete.

Enter the following command, you can see the number of keys in Redis at the moment.

redis-cli

dbsize

<font color='red'>The user can cut off the above result picture when doing the experiment and send it to the teacher, indicating that the part of the current chapter has been completed.</font>

For now, you need to enter the command exit to exit the client.

4. Prepare Alibaba Cloud Redis environment

4.1 Set the whitelist

Go to Alibaba Cloud Redis console as showing in the figure below .

Select the US (Silicon Valley) area, you can see that there is an existed Redis instance.

Click on the instance ID.

To set the whitelist, click on Modify.

Add the internal network address of ECS.

Now you can see the address is added.

4.2 Create Account

Click on Create as showing in the figure below.

Set the account password to labex/Aliyun-test as showing in the figure below, and click on OK.

The account is activating.

Now the account activation is complete.

The other account appears here is the default account, which password has been set to “Aliyun-test” when creating the instance.

Now you can check the intranet connection address of Redis.

Now go back to the ECS command line.

Enter the following command to connect to Alibaba Cloud Redis. Please note to replace YOUR-ALI-REDIS-ADDR with your own Alibaba Cloud Redis intranet connection address.

redis-cli -h YOUR-ALI-REDIS-ADDR -p 6379 -a Aliyun-test

dbsize

Now the account is successfully connect to Alibaba Redis.

Enter the command exit to exit the client.

5. Use Redis-shake to migrate data

Enter the following command to download the Redis-shake installation package.

wget http://labex-ali-data.oss-us-west-1.aliyuncs.com/redis/redis-shake-v2.0.3.tar.gz

Enter the following command to decompress the installation package.

tar -zxvf redis-shake-v2.0.3.tar.gz

Enter the following command to enter the decompressed directory.

cd redis-shake-v2.0.3/

ls

Enter the command vim redis-shake.conf to open the configuration file of Redis-shake.

Refer to the setting parameters below,

source.type = standalone

source.address = 127.0.0.1:6379

source.passwrod_raw = 

source.auth_type = auth

Please note to replace YOUR-ALI-REDIS-ADDR with your own Alibaba Cloud Redis intranet connection address

target.type = standalone

target.address = YOUR-ALI-REDIS-ADDR:6379

target.password_raw = labex:Aliyun-test

target.auth_type = auth

Enter the following command to start the migration task.

./redis-shake.linux -type=sync -conf=redis-shake.conf

As shown in the figure above, Redis-shake will migrate stock data first, and then to migrate incremental data.

Now you need to create the second command line to connect to ECS.

Enter the following command to connect to Alibaba Cloud redis. Please note to replace YOUR-ALI-REDIS-ADDR with your own Alibaba Cloud Redis intranet connection address

redis-cli -h YOUR-ALI-REDIS-ADDR -p 6379 -a Aliyun-test

dbsize

You can see that the existing data has been migrated.

Now you need to create the third ECS command line.

Enter the following command to insert 1 million pieces of data into local Redis instance.

python3 data.py

Go back to the first command line, you can see that the incremental data is migrating.

<font color='red'>Users can cut off the above result picture when they are doing the experiment and send it to the teacher, indicating that the current experiment has been completed.</font>

After the data insertion complete, enter the dbsize command in the second command line, you can see that the newly added data has been migrated, and there are 2 million pieces of data in total.

Reminder:
Before you leave this lab, remember to log out your Alibaba RAM account before you click the ‘stop’ button of your lab. Otherwise you’ll encounter some issues when opening a new lab session in the same browser:

6. Experiment summary

The experiment introduced the process of using the Redis-shake migration tool to migrate your local Redis inventory and incremental data to Alibaba Cloud Redis. ApsaraDB for Redis supports product configurations for products contain multiple memory specifications. Users can upgrade memory specifications according to business volume, and support elastic expansion of the storage space and throughput performance of the database system under the cluster architecture, breaking through the high QPS performance bottleneck of massive data, therefore, to deal with the requirements of read and write for millions pieces of data every second.