Deploying a Standalone Databend

Deploying a Standalone Databend

Databend works with both self-hosted and cloud object storage solutions. This topic explains how to deploy Databend with your object storage. For a list of supported object storage solutions, see Understanding Deployment Modes.

Setting up Your Object Storage

MinIO

a. Follow the MinIO Quickstart Guide to download and install the MinIO package to your local machine.

b. Open a terminal window and navigate to the folder where MinIO is stored.

c. Run the command vim server.sh to create a file with the following content:

~/minio$ cat server.sh
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=minioadmin
./minio server --address :9900 ./data

d. Run the following commands to start the MinIO server:

chmod +x server.sh
./server.sh

e. In your browser, go to http://127.0.0.1:9900 and enter the credentials (minioadmin / minioadmin) to log in to the MinIO Console.

f. In the MinIO Console, create a bucket named databend.

Amazon S3

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
• https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html

Tencent COS

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://cloud.tencent.com/document/product/436/13309
• https://cloud.tencent.com/document/product/436/68282

Alibaba OSS

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://www.alibabacloud.com/help/zh/object-storage-service/latest/create-buckets-2
• https://help.aliyun.com/document_detail/53045.htm

Wasabi

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://wasabi.com/wp-content/themes/wasabi/docs/Getting_Started/index.html#t=topics%2FGS-Buckets.htm%23TOC_Creating_a_Bucketbc-1&rhtocid=_5_0
• https://wasabi.com/wp-content/themes/wasabi/docs/Getting_Started/index.html#t=topics%2FAssigning_an_Access_Key.htm

QingCloud QingStore

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docsv3.qingcloud.com/storage/object-storage/manual/console/bucket_manage/basic_opt/
• https://docs.qingcloud.com/product/api/common/overview.html

Azure Blob Storage

Before deploying Databend, make sure you have successfully set up your object storage environment in the cloud, and the following tasks have been completed:

• Create a bucket or container named databend.
• Get the endpoint URL for connecting to the bucket or container you created.
• Get the Access Key ID and Secret Access Key for your account.

For information about how to manage buckets and Access Keys for your cloud object storage, refer to the user manual from the solution provider. Here are some useful links you may need:

• https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container
• https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys

Downloading Databend

a. Create a folder named databend in the directory /usr/local.

b. Download and extract the latest Databend package for your platform from https://github.com/datafuselabs/databend/releases.

c. Move the extracted folders bin and etc to the folder /usr/local/databend.

Deploying a Meta Node

a. Open the file databend-meta-node.toml in the folder /usr/local/databend/etc, and replace 0.0.0.0 with 127.0.0.1 within the whole file.

b. Open a terminal window and navigate to the folder /usr/local/databend/bin.

c. Run the following command to start the Meta node:

./databend-meta -c ../etc/databend-meta.toml > meta.log 2>&1 &

d. Run the following command to check if the Meta node was started successfully:

curl -I  http://127.0.0.1:28101/v1/health

Deploying a Query Node

a. Open the file databend-query-node.toml in the folder /usr/local/databend/etc, and replace 0.0.0.0 with 127.0.0.1 within the whole file.

b. In the file databend-query-node.toml, set the parameter type in [storage] block to s3 if you're using a S3 compatible object storage, or azblob if you're using Azure Blob storage.

[storage]
# fs | s3 | azblob
type = "s3"

c. Comment out the [storage.fs] block first, and then uncomment the [storage.s3] block if you're using a S3 compatible object storage, or uncomment the [storage.azblob] block if you're using Azure Blob storage.

# Set a local folder to store your data.
# Comment out this block if you're NOT using local file system as storage.
#[storage.fs]
#data_path = "benddata/datas"

# To use S3-compatible object storage, uncomment this block and set your values.
[storage.s3]
bucket = "<your-bucket-name>"
endpoint_url = "<your-endpoint>"
access_key_id = "<your-key-id>"
secret_access_key = "<your-account-key>"

# To use Azure Blob storage, uncomment this block and set your values.
# [storage.azblob]
# endpoint_url = "https://<your-storage-account-name>.blob.core.windows.net"
# container = "<your-azure-storage-container-name>"
# account_name = "<your-storage-account-name>"
# account_key = "<your-account-key>"

d. Set your values in the [storage.fs] or [storage.azblob] block. Please note that the field endpoint_url refers to the service URL of your storage region and varies depending on the object storage solution you use:

MinIO

endpoint_url = "http://127.0.0.1:9900"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

Amazon S3

endpoint_url = "https://s3.amazonaws.com"

Tencent COS

You can get the URL from the bucket detail page.

For example,

endpoint_url = "https://cos.ap-beijing.myqcloud.com"

Alibaba OSS

Follow this format:

https://<bucket-name>.<region-id>[-internal].aliyuncs.com

For example,

endpoint_url = "https://databend.oss-cn-beijing-internal.aliyuncs.com"

For information about the region ID, see https://help.aliyun.com/document_detail/31837.htm

Wasabi

To find out the service URL for your storage region, go to https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions-

For example,

endpoint_url = "https://s3.us-east-2.wasabisys.com"

QingCloud QingStore

To find out the service URL for your storage region, go to https://docsv3.qingcloud.com/storage/object-storage/intro/object-storage/#zone

For example,

endpoint_url = "https://pek3b.qingstor.com"

Azure Blob Storage

Follow this format:

https://<your-storage-account-name>.blob.core.windows.net

e. Open a terminal window and navigate to the folder /usr/local/databend/bin.

f. Run the following command to start the Query node:

./databend-query -c ../etc/databend-query.toml > query.log 2>&1 &

g. Run the following command to check if the Query node was started successfully:

curl -I  http://127.0.0.1:8081/v1/health

Verifying Deployment

In this section, we will run MySQL queries from a SQL client installed on your local machine.

a. Create a connection to 127.0.0.1 from your SQL client. In the connection, set the port to 3307, and set the username to root.

b. Run the following commands to check if the query is successful:

CREATE TABLE t1(a int);

INSERT INTO t1 VALUES(1), (2);

SELECT * FROM t1;