Big Data

How to migrate the PostgreSQL data directory

Check the current installed data directory [ec2-user@ip-192-168-93-151 ~]$ psql -h 192.168.28.223 -U postgres -d postgres Password for user postgres: postgres=# SHOW config_file; config_file ------------------------------------- /var/lib/pgsql/data/postgresql.conf (1 row) postgres=# SHOW data_directory; data_directory --------------------- /var/lib/pgsql/data (1 row)

How to install and configure PostgreSQL on Amazon Linux

Downloand and install updates [ec2-user@ip-192-168-28-223 ~]$ cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" [ec2-user@ip-192-168-28-223

How to set up etcd cluster

Install etcd and etcdctl On vm1, vm2, and vm3, run the following script to install etcd and etcdctl: $ cat etcd_install ETCD_VER=v3.4.25 # choose either URL GOOGLE_URL=https://storage.googleapis.com/etcd GITHUB_URL=https://github.com/etcd-io/etcd/releases/download DOWNLOAD_URL=${GOOGLE_URL} rm

Getting started with PostgreSQL

PostgreSQL is a powerful, open source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads.

Using ANOVA in R to analyze US COVID data to understand age impact to death rate

A great example to use ANOVA in R to analyze complicated data.

Using R to analyze US COVID pandemic waves and peaks

Goal According to weekly data between 2020 to 2022, we want to get to know the waves and peaks of COVID pandemic in these years. Download the data We will continue to use NCHS(National Center for Health Statistics) as our data source. Visit https://data.cdc.gov/browse?category=

Using R to analyze the quarterly US COVID data

An R example to analyze big amount of data

Getting started with R

Installation The simplest way is to use Homebrew: $ brew install r Another way is to download installation package from https://cloud.r-project.org/ "Hello world" of R Run it from R console Command R will start a R console, and you can run R code inside R console. (base) ➜ benchling

Using virtctl to access virtual machine in Kubernetes

Install the virtctl client tool Basic VirtualMachineInstance operations can be performed with the stock kubectl utility. However, the virtctl binary utility is required to use advanced features such as: * Serial and graphical console access It also provides convenience commands for: * Starting and stopping VirtualMachineInstances * Live migrating VirtualMachineInstances * Uploading virtual machine

Using node selector to assign virtual machines to a node

In some circumstances, we want to control which node the virtual machine or pod deploys to. The node selector can be used to assign virtual machine or pod to a node. Add label to a node The label can be added to a node from either command line or openshift

Deploy application with kubernetes statefulset

StatefulSet is the workload API object used to manage stateful applications. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are

Keep a docker container running and not exiting

In the kubernetes environment, we can keep a container(pod) alive and avoid it exits immediately after starting. Method one 1. Use a long-time-run command in Dockerfile CMD ["sh", "-c", "tail -f /dev/null"] or CMD ["sh", "-c", "sleep infinity"] 2. Build the docker image and push to docker repository

Pull an image from a private docker registry in Kubernetes Pod

Log in to Docker Hub In order to pull a image from Docker Hub, you must authenticate with a registry. Use docker tool to log in to the Docker Hub as below. A username and password is needed to log in. $ docker login The login process creates or updates the

Deploy kubernetes statefulset with affinity

Here is an example to demonstrate how to assign statefulset and pod to the target worker node. affinity helps define which node will be assigned for the pod by using the node labels. $ kubectl get nodes -o wide --show-labels $ kubectl label nodes portworx.io/fio=true $ cat statefulset-node-select.yaml apiVersion:

Using kubeconfig to configure access to remote Kubernetes cluster

Install kubectl binary The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs. [root@localhost ~]# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)

Lambda architecture in big data system

What is Lambda Architecture When working with very large data sets, it can take a long time to run the sort of queries that clients need. These queries can't be performed in real time, and often require algorithms such as MapReduce that operate in parallel across the entire data set.

Deploy CockroachDB in kubernetes cluster

Deploy Kubernetes cluster $ kubectl get nodes NAME STATUS ROLES AGE VERSION node0 Ready 115d v1.19.2 node1 Ready 115d v1.19.2 node2 Ready 115d v1.19.2 node3 Ready master 115d v1.19.2 Start CockroachDB cluster Start the CockroachDB nodes with a configuration file that has been

Setup CockroachDB in docker container

Create docker volume Create docker volume on each host: [root@host1 ~]# docker volume create --driver local --opt type=ext4 --opt device=/dev/nvme2n1 vol1 [root@host1 ~]# mkfs.ext4 /dev/nvme2n1 [root@host1 ~]# docker inspect vol1 [ { "CreatedAt": "2022-09-13T01:10:50Z", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/vol1/_data", "Name"

Setup CockroachDB cluster with HAProxy load balancing

Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing. * Performance: Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per

Useful Elastic cluster APIs

Useful Elastic cluster APIs

Getting started with Elasticsearch and Kibana

Install elasticsearch Add user for elasticsearch [root@vm1 home]# groupadd es [root@vm1 home]# useradd es -g es [root@vm1 home]# passwd es [root@vm1 home]# cd es Download elasticsearch [root@vm1 es]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.3-linux-x86_64.tar.gz [root@vm1 es]# tar

How to setup the Grafana and Prometheus dashboard

Install node-exporter on each target nodes $ docker run --restart=always --name nd-export -d -p 9100:9100 -v /proc:/host/proc -v /sys:/host/sys -v /:/rootfs:ro,rslave --net=host quay.io/prometheus/node-exporter $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d8c7fc8a4e40 quay.io/prometheus/node-exporter

Understanding MapReduce with an example

What is MapReduce MapReduce is a programming framework that allows us to perform distributed and parallel processing on large data sets in a distributed environment. * MapReduce consists of two distinct tasks — Map and Reduce. * As the name MapReduce suggests, reducer phase takes place after the mapper phase has been completed.

How to deploy MySQL and phpMyAdmin with Docker

Create the MySQL docker container $ docker pull mysql/mysql-server:5.7 $ docker run -d --name=mysqldb -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=perfdb -v /var/lib/osd/mounts/mysql_data:/var/lib/mysql -p 3306:3306 mysql/mysql-server:5.7 $ docker logs mysqldb $ docker

Database concepts you should know

SQL vs. NoSQL Source Benefits of NoSQL databases NoSQL databases offer many benefits over relational databases. NoSQL databases have flexible data models, scale horizontally, have incredibly fast queries, and are easy for developers to work with. * Flexible data models NoSQL databases typically have very flexible schemas. A flexible schema allows

Distributed system concepts you should know

This post contains basic explanations for concepts you should know related to distributed system. API and REST API Application Programming Interface, abbreviated as API, enables connection between computers or computer programs. It is a Software Interface that offers services to other software to enhance the required functionalities. REST API is

Estimate the scale of the system

It is always a good idea to estimate the scale of the system as it help reflect if the designed system could fulfill the functional requirements. The requirements might include: * Number of users * Number of active users(NAU) * Requrests per second(RPS) * Logins per seconds * Transactions per second(TPS) for

What is high availability

What is high availability? High availability refers to a system or component that is operational without interruption for long periods of time. High availability is measured as a percentage, with a 100% percent system indicating a service that experiences zero downtime. This would be a system that never fails. It’

Backup and restore MySQL database

Backup a database mysqldump is a command-line utility which can be used to generate backups of MySQL database. $ mysqldump -u root --password= mydb > mydb_dump_`date +"%Y%m%d_%H%M%S"`.sql $ ls -ltr | grep mydb -rw-r--r--. 1 root root 4834575 Sep 28 21:11 mydb_dump_20210928_144610.

An easy guide to install kubernetes cluster with kubeadm

Kubernetes can be installed with the following deployments tools. * Bootstrapping clusters with kubeadm * Installing Kubernetes with kops * Installing Kubernetes with Kubespray In this article, we learn to install Kubernetes cluster with kubeadm. Installing kubeadm Prepare the cluster nodes We have three Centos nodes to install Kubernetes cluster. $ cat /etc/centos-release

Migrate docker data to different directory

This post guides us how to migrate the docker data from the existing directory to a target directory in the case that the old directory runs out of space. Stop the docker daemon $ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ systemctl stop docker.service $ ps aux | grep

Using gstack for docker container process

In a Docker container environment, we won't get a valid stack trace directly on the container host as below. $ ps -ef |grep smbd root 171118 167977 0 Apr22 ? 00:00:02 /usr/sbin/smbd --foreground --no-process-group root 171166 171118 0 Apr22 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group root

Tuning kernel parameters in Docker

Configure namespaced kernel parameters(sysctl) at runtime The --sysctl sets namespaced kernel parameters (sysctls) in the container. For example, to turn on IP forwarding in the containers network namespace, run this command: $ docker run --sysctl net.ipv4.ip_forward=1 someimage Note Not all sysctls are namespaced. Docker does not

CBT - Changed Block Tracking

Introduction Changed Block Tracking is an incremental backup technology for virtual machines. It helps create faster and smaller backups. It has the following advantages. * Reduce backup time * Save disk space by only storing changed data to the previous backup Block changes are tracked in the virtualization layer, outside the virtual

Monitor RMAN backup and restore progress

The following script will allow you to monitor progress of an RMAN backup or Restore from 4 different perspectives (channel, session wait events, datafiles, backuppieces). The script is run from SQLPlus and takes a date input value in the format ‘dd-mon-rr hh24:mi:ss’. The date supplied does not have

RMAN backup set, pieces, and datafiles

RMAN backs up the datafiles, control file, archived log files, and server parameter files in a RMAN specific format called a backup piece. In a nutshell, a backup set is a bundle of dbf, ctl and redo file that can restore a database. A set of one or more such

Level 0 and Level 1 Incremental Backups

Incremental backups can be either level 0 or level 1. A level 0 incremental backup, which is the base for subsequent incremental backups, copies all blocks containing data, backing the datafile up into a backup set just as a full backup would. The only difference between a level 0 incremental