Google Cloud Build connect to VM on premise

GCP Cloud Build is so cool with 120 minutes free per day.

I want to use Cloud Build to execute a script to deploy a NodeJS project on my private VM.

Here is what i have done in my cloudbuild.yaml

steps:
# copy configuration bucket from GCS to cloudbuild
- name: gcr.io/cloud-builders/gsutil
  args: ['cp',
    '-r',
    'gs://${_GCS_CONFIGRATION_BUCKET}',
    '.']
# Set 400 to private key.
- name: 'kroniak/ssh-client'
  args: ['chmod',
    '400',
    '${_GCS_CONFIGRATION_BUCKET}/ssh/cloudbuild_id_rsa']
# ssh into remote instance and run a script.
- name: 'kroniak/ssh-client'
  args: ['ssh',
    '-i',
    '${_GCS_CONFIGRATION_BUCKET}/ssh/cloudbuild_id_rsa',
    '-o',
    'UserKnownHostsFile=/dev/null',
    '-o',
    'StrictHostKeyChecking=no',
    '-p',
    '${_SSH_REMOTE_PORT}',
    '${_SSH_REMOTE_USER_HOST}',
    '${_SSH_REMOTE_COMMAND}']

Because CloudBuild is stateless, you need to create your RSA keypairs and store the keys on a private GCS.


You need to add your RSA public key into ~/.ssh/authorized_keys on your server, tutorial here

And the script to pulling code and restart server.

#!/bin/sh
 
# It is good practice to print the required versions on server. 
# cause the code will execute in SSH non interactively mode.
# https://stackoverflow.com/questions/17089808/how-to-do-remote-ssh-non-interactively
 
echo "NodeJS: "$(node -v)
echo "NPM: "$(npm -v)
 
 
WORKSPACE=/working_directory/public_html
 
echo "Working directory: " $WORKSPACE
 
cd $WORKSPACE
 
git checkout develop
git pull -Xtheirs
 
echo "===================Install dependencies ==========="
npm install
echo "===================Finished install dependencies ========"
 
echo "RELOAD ENV"
pm2 reload $WORKSPACE/ecosystem.config.js production  --update-env

Here are example values of the variable.

_GCS_CONFIGRATION_BUCKET : my-private-bucket-configuration
_SSH_REMOTE_PORT :  2202 // hacker will sad.
_SSH_REMOTE_USER_HOST : aduckdev@8.8.8.8
_SSH_REMOTE_COMMAND :  /home/aduckdev/deployment_script.sh

Stupid errors that took me more than 1 hour to solve

Here is my follows up on the original post here: https://blog.juliobiason.net/thoughts/things-i-learnt-the-hard-way/#code-reviews-are-not-for-style

The list will keep increasing day by day but hopefully i wont repeat any of the items here.

Soft-Hyphen Characters

https://en.wikipedia.org/wiki/Soft_hyphen

There was a request from my client to remove Soft-Hyphen character from a JSON.
A JSON use to render a page on Android and iOS.
Client did not specify the devices or OS, And I did not get the issue at first on Android.
But I could see some red-dot in JSON, I did a replace

replace({ '­':'', '­':'', '­': '' }) 

Sadly, It was not enough,
Took me some hours to find a solution here: https://stackoverflow.com/questions/10148184/php-check-if-string-contains-soft-hypen-and-replace-it/56885146#56885146

Even my friend told me that i should convert the U+00AD into a character and use that character to replace

And yes!, if you're checking the solution on StackOverflow, the answer is

$str = str_replace('­', '', $str);

And here was the diff on Stash

Throw Error in Promise.

What did i do?
I was trying to throw some error inside a Promise, but I forgot to put catch in the end of the end. My web server was hanging for several minutes before it responded.

function doAsync (){
  return new Promise( (resolve, reason ) =>{
      // do something to make error happens.
      if(error){
       throw new Error("ABC")
      }
     resolve("ok");
  });
}

For the code, we have two solutions. 1.1 Use resolve with the error.

function doAsync (){
  return new Promise( (resolve, reason ) =>{
      // do something to make error happens.
      if(error){
       return reason( new Error("ABC") );
      }
     resolve("ok");
  });
}

1.2 Use catch to return the error value.

function doAsync (){
  return new Promise( (resolve, reason ) =>{
      // do something to make error happens.
      if(error){
       throw new Error("ABC");
      }
     resolve("ok");
  }).catch(err => errorHandling(err))
}

Setup SSH with RSA keypairs

Firstly, you can easily remote to server with 

ssh your_username@server_address

But you can remove the annoying by being asked typing password many times, using RSA keypairs, it is also more secured than using password.

1. Creating a key pairs on Workstation

ssh-keygen -t rsa - b 4096

Add passphrase is recommended

To read your public key

cat ~/.ssh/id_rsa.pub

2. Uploading public keys to the Server

SSH to the Server

ssh your_username@server_address 

Create a file in .ssh/authorized_keys  and paste the public key into

you can do it from Workstation with one command using 

cat ~/.ssh/id_rsa.pub | ssh your_username@server_address 'cat >> .ssh/authorized_keys'

And change the permission

ssh your_username@server_address ‘chmod 700 .ssh; chmod 640 .ssh/authorized_keys’

From now we can ssh with being asked for password

ssh your_username@server_address

3. Turn off password authentication

On Server Machine

sudo vim /etc/ssh/sshd_config

Uncomment Passwordauthentication , change the value to No

And  restart ssh 

sudo systemctl restart ssh

When we need to use swap partition in Linux?

Recently i have an issue on my LightSail small instance which have 512MB of RAM,

I can not run react-script build on it

npm run build

Because of the issue: https://github.com/kitze/custom-react-scripts/blob/master/packages/react-scripts/template/README.md#npm-run-build-exits-too-early

So create a Linux swap is life saver.

Firstly, check if swap available :

sudo swapon --show

if it show nothing, then you don't have any swap yet.

Now, create a swap file, it should be 2x or 4x of your RAM capacity

sudo fallocate -l 1G /swapfile

To verify the swap is created

ls -lh /swapfile

To enable swap, you may need to change the permission

sudo chmod 600 /swapfile

make swap file

          sudo mkswap /swapfile        

then

sudo swapon /swapfile

check if it is ready

sudo swapon --show

 To enable swap permanent

Backup file

sudo cp /etc/fstab /etc/fstab.bak

Add the line at the end of file

/swapfile swap swap defaults 0 0

Benchmarking web application tools

1. Apache benchmarking 

Home page https://httpd.apache.org/docs/2.4/programs/ab.html

• c: ("Concurrency"). Indicates how many clients (people/users) will be hitting the site at the same time. While ab runs, there will be -c clients hitting the site. This is what actually decides the amount of stress your site will suffer during the benchmark.
• n: Indicates how many requests are going to be made. This just decides the length of the benchmark. A high -n value with a -c value that your server can support is a good idea to ensure that things don't break under sustained stress: it's not the same to support stress for 5 seconds than for 5 hours.
• k: This does the "KeepAlive" functionality browsers do by nature. You don't need to pass a value for -k as it it "boolean" (meaning: it indicates that you desire for your test to use the Keep Alive header from HTTP and sustain the connection). Since browsers do this and you're likely to want to simulate the stress and flow that your site will have from browsers, it is recommended you do a benchmark with this.
  The final argument is simply the host. By default it will hit http:// protocol if you don't specify it.

ab -k -c 350 -n 20000 example.com/ 

By issuing the command above, you will be hitting http://example.com/ with 350 simultaneous connections until 20 thousand requests are met. It will be done using the keep alive header.

E.g: Get user profile

ab -n2000 -c10 -k -m GET -T "application/json" -H "${accessToken}" http://localhost:8080/api/v1/users/profile

Limitation : ab does not support dynamic body per request -  https://stackoverflow.com/questions/2602841/changing-post-data-used-by-apache-bench-per-iteration

2. Apache Jmeter

Home Page: http://jmeter.apache.org/

Create many users using __UUID() random method example:

References:

https://stackoverflow.com/questions/12732182/ab-load-testing

https://stackoverflow.com/questions/44800557/apache-jmeter-add-random-data-in-body-for-request/54059660#54059660

https://www.roseindia.net/jmeter/using-jmeter.shtml

MacOS basic commands

1. Copy a folder into another folder

cp -r /path/folder/a /path/folder/b

result:

/path/folder/b
/path/folder/b/a

2. Move a folder into another folder

mv /path/folder/a /path/folder/b

result:

/path/folder/b
/path/folder/b/a

3. Remove files

rm -rf /path/to/file
rm -rf /path/to/file*

4. Process and ports

lsof -iTCP -sTCP:LISTEN -n -P
lsof -iTCP -sTCP:LISTEN -n -P | grep {PORT}

5. Kill a process

kill -9 ${PID}

6. Change owner of a folder

# for current user
sudo chown -R `id -un` /path/to/folder
# a user
sudo chown -R ${username} /path/to/folder

MongoDB basic commands

1. Setup MongoDB

• Download: https://www.mongodb.com/download-center/community


• Extract folder
  

  tar -zxvf mongodb-osx-x86_64-xx.tgz -C /path/to/container/folder



• Add to $PATH
  Find out which current shell you are working on.
  zsh open ~/.zshrc
  bash open ~/.bashrc
  add lines at the end of the file.
  

  export MONGODB_HOME=~/Workspace/mongodb-osx-x86_64-xx/
  export PATH=$MONGODB_HOME/bin:$PATH

  
  Reload the environment variables for current shell
  

  # e.g
  source ~/.zshrc



• Create the directory where MongoDB stores data
  

  sudo mkdir /data/db
  sudo chown -R `id -un` /data/db



• Start MongoDB
  

  mongod

  
  For now , you have a mongodb server is running on default port 2017 without authentication, the data is stored on /data/db

2. Basic client commands

# connect to server
mongo

# list databases;
show databases;

# use a database
use ${db_name}

# list collections;
show collections;

# form now we can perform collection methods
db.${collectionName}.${command}

find({})
findOne()
find({"_id": new IdObject("${docId}")})
find({ <field>: { $regex: /pattern/, $options: '<options>' }})
deleteMany({})
deleteOne({})
remove({})
drop()

Checkout collection methods

2.Delete database

# from commandline
mongo ${dbName} --eval "db.dropDatabase()"

# inside mongoClient
use ${dbName}
db.dropDatabase()