Difference between revisions of "Running a full DigiAssets stack"

From DigiByte Wiki
Jump to navigation Jump to search
m (Moved TOC to the right-hand side, floating)
m (Fixed some formatting)
(2 intermediate revisions by the same user not shown)
Line 42: Line 42:
  
 
<pre>mkdir ~/.digibyte
 
<pre>mkdir ~/.digibyte
cat <<eof> ~/.digibyte/digibyte.conf
+
cat <<EOF> ~/.digibyte/digibyte.conf
rpcuser=user
+
rpcuser=user
rpcpassword=password
+
rpcpassword=password
server=1
+
server=1
listen=1
+
listen=1
daemon=1
+
daemon=1
txindex=1
+
txindex=1
rpcallowip=127.0.0.1
+
rpcallowip=127.0.0.1
EOF</pre>
+
EOF</pre>
  
  
Line 57: Line 57:
  
 
<pre>wget -c https://github.com/digibyte/digibyte/releases/download/v7.17.2/digibyte-7.17.2-x86_64-linux-gnu.tar.gz -O - | tar xz
 
<pre>wget -c https://github.com/digibyte/digibyte/releases/download/v7.17.2/digibyte-7.17.2-x86_64-linux-gnu.tar.gz -O - | tar xz
./digibyte-7.17.2/bin/digibyted</pre>
+
./digibyte-7.17.2/bin/digibyted</pre>
  
  
Line 70: Line 70:
  
 
When the "block height for reward is...." matches the latest block on DigiExplorer.info, you know it's fully sync'd.
 
When the "block height for reward is...." matches the latest block on DigiExplorer.info, you know it's fully sync'd.
 
 
 
  
 
=Step 4: Installing node.js=
 
=Step 4: Installing node.js=
Line 115: Line 112:
  
  
<pre>cat <<eof> ~/pg_hba.conf
+
<pre>cat <<EOF> ~/pg_hba.conf
 
# Database administrative login by Unix domain socket
 
# Database administrative login by Unix domain socket
 
local all postgres peer
 
local all postgres peer
Line 147: Line 144:
 
psql
 
psql
 
create database digiassets;
 
create database digiassets;
create user digiassetsuser with encrypted password ******;
+
create user digiassetsuser with encrypted password '******';
 
grant all privileges on database digiassets to digiassetsuser;
 
grant all privileges on database digiassets to digiassetsuser;
 
\q
 
\q
Line 181: Line 178:
 
<pre>sudo mkdir /etc/minio
 
<pre>sudo mkdir /etc/minio
 
sudo chown minio-user:minio-user /etc/minio
 
sudo chown minio-user:minio-user /etc/minio
cat <<eof> minio-conf
+
cat <<EOF> minio-conf
 
MINIO_VOLUMES="/usr/local/share/minio/"
 
MINIO_VOLUMES="/usr/local/share/minio/"
 
MINIO_OPTS="-C /etc/minio --address 127.0.0.1:9000"
 
MINIO_OPTS="-C /etc/minio --address 127.0.0.1:9000"
Line 260: Line 257:
 
cd DigiAssets-Block-Explorer
 
cd DigiAssets-Block-Explorer
 
npm install
 
npm install
nano -w config/properties_development.conf
+
nano -w config/properties_development.conf</pre>
 +
 
  
 +
Under [server] you will want to set the http_port=9003
  
You will want to set the http_port=9003, as well as the [db] leaving #debug commented out
+
Under [db] you will want to set:
  
 +
<pre>
 +
[db]
 +
host=localhost
 +
port=5432
 +
name=digiassets
 +
dir=node_modules/digiasset-block-parser/models/
 +
user=digiassetsuser
 +
pass=MYPASSWORD
 +
#debug=true
 +
dialect=postgres</pre>
 +
 +
Near the bottom of the file you will also want to adjust the following in the [bitcoin_rpc] area:
  
 
<pre>dir=node_modules/digiasset-block-parser/models/
 
<pre>dir=node_modules/digiasset-block-parser/models/
Line 296: Line 307:
  
 
(This will be committed in a future update)
 
(This will be committed in a future update)
 
 
 
  
 
=Step 9: Installing digiassetsd=
 
=Step 9: Installing digiassetsd=
Line 341: Line 349:
  
  
<pre>cat <<eof> ~/digiassetsd.nginx
+
<pre>cat <<EOF> ~/digiassetsd.nginx
 
server {
 
server {
 
listen 80;
 
listen 80;
Line 356: Line 364:
  
  
<pre>cat <<eof> ~/metadata.nginx
+
<pre>cat <<EOF> ~/metadata.nginx
 
server {
 
server {
 
listen 80;
 
listen 80;
Line 371: Line 379:
  
  
<pre>cat <<eof> ~/explorer.nginx
+
<pre>cat <<EOF> ~/explorer.nginx
 
server {
 
server {
 
listen 80;
 
listen 80;
Line 403: Line 411:
 
sudo certbot --nginx -d explorer.domainname.tld
 
sudo certbot --nginx -d explorer.domainname.tld
 
sudo certbot --nginx -d metadata.domainname.tld
 
sudo certbot --nginx -d metadata.domainname.tld
sudo service nginx restart</eof></eof></eof></enter></eof></eof></pre>
+
sudo service nginx restart</pre>

Revision as of 15:54, 31 May 2020

The full DigiAssets stack will include:

  • Minio
  • Postgresql
  • Node.js
  • nginx
  • DigiAssets-Metadata-Server
  • DigiAssets-Block-Explorer
  • DigiAssetsd
  • pm2 (optional)
  • certbot (LetsEncrypt)

Step 1: OS, presumptions and initial preparation

This guide presumes a completed installation on Ubuntu 18.04 (LTS) with the user "digibyte" that you have sudo access from.It also presumes you are moderately familiar with server administration through SSH, and have full access to either the VM on which this will be all running or the bare-metal system, with a dedicated static IP address. If you are doing this from behind a corporate firewall or at home, it presumes you have the technical knowledge and the access to be able to forward the relevant ports.There are no minimum CPU requirements, though you'll want 8GB of RAM (Core Wallet will use ~4GB on it's own) with 16GB+ being good for future-proofing, as well as 1TB drive-space free, despite it currently only using ~250GB for a full sync (prior to a Postgresql vacuum) as of December 2019.This whole process is expected to only require around an hour in front of a computer (excluding sync time), but will likely take ~4-5 hours for DigiByte Core to sync, and then a few days for the DigiAssets-MetaData-Server to sync (Depending on your CPU and *especially* if you have a high-speed SSD etc).If you have not installed Ubuntu, go through with it now. There is no need for any GUI, so the basic Ubuntu 18.04 server install is sufficient.Once you have completed the installation, ssh in to the system as "digibyte" user.If required, you may need to:

sudo apt-get update && sudo apt-get install openssh-server
sudo ufw allow 22

Step 2: Domain Name setup

First things first, to give time for everything to propogate, you'll want to create a domain name with your hosting provider. In this case we are using digiassets.digibyteservers.io


Create an A-Record and point it to the IP address of your Ubuntu server


You'll also want to create the following additional 3x additional A-records: explorer.domainname.tld metadata.domainname.tld digiassetsd.domainname.tld


If you want to host the varying parts of your DigiAssets stack, you will want to ensure that relevant domain records are setup now for each of the above parts, though the scope of this guide will only account for it occurring on one system


You can also go through this document now and find / replace "domainname.tld" with yourdomain.com that you will be hosting it under.

Step 3: Installing digibyted

We're going to use the pre-made binaries, as these have all been through the gitian-build process. If you would like to compile your own, you may do-so following the instructions in src/build/unix.md, however that is outside of the scope of this document


We're first of all going to create the DigiByte config file:


mkdir ~/.digibyte
cat <<EOF> ~/.digibyte/digibyte.conf
rpcuser=user
rpcpassword=password
server=1
listen=1
daemon=1
txindex=1
rpcallowip=127.0.0.1
EOF


We will then download and run DigiByte version 7.17.2 (The current version at the time of writing, December 2019)


wget -c https://github.com/digibyte/digibyte/releases/download/v7.17.2/digibyte-7.17.2-x86_64-linux-gnu.tar.gz -O - | tar xz
./digibyte-7.17.2/bin/digibyted


You can follow the logs with:


tail -f ~/.digibyte/debug.log


This step will likely take ~4 hours on a moderately specced machine


When the "block height for reward is...." matches the latest block on DigiExplorer.info, you know it's fully sync'd.

Step 4: Installing node.js

We'll be using nvm as the Node Version Manager to install a specific version of node.js:


curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.2/install.sh | bash
	export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
	[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"


We then want to install and use version 10.17.0 (LTS):


nvm install 10.17.0


We then run the following in case this isn't a fresh server that had an older version:


nvm use 10.17.0



Step 5: Installing postgres

Postgresql is what we will be using for the database, so we'll create a user and password. You should not use this default password, please for the love of all things secure change it.


Database Name: digiassets Database User: digiassetsuser Database Pass: ********


So first we will install postgres:


sudo apt install postgresql postgresql-contrib


We're then going to allow anything locally to authenticate (This needs to be locked down more later):


cat <<EOF> ~/pg_hba.conf
	# Database administrative login by Unix domain socket
	local all postgres peer
	#"local" is for Unix domain socket connections only
	local all all trust
	# IPv4 local connections:
	host all all 127.0.0.1/32 trust
	# IPv6 local connections:
	host all all  ::1/128 trust
	# reject all other connection attempts
	host all all 0.0.0.0 0.0.0.0 reject
	EOF


sudo mv ~/pg_hba.conf /etc/postgresql/10/main/


Then restart the postgres service:


sudo /etc/init.d/postgresql restart


Now run the following line-by-line to log you in as the postgres user, and create the database + permissions.


Change only the password '******' (Just type what you want as a secure password instead into it):


sudo -i -u postgres
	psql
	create database digiassets;
	create user digiassetsuser with encrypted password '******';
	grant all privileges on database digiassets to digiassetsuser;
	\q
	exit


You should now be logged out of postgres and logged in as the digibyte user again.



Step 6: Installing Minio

Rather than using Amazon S3 storage, we're instead going to run this all locally:


curl -O https://dl.minio.io/server/minio/release/linux-amd64/minio
	chmod +x minio
	sudo mv minio /usr/local/bin
	sudo useradd -r minio-user -s /sbin/nologin
	sudo chown minio-user:minio-user /usr/local/bin/minio


We will then create the relevant directories to hold our data buckets. You can adjust the location of this yourself if required.


sudo mkdir /usr/local/share/minio
	sudo chown minio-user:minio-user /usr/local/share/minio


Next we will make a config file for minio. Replace the IP address on the MINIO_OPTS line and the password on the MINION_SECRET_KEY line with something more secure:


sudo mkdir /etc/minio
	sudo chown minio-user:minio-user /etc/minio
	cat <<EOF> minio-conf
	MINIO_VOLUMES="/usr/local/share/minio/"
	MINIO_OPTS="-C /etc/minio --address 127.0.0.1:9000"
	MINIO_ACCESS_KEY="minio"
	MINIO_SECRET_KEY="*****"
	EOF
	sudo mv minio-conf /etc/default/minio


We want minio to start on boot:


curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service
	sudo mv minio.service /etc/systemd/system
	sudo systemctl daemon-reload
	sudo systemctl enable minio
	sudo systemctl start minio
	sudo ufw allow 9000


Check and ensure that minio has started correctly:


sudo systemctl status minio


(Push "q" to stop seeing the status)


You should see "Active: active (running)"


We will temporarily be using a browser now to complete the minio setup. Go to http://127.0.0.1:9000 and log in with the access key and secret key you defined in your minio conf above.


Once successfully logged in, click on the "+" down the bottom-right, click on "Create bucket" and then call it "digiassets". It should immediately appear on the left-hand column.


We are now done with our browser, back to the ssh session.


We're now going to edit the minio opts and change the IP address:


sudo nano -w /etc/default/minio


Set it to:


MINIO_OPTS="-C /etc/minio --address 127.0.0.1:9000"


Ctrl + X, y, <enter>


sudo systemctl restart minio



Step 7: Installing DigiAssets-Metadata-Server

cd ~/
	git clone https://github.com/DigiByte-Core/DigiAssets-Metadata-Server
	cd DigiAssets-Metadata-Server
	npm install
	nano -w config/properties_development.conf


You will want to enable minio accessKey, secretKey, change the bucket to "digiassets", and the http_port to 9002



Step 8: Installing DigiAssets-Block-Explorer

cd ~/
	git clone https://github.com/DigiByte-Core/DigiAssets-Block-Explorer
	cd DigiAssets-Block-Explorer
	npm install
	nano -w config/properties_development.conf


Under [server] you will want to set the http_port=9003

Under [db] you will want to set:

[db]
host=localhost
port=5432
name=digiassets
dir=node_modules/digiasset-block-parser/models/
user=digiassetsuser
pass=MYPASSWORD
#debug=true
dialect=postgres

Near the bottom of the file you will also want to adjust the following in the [bitcoin_rpc] area:

dir=node_modules/digiasset-block-parser/models/
	[bitcoin_rpc]
	ssl=false
	url=localhost
	path=/
	username=user
	password=password
	port=14022
	timeout=3000


We also need to ensure that we're going to correctly handle Burn instructions, so run:


nano +16 ~/DigiAssets-Block-Explorer/node_modules/digiasset-block-parser/models/assetstransactions.js


Change from (IF yours still has the older types):


type: DataTypes.ENUM('issuance', 'transfer'),


to:


type: DataTypes.ENUM('issuance', 'transfer', 'burn'),


(This will be committed in a future update)

Step 9: Installing digiassetsd

cd ~/
	git clone https://github.com/DigiByte-Core/digiassetsd
	cd digiassetsd
	npm install
	nano -w config.js


You will want to modify the machineurl, bitcoind-port (14022) blockexplorer and metadataserver URLs



Step 10: Installing pm2

We will use pm2 to look after our node.js services for logging / control etc


cd ~/
	npm install -g pm2
	cd ~/DigiAssets-Block-Explorer
	pm2 start npm --name "explorer" -- start
	#you may want to pause here for a week while explorer syncs
	cd ~/DigiAssets-Metadata-Server
	pm2 start npm --name "metadata" -- start
	cd ~/digiassetsd
	pm2 start npm --name "digiassetsd" -- start



Step 11: Installing nginx webserver

We will then install nginx and allow it out the firewall


sudo apt install nginx
	sudo ufw allow 'Nginx Full'


Then lots of copy / paste for each of the instances, you should find / replace "domainname.tld" with your own domain if you haven't already:


cat <<EOF> ~/digiassetsd.nginx
	server {
	listen 80;
	server_name digiassetsd.domainname.tld;
	location / {
	proxy_pass [http://127.0.0.1:8080];
	proxy_http_version 1.1;
	proxy_set_header Upgrade \$http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host \$host;
	}
	}
	EOF


cat <<EOF> ~/metadata.nginx
	server {
	listen 80;
	server_name metadata.domainname.tld;
	location / {
	proxy_pass [http://127.0.0.1:9002];
	proxy_http_version 1.1;
	proxy_set_header Upgrade \$http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host \$host;
	}
	}
	EOF


cat <<EOF> ~/explorer.nginx
	server {
	listen 80;
	server_name explorer.domainname.tld;
	location / {
	proxy_pass [http://127.0.0.1:9003];
	proxy_http_version 1.1;
	proxy_set_header Upgrade \$http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host \$host;
	}
	}
	EOF


sudo mv digiassetsd.nginx /etc/nginx/sites-available/digiassetsd.domainname.tld
	sudo mv metadata.nginx /etc/nginx/sites-available/metadata.domainname.tld
	sudo mv explorer.nginx /etc/nginx/sites-available/explorer.domainname.tld
	sudo ln -s /etc/nginx/sites-available/* /etc/nginx/sites-enabled/



Step 12: Installing certbot for LetsEncrypt

When prompted, select option 2 to Redirect all requests to HTTPS:


sudo add-apt-repository ppa:certbot/certbot
	sudo apt install python-certbot-nginx
	sudo certbot --nginx -d digiassets.domainname.tld
	sudo certbot --nginx -d explorer.domainname.tld
	sudo certbot --nginx -d metadata.domainname.tld
	sudo service nginx restart