Optimize your website with a few linux command line tools

When optimizing your website you will notice that images, and static resources are the most delaying the load of your website. You can use numerous tool to optimize them on the fly and cache them in WordPress or Joomla cache. I personally prefer to optimize them for once and save them on the disk.

The latter is requiring some access to the Linux command line and is therefore reserved for the most tech oriented people. I am presenting here some tools: Guetzli and PNGQuant, uglifyJS but there are more available. At this point of time these tools are the best in their category.

Optimizing JPEG files

Guetzli is a Google JPEG encoder that aims for excellent compression density at high visual quality. Guetzli-generated images are typically 20-30% smaller than images of equivalent quality generated by libjpeg. Guetzli generates only sequential (non progressive) JPEG due to faster decompression speeds they offer.

Install it

apt install guetzli # linux
brew install guetzli # macos
# windows user should use binary releases https://github.com/google/guetzli/releases

It is recommended to always do a backup of the folder containing your images, just in case. (real men don’t do backup but they often cries). I run Guetzli recursively in all sub-folder using find

find . -type f -name "*.jpg" -exec guetzli {} {} \;

Optimizing PNG files

pngquant is a command-line utility and a library for lossy compression of PNG images. The conversion reduces file sizes significantly (often as much as 70%) and preserves full alpha transparency. Generated images are compatible with all web browsers and operating systems.

apt install pngquant # linux
wget https://pngquant.org/pngquant.tar.bz2 # macos
# windows user should use binary releases https://pngquant.org/pngquant-windows.zip

I use the same technique, running pngquant recursively in all sub-folder using find

find . -iname "*.png" -exec pngquant -f -ext .png --verbose {} {} \;

Optimizing JavaScript files

Warning: i do not recommend to optimize official files of WordPress or Joomla. Some build int scanner that checksum internal file may detect this changes as a security breach: new code checksum do not match official WordPress expect checksum anymore.
It can be use for plugins and your own custom JavaScript code through.

UglifyJS is a general-purpose JavaScript parser/compressor/beautifier toolkit.

find . -maxdepth 1 -iname "*.js" -exec uglifyjs --compress --mangle -o {} -- {} \;

XMR-Stak-Nvidia miner running in docker

XMR-Stak is a universal Stratum pool miner. This is the NVIDIA GPU mining version; there is also an AMD GPU version, and a CPU version.

Docker® containers are often used to seamlessly deploy CPU-based applications on multiple machines. But Docker Engine does not natively support NVIDIA GPUs with containers.

I have dockerized the Nvidia GPU-mining version, so it is now running in nvidia-docker.

Containerizing GPU applications provides several benefits, among them:

Reproducible builds
Ease of deployment
Isolation of individual devices
Run across heterogeneous driver/toolkit environments
Requires only the NVIDIA driver to be installed
Enables “fire and forget” GPU applications
Facilitate collaboration

Check my code at https://github.com/cedricwalter/docker-xmr-stak-nvidia

or pull the image from docker hub https://hub.docker.com/r/cedricwalter/xmr-stak-nvidia/

Docker essential tools: Dive to navigate docker image content

If you want to explore the content of a docker image, Dive is THE tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker image.

Note that we will be exploring the image not the container: viewing the container and the image are not the same thing. You can get most of the time a bash access to a running container with

docker run -it image_name sh

Or following for images with an entrypoint

docker run -it --entrypoint sh image_name

NOTE: there is no guarantee that an image will have any sort of interactive shell. many minimal images contain only the binaries necessary to support a service.

An image is not running, consider it like a definition of a future container. You can still view the content, this is possible since each “layerid” directory contains json file describing layer property and filesystem associated with that layer. Docker stores Container images as layers to optimize storage space by reusing layers across images.

Beside Dive, you can also use docker inspect

docker image inspect image_id

But Dive is a lot better as it allow to navigate and see the filesystem changes in a nice norton commander interface.

Features

Show Docker image contents broken down by layer

As you select a layer on the left, you are shown the contents of that layer combined with all previous layers on the right. Also, you can fully explore the file tree with the arrow keys.

Indicate what’s changed in each layer

Files that have changed, been modified, added, or removed are indicated in the file tree. This can be adjusted to show changes for a specific layer, or aggregated changes up to this layer.

Estimate “image efficiency”

The lower left pane shows basic layer info and an experimental metric that will guess how much wasted space your image contains. This might be from duplicating files across layers, moving files across layers, or not fully removing files. Both a percentage “score” and total wasted file space is provided.

Quick build/analysis cycles

You can build a Docker image and do an immediate analysis with one command: dive build -t some-tag .

You only need to replace your docker build command with the same dive build command.

CI Integration

Analyze and image and get a pass/fail result based on the image efficiency and wasted space. Simply set CI=true in the environment when invoking any valid dive command.

To install it:

docker pull wagoodman/dive

then just provide your image tag/id/digest

docker run –rm -it -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive:latest <your-image>

Get more by visiting the project page https://github.com/wagoodman/dive

git-branch-renamer-maven-plugin

When working with many feature/release/bugix/hotfix branches, it is a bad idea to start changing the pom version as this will create merge conflicts using pull request. this plugin allow you to keep in ALL branches the same pom version for all your projects, for example MASTER-SNAPSHOT the version will be derived from branch name automagically 🙂

You may want to read more first these 2 short articles

git-branch-renamer-maven-plugin allow you to keep in ALL branches the same pom version for all your projects: for example MASTER-SNAPSHOT and never change it again.

the project version will be derived from branch name automatically when running in your continuous integration server.

branch name feature/xxxx

<version>xxxx-SNAPSHOT</version> (default)
<version>xxxx</version> (release = true)
<version>0-xxxx-SNAPSHOT</version> (forceNumericalVersion = true)
<version>feature-xxxx-SNAPSHOT</version> (filterOutBranchQualifier = false)

The project is hosted at Github https://github.com/cedricwalter/git-branch-renamer-maven-plugin

Running an Ethereum Node with Docker

Docker is a powerful tool for managing containers and run-time environments and, besides its many advantages, Docker can also be handy to keep your server tidy and secure.

#Docker allows to run operating systems, applications and tools in so called Containers. A #Container is an isolated environments that represents a autonomous host on its own – a bit in the same way a Virtual Machine does. Yet, Docker Containers are much lighter. They do not start an entire full-blown operating system for each Container instance. Instead, Docker uses #Linux kernel isolation mechanisms to run applications on the top of the host’s operating systems, yet keeping them isolated.

The Ethereum Go (language) team builds a Docker image of a “geth” node as part of their continuous build chain. Their Howto is more then enough to run your full node, mine below is just an enhanced example with volume, name, .. nothing fancy.

# i want to persist the blockchain in a volume

docker volume create --name=ethereum-data

# and limit cpu usage to 20% of all 8 cores –cpus=”.2″, give a name to container, more command line options

docker run --cpus=".2" -d -p 8545:8545 -p 30303:30303 \
 --name=ethereum-node \
 -v ethereum-data:/root/.ethereum ethereum/client-go \
 --rpc --rpcaddr "127.0.0.1"

to stop and recreate the container

docker stop ethereum-node && docker rm ethereum-node

to go inside the container

docker exec -it ethereum-node bash

to test the RPC api

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' localhost:8545

curl -H "Content-Type: application/json" -X POST \
 --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", true],"id":1}' http://127.0.0.1:8545


see https://ethereum.gitbooks.io/frontier-guide/content/rpc.html

You may want to register your node at The Ethereum (centralised) network status monitor , in that case just follow https://github.com/ethereum/wiki/wiki/Network-Status

My Ethereum node is now running at http://ethereum.galaxiis.com

A better status page is in development using PHP with RPC

Git and GitHub Training

I went today to a Git/GitHub training organized by Canoo and lead by Tim Berglund (GitHub.com) The online training presentation is available at http://teach.github.com/presentations/git-foundations.html use first left key then left/right/up and bottom keys to navigate through the slides.

My notes are still under reviews, but are available at http://gittraining.waltercedric.com

GitHub also offers free, short, topical online classes about GitHub, Git and the union of these two technologies Here.

Continue reading Git and GitHub Training

Nginx Specify a Vary: Accept-Encoding header

nginx (pronounced “engine-x”) is an open source Web server and a reverse proxy server for HTTP, SMTP, POP3 and IMAP protocols, with a strong focus on high concurrency, performance and low memory usage. It is licensed under a BSD-like license and it runs on Unix, #Linux, BSD variants, Mac OS X, Solaris, AIX and Microsoft Windows [WikiPedia]

Instructs proxy servers to cache two versions of the resource: one compressed, and one uncompressed. This helps avoid issues with public proxies that do not detect the presence of a Content-Encoding header properly.

Configuration files are provided using Gist https://gist.github.com/1620307 and are CONSTANTLY updated for added security and speed. Gist is a simple way to share snippets and pastes with others. All gists are git repositories, so they are automatically versioned, forkable and usable as a git repository. I recommend you to starred them to stay up to date.

Best nginx configuration for Joomla

These are my reusable settings for any #Joomla hosting, these are the most secure, and fastest settings to the best of my knowledge.

Configuration files are provided using Gist and are CONSTANTLY updated for added security and speed. Gist is a simple way to share snippets and pastes with others. All gists are git repositories, so they are automatically versioned, forkable and usable as a git repository. I recommend you to starred them to stay up to date.

Joomla.conf for nginx

Create a new directory nginx/conf to be able to place reusable nginx settings:

mkdir -p /etc/nginx/conf
vi /etc/nginx/conf/joomla.conf

Edit or create joomla.conf, you can find the latest joomla.conf documented version in one of my Gist at https://gist.github.com/1620307 https://gist.github.com/1620307.js?file=joomla.conf

Adding a new Joomla Site to nginx

Create required directory anywhere on your disk, here is an example with a domain www.example.com

mkdir -p /var/www/vhosts/example.com/httpdocs mkdir -p /var/www/vhosts/example.com/logs

Set the right permission to the user and group you have defined in nginx.conf

chown -fR www-data:www-data /var/www/vhosts/example.com/httpdocs

Copy the nginx template and adapt to your liking

cp /etc/nginx/sites-available/default /etc/nginx/sites-available/example vi /etc/nginx/sites-available/example

Edit or create example, you can find the latest file example documented version in one of my Gist at https://gist.github.com/1620307 https://gist.github.com/1620307.js?file=www.example.com

this file include Joomla.conf to avoid duplicating nginx settings

Activate the new domain

ln -s /etc/nginx/sites-available/example /etc/nginx/sites-enabled/example
service nginx restart