Docker is a form of OS-level virtualisation whose purpose is, to quote:
Package up code and all its dependencies so the application runs quickly and reliably from one computing environment to another
Speed: isolated, consistent and repeatable environments for your developers to fire up a new version of the application that a collogue may have written requires muddling around in Git and using branches – this interrupts work flows and can lead to lose of work if “git stash” isn’t used right. With a container: they just run it.
Portability: Docker images define abstract and immutable run time environment independent of the underlying hardware platform and packages it into a well defined and efficient image specification. With Dockerhub, we have a global distribution network First up: portability and allowing anyone, on any hardware, and any OS to run the application with a single command. All they need to do is install Docker and “docker run” will do the rest. With a TAR file based deployment, they need TAR, they need to know how-to deploy the application and manage the files; they may need some runtime on their system like Ruby, Python or NodeJS; and they cannot restrict the resources the application is using…
Simplifies your infrastructure requirements you can restrict the resources the application is using inside of a container much easier than you can with one that is not.
Automation is first class citizen: Dockerfile, DockerCompose one final point here: Docker Compose. You’re missing out on a lot of free stuff here. It’s so easy to write a Docker Compose file that brings up everything needed by the application - databases, caches, mocked APIs - allowing anyone in your organisation to bring up the application on their laptop and play with it, including the CEO, DBAs, QA, the receptionist.
Example: You don’t have to do these things (installing build tools, dependencies to build an application) on every new deployment! Instead you can build the applicationone and distribute it as an “executable” (without the build tools and dependencies)
Source Openrc-hgi.sh
Install python3
Install pip
Install setuptools wheel
Pip install -r requirements.txt (openstack, dateutil, json,….)
Npm install (vue, bootstrapvue, …)
How is Docker better than just packaging the app with tar?
| runs natively on Linux and shares the kernel of the host machine with other containers | runs a full-blown “guest” operating system with virtual access to host resources through a hypervisor
| runs a discrete process, taking no more memory than any other executable, making it lightweight. With Docker, scaling your application is a matter of spinning up new executables, not running heavy | VM hosts: applications have no system dependencies, updates can be pushed to any part of a distributed application. provide an environment with more resources than most applications need.
Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface. When people say “Docker” they typically mean Docker Engine, the client-server application made up of the Docker daemon, a REST API that specifies interfaces for interacting with the daemon, and a command line interface (CLI) client that talks to the daemon (through the REST API wrapper). Docker Engine accepts docker commands from the CLI, such as docker run
Docker Engine is a client-server application with these major components:
The CLI uses the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands. Many other Docker applications use the underlying API and CLI. The daemon creates and manages Docker objects, such as images, containers, networks, and volumes.  The Docker daemon (dockerd) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services.
A Docker registry stores docker images. Docker Hub is a public registry that anyone can use, and Docker is configured to look for images on Docker Hub by default. You can even run your own private registry. If you use Docker Datacenter (DDC), it includes Docker Trusted Registry (DTR).
When you use the docker pull or docker run commands, the required images are pulled from your configured registry. When you use the docker push command, your image is pushed to your configured registry
The copy-on-write (CoW) strategy Copy-on-write is a strategy of sharing and copying files for maximum efficiency. If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to it, it just uses the existing file. The first time another layer needs to modify the file (when building the image or running the container), the file is copied into that layer and modified. This minimizes I/O and the size of each of the subsequent layers.
When you start a container, a thin writable container layer is added on top of the other layers. Any changes the container makes to the filesystem are stored here. Any files the container does not change do not get copied to this writable layer. This means that the writable layer is as small as possible.Not only does copy-on-write save space, but it also reduces start-up time. When you start a container (or multiple containers from the same image), Docker only needs to create the thin writable container layer.If Docker had to make an entire copy of the underlying image stack each time it started a new container, container start times and disk space used would be significantly increased. This would be similar to the way that virtual machines work, with one or more virtual disks per virtual machine.
The docker filesystem uses copy-on-write with it’s layered union fs. So when you write to a file that’s part of the image, it will first make a copy of that file to the container filesystem which is a layer above all the image layers. What that means is when you append a line to the /var/log/cron.log, it will get a new inode in the filesystem and the file that the tail command is following at is no longer the one you see when you docker exec into the container.
For deployment of code
docker build -t mercury/openstack_report_backend:prod .
docker login --username=mercury --password=****
docker push mercury/openstack_report_backend:prod
# You need to tag your image correctly first with your registryhost:
docker tag [OPTIONS] IMAGE[:TAG] [REGISTRYHOST/][USERNAME/]NAME[:TAG]
#Then docker push using that same tag.
docker push NAME[:TAG]
docker tag 518a41981a6a myRegistry.com/myImage
docker push myRegistry.com/myImage
For debugging
# You can see a list of your running containers
docker ps
docker ps -a -q
# -q prints just the container ids (without column headers)
# -f allows you to filter your list of printed containers
# (in this case we are filtering to only show exited containers)
# -v to avoid dangling volumes
journalctl -u docker.service | tail -n 50
# to get a bash shell in a container
docker exec -it <container name> /bin/bash
# Example: to get environmental variables
docker exec container bash -c 'echo "$ENV_VAR"'
docker exec container printenv VARIABLE
docker inspect <container name>
# You can display the long-form Id for a container with the command:
docker inspect --format '{{ .Id }}>' <container name>
# in a swarm
docker service logs —details --tail 10 <service name>
docker service ps --no-trunc # to find out which node the service is running on
docker service ps my-ngx # returns a task id
docker service inspect --pretty <SERVICE-ID>
# to get the container id
docker inspect -f "{{.Status.ContainerStatus.ContainerID}}" <task_id>
# exec inside a swarm service
docker exec -it $(docker ps --filter "name=my-service." -q) cat /run/secrets/my-service-secret
For Cleanup
# This will remove all stopped containers
docker container prune
# To clean up all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes.
docker system prune
docker rm `docker ps --no-trunc -aq`
docker rm $(docker ps -a -q --no-trunc) # to avoid id clashes
docker ps -a -q | xargs docker rm
docker rm -v $(docker ps -q -f status=exited)
docker rm $(docker ps -v -q -f status=exited)
An image is an executable package that includes everything needed to run an application–the code, a runtime, libraries, environment variables, and configuration files.
A container is a runtime instance of an image–what the image becomes in memory when executed (that is, an image with state, or a user process). A container is launched by running an image.
Docker images are stored as series of read-only layers. When we start a container, Docker takes the read-only image and adds a read-write layer on top. Images are frozen immutable snapshots of live containers. Containers are running (or stopped) instances of some image. Image vs Container
A Task definition is a collection of 1 or more container configurations. Some Tasks may need only one container, while other Tasks may need 2 or more potentially linked containers running concurrently. The Task definition allows you to specify which Docker image to use, which ports to expose, how much CPU and memory to allot, how to collect logs, and define environment variables.
A Task is created when you run a Task directly, which launches container(s) (defined in the task definition) until they are stopped or exit on their own, at which point they are not replaced automatically. Running Tasks directly is ideal for short running jobs, perhaps as an example things that were accomplished via CRON.
A service is used to guarantee that you always have some number of Tasks running at all times. If a Task’s container exits due to error, or the underlying EC2 instance fails and is replaced, the ECS Service will replace the failed Task. This is why we create Clusters so that the Service has plenty of resources in terms of CPU, Memory and Network ports to use. To us it doesn’t really matter which instance Tasks run on so long as they run. A Service configuration references a Task definition. A Service is responsible for creating Tasks. Can define as a group of containers representing the same function (e.g. container replicas in a swarm)
Services are typically used for long running applications like web servers. For example, if I deployed my website powered by Node.JS in Oregon (us-west-2) I would want say at least three Tasks running across the three Availability Zones (AZ) for the sake of High-Availability; if one fails I have another two and the failed one will be replaced (read that as self-healing!). Creating a Service is the way to do this. If I had 6 EC2 instances in my cluster, 2 per AZ, the Service will automatically balance Tasks across zones as best it can while also considering cpu, memory, and network resources.
Services and container are related but both are different things. A service can be run by one or multiple containers. With docker you can handle containers and with docker-compose you can handle services. For example: Let’s say that we have this docker-compose.yml file:
web:
image: example/my_web_app:latest
expose:
- 80
links:
- db
db:
image: postgres:latest
This compose file defines two services, web and db. When you run docker-compose up, Asuming that the project directory is test1 then compose will start 2 containers named myapp_db_1 and myapp_web_1.
$ docker ps -a
CONTAINER ID IMAGE COMMAND ... NAMES
1c1683e871dc test1_web "nginx -g" ... test1_web_1
a41360558f96 test1_db "postgres -d" ... test1_db_1
So, in this point you have 2 services and 1 container for each.
But you could scale the service named web to use 5 containers.
Another very important point is that a Service can be configured to use a load balancer, so that as it creates the Tasks—that is it launches containers defined in the Task Defintion—the Service will automatically register the container’s EC2 instance with the load balancer. Tasks cannot be configured to use a load balancer, only Services can.
$ docker-compose scale web=5
Creating and starting 2 ... done
Creating and starting 3 ... done
Creating and starting 4 ... done
Creating and starting 5 ... done
In this point you have 2 services and 6 containers
Docker compose allows an application developer to define their application stack as a set of distributed applications. Compose allows it work just the same whatever infrastructure it runs on. Decoupling the infrastructure from the application significantly increases the portability of the distributed application. A service definition contains configuration that is applied to each container started for that service, much like passing command-line parameters to docker container create Likewise, network and volume definitions are analogous to docker network create and docker volume create As with docker container create, options specified in the Dockerfile, such as CMD, EXPOSE, VOLUME, ENV, are respected by default - you don’t need to specify them again in docker-compose.yml
Node: a networking address: could be a container, a host etc.
Swarm : a group of nodes acting as one
Stack: a group of services + networks + volumes
About itd
-i (interactive) is about whether to keep stdin open (some programs, like bash, use stdin and other programs don’t).
-d (detached) is about whether the docker run command waits for the process being run to exit. Thus, they are orthogornal and not inherently contradictory. A program like bash exits when stdin in closed, so without -i, it exits immediately.
| -t allocates a pseudo-tty. You can see the difference from running bash with -it vs with just -i. For example, without -t, you don’t get any prompt and ls shows results in one column. This difference is like the difference between running ls and running ls | cat, where cat does not have a pseudo-tty. |
Processes, exec and entrypoint
exec “$@” is typically used to make the entrypoint a pass through that then runs the docker command.
exec will replace the parent process, rather than have two processes running. Example, At the exec line entrypoint.sh, the shell running as pid 1 will replace itself with the command server start. For example, if Redis was started without exec, it will not receive a SIGTERM upon docker stop and will not get a chance to shutdown cleanly. In some cases, this can lead to data loss or zombie processes. Example: This is critical for signal handling. Without using exec, the server start in the above example would run as another pid, and after it exits, you would return to your shell script. With a shell in pid 1, a SIGTERM will be ignored by default. That means the graceful stop signal that docker stop sends to your container, would never be received by the serverprocess. After 10 seconds (by default), docker stop would give up on the graceful shutdown and send a SIGKILL that will force your app to exit, but with potential data loss or closed network connections, that app developers could have coded around if they received the signal. It also means your container will always take the 10 seconds to stop. If you do start child processes (i.e. don’t use exec), the parent process becomes responsible for handling and forwarding signals as appropriate. This is one of the reasons it’s best to use supervisord or similar when running multiple processes in a container, as it will forward signals appropriately.
without exec
After all, at the end of the day a container is a set of virtualized processes.
In docker you should only execute one process per container because if you don’t, the process that forked and went background is not monitored and may stop without you knowing it.
When you use CMD cron && tail -f /var/log/cron.log the cron process basically fork in order to execute cron in background, the main process exits and let you execute tailf in foreground. The background cron process could stop or fail you won’t notice, your container will still run silently and your orchestration tool will not restart it.
You can avoid such a thing by redirecting directly the cron’s commands output into your docker stdout and stderr which are located respectively in /proc/1/fd/1 and /proc/1/fd/2 (The adopted solution may be dangerous in a production environment)
Using basic shell redirects you may want to do something like this :
ENTRYPOINT should be used in scenarios where you want the container to behave exclusively as if it were the executable it’s wrapping. That is, when you don’t want or expect the user to override the executable you’ve specified, e.g. sometimes. convenient to use Docker as portable packaging for a specific executable. a utility implemented as a Python script you need to distribute but don’t want to burden the end-user with installation of the correct interpreter version and dependencies. You could package everything in a Docker image with an ENTRYPOINT referencing your script. Now the user can simply docker run your image and it will behave as if they are running your script directly.
you can achieve this same thing with CMD, but the use of ENTRYPOINT sends a strong message that this container is only intended to run this one command.
CMD is used where you want the user to be able to override from the command line. CMD value can be easily overridden by supplying one or more arguments to docker run after the name of the image. e.g. Docker run ping localhost, Docker run ping docker.io. Running the image starts to feel like running any other executable – you specify the name of the command you want to run followed by the arguments you want to pass to that command. the -c 3 argument that was included as part of the ENTRYPOINT essentially becomes a “hard-coded” argument for the ping command (the -c flag is used to limit the ping count to the specified number). It’s included in each invocation of the image and can’t be overridden in the same way as the CMD parameter.
CMD executable param1 param2. shell form: problematic if we need to send POSIX signals to the container since /bin/sh doesn’t dforward signals to child processes. Beyond the PID 1 issue, you may also run into problems with the shell form if you’re building a minimal image which doesn’t even include a shell binary. )
OR CMD [“executable”,”param1”,”param2”] (exec form: When the exec form of the CMD instruction is used the command will be executed without a shell.)
You’re using the exec form of ENTRYPOINT. Unlike the shell form, the exec form does not invoke a command shell. This means that normal shell processing does not happen. For example, ENTRYPOINT [ “echo”, “$HOME” ] will not do variable substitution on $HOME. If you want shell processing then either use the shell form or execute a shell directly, for example: ENTRYPOINT [ “sh”, “-c”, “echo $HOME” ]. When using the exec form and executing a shell directly, as in the case for the shell form, it is the shell that is doing the environment variable expansion, not docker.(from Dockerfile reference) In your case, I would use shell form ENTRYPOINT ./greeting –message “Hello, $ADDRESSEE!”
When both an ENTRYPOINT and CMD are specified, the CMD string(s) will be appended to the ENTRYPOINT in order to generate the container’s command string. When using ENTRYPOINT and CMD together it’s important that you always use the exec form of both instructions.
You can use Docker Remote API to extract the log messages or run commands remotely
To enable the Docker Remote API, add the following line to the file /etc/default/docker: DOCKEROPTS=’-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock’
When Docker is restarted, it listens for HTTP API requests on port 4243 (you can specify a different port) and also on a socket. To get all the messages, you can issue the GET request:
http://
For running a command inside a docker, remotely, edit the file /lib/systemd/system/docker.service
Modify the line that starts with ExecStart to look like this:
ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:4243
Where the addition is the “-H tcp://0.0.0.0:4243” part.
Make sure the Docker service notices the modified configuration:
systemctl daemon-reload
Restart the Docker service:
sudo service docker restart
Test that the Docker API is indeed accessible:
curl http://localhost:4243/version