Sunday, January 22, 2017

Wrapup: Docker Developer - Beginner Linux Containers course

So, I did another quick run-through of the subject course this morning prior to taking the test, and here are my thoughts.

  • I'd forgotten about a few potentially useful commands like "docker inspect", so the review was useful.
  • Lesson 3 (docker-compose) still feels ... compressed. There could have been more content, or it could have been a complete course all by itself. The overall structure of the lesson was 1) download the files from git; 2) bring up the swarm; 3) a brief explanation of some of the lines in the yml file; and 4) here's how to publish the swarm. I would like to see the lesson expanded to get a better sense of how the yml file was developed step by step. In fact, the example included four other yml files, so that might have been where the direction the author(s) was (were) originally headed. Update: found the tutorial which this lesson appears to be a summary of: https://docs.docker.com/engine/getstarted-voting-app/
I scored 13/14 on the quiz, so yay me.

I'm going to take the rest of the free courses offered by Docker, just to see if they issues like resource control.

Saturday, January 21, 2017

More fun with the Docker Developer Beginner Linux Containers course

This time in lesson 3. Problems bringing up the docker swarm.

docker compose up -d

Only redis stayed up, according to docker ps -a. The other four containers exited prematurely.

First action: run compose up again without the -d option. This results in more output from each container. Summary of the output:

vote_1: python can't open file 'app.py': [Errno 13] Permission denied
worker_1: System.AggregationException: One or more errors occurred. (No such device or address)
db: exited with code 1

Well, after spending about a day debugging this, I found some comments on github that fedora uses a fork of docker, and the version may be out of date. So I uninstalled docker:

sudo dnf remove docker-engine docker-compose docker

Reinstalled from the official repos using instructions from:

https://docs.docker.com/engine/installation/linux/fedora/
https://docs.docker.com/compose/install/

And rebooted the machine because systemctl start docker-engine complained "docker-engine.service not found". After all this, the docker-compose tutorial worked as expected.

Here are some debugging notes, even if the correct solution ultimately turned out to be "reinstall from official sources."

As always, focus on one problem at a time. I did this by commenting out all but one member of the swarm in docker-compose.yml, and running "docker-compose up" to debug. The "command:" line in the yml file can be changed to try different things in the container. For example, replacing "command: python app.py" with "command: ls -l /" allowed me to see that the ownership of the app directory (1000:1000) did not match the user (root), which explains the permission denied error.

Learn how to clean up and run things manually.

docker ps # show running containers
docker ps -a # show all containers
docker rm <container id> # remove containers listed in docker ps -a
docker rm -f <container id>
docker images
docker rmi <image id> # remove images

A neat shortcut is you only need to pass the first three digits of a container or image id.

Note that running "docker run -it examplevotingapp_vote /bin/sh" produced different results (i.e. /app was owned by root instead of 1000). My guess is it's because the docker-compose.yml file has this line:

volumes:
  - ./vote:/app

This tells docker-compose to mount the "vote" directory in the cwd on the host as "/app". Of course, the vote directory is owned by me (uid 1000) instead of root, so that probably explains the ownership weirdness.

The difference from running the container manually and using docker-compose ultimately led me to realize that docker-compose starts up container examplevotingapp_vote_1. My current theory is that docker-compose takes the base _vote image and adds another layer to it according to the configuration in the yml file (e.g. mounts the /app volume, connects to the swarm networks, etc). Using layers is consistent with the docker architecture from what I've seen so far, so let's run with that for now.

One final note about the last build-push-pull steps from the end of the lesson. The steps do allow you to access (pull) your voting app containers from another machine. But they fail to mention what else you'll to bring up the swarm. Certainly, docker-compose up failed to do anything until I cloned the example app from github. And running docker-compose up now, it seems to be rebuilding the swarm instead of using the containers I already pulled down. And yes, docker images confirms there is now images beginning with examplevotingapp* as well as bobthesquirrel/votingapp*. Needs further investigation.

Thursday, January 19, 2017

Notes on Docker Course - Developer Beginner Linux Containers

I walked through the Developer Beginner Linux Containers course offered by Docker today, and hit a few snags. There's no obvious way to post comments or feedback to get the instructions updated, so here are my notes, for what they're worth.

Lesson 2.0 Webapps with Docker

This is the first snag:

$ sudo docker build -t bob_the_squirrel/myfirstapp .
[sudo] password for bob:
Sending build context to Docker daemon 8.192 kB
Step 1 : FROM alpine:latest
---> 88e169ea8f46
Step 2 : RUN apk add --update py-pip
---> Running in 15deabd22845
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
 py-pip (virtual):
   provided by: py2-pip
   required by: world[py-pip]
The command '/bin/sh -c apk add --update py-pip' returned a non-zero code: 1

Easily fixed by modifying the Dockerfile:

# install the pip package and any dependencies
RUN apk add --update py2-pip # the lesson says to use py-pip


The next snag was this:

$ sudo docker build -t bob_the_squirrel/myfirstapp .
[sudo] password for bob:
Sending build context to Docker daemon 8.192 kB
Step 1 : FROM alpine:latest
---> 88e169ea8f46
Step 2 : RUN apk add --update py2-pip
---> Running in 33207dfe8efd
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
(1/12) Installing libbz2 (1.0.6-r5)
(2/12) Installing expat (2.2.0-r0)
(3/12) Installing libffi (3.2.1-r2)
(4/12) Installing gdbm (1.12-r0)
(5/12) Installing ncurses-terminfo-base (6.0-r7)
(6/12) Installing ncurses-terminfo (6.0-r7)
(7/12) Installing ncurses-libs (6.0-r7)
(8/12) Installing readline (6.3.008-r4)
(9/12) Installing sqlite-libs (3.15.2-r0)
(10/12) Installing python2 (2.7.13-r0)
(11/12) Installing py-setuptools (29.0.1-r0)
(12/12) Installing py2-pip (9.0.0-r0)
Executing busybox-1.25.1-r0.trigger
OK: 61 MiB in 23 packages
---> 840b9af43db6
Removing intermediate container 33207dfe8efd
Step 3 : COPY requirements.txt /usr/src/app
---> 0eac1e0f7a53
Removing intermediate container e9c368692b72
Step 4 : RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt
---> Running in cd9acc93e4f3
Could not open requirements file: [Errno 20] Not a directory: '/usr/src/app/requirements.txt'
You are using pip version 9.0.0, however version 9.0.1 is available.


You should consider upgrading via the 'pip install --upgrade pip' command. 


The command '/bin/sh -c pip install --no-cache-dir -r

/usr/src/app/requirements.txt' returned a non-zero code: 1

Turns out this was my fault because I forgot to add a trailing slash to the Dockerfile COPY command:

# Add files that make up the Flask application
COPY requirements.txt /usr/src/app # oops
RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt


If the trailing slash is included (/usr/src/app/), it looks like the COPY command will make the directory if it doesn't already exist.

Sunday, January 6, 2013

OS By Petzl - Mixed Results

Tried using the OS By Petzl software on a Core battery connected to a Tikka XP2:

  • The software doesn't work on my x220 tablet running Windows 7 (64-bit): it fails to detect that the battery is plugged in.
  • Even after trying to run the software in various Windows Compatibility modes (Vista, XP SP2, XP SP3, Windows 7).
  • It does work on another computer which is running Windows XP: can name the battery, tell it which lamp is being used, and set profiles.
  • But I can't create a custom profile. The instructions are just plain wrong.
  • Maybe a conflict with Lenovo drivers? Who knows.

Lessons learned:

  • If I had been using Windows 7 System Restore, it would have saved me a lot of trouble trying to uninstall the Petzl software and drivers.
  • Restoring from a previous backup (Windows 7 Backup and Restore) works fine, and only took about 90 minutes.
  • Was also able to use Areca to restore user documents. (Right-click on an archive, choose "Recover...", and choose a recovery option.)
  • Drag and drop of a VMWare Player Linux VM to/from a USB drive worked fine as a way to backup it up.

So, be warned. The OS By Petzl software kinda sorta works, but don't count on it. Bet the battery controller could be hacked though.

Tuesday, January 1, 2013

Bus Pirate meets the 93LC46B

One of the toys I acquired last year (or possibly even the year before) was a DangerousPrototypes Bus Pirate. Among the many things it can do is read/write EEPROM chips, which is a type of memory. They are used in electronics to store things like programs and state information. People have used Bus Pirates to recover motherboards that have been bricked by a bad BIOS flash. Some laptops have start-up password protection. The password is stored on an EEPROM chip, and if the user should become forgetful, reading the chip is one way to recover it.

I found an old EEPROM chip in my spare parts collection, the 93LC46B. The data sheet says it communicates using 3-wire serial I/O, and can store 1024 bits organized either as 8- or 16-bit words.

Next was the question of how to wire the bus pirate to the chip. I liked the idea of plugging some headers into a breadboard:

These give the Bus Pirate's clips something to grab onto.

Then, I wasn't sure if I needed to create a separate power supply. Turns out one can simply power the chip directly from the Bus Pirate's 3.3V connector. This is undoubtedly one of the reasons the Bus Pirate is such a well-regarded tool.

Finally, I hooked up the CS (chip select), CLK, MOSI (to the DI or data in pin), and MISO (to the DO or data out pin) connectors.

I used putty as a terminal to control the Bus Pirate. Configuring the Bus Pirate in 3-wire mode took a little trial-and-error, but I finally found settings that worked:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2013.01.01 19:28:55 =~=~=~=~=~=~=~=~=~=~=~=

HiZ>m
1. HiZ
2. 1-WIRE
3. UART
4. I2C
5. SPI
6. 2WIRE
7. 3WIRE
8. LCD
9. DIO
x. exit(without change)

(1)>7
Set speed:
 1. ~5KHz
 2. ~50KHz
 3. ~100KHz
 4. ~400KHz

(1)>1
CS:
 1. CS
 2. /CS *default

(2)>1
Select output type:
 1. Open drain (H=Hi-Z, L=GND)
 2. Normal (H=3.3V, L=GND)

(1)>2
Ready
3WIRE>W
Power supplies O

The 93LC46B's data sheet says the chip select needs to be held high before sending any commands, which is why I chose "CS" instead of "/CS" from the menu ("/CS" means the chip select pin is held low).

The other thing that got me was the output type. At one point, I selected "Open drain" and then couldn't figure out why CS wouldn't go to 3.3V when it was enabled. Something I found out while debugging this is the Bus Pirate's ADC connector can be used to measure voltages, up to 6V. At any rate, after realizing the CS problem was caused by the wrong output type, things started working a lot better.

The 93LC46B's commands take the form of a start bit, 2 op code bits, 7 address bits, and optionally 8 data bits for some write commands. This means having to tell Bus Pirate to write 10 or 18 bits. This is one way to from address 0:

3WIRE>[0b110;3 0b0000000;7 r;8]
CS ENABLED
WRITE: 0x06;3 
WRITE: 0x00;7 
READ: 0xFF 
CS DISABLED

The command broken down:

[            Enable CS
0b110;3      Write 3 bits: start bit '1', opcode '10' (read)
0b0000000;7  Write 7 bits: address 0x00
r;8          Read 8 bits
]            Disable CS

So far so good. I am able to issue 10-bit commands to the chip. These commands include write-enable and erase-all. (Erase writes '1' into each bit position). Unfortunately, I haven't got any 18-bit commands, which take data, to work. So I'm not yet able to write arbitrary data into the eeprom.

Lessons learned:

  • Learned how to wire up the Bus Pirate to a chip.
  • Got some experience with driving the BP in 3-wire mode.
  • Learned how to write words which are not 8-bits in length to the device.
  • The BP's ADC connector can be used to confirm the CS goes to the correct level.
  • Read the datasheet carefully, watch for gotchas like "write-enable must be sent before writing data or erasing cells".

Update

Finally found the correct data sheet. Turns out the 93LC46B only operates in 16-bit mode. So that jumper in the first picture has no effect on the chip's organization. The correct command format takes 6-bit addresses and 16-bit data:

3WIRE>[0b110;3 0b000000;6 r:2;8]  // read two 8-bit words from address 0x00
CS ENABLED
WRITE: 0x06;3 
WRITE: 0x00;6 
READ: 0xFF 0xFF 
CS DISABLED
3WIRE> [2K
3WIRE>[0b100;3 0b110000;6]  // enable write
CS ENABLED
WRITE: 0x04;3 
WRITE: 0x30;6 
CS DISABLED
3WIRE>[0b101;3 0b000000;6 0x12;8 0x34;8]  // write 0x1234
CS ENABLED
WRITE: 0x05;3 
WRITE: 0x00;6 
WRITE: 0x12 
WRITE: 0x34 
CS DISABLED
3WIRE>[0b110;3 0b000000;6 r:2;8]
CS ENABLED
WRITE: 0x06;3 
WRITE: 0x00;6 
READ: 0x12 0x34 
CS DISABLED
3WIRE>w
Power supplies OFF
3WIRE>W
Power supplies ON
3WIRE>[0b110;3 0b000000;6 r;16]  // another way to read 16 bits
CS ENABLED
WRITE: 0x06;3 
WRITE: 0x00;6 
READ: 0x1234 
CS DISABLED

No wonder the chip wasn't accepting any data. Lessons learned:

  • Find the right datasheet. Be excruciatingly exact. A single incorrect letter or digit can waste a lot of your time.
  • Hardware doesn't return error codes. Either it works or you made a mistake. Figuring out where you went wrong is hard.

Monday, December 31, 2012

The Law of Comparative Advantage

Heard about Comparative Advantage on the CBC Ideas program, episode The Screw That Changed The World. The colourful analogy was: although Winston Churchill was a good bricklayer, he is better off to run the country and pay someone else to build his walls. Similarly, the brick layer is better off building walls, whilst paying W.C. to run the country. It's food for thought for someone like myself, who likes to dabble a bit in everything.

Monday, December 24, 2012

Nokia N8 Upgrade to Belle 111.040.1511

I've been putting off any changes to my N8 until I had a lot of spare time in case anything should go wrong. Up to today, I was running a custom Belle firmware (don't remember the details) that I had modified slightly (default analog clock, fixed some errors with the little icons along the top, etc). It was reliable and fast, and I loved the customizability. But a few months ago, the Store stopped working. It would load, but I couldn't go to the screen with My Stuff to check for updates, and I couldn't download new content.

Removing the custom firmware took a little research. The generally accepted procedure is to flash the N8 with the 11.5 Scandinavian version, after which you can install whatever you like: Anna, Belle, etc. Problem is, v11.5 no longer appears in Navifirm. After some hunting, I finally found a copy on the N8FanClub site.

Flashing the N8 did not take long at all.

  • Close all applications and make a full backup using Nokia Suite.
  • Power down the N8.
  • Remove the microSD card and the SIM (this may not be really necessary because I forgot once, and nothing happened. Still, may not be a bad idea).
  • Uninstall Nokia Suite to prevent it from popping up every time the N8 is reconnected to the computer.
  • Install Phoenix.
  • Copy the Scandi firmware to Phoenix\Products\RM-596.
  • Start Phoenix, select the port the N8 is connected to.
  • Press ctrl-R to rescan.
  • Select Flashing -> Firmware Update and press the button with the three dots.
  • Choose firmware 0598984 Scandinavian DG.
  • Press the Refurbish button.
  • After programming the phone is complete, it was necessary to manually power up the phone and abort Phoenix. For some reason, it was waiting for a communications check with the phone, but the phone stayed powered down. Fortunately, the phone didn't brick. Once the N8 powered up, it was clearly running older firmware and was ok. Yay.
  • Rename the Scandi RM-596 directory, to save it for next time.
  • Download the new firmware and put it in Phoenix\Products\RM-596. At first I downloaded and flashed 111.030.609 since that was the most recent version for my phone (059C8T6), but when I realized that it was an old version of Belle, I threw it out. Reflashed to Scandi 11.5, and grabbed nam_0599218. The product code doesn't match (0599218 vs 059C8T6), but I did confirm that nam = North America. It seemed like a low risk substitute. If anything went wrong, I could just reflash, right?
  • With the new firmware in the RM-596 directory, restart Phoenix and connect the N8.
  • Select the right USB port, press ctrl-R, and select Flashing -> Firmware Update again.
  • Select the button with the three dots. The new firmware (0599218 nam) should appear).
  • This time select Update Software.
  • After about 5 minutes, the phone has been updated.
  • Restore the phone from backup, and enjoy setting up your N8 again.

The curious thing is that after I upgraded with the nam 599218 firmware, my phone still shows Product Code = 059C8T6. Unexpected, but I can live with that.