Posted on 2 Comments

Determining Boot Duration Issues on Our Ubuntu 18.04 (Bionic Beaver) Preview Images for AML-S905X-CC (Le Potato)

One of the major issues with our previous Ubuntu 18.04 preview release images was the long 5 minute boot time. We did a little debugging on that front to find out the exact reason and to derive a resolution. In this article we go over the steps that we took to help people understand how to approach similar issues. Please note that the problem is not related or relevant for official Ubuntu images for x86 and Armbian Ubuntu 18.04 images.

Background

To start off with a background of the boot process for our images, we have to begin with the disk layout for our Ubuntu images. Our images are released as a zip file of the raw block data. The raw image is 4GB in size using a MBR partition system composed of two primary partitions.

Disk Image (4GB):

  • MBR Partition System (512 Byte Sectors)
    • Empty Space – Sector 0 to 2047 (0MB to 1MB) – MBR and u-boot
    • Partition 1 – Sector 2048 to 524287 (1MB to 256MB) – FAT (EFI) – /boot
    • Partition 2 – Sector 524288 to 8191999 (256MB to End) – btrfs

When you flash this image onto a MicroSD card using dd or Win32DiskImager, you only have to flash 4GB even if the MicroSD card is much larger. Upon boot, the image has a run-once script, lc_repart_disk_once, that determines the actual MicroSD card size and re-formats the disk to make use of the empty space > 4GB.

It computes the last incomplete gigabyte (1024 ^ 3) and creates primary partition 3 in that space for a swap partition. Then it extends primary partition 2 to use all of the intermediary space. BTRFS can be resized online so there no need to reboot to extend it. The swap partition is added to /etc/fstab and turned on.

Setup

Since most of the early kernel actions happen when there isn’t a GUI, we used our trusty UART to USB adapter to get early access to the system. We connected it to the three pin 2J1 connector which is highlighted in red on the picture below.

AML-S905X-CC UART 2J1 Connector
2J1 Connector Highlighted in Red: Ground (BLACK) TX (WHITE) RX (GREEN)

We use Ubuntu/Debian internally so we ran sudo minicom -b 115200 -D /dev/ttyUSB0 on our computers after plugging in the UART cable. The baud rate for the board is set to 115200 in software. We had to disable the hardware flow control by pressing Control+A, O, Serial port setup, F.

Problem Isolation and Resolution

Ubuntu 18.04 like Ubuntu 16.04 before it uses systemd as the init system which allows for clear dependencies and parallel process execution. There are two valuable tools: journalctl for reading logs and systemd-analyze for determining the process tree that took the longest.

It takes about 20 seconds for the board to get to UART TTY prompt. We login using libre and computer as the username and password respectively. We crawled through the boot logs using sudo journalctl to find that lc_repart_disk_once was timing out after 5 minutes and getting restarted. The other way to determine this was by running sudo systemd-analyze criticial-chain. After a system is fully booted (or timed out), this will give you the process tree that took the longest time with each process itemized.

Next, we enabled debugging on the lc_repart_disk_once shell script by adding set -x to enable verbose output that can be examined via sudo journalctl -u lc_repart_disk_once. We noticed that the mkswap command in the script seems to hang for a few minutes even though it should only take a few seconds.

To trace what goes wrong with a process, we installed the handy strace utility via sudo apt-get install strace. This utility will report userspace and kernel interactions. By prefixing the mkswap command with strace a script, we were able to determine exactly what mkswap was doing.

After restoring the filesystem to its original state, we restarted the system. sudo journalctl -u lc_repart_disk_once reported that mkswap was hanging on a read from /dev/random which is a system entropy issue. The annoying thing with /dev/random is that reading from it is a blocking call when system entropy gets low and won’t unblock until system entropy recovers, which can be quite slow. We checked sudo cat /proc/sys/kernel/random/entropy_avail and sure enough it was below 100, which will cause reads from /dev/random to block.

Luckily, the Amlogic S905X found in Le Potato has a built-in hardware random number generator (RNG) and BayLibre upstreamed support for it in Linux 4.8. All that was missing is the rng-tools daemon that will back the /dev/random with the HWRNG in /dev/hwrng. By installing it via sudo apt-get install rng-tools, we were able to let mkswap finish within seconds instead of hanging on entropy.

This problem is not readily transparent or an issue with the application code. It is sometimes critical for an user or developer to understand how Linux (and system level design) works in order to develop an effective solution and not resort to workarounds like patching base utilities or working around system level problems in application level logic.

Results

With this all being said and done, we have released our Ubuntu 18.04 Preview Image 3 which now boots in two minutes to the Gnome Display Manager instead of five minutes on the first run. Second boots takes less than 45 seconds. The headless boot times have not changed from the previous 20 seconds. This is a tremendous improvement to usability.

Other Thoughts

We have started putting images in Google Drive for faster downloads. You can find them in the README.txt. Other changes in PI3 include Linux LTS 4.14.50, defaulting to Wayland in GDM for the libre user, increased compressed memory pool, and a few more resolutions. Outstanding issues include overlay implementation, upstream Linux support for 2K/4K HDMI output, and VPU work for accelerated video decoding. Ely, a community member, has contributed work towards open source hardware video decoding which is very exciting.

We expect another preview release before we have a formal release. We are currently getting infrastructure in place to host repositories for the formal release so you can sudo apt-get update && sudo apt-get upgrade to keep everything up to date instead of re-flashing MicroSD cards.

By the time the next Linux LTS rolls around (4.19 in October), we should have an unified image for all three of the current CC and CM platforms.

Posted on Leave a comment

Ubuntu 18.04 (Bionic Beaver) LTS for AML-S905X-CC (Le Potato) Preview Image 2

This is a quick update to the previous preview release 1 with a newer 4.14.49 kernel.

We fixed the USB/mali conflict and the USB ports will not hang if something is plugged into the OTG port while mali is enabled.

The reason for the long boot process is due mkswap taking what seems like forever on 18.04. We have not isolated the cause yet so we will continue to address that issue as well as many other kernel side features.

We will be creating a linux flashing utility to this platform for eMMC much akin to the fel-mass-storage utility for Allwinner boards. You will be able to directly flash SD, eMMC, and NOR via a Type A to Type A cable that is included with our eMMC modules.

Le Potato will be getting a little brother soon…

Posted on 1 Comment

GPIO Headers Reference for AML-S905X-CC

GPIOs (General Purpose Input/Output) are single-bit pins capable of digital input or output typically used for controlling LEDs or signaling. There is a total of 35 GPIO pins operating at 3.3V logic level on Le Potato which is 7 more than what is found on the Raspberry Pi series of boards. GPIOs are half duplex which means they can only be in either input or output mode but not both simultaneously.

When setup for input, applying 3.3V or 0V to the pin will correspond to 1 or 0 respectively when read in software. When setup for output, the pins will be either 3.3V or 0V depending on whether it is set as 1 or 0 in software. GPIOs typically provide only a few milli-amps of current so they should only directly drive low power things like LEDs. They cannot drive power hungry things like DC motors, which need power transistors to deliver adequate current.

Some of the GPIO pins have secondary functionality for signaling SPDIF, I2S, I2C, SPI, SDIO, UART, PCM, clock-generation, and more. They still operate at the same 3.3V logic level but these pins are usually connected to internal specialized hardware that can generate precisely timed signals without using the CPU to bit bang. This frees up the CPU to do other things like running your operating system and software.

There are two interfaces in the Linux kernel for controlling and reading from GPIO pins. Since the interfaces are standardized, you do not need to use libraries to read from and write to GPIOs. We have put together a simple guide for utilizing these interfaces. As an addition, we also mapped out the functionality and software pin numbers on Le Potato for easy access.

Screenshot from 2018-05-21 11-48-00

On the picture at the top of the page, GPIOs on the Le Potato are highlighted in green. 5V pins are red, 3.3V pins are orange, ADC pins are blue. Other un-highlighted pins are ground with the exception of the pins on the top right header.

Posted on 3 Comments

Ubuntu 18.04 (Bionic Beaver) LTS for AML-S905X-CC (Le Potato) Preview Image 1 with Wayland

We are happy to announce our first preview image for Ubuntu 18.04 Bionic Beaver. It features the latest Linux tools and utilities along with Gnome 3 as the replacement for Unity. They back tracked on defaulting to Wayland from 17.10 due to maturity issues but we are excited to bring some GBM groundwork in the Linux kernel for you to play with Wayland.

We recommend testing new LTS releases to get a feel for the changes and upgrading only after its first point release since new software is often different, unstable, and bug ridden. For stability, stay with Ubuntu 16.04 which is on its fourth point release (denoted by 16.04.4) and on well on its way to a fifth point release.

This Bionic PI1 image release is for people to play with Wayland/Weston on top of Gnome 3’s Mutter window manager on Le Potato. This image takes around 5 minutes for first boot because it has to re-partition and create swap files so be patient and don’t interrupt it. When you get to the login screen and select the Libre Computer user, make sure to click the little settings icon (looks like a gear) and select “Ubuntu on Wayland”. From there, you can download es2gears_wayland (sudo apt-get install mesa-utils-extra) and glmark2-es2-wayland (sudo apt-get install glmark2-es2-wayland). Le Potato should be about 50% faster than the Raspberry Pi 3 Model B+ in glmark2. Please note that the 3D acceleration feature is neither stable or fully featured at this point and a lot of work remains. Having an open-source Mali driver like lima would help greatly. The hardware is OpenGL ES 2.0 only so needs something like gl4es shim for applications that rely on OpenGL.

The other big change in Bionic is the move away from /etc/network/interfaces and to netplan which renders configurations to other backends like interfaces and NetworkManager. We have included the appropriate yaml files for the headless and desktop images. Ubuntu also removed ifconfig so you have to use the ip command instead.

Major Changes:

  • Unity -> Gnome 3 (Mutter WM)
  • /etc/network/interfaces -> netplan
  • Linux Kernel 4.14.38 with GBM for Mali
  • Virtualization and container accounting improvements
  • Default to disable UHS and CVBS for compatibility

You can still use the lc_distro_transfer tool for flashing eMMC from MicroSD card just like with the Ubuntu 16.04 Preview Image 8. The image can be downloaded here.

Known Issues:

  • Top left USB port (OTG USB) conflicts with Mali driver and will cause all USB ports to stop working if you plug in a USB device into that port, blacklist the mali module before using the port
  • Mutter packages are held back from upgrading automatically due to out-of-tree patches
  • ARM64 Linux kernel are not part of Ubuntu ports repository, a separate repository needs to be set up in order to enable automatic kernel updates
  • No video for monitors and TVs with 1366×768 and some other odd resolutions

Other Project Notes:

Tritium boards will be mailed out next week for Kickstarter backers and we will begin the engineering effort for unifying board support in software so you can switch between ALL-H3-CC H5, AML-S905X-CC, and ROC-RK3328-CC with one command. Hopefully by the end of 2018, the fruits of our software efforts will offer a seemless experience on all of our boards.

Posted on 2 Comments

The Sub-1-Watt 64-bit Ubuntu Linux ARM Server

Today, we follow up the release of our Ubuntu desktop image with our headless server image for AML-S905X-CC. The headless server image is based on the latest Linux LTS 4.14 along with some of our customizations. It utilizes the same infrastructure we’ve created for our desktop image while consuming considerably less power.

  • Linux 4.14 LTS
  • u-boot 2017.11
  • Custom Partition Layout
    • 256MB FAT EFI Partition
    • BTRFS Partition
      • Copy-on-Write for greater reliability
      • Facebook’s zstd compression
      • @ root subvolume and @lc-ubuntu-16-headless release snapshot
      • Self expands on startup to full disk size
    • Auto generated Swap
      • 1-2GB in side located at end of disk
      • offload pages from zswap

In testing, power consumption is around 180mA with network, eMMC, and microSD card connected for a total consumption of 0.91W! Did we mention this thing has quad 64-bit ARM Cortex-A53 processors? Just to give you an idea, your “energy efficient” 13W LED light bulb uses the same power as 15 of these servers.

We performed no optimization or other funny business in our images to achieve these results. You can get a Le Potato yourself and compile a mainline kernel to verify the results. If you have one already, you can download the latest images here. If you are serious about minimizing power consumption, more saving can probably be extracted.

Posted on 8 Comments

AML-S905X-CC Mainline Linux Preview Image 8 with eMMC Support

It has been over three months since our last preview image 7 for the AML-S905X-CC Le Potato platform and there has been a lot of working going into mainline Linux and u-boot by our partner BayLibre. Preview image 8 brings all of the work together into a flash-able image for our end-users.

PI8 Core feature upgrades:

  • Upgraded to mainline u-boot 2017.11
    • No longer using Amlogic’s u-boot
    • eMMC support
    • Saving u-boot environment
    • cvbs enablement through UART via “setenv enable-cvbs 1; env save”
  • Upgraded to mainline Linux LTS 4.14.29
    • New DMT Display Modes support
      • 640×480@60Hz
      • 800×600@60Hz
      • 1024×768@60Hz
      • 1152×864@75Hz
      • 1280×1024@60Hz
      • 1600×1200@60Hz
      • 1920×1080@60Hz
    • Fixed top USB Port by Ethernet causing USB subsystem to hang
    • Utilize kernel and kernel module deb package install
    • Power optimizations
  • Additional image work
    • eMMC support via lc_distro_transfer utility
      • Reworked fstab mount descriptors
      • Reworked partition resize utility
    • Preliminary EFI support
    • Headless and XFCE Desktop snapshots in one image

Some known issues with this release and additional improvement work:

  • shutdown/suspend/wakeup
  • MicroSD card may hang on reboot command
  • package u-boot and board specific utilities and binaries into deb files
  • add repository to apt for board support deb files
  • display pipeline improvements and rework for additional resolutions
  • wayland support
  • Ubuntu 18.04 LTS
  • video codec acceleration support
  • overlay auto-loading in u-boot
  • eMMC performance improvements
  • EFI support improvements

eMMC Support

eMMC modules are solid-state flash devices created for embedded systems. They offer higher reliability and additional performance compared to MicroSD cards. They are purchased separately and attached to the eMMC connector on the bottom side of the board.

We currently have eMMC 5.x modules and eMMC 4.x modules. You can attach an eMMC 5.x module on a board that supports eMMC 5.x only. If you attach an eMMC 5.x module on a board with only eMMC 4.x support like the ALL-H3-CC, it will not work. The reverse is also true. The performance differences between the two module types are small to negligible.

lc_distro_transfer utility

This image includes the release state of our distro as snapshots on top of the BTRFS filesystem. We added a new script called lc_distro_transfer that utilizes core design features of our image to transfer system snapshots created during image building to eMMC and back to MicroSD card.

sudo lc_distro_transfer –help
USAGE: lc_distro_transfer VENDOR/MODEL DEVICE DEVICE_TYPE DISTRO [FIRMWARE_TYPE]

For example, you can flash the headless or desktop image from a MicroSD card to eMMC after booting up. After login and boot-up, simply run the following command to flash the headless image to eMMC:

sudo lc_distro_transfer libre-computer/aml-s905x-cc /dev/mmcblk0 emmc lc-ubuntu-16-headless

If you want to flash the desktop image to eMMC, run the following:

sudo lc_distro_transfer libre-computer/aml-s905x-cc /dev/mmcblk0 emmc lc-ubuntu-16-xfce

Once the image is flashed to eMMC, you can shutdown and remove the MicroSD card. The devices will then boot to eMMC. If you want to flash a blank MicroSD card from eMMC, you can run the following:

sudo lc_distro_transfer libre-computer/aml-s905x-cc /dev/mmcblk1 sd lc-ubuntu-16-xfce

Please note that you can only flash the image that you original flashed to the eMMC. You cannot flash the headless image back onto the MicroSD card if you originally flashed the xfce image onto the eMMC.

The image release can be downloaded here.

You can let us know your thoughts on issues for this image on the LoveRPi forum. Once Ubuntu 18.04 LTS is out, we will have a lot of additional work lined up such as Wayland and Mali 3D support.

Posted on Leave a comment

First Preview Image for ROC-RK3328-CC (Renegade)

We are happy to announce the first preview image for the ROC-RK3328-CC (Renegade) board that was released on Indiegogo courtesy of the Firefly team.

You can find the latest downloads for ROC-RK3328-CC on the product page’s Downloads section.

The first image is based on Rockchip’s Linux 4.4 LTS kernel running Ubuntu 16.04 LTS. It currently does not offer Mali 3D acceleration or hardware codec support.

Android will be coming within the next two weeks.

Once Ubuntu 18.04 is released, we will begin work on a repository for bootloaders and kernels housed in deb files so that bootstrapping images will be easy for all of our boards. Once that infrastructure is in place, non-preview images for our boards will be released. Updating bootloaders and kernels will be as simple as an apt-get.

Posted on 13 Comments

Retro Gaming on AML-S905X-CC with Lakka!

One of the most popular uses for single board computers is playing retro games. AML-S905X-CC now has a Lakka image (Thanks kszaq for the tip and the Lakka team for their great work!) If you don’t know what Lakka is, it is basically one of two popular RetroArch distros, the other being RetroPie. Lakka is built on top of LibreELEC and has a very friendly UI that lets you dive into retro gaming very quickly. Hopefully we get formal support soon but we are very excited to have this feature.

AML-S905X-CC is one of our fastest low cost and low power platforms. It should run emulators significantly faster than the Raspberry Pi 3. Please try it out and let us know your thoughts!

Posted on 4 Comments

AML-S905X-CC Mainline Linux Preview Image 7

After much ado, the AML-S905X-CC ethernet issue has finally been resolved. This images has the following fixes/features:

  • Swap partition generation (1GB-2GB in size determined by SD card size)
  • Fixed ethernet EEE causing link drop
  • Updated to Linux kernel 4.14.11
  • Overlay support in the Linux kernel
  • Zswap for larger memory foot-print applications

Issues still outstanding for official image release:

  • shutdown/suspend/wakeup
  • add kernel, u-boot, and board specific binaries into deb files
  • add repository to apt for board support deb files
  • display pipeline support
  • wayland support
  • codec acceleration support
  • overlay auto-loading in u-boot
  • boot device detection and selection
  • eMMC support

The image release can be downloaded here.

Posted on 2 Comments

Get your Kodi on! LibreELEC for AML-S905X-CC by kszaq and Image Preview Release 6

LibreELEC fork developer kszaq recently made images for AML-S905X-CC (Le Potato) based on LibreELEC 8.x. He hopes that his work will eventually be upstreamed to LibreELEC. This is wonderful news as it features hardware acceleration built around Linux kernel 3.14. This is a nice Christmas present for everybody and we greatly appreciate your work kszaq! You can now get Kodi media center on Linux running on our boards.

Community is a hard thing to get right and we are glad to have people contribute to the ecosystem. We are focused on proper upstream support for the technologies we use in hopes that the work will have a great impact on the open-source community and people building on top of it.

We hope to have Wayland and V4L2 support in mainline Linux with BayLibre’s assistance in 2018 so that we can run the latest and greatest like Ubuntu 18.04. Ubuntu Linux preview image 6 is available based on Linux 4.14.7 LTS.

Hope you all have a jolly holiday season and see you next year! We will be seeing what’s new at CES 2018. Hopefully we will see you there!