Since we started our endeavor, we have made tremendous progress on the software side of the AML-S905X-CC (Le Potato) platform through our collaboration with BayLibre. We are happy to announce that the platform will be getting a mini-me, the AML-S805X-AC (La Frite). This smaller and more cost-oriented board shares the same underlying technology as its bigger brother and will utilize the upstream work completed so far. It is limited to 1080P instead of 4K60 like Le Potato. The crowdfunding will occur at the end of July after we release the images for ALL-H3-CC series (Tritium). This will coincide with the Indiegogo campaign for ROC-RK3399-PC (Renegade Elite). More details to come about both of these projects.
Meanwhile, we have released a new preview image for the current AML-S905X-CC. This image fixes two outstanding issues:
floating MAC address causing new DHCP IP leases
support for automatically generating timings for DMT resolutions
The images slipstreams the latest Linux 4.14.52. If you already have preview image 2, this is not much of an upgrade.
We have been quietly working on support for Raspbian. The work is relatively simple and we have been testing it internally for release in the next 10 days. It will come in two forms: an image and a script to install on top of existing Raspbian for Raspberry Pi. This will work for both Le Potato and Tritium platforms.
LXDE is much faster than Gnome 3 on these low power ARM hardware. It is the basis for the latest desktop UI for Raspbian. We will start rolling Ubuntu with LXDE on X11 along side the Gnome 3 with Wayland images for usability reasons.
Upstream roadmap items being worked on include the following:
4K30 support in Linux (4K60 requires a lot of work still)
single resolution video output and framebuffer in u-boot
SPI-FC support for AML-S805X-AC in u-boot
DRM overlays in Linux
Device tree overlays (like HAT support in Raspbian for Raspberry Pi boards)
Logistical roadmap items being worked on include the following:
Unified kernel for ALL-H3-CC H5, AML-S905X-CC, ROC-RK3328-CC based on Linux 4.18
Debian repo infrastructure and packaging for kernel and u-boot for all of our boards
Continuous integration and image buildbot
These two lists are not comprehensive. It will be a fun summer with lots to do.
One of the major issues with our previous Ubuntu 18.04 preview release images was the long 5 minute boot time. We did a little debugging on that front to find out the exact reason and to derive a resolution. In this article we go over the steps that we took to help people understand how to approach similar issues. Please note that the problem is not related or relevant for official Ubuntu images for x86 and Armbian Ubuntu 18.04 images.
To start off with a background of the boot process for our images, we have to begin with the disk layout for our Ubuntu images. Our images are released as a zip file of the raw block data. The raw image is 4GB in size using a MBR partition system composed of two primary partitions.
Disk Image (4GB):
MBR Partition System (512 Byte Sectors)
Empty Space – Sector 0 to 2047 (0MB to 1MB) – MBR and u-boot
Partition 1 – Sector 2048 to 524287 (1MB to 256MB) – FAT (EFI) – /boot
Partition 2 – Sector 524288 to 8191999 (256MB to End) – btrfs
When you flash this image onto a MicroSD card using dd or Win32DiskImager, you only have to flash 4GB even if the MicroSD card is much larger. Upon boot, the image has a run-once script, lc_repart_disk_once, that determines the actual MicroSD card size and re-formats the disk to make use of the empty space > 4GB.
It computes the last incomplete gigabyte (1024 ^ 3) and creates primary partition 3 in that space for a swap partition. Then it extends primary partition 2 to use all of the intermediary space. BTRFS can be resized online so there no need to reboot to extend it. The swap partition is added to /etc/fstab and turned on.
Since most of the early kernel actions happen when there isn’t a GUI, we used our trusty UART to USB adapter to get early access to the system. We connected it to the three pin 2J1 connector which is highlighted in red on the picture below.
We use Ubuntu/Debian internally so we ran sudo minicom -b 115200 -D /dev/ttyUSB0 on our computers after plugging in the UART cable. The baud rate for the board is set to 115200 in software. We had to disable the hardware flow control by pressing Control+A, O, Serial port setup, F.
Problem Isolation and Resolution
Ubuntu 18.04 like Ubuntu 16.04 before it uses systemd as the init system which allows for clear dependencies and parallel process execution. There are two valuable tools: journalctl for reading logs and systemd-analyze for determining the process tree that took the longest.
It takes about 20 seconds for the board to get to UART TTY prompt. We login using libre and computer as the username and password respectively. We crawled through the boot logs using sudo journalctl to find that lc_repart_disk_once was timing out after 5 minutes and getting restarted. The other way to determine this was by running sudo systemd-analyze criticial-chain. After a system is fully booted (or timed out), this will give you the process tree that took the longest time with each process itemized.
Next, we enabled debugging on the lc_repart_disk_once shell script by adding set -x to enable verbose output that can be examined via sudo journalctl -u lc_repart_disk_once. We noticed that the mkswap command in the script seems to hang for a few minutes even though it should only take a few seconds.
To trace what goes wrong with a process, we installed the handy strace utility via sudo apt-get install strace. This utility will report userspace and kernel interactions. By prefixing the mkswap command with strace a script, we were able to determine exactly what mkswap was doing.
After restoring the filesystem to its original state, we restarted the system. sudo journalctl -u lc_repart_disk_once reported that mkswap was hanging on a read from /dev/random which is a system entropy issue. The annoying thing with /dev/random is that reading from it is a blocking call when system entropy gets low and won’t unblock until system entropy recovers, which can be quite slow. We checked sudo cat /proc/sys/kernel/random/entropy_avail and sure enough it was below 100, which will cause reads from /dev/random to block.
Luckily, the Amlogic S905X found in Le Potato has a built-in hardware random number generator (RNG) and BayLibre upstreamed support for it in Linux 4.8. All that was missing is the rng-tools daemon that will back the /dev/random with the HWRNG in /dev/hwrng. By installing it via sudo apt-get install rng-tools, we were able to let mkswap finish within seconds instead of hanging on entropy.
This problem is not readily transparent or an issue with the application code. It is sometimes critical for an user or developer to understand how Linux (and system level design) works in order to develop an effective solution and not resort to workarounds like patching base utilities or working around system level problems in application level logic.
With this all being said and done, we have released our Ubuntu 18.04 Preview Image 3 which now boots in two minutes to the Gnome Display Manager instead of five minutes on the first run. Second boots takes less than 45 seconds. The headless boot times have not changed from the previous 20 seconds. This is a tremendous improvement to usability.
We have started putting images in Google Drive for faster downloads. You can find them in the README.txt. Other changes in PI3 include Linux LTS 4.14.50, defaulting to Wayland in GDM for the libre user, increased compressed memory pool, and a few more resolutions. Outstanding issues include overlay implementation, upstream Linux support for 2K/4K HDMI output, and VPU work for accelerated video decoding. Ely, a community member, has contributed work towards open source hardware video decoding which is very exciting.
We expect another preview release before we have a formal release. We are currently getting infrastructure in place to host repositories for the formal release so you can sudo apt-get update && sudo apt-get upgrade to keep everything up to date instead of re-flashing MicroSD cards.
By the time the next Linux LTS rolls around (4.19 in October), we should have an unified image for all three of the current CC and CM platforms.
This is a quick update to the previous preview release 1 with a newer 4.14.49 kernel.
We fixed the USB/mali conflict and the USB ports will not hang if something is plugged into the OTG port while mali is enabled.
The reason for the long boot process is due mkswap taking what seems like forever on 18.04. We have not isolated the cause yet so we will continue to address that issue as well as many other kernel side features.
We will be creating a linux flashing utility to this platform for eMMC much akin to the fel-mass-storage utility for Allwinner boards. You will be able to directly flash SD, eMMC, and NOR via a Type A to Type A cable that is included with our eMMC modules.
GPIOs (General Purpose Input/Output) are single-bit pins capable of digital input or output typically used for controlling LEDs or signaling. There is a total of 35 GPIO pins operating at 3.3V logic level on Le Potato which is 7 more than what is found on the Raspberry Pi series of boards. GPIOs are half duplex which means they can only be in either input or output mode but not both simultaneously.
When setup for input, applying 3.3V or 0V to the pin will correspond to 1 or 0 respectively when read in software. When setup for output, the pins will be either 3.3V or 0V depending on whether it is set as 1 or 0 in software. GPIOs typically provide only a few milli-amps of current so they should only directly drive low power things like LEDs. They cannot drive power hungry things like DC motors, which need power transistors to deliver adequate current.
Some of the GPIO pins have secondary functionality for signaling SPDIF, I2S, I2C, SPI, SDIO, UART, PCM, clock-generation, and more. They still operate at the same 3.3V logic level but these pins are usually connected to internal specialized hardware that can generate precisely timed signals without using the CPU to bit bang. This frees up the CPU to do other things like running your operating system and software.
On the picture at the top of the page, GPIOs on the Le Potato are highlighted in green. 5V pins are red, 3.3V pins are orange, ADC pins are blue. Other un-highlighted pins are ground with the exception of the pins on the top right header.
We recommend testing new LTS releases to get a feel for the changes and upgrading only after its first point release since new software is often different, unstable, and bug ridden. For stability, stay with Ubuntu 16.04 which is on its fourth point release (denoted by 16.04.4) and on well on its way to a fifth point release.
This Bionic PI1 image release is for people to play with Wayland/Weston on top of Gnome 3’s Mutter window manager on Le Potato. This image takes around 5 minutes for first boot because it has to re-partition and create swap files so be patient and don’t interrupt it. When you get to the login screen and select the Libre Computer user, make sure to click the little settings icon (looks like a gear) and select “Ubuntu on Wayland”. From there, you can download es2gears_wayland (sudo apt-get install mesa-utils-extra) and glmark2-es2-wayland (sudo apt-get install glmark2-es2-wayland). Le Potato should be about 50% faster than the Raspberry Pi 3 Model B+ in glmark2. Please note that the 3D acceleration feature is neither stable or fully featured at this point and a lot of work remains. Having an open-source Mali driver like lima would help greatly. The hardware is OpenGL ES 2.0 only so needs something like gl4es shim for applications that rely on OpenGL.
The other big change in Bionic is the move away from /etc/network/interfaces and to netplan which renders configurations to other backends like interfaces and NetworkManager. We have included the appropriate yaml files for the headless and desktop images. Ubuntu also removed ifconfig so you have to use the ip command instead.
Unity -> Gnome 3 (Mutter WM)
/etc/network/interfaces -> netplan
Linux Kernel 4.14.38 with GBM for Mali
Virtualization and container accounting improvements
Top left USB port (OTG USB) conflicts with Mali driver and will cause all USB ports to stop working if you plug in a USB device into that port, blacklist the mali module before using the port
Mutter packages are held back from upgrading automatically due to out-of-tree patches
ARM64 Linux kernel are not part of Ubuntu ports repository, a separate repository needs to be set up in order to enable automatic kernel updates
No video for monitors and TVs with 1366×768 and some other odd resolutions
Other Project Notes:
Tritium boards will be mailed out next week for Kickstarter backers and we will begin the engineering effort for unifying board support in software so you can switch between ALL-H3-CC H5, AML-S905X-CC, and ROC-RK3328-CC with one command. Hopefully by the end of 2018, the fruits of our software efforts will offer a seemless experience on all of our boards.
Today, we follow up the release of our Ubuntu desktop image with our headless server image for AML-S905X-CC. The headless server image is based on the latest Linux LTS 4.14 along with some of our customizations. It utilizes the same infrastructure we’ve created for our desktop image while consuming considerably less power.
Linux 4.14 LTS
Custom Partition Layout
256MB FAT EFI Partition
Copy-on-Write for greater reliability
Facebook’s zstd compression
@ root subvolume and @lc-ubuntu-16-headless release snapshot
Self expands on startup to full disk size
Auto generated Swap
1-2GB in side located at end of disk
offload pages from zswap
In testing, power consumption is around 180mA with network, eMMC, and microSD card connected for a total consumption of 0.91W! Did we mention this thing has quad 64-bit ARM Cortex-A53 processors? Just to give you an idea, your “energy efficient” 13W LED light bulb uses the same power as 15 of these servers.
We performed no optimization or other funny business in our images to achieve these results. You can get a Le Potato yourself and compile a mainline kernel to verify the results. If you have one already, you can download the latest images here. If you are serious about minimizing power consumption, more saving can probably be extracted.
It has been over three months since our last preview image 7 for the AML-S905X-CC Le Potato platform and there has been a lot of working going into mainline Linux and u-boot by our partner BayLibre. Preview image 8 brings all of the work together into a flash-able image for our end-users.
PI8 Core feature upgrades:
Upgraded to mainline u-boot 2017.11
No longer using Amlogic’s u-boot
Saving u-boot environment
cvbs enablement through UART via “setenv enable-cvbs 1; env save”
Upgraded to mainline Linux LTS 4.14.29
New DMT Display Modes support
Fixed top USB Port by Ethernet causing USB subsystem to hang
Utilize kernel and kernel module deb package install
Additional image work
eMMC support via lc_distro_transfer utility
Reworked fstab mount descriptors
Reworked partition resize utility
Preliminary EFI support
Headless and XFCE Desktop snapshots in one image
Some known issues with this release and additional improvement work:
MicroSD card may hang on reboot command
package u-boot and board specific utilities and binaries into deb files
add repository to apt for board support deb files
display pipeline improvements and rework for additional resolutions
Ubuntu 18.04 LTS
video codec acceleration support
overlay auto-loading in u-boot
eMMC performance improvements
EFI support improvements
eMMC modules are solid-state flash devices created for embedded systems. They offer higher reliability and additional performance compared to MicroSD cards. They are purchased separately and attached to the eMMC connector on the bottom side of the board.
We currently have eMMC 5.x modules and eMMC 4.x modules. You can attach an eMMC 5.x module on a board that supports eMMC 5.x only. If you attach an eMMC 5.x module on a board with only eMMC 4.x support like the ALL-H3-CC, it will not work. The reverse is also true. The performance differences between the two module types are small to negligible.
This image includes the release state of our distro as snapshots on top of the BTRFS filesystem. We added a new script called lc_distro_transfer that utilizes core design features of our image to transfer system snapshots created during image building to eMMC and back to MicroSD card.
Please note that you can only flash the image that you original flashed to the eMMC. You cannot flash the headless image back onto the MicroSD card if you originally flashed the xfce image onto the eMMC.
The first image is based on Rockchip’s Linux 4.4 LTS kernel running Ubuntu 16.04 LTS. It currently does not offer Mali 3D acceleration or hardware codec support.
Android will be coming within the next two weeks.
Once Ubuntu 18.04 is released, we will begin work on a repository for bootloaders and kernels housed in deb files so that bootstrapping images will be easy for all of our boards. Once that infrastructure is in place, non-preview images for our boards will be released. Updating bootloaders and kernels will be as simple as an apt-get.
One of the most popular uses for single board computers is playing retro games. AML-S905X-CC now has a Lakka image (Thanks kszaq for the tip and the Lakka team for their great work!) If you don’t know what Lakka is, it is basically one of two popular RetroArch distros, the other being RetroPie. Lakka is built on top of LibreELEC and has a very friendly UI that lets you dive into retro gaming very quickly. Hopefully we get formal support soon but we are very excited to have this feature.
AML-S905X-CC is one of our fastest low cost and low power platforms. It should run emulators significantly faster than the Raspberry Pi 3. Please try it out and let us know your thoughts!