Since we started our endeavor, we have made tremendous progress on the software side of the AML-S905X-CC (Le Potato) platform through our collaboration with BayLibre. We are happy to announce that the platform will be getting a mini-me, the AML-S805X-AC (La Frite). This smaller and more cost-oriented board shares the same underlying technology as its bigger brother and will utilize the upstream work completed so far. It is limited to 1080P instead of 4K60 like Le Potato. The crowdfunding will occur at the end of July after we release the images for ALL-H3-CC series (Tritium). This will coincide with the Indiegogo campaign for ROC-RK3399-PC (Renegade Elite). More details to come about both of these projects.
Meanwhile, we have released a new preview image for the current AML-S905X-CC. This image fixes two outstanding issues:
floating MAC address causing new DHCP IP leases
support for automatically generating timings for DMT resolutions
The images slipstreams the latest Linux 4.14.52. If you already have preview image 2, this is not much of an upgrade.
We have been quietly working on support for Raspbian. The work is relatively simple and we have been testing it internally for release in the next 10 days. It will come in two forms: an image and a script to install on top of existing Raspbian for Raspberry Pi. This will work for both Le Potato and Tritium platforms.
LXDE is much faster than Gnome 3 on these low power ARM hardware. It is the basis for the latest desktop UI for Raspbian. We will start rolling Ubuntu with LXDE on X11 along side the Gnome 3 with Wayland images for usability reasons.
Upstream roadmap items being worked on include the following:
4K30 support in Linux (4K60 requires a lot of work still)
single resolution video output and framebuffer in u-boot
SPI-FC support for AML-S805X-AC in u-boot
DRM overlays in Linux
Device tree overlays (like HAT support in Raspbian for Raspberry Pi boards)
Logistical roadmap items being worked on include the following:
Unified kernel for ALL-H3-CC H5, AML-S905X-CC, ROC-RK3328-CC based on Linux 4.18
Debian repo infrastructure and packaging for kernel and u-boot for all of our boards
Continuous integration and image buildbot
These two lists are not comprehensive. It will be a fun summer with lots to do.
One of the major issues with our previous Ubuntu 18.04 preview release images was the long 5 minute boot time. We did a little debugging on that front to find out the exact reason and to derive a resolution. In this article we go over the steps that we took to help people understand how to approach similar issues. Please note that the problem is not related or relevant for official Ubuntu images for x86 and Armbian Ubuntu 18.04 images.
To start off with a background of the boot process for our images, we have to begin with the disk layout for our Ubuntu images. Our images are released as a zip file of the raw block data. The raw image is 4GB in size using a MBR partition system composed of two primary partitions.
Disk Image (4GB):
MBR Partition System (512 Byte Sectors)
Empty Space – Sector 0 to 2047 (0MB to 1MB) – MBR and u-boot
Partition 1 – Sector 2048 to 524287 (1MB to 256MB) – FAT (EFI) – /boot
Partition 2 – Sector 524288 to 8191999 (256MB to End) – btrfs
When you flash this image onto a MicroSD card using dd or Win32DiskImager, you only have to flash 4GB even if the MicroSD card is much larger. Upon boot, the image has a run-once script, lc_repart_disk_once, that determines the actual MicroSD card size and re-formats the disk to make use of the empty space > 4GB.
It computes the last incomplete gigabyte (1024 ^ 3) and creates primary partition 3 in that space for a swap partition. Then it extends primary partition 2 to use all of the intermediary space. BTRFS can be resized online so there no need to reboot to extend it. The swap partition is added to /etc/fstab and turned on.
Since most of the early kernel actions happen when there isn’t a GUI, we used our trusty UART to USB adapter to get early access to the system. We connected it to the three pin 2J1 connector which is highlighted in red on the picture below.
We use Ubuntu/Debian internally so we ran sudo minicom -b 115200 -D /dev/ttyUSB0 on our computers after plugging in the UART cable. The baud rate for the board is set to 115200 in software. We had to disable the hardware flow control by pressing Control+A, O, Serial port setup, F.
Problem Isolation and Resolution
Ubuntu 18.04 like Ubuntu 16.04 before it uses systemd as the init system which allows for clear dependencies and parallel process execution. There are two valuable tools: journalctl for reading logs and systemd-analyze for determining the process tree that took the longest.
It takes about 20 seconds for the board to get to UART TTY prompt. We login using libre and computer as the username and password respectively. We crawled through the boot logs using sudo journalctl to find that lc_repart_disk_once was timing out after 5 minutes and getting restarted. The other way to determine this was by running sudo systemd-analyze criticial-chain. After a system is fully booted (or timed out), this will give you the process tree that took the longest time with each process itemized.
Next, we enabled debugging on the lc_repart_disk_once shell script by adding set -x to enable verbose output that can be examined via sudo journalctl -u lc_repart_disk_once. We noticed that the mkswap command in the script seems to hang for a few minutes even though it should only take a few seconds.
To trace what goes wrong with a process, we installed the handy strace utility via sudo apt-get install strace. This utility will report userspace and kernel interactions. By prefixing the mkswap command with strace a script, we were able to determine exactly what mkswap was doing.
After restoring the filesystem to its original state, we restarted the system. sudo journalctl -u lc_repart_disk_once reported that mkswap was hanging on a read from /dev/random which is a system entropy issue. The annoying thing with /dev/random is that reading from it is a blocking call when system entropy gets low and won’t unblock until system entropy recovers, which can be quite slow. We checked sudo cat /proc/sys/kernel/random/entropy_avail and sure enough it was below 100, which will cause reads from /dev/random to block.
Luckily, the Amlogic S905X found in Le Potato has a built-in hardware random number generator (RNG) and BayLibre upstreamed support for it in Linux 4.8. All that was missing is the rng-tools daemon that will back the /dev/random with the HWRNG in /dev/hwrng. By installing it via sudo apt-get install rng-tools, we were able to let mkswap finish within seconds instead of hanging on entropy.
This problem is not readily transparent or an issue with the application code. It is sometimes critical for an user or developer to understand how Linux (and system level design) works in order to develop an effective solution and not resort to workarounds like patching base utilities or working around system level problems in application level logic.
With this all being said and done, we have released our Ubuntu 18.04 Preview Image 3 which now boots in two minutes to the Gnome Display Manager instead of five minutes on the first run. Second boots takes less than 45 seconds. The headless boot times have not changed from the previous 20 seconds. This is a tremendous improvement to usability.
We have started putting images in Google Drive for faster downloads. You can find them in the README.txt. Other changes in PI3 include Linux LTS 4.14.50, defaulting to Wayland in GDM for the libre user, increased compressed memory pool, and a few more resolutions. Outstanding issues include overlay implementation, upstream Linux support for 2K/4K HDMI output, and VPU work for accelerated video decoding. Ely, a community member, has contributed work towards open source hardware video decoding which is very exciting.
We expect another preview release before we have a formal release. We are currently getting infrastructure in place to host repositories for the formal release so you can sudo apt-get update && sudo apt-get upgrade to keep everything up to date instead of re-flashing MicroSD cards.
By the time the next Linux LTS rolls around (4.19 in October), we should have an unified image for all three of the current CC and CM platforms.
This is a quick update to the previous preview release 1 with a newer 4.14.49 kernel.
We fixed the USB/mali conflict and the USB ports will not hang if something is plugged into the OTG port while mali is enabled.
The reason for the long boot process is due mkswap taking what seems like forever on 18.04. We have not isolated the cause yet so we will continue to address that issue as well as many other kernel side features.
We will be creating a linux flashing utility to this platform for eMMC much akin to the fel-mass-storage utility for Allwinner boards. You will be able to directly flash SD, eMMC, and NOR via a Type A to Type A cable that is included with our eMMC modules.
GPIOs (General Purpose Input/Output) are single-bit pins capable of digital input or output typically used for controlling LEDs or signaling. There is a total of 35 GPIO pins operating at 3.3V logic level on Le Potato which is 7 more than what is found on the Raspberry Pi series of boards. GPIOs are half duplex which means they can only be in either input or output mode but not both simultaneously.
When setup for input, applying 3.3V or 0V to the pin will correspond to 1 or 0 respectively when read in software. When setup for output, the pins will be either 3.3V or 0V depending on whether it is set as 1 or 0 in software. GPIOs typically provide only a few milli-amps of current so they should only directly drive low power things like LEDs. They cannot drive power hungry things like DC motors, which need power transistors to deliver adequate current.
Some of the GPIO pins have secondary functionality for signaling SPDIF, I2S, I2C, SPI, SDIO, UART, PCM, clock-generation, and more. They still operate at the same 3.3V logic level but these pins are usually connected to internal specialized hardware that can generate precisely timed signals without using the CPU to bit bang. This frees up the CPU to do other things like running your operating system and software.
On the picture at the top of the page, GPIOs on the Le Potato are highlighted in green. 5V pins are red, 3.3V pins are orange, ADC pins are blue. Other un-highlighted pins are ground with the exception of the pins on the top right header.
ROC-RK3328-CC (Renegade) is a powerful SBC platform powered by the Rockchip RK3328 SoC and equipped with high bandwidth DDR4. It features high performance IO like Gigabit Ethernet and USB 3.0 operating near native speeds. It is perfect for media center and IO intensive applications.
Since the board has only been out for two months, there is only sparse documentation available on the web so this is a great central resource for getting started with the Renegade. Did we mention that it is backed by GitHub it can be improved upon?
The documentation covers images for Android, Ubuntu, and LibreELEC by the Firefly team based on Rockchip’s SDK (Linux 4.4 LTS). We will also be rolling out Ubuntu 16.04 Xenial and Ubuntu 18.04 Bionic images for Renegade based on the latest upstream Linux LTS as soon as we are able to digest and test patches that are needed. We have no concrete timeline yet but will provide updates when we are close.
We recommend testing new LTS releases to get a feel for the changes and upgrading only after its first point release since new software is often different, unstable, and bug ridden. For stability, stay with Ubuntu 16.04 which is on its fourth point release (denoted by 16.04.4) and on well on its way to a fifth point release.
This Bionic PI1 image release is for people to play with Wayland/Weston on top of Gnome 3’s Mutter window manager on Le Potato. This image takes around 5 minutes for first boot because it has to re-partition and create swap files so be patient and don’t interrupt it. When you get to the login screen and select the Libre Computer user, make sure to click the little settings icon (looks like a gear) and select “Ubuntu on Wayland”. From there, you can download es2gears_wayland (sudo apt-get install mesa-utils-extra) and glmark2-es2-wayland (sudo apt-get install glmark2-es2-wayland). Le Potato should be about 50% faster than the Raspberry Pi 3 Model B+ in glmark2. Please note that the 3D acceleration feature is neither stable or fully featured at this point and a lot of work remains. Having an open-source Mali driver like lima would help greatly. The hardware is OpenGL ES 2.0 only so needs something like gl4es shim for applications that rely on OpenGL.
The other big change in Bionic is the move away from /etc/network/interfaces and to netplan which renders configurations to other backends like interfaces and NetworkManager. We have included the appropriate yaml files for the headless and desktop images. Ubuntu also removed ifconfig so you have to use the ip command instead.
Unity -> Gnome 3 (Mutter WM)
/etc/network/interfaces -> netplan
Linux Kernel 4.14.38 with GBM for Mali
Virtualization and container accounting improvements
Top left USB port (OTG USB) conflicts with Mali driver and will cause all USB ports to stop working if you plug in a USB device into that port, blacklist the mali module before using the port
Mutter packages are held back from upgrading automatically due to out-of-tree patches
ARM64 Linux kernel are not part of Ubuntu ports repository, a separate repository needs to be set up in order to enable automatic kernel updates
No video for monitors and TVs with 1366×768 and some other odd resolutions
Other Project Notes:
Tritium boards will be mailed out next week for Kickstarter backers and we will begin the engineering effort for unifying board support in software so you can switch between ALL-H3-CC H5, AML-S905X-CC, and ROC-RK3328-CC with one command. Hopefully by the end of 2018, the fruits of our software efforts will offer a seemless experience on all of our boards.
Today, we follow up the release of our Ubuntu desktop image with our headless server image for AML-S905X-CC. The headless server image is based on the latest Linux LTS 4.14 along with some of our customizations. It utilizes the same infrastructure we’ve created for our desktop image while consuming considerably less power.
Linux 4.14 LTS
Custom Partition Layout
256MB FAT EFI Partition
Copy-on-Write for greater reliability
Facebook’s zstd compression
@ root subvolume and @lc-ubuntu-16-headless release snapshot
Self expands on startup to full disk size
Auto generated Swap
1-2GB in side located at end of disk
offload pages from zswap
In testing, power consumption is around 180mA with network, eMMC, and microSD card connected for a total consumption of 0.91W! Did we mention this thing has quad 64-bit ARM Cortex-A53 processors? Just to give you an idea, your “energy efficient” 13W LED light bulb uses the same power as 15 of these servers.
We performed no optimization or other funny business in our images to achieve these results. You can get a Le Potato yourself and compile a mainline kernel to verify the results. If you have one already, you can download the latest images here. If you are serious about minimizing power consumption, more saving can probably be extracted.
It has been over three months since our last preview image 7 for the AML-S905X-CC Le Potato platform and there has been a lot of working going into mainline Linux and u-boot by our partner BayLibre. Preview image 8 brings all of the work together into a flash-able image for our end-users.
PI8 Core feature upgrades:
Upgraded to mainline u-boot 2017.11
No longer using Amlogic’s u-boot
Saving u-boot environment
cvbs enablement through UART via “setenv enable-cvbs 1; env save”
Upgraded to mainline Linux LTS 4.14.29
New DMT Display Modes support
Fixed top USB Port by Ethernet causing USB subsystem to hang
Utilize kernel and kernel module deb package install
Additional image work
eMMC support via lc_distro_transfer utility
Reworked fstab mount descriptors
Reworked partition resize utility
Preliminary EFI support
Headless and XFCE Desktop snapshots in one image
Some known issues with this release and additional improvement work:
MicroSD card may hang on reboot command
package u-boot and board specific utilities and binaries into deb files
add repository to apt for board support deb files
display pipeline improvements and rework for additional resolutions
Ubuntu 18.04 LTS
video codec acceleration support
overlay auto-loading in u-boot
eMMC performance improvements
EFI support improvements
eMMC modules are solid-state flash devices created for embedded systems. They offer higher reliability and additional performance compared to MicroSD cards. They are purchased separately and attached to the eMMC connector on the bottom side of the board.
We currently have eMMC 5.x modules and eMMC 4.x modules. You can attach an eMMC 5.x module on a board that supports eMMC 5.x only. If you attach an eMMC 5.x module on a board with only eMMC 4.x support like the ALL-H3-CC, it will not work. The reverse is also true. The performance differences between the two module types are small to negligible.
This image includes the release state of our distro as snapshots on top of the BTRFS filesystem. We added a new script called lc_distro_transfer that utilizes core design features of our image to transfer system snapshots created during image building to eMMC and back to MicroSD card.
Please note that you can only flash the image that you original flashed to the eMMC. You cannot flash the headless image back onto the MicroSD card if you originally flashed the xfce image onto the eMMC.
Raspberry Pi 3 Model B+ is the newest offering from the Raspberry Pi Foundation sharing much of the same features as the Raspberry Pi 3 Model B with small but significant improvements on many fronts while maintaining the same price. Below is some highlights.
Increased CPU clock speed from 1.2GHz to 1.4GHz
Increased memory throughput
Integrated Heat Spreader (IHS) for SoC
Integrated MxL7704 PMIC for power management and delivery
Modularized WiFi/BT Radio with 5GHz and improved performance
Gigabit Ethernet (albeit still over USB 2.0)
Ethernet headers for PoE addon
We ran a comprehensive set of benchmarks on the new model, old model, and our boards to compare performance and power consumption.
We begin with heavily optimized C applications like C-Ray and SciMark2. Raspberry Pi 3 Model B+ is still utilizing 32-bit kernel and userland like its predecessor. Raspbian, the official OS of Raspberry Pi, has not moved to 64-bit ARMv8 despite the ARM Cortex-A53 CPU cores supporting it. Legacy 32-bit can help performance for this specific benchmark since some data structures and pointers are smaller than in 64-bit ARMv8 mode. Performance increases around 20% from the Model B, which means that the Raspberry Pi 3 Model B+ matches the performance of similarly clocked ROC-RK3328-CC. It is still slightly behind the AML-S905X-CC since that is about 100MHz faster.
Now we move onto server based workloads. Redis is a good test of overall system performance as it stresses not just the CPU but also the interrupt and memory subsystems. We see the Raspberry Pi 3 Model B+ improving greatly in performance over the previous Model B but still not enough to catch up with modern 28nm SoCs with faster DDR3/4 running ARMv8 kernel and userland.
Java benefits heavily from having native 64-bit arithmetic and performs signficantly faster on the boards running true 64-bit OSes. Despite nearly a 30% increase in performance, the Raspberry Pi 3 Model B+ is left behind.
Sysbench results should be taken with a grain of salt when comparing different binaries but this demonstrates the necessity of true 64-bit ARMv8 kernel and userland. Even with the performance gains, Raspberry Pi products are still held back by 32-bit ARMv7 Raspbian OS. Both the Renegade and Le Potato boards deliver more than 10x the performance.
Raspberry Pi 3 Model B+ uses a newly revised BCM2837B0. There are four limiting features of this SoC just like the previous BCM2837 in the Model B. First, it is missing ARM ISA’s Crypto Extensions. For encryption and decryption workloads such as VPN, SSL, SSH, and HTTPS, it’s NEON accelerated implementation is roughly 15x slower. This is one of the critical missing features that make the Raspberry Pi 3 Model B+ a poor choice for server based workloads that depend on these security instructions.
The second limiting feature has to do with the GPU which is a 30-bit design limited to 512MB of RAM. It only supports DDR2 so we haven’t seen the Raspberry Pi move to faster memory like LP/DDR3 like on Le Potato or LP/DDR4 on Renegade. As a result, memory intensive workloads will be much slower although the Model B+ is marginally faster than the Model B.
Like the previous Model B, the Model B+ has not implemented UHS support for MicroSD cards. It is still limited to 25MB/s while other boards are more than twice as fast.
As mentioned by Eben Upton, the Raspberry Pi 3 Model B+ has a single USB 2.0 channel which is shared between the ethernet and four port USB hub. Despite having a physical gigabit ethernet, performance is limited to 320Mb/s (40MB/s) peak. If you are using a USB hard drive serving files over ethernet, the effective throughput is reduced to 160Mb/s (20MB/s). Intensive NAS based use-cases for the Pi continues to be ruled out. ROC-RK3328-CC (Renegade) has both dedicated Gigabit Ethernet and dedicated USB 3.0 so it can deliver an order of magnitude more throughput.
The most horrifying aspect of the Raspberry Pi 3 Model B+ is the power consumption. They’ve learned the wrong lesson from ASUS Tinker Board, Orange Pi, et al. While the new PMIC addressed the voltage drop issues, power consumption shot up 50% for marginal increases in performance. In our previous guess of the Raspberry Pi Foundation’s plans, we assumed Broadcom would help make a power efficient Cortex-A35 design. Instead, BCM2837B0 went in the exact opposite direction.
In our CPU burn tests, the board consumed nearly 1.8A without any peripheral or screen connected. This is at the borderline of the MicroUSB power spec and will un-doubted create new power related headaches for many end-users. Most cell phone power supplies simply will not work for this board.
It would be fair to say that the Raspberry Pi 3 Model B+ will not be winning any performance per watt benchmarks especially compared to the super fast and efficient Le Potato board. However, it is a significant step in the right direction compared to its predecessor. The hardware designers have addressed quite a few long-standing issues and we expect the next generation Raspberry Pi 4 to further amortize design issues.
The new integrated heat spreader (IHS) will allow the Raspberry Pi 3 Model B+ to sustain performance for longer and perform more reliably than the previous Model B. It will help the temperature sensor adequately throttle performance when a specific area of the chip becomes too hot and extend the useful life of the board specially in industrial conditions.
The new WiFi/Bluetooth module performs with excellence. It also uses a module design which saves companies from having to go through expensive radio certification process. In our tests, the WiFi performance on the 5GHz band exceeded performance of the Model B on the 2.4GHz band by five times.
The power delivery and management IC has eliminated the voltage drop across the previous poorly-designed power delivery circuit that was causing power warnings with compliant power supplies. While the added power consumption exacerbates the problem, we still feel that this is a step in the right direction.
You can find all of the performance data that we aggregated on this Google Sheet.
Ever want to compare features of our boards to know what the differences are? Want to compare our Tritium, Le Potato, and Renegade boards with newly released popular boards like the Raspberry Pi 3 Model B+ and ASUS Tinker Board S?
We have just made it super easy for you with our spreadsheet. This covers hardware features found on the boards. We will throw together another spreadsheet with software and performance numbers for each board in the future.