My third book is released! Learn what you'll need to know in order to become an embedded engineer.
Check out my second book; learn practical stuff about building robots and control systems around Linux PCs and the Atmel AVR.
My first book gives you all the intro you need on developing 32-bit embedded systems on a hobbyist budget.
Installing Linux on Geode-based Single-Board Computers
Thursday 9/11/2003 - This document has been revised, cleaned-up and extended. Most of this material will eventually appear in my second book, which you can expect to see around the middle of 2004. Feel free to buy my first embedded systems book, while you wait :)
My daytime employer builds embedded Linux appliances based around the Advantech PCM-5820-G0xx 3.5" miniature SBC, and I've found several issues with it that require tweaking. Over the last two weeks, in fact, I have spent almost all my time reverse-engineering the BIOS and creating hybrid firmware to fix some ongoing issues with it.
Before continuing with the meat of this information, let me pause to mention that several readers have commented on the relevance, or lack thereof, of this document to other Geode SBCs. By and large, everything you read here is relevant to most Geode boards with an Award BIOS, which includes the following specific products (all of which are biscuit PCs in the 3.5" or 5.25" form factors):
As of the time of writing, we are using Linux kernel 2.4.21. I have heard positive results from people using the 2.6 kernel betas, but we don't wish to ride the bleeding edge in our application, so we will defer testing on 2.6 until it is primetime. Our distribution has very heavy "eye candy" requirements, so it is based on Red Hat 8.0, with significant customizations. In fact, there is not much Red Hat left.
Our chosen SBC is based around a 300MHz National Semiconductor Geode processor with the CS5530A companion IC providing interface to CRT SVGA monitors and/or TFT LCD panels. It also features an IrDA port, two OHCI-compliant USB ports, PS/2 keyboard/mouse port, one parallel port, two RS-232 ports (one of which can be configured for RS-485), floppy controller, single IDE bus, CompactFlash slot residing on the second IDE bus, 10/100 Ethernet via a Realtek RTL8139 and analog audio I/O. Some of the other devices listed above differ in peripherals; dual or more LANs, TV-out, etc, but they are all the same core hardware and share a lot of the same quirks.
I'll get some of the on-board peripherals out of the way immediately by stating that they work without special modifications:
The first problem you may encounter if you install your Linux distribution on another computer is that you'll get an i686 kernel installed, and it will not boot. It seems to have problems finding a Pentium-style TSC when booting on a Geode. GX1 does in fact have a TSC at MSR 10, but the i686 kernel installed on an Intel/Via/AMD on 7.x's install CD will fail to find it, and will panic. To rectify this, use the kernel-x.x.x-i586.config configuration as a starting point, then make config and make bzImage to build your kernel.
Geode is descended from the Cyrix MediaGX processor. It is probably most directly comparable with a Pentium-MMX, but it is by no means 100% identical. Overall performance of a 300MHz Geode is very roughly equivalent to a 200MHz Pentium-MMX, but any meaningful comparison must be made on a function-by-function basis. In particular, Geode's performance is significantly limited by the fact that a lot of the functionality, particularly video, is emulated in what NS calls "Virtual System Architecture"; special firmware accessed in System Management mode. But, more on that topic later.
The audio feature built into the CS5530 is supported using the SoundBlaster 16 driver with the "mpu_io=0" parameter. Note that you must disable ISA Plug'n'Play in the kernel. Recent(ish - i.e. contemporaneous Red Hat 7.x and later) versions of sndconfig will detect the 5530 as "CS5530 [Kahlua]", but they will inform you that the chip is disabled. On some boards (perhaps those using National's XPressROM) it may be possible for you to fix this error with CMOS settings and get the native driver loaded, but this doesn't seem to be possible on the PCM-5820. Just use the part in SB16 emulation mode.
Now for the longest part of this discussion - how to get XFree86 running. Our application is graphics-intensive. For various reasons, it is advantageous for us to implement our embedded app as an X client application, but getting X to work on this board is quite a chore. I'll skip the story of every combination I tested, and just give you the answers here.
Before going any further, note that there is a special issue with the BIOS on this board, which affects all the Geode boards I have tested. This board supports two displays directly (CRT and LCD). The BIOS allows you to select the LCD resolution - 640x480, 800x600 or 1024x768. Even if you have the LCD disabled in BIOS and only use the CRT outputs, you MUST set the LCD resolution option in CMOS setup at least as high as the highest resolution you want to use with X. Otherwise the system will hang with a black screen when you attempt to enter graphics mode. Also ensure that you allocate enough video RAM (in CMOS setup) to accommodate your desired video mode.
Now, using out-of-box Linux components from major distributions, there are basically three ways to get XFree86 working on the CS5530:
Method 1. Use the framebuffer console and the XF86_FBDev server.
This method might not work on all Geode-based systems because it relies on the presence of a VESA BIOS extension. It works on the PCM-5820 (but note the important drawbacks below). For testing purposes, you can simply type "video=vesa vga=xxx" at the LILO boot: prompt, or make whatever changes are appropriate to your bootloader if you're not using LILO. This will allow you to check various video modes (xxx = video mode; look in Documentation/vb/vesafb.txt inside your Linux kernel source directory for more information on this). Once you've established which mode works best for you, adding these lines to the appropriate paragraph in lilo.conf will make your choice permanent:
Note that this feature requires kernel framebuffer console support. Assuming you have everything working for the framebuffer console, simply install the XF86_FBDev server and link it to /usr/X11R6/bin/X and you'll be set. An example XF86Config file for framebuffer operation is below. Using this file, X will start up in the resolution you selected at boot time.
There are two major disadvantages to this setup (and some minor ones). The two big problems are (a) it's slow, because you have no access to any accelerated hardware features, and (b) you can't switch resolutions on the fly. In our application, we want to run at 1024x768 for still images, but use lower resolutions for full-screen motion video (MPEG-1 playback) since the source material is lo-res anyway and stretching it out to fill a large screen is a complete waste of CPU bandwidth.
Important note: If you are using the VESA framebuffer console, it is not possible to use non-FBDev X servers, nor graphics libraries like SVGAlib. The system will go into an undefined video mode as soon as you attempt to switch modes away from the boot-time default.
Method 2. Use the XF86_SVGA server from version 3.x.
The best match I have found so far is the XF86_SVGA server, version 3.3.6a. If you install Red Hat 7.2 and select Geode, this server will be installed but it will _NOT_ be enabled! The Red Hat 7.2 install process by default links /etc/X11R6/bin/X to /etc/X11R6/bin/XFree86, which is the XFree86 4.1.0 "mega-wrapper"; it attempts to install the Cyrix MediaGX driver, which does not work with Geode.
Note that the 3.3.6a server I mention above is NOT the same version Advantech supplies on their driver CD-ROM. The Advantech version seems to calculate dot clocks and other timings quite bizarrely, and I've had trouble getting properly centered video with their server. Unfortunately, there's a catch-22 lurking here, which I describe below.
One essential point to note is that there is a bug in the CS5530 which can make the SVGA server extremely unstable. You can work around this bug by specifying an odd virtual screen size using the Virtual x y keyword (I use 1024 769 because our system normally runs at 1024x768. If you are using a lower screen resolution like 640x480 and want to conserve video memory requirements, use an appropriately odd-sized virtual buffer such as 640x481). The cause of this bug is a combination of XF86_SVGA's behavior and a quirk of the CS5530. The Geode system relies heavily on I/O traps and "faked" hardware emulation for some of its functionality, particularly video. In the "standard" resolutions, XF86_SVGA attempts to enable display compression, which causes problems. In the best case, you will get some garbage on the display; in the worst case, the system will lock hard and require a hard reset or power-cycle. Specifying the strange virtual resolution implicitly disables the compressed display feature and works around the problem. This workaround may or may not be ideal for you depending on your system setup; in our case, the system has no pointing device so it's not possible for the user to scroll the display window and see the extra "phantom" scanline.
Using XF86_SVGA you will have access to some hardware acceleration features such as a hardware mouse cursor. More importantly, you will be able to change resolutions on the fly. Here is a suitable XF86Config file for the three standard resolutions, 640x480, 800x600 and 1024x768. It is possible to get the low-res video modes working also, but only with the Advantech-supplied server. (Argh! This is the catch-22 I mentioned) If you don't need low-res I suggest you stick to the Red Hat-supplied server. It's much easier to get stable, easily- centered video with this server.
One final note on XFree86 3.x - National has supplied numerous subtly different versions of the 3.x SVGA server for Geode; some with sourcecode, some as binary-only. Since 3.x is officially dead, I have not experimented with all of these. If you're desperate to fix some particular issue, feel free to go the trial-and-error route!
Method 3. Use XFree86 4.x
Modern distributions of Linux, such as Red Hat 9.0, ship with XFree86 4.x. However, the installation process doesn't correctly detect the GX/5530 chipset; it installs the Cyrix MediaGX driver, again. To get the system working, use this XF86Config, which uses the nsc_drv.o driver. Note: This XF86Config has only been tested on XFree86 220.127.116.11 beta and the current release, 4.3.0.
There are many good reasons to switch to using XFree86 4 if you can, including:
Now, that's only half the picture. Due to a bug, the XFree86.org official release code does not support scandoubled modes (e.g. 320x240) on Geode. I have generated a patch for this bug; you can download a precompiled version of the patched nsc_drv.o here. If you are interested in the details of the patch, for example if you wish to apply it to a newer sourcecode release, please refer to my posting on the topic, at email@example.com.
Using my patched driver enables 400x300, 320x240 and 320x200 graphics modes, which are useful if you need to play VCD content on a Geode.
A final note: Of BIOSes and Quirks
Recently, we discovered that our PCM-5820-based systems would intermittently fail to go into X properly, mostly after a warm boot. There was no pattern to this behavior; the symptom would happen once. The system would actually be running - we could hear audio features playing, and communicate with the device over Ethernet - but the CS5530 was putting out very odd syncrates. Further investigation linked this problem to the currently-shipping BIOS (as of September, 2003) - version 1.23. Flashing the boards down to version 1.14 fixed the problem. Unfortunately, coincident with BIOS release 1.23, Advantech also changed the AC97 audio codec on the board from an Analog Devices chip to a Realtek ALC201 part. BIOS code 1.22 and earlier will play back audio approximately 10% too fast on new hardware. BIOS 1.22 also shows the video hanging problem. The last BIOS that doesn't have the video problem is 1.21, which can't be used on current hardware because of the audio issue.
This was the start of panic mode. Our mechanical design is based around the PCM-5820, so we couldn't quickly adapt to a new board. So, I experimented with numerous "alien" BIOSes from other vendors. All of them either had the audio problem, or had the intermittent hanging problem, or didn't work correctly in some other way (e.g. no network support, USB ports not working, &c). Clearly, the shotgun approach wasn't going to work. So, it was time to hack a little deeper.
The first step is to extract various parts of the BIOS code to compare them. The PCM-5820 uses Award BIOS 4.51G. This is comprised of a tiny decompressor/bootloader module, and an internal LZH-compressed filesystem containing various modules. One tool to manipulate this internal filesystem is AwardMod (a GNU tool); you can download it here. AwardMod (and the official Award utilities) work with raw dumps of the BIOS ROM file; if you don't have a copy of your BIOS, you can write one out using the /SY command-line switch to the AWDFLASH flash-update utility (search for this utility with Google or download it from BiosMods.com).
Splitting out the BIOSes into their component modules reveals that they contain the following parts:
The EPA logo, LCD parameters, Award PnP extension and PXE ROM are identical across all affected versions, so they're not part of our problem. Using AwardMod, I created various hybrid BIOSes with combinations of modules from different Advantech releases. I quickly determined that both the video problem and the audio fix reside in the VSA code.
My attack on this problem was three-pronged. Success along any of these routes would satisfy us:
In pursuit of the first attack, I started to reverse-engineer the VSA code from 1.21 and 1.23 side by side. I quickly determined that there are major differences between the two; it's not just a timing constant or two that needs to be hacked. Due to the large number of Geode-specific instructions and the fact that I would basically need to create a full reverse-engineered disassembly of the entire 128K of VSA code, I decided that I would never be able to finish this inside a useful timeframe.
The second attack method has yielded an apparently working BIOS. It is comprised of Advantech's 1.21 code, with VSA code from a different board. This has also fixed another problem with Advantech's code - high interrupt latency causing glitches in audio playback. There are political reasons why this code is not a desirable solution, and for copyright reasons I obviously can't post it for download.
The third attack seems to be the safest. I have had promising results by disabling SMA trapping on the VGA text-mode addresses. (Note that this will partly break text video modes. This is not an issue for us). This code has not yet been through rigorous testing, but it seems to be working. Once I have verified that it is "the" fix, I will post the sourcecode and precompiled binary here.
One final note: Yes, I am available for contract help with custom BIOS modifications and Linux distributions for embedding on these SBCs ;)