# Known issue - [Network issue in quick start](#Network-issue-in-quick-start) - [EXT4 fs error rootfs issue](#EXT4-fs-error-rootfs-issue) - [EXT4 fs error rootfs issue while booting on zcu102](#EXT4-fs-error-rootfs-issue-while-booting-on-zcu102) - [antsdr e200 UART console](#antsdr-e200-UART-console) - [Client can not get IP](#Client-can-not-get-IP) - [Big packet loss while slow ping](#Big-packet-loss-while-slow-ping) - [No space left on device](#No-space-left-on-device) - [Ping issue due to hostname resolving issue caused by DNS server change](#Ping-issue-due-to-hostname-resolving-issue-caused-by-DNS-server-change) - [FMCOMMS board eeprom issue causes Linux crash](#FMCOMMS-board-eeprom-issue-causes-Linux-crash) - [Not booting due to SPI flash](#Not-booting-due-to-SPI-flash) - [Kernel compiling issue like GCC plugins](#Kernel-compiling-issue-like-GCC-plugins) - [Missing libidn.so.11 while run boot_bin_gen.sh](#Missing-libidn.so.11-while-run-boot_bin_gen.sh) - [Zcu102 booting kernel panic due to RTC](#Zcu102-booting-kernel-panic-due-to-RTC) - [Kernel panic due to hardware capacitor and current load](#Kernel-panic-due-to-hardware-capacitor-and-current-load) - [lightdm memory leakage leads to issue after long run](#lightdm-memory-leakage-leads-to-issue-after-long-run) - [Wrong memory size on adrv9361z7035 SoM](#Wrong-memory-size-on-adrv9361z7035-SoM) - [Unsupported PRODUCT_ID 0xFF](#Unsupported-PRODUCT_ID-0xFF) OpenWrt specific: - [No UART output on zcu102](#no-uart-output-on-zcu102) ## Network issue in quick star - OS: Ubuntu 22 LTS - image: [openwifi img](https://drive.google.com/file/d/1fb8eJGJAntOciCiGFVLfQs7m7ucRtSWD/view?usp=share_link) If can't ssh to the board via Ethernet for the 1st time, you might need to delete /etc/network/interfaces.new on SD card (on your computer). If still can't ssh the board via Ethernet, you should use UART console (/dev/ttyUSBx, /dev/ttyCH341USBx, etc.) to monitor what happened during booting. ## EXT4 fs error rootfs issue Sometimes, the 1st booting after flashing SD card might encounter "EXT4-fs error (device mmcblk0p2): ..." error on neptunesdr, changing SD card flashing tool might solve this issue. Some tool candidates: - gnome-disks - Startup Disk Creator - win32diskimager ## EXT4 fs error rootfs issue while booting on zcu102 Issue description: same SD card can boot normally on some zcu102 boards but not on some boards else. Many reportings on internet (while booting zcu102): ``` Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2) ... ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2) ]--- ``` Need to add following blocks into the mmc entry or sdhci entry of the zcu102 devicetree: ``` xlnx,has-cd = <0x1>; xlnx,has-power = <0x0>; xlnx,has-wp = <0x1>; disable-wp; no-1-8-v; broken-cd; xlnx,mio-bank = <1>; /* Do not run SD in HS mode from bootloader */ sdhci-caps-mask = <0 0x200000>; sdhci-caps = <0 0>; max-frequency = <19000000>; ``` Suspect the main reason: sdcard speed needs to be limited by above. Might be due to that the sd card interface degrades and becomes unstable after years. ## antsdr e200 UART console If can't see the UART console in Linux (/dev/ttyUSB0 or /dev/ttyCH341USB0), according to https://github.com/juliagoda/CH341SER, you might need to do `sudo apt remove brltty` ## Client can not get IP If the client can not get IP from the openwifi AP, just re-run "service isc-dhcp-server restart" on board and do re-connect from the client. ## Big packet loss while slow ping When ping from openwifi to COTS device (such as a laptop), if you see big packet loss with normal/slow ping interval (like 1s) but less and less packet loss while decreasing ping interval to 0.01 and 0.001s, the reason most probably is power save behavior of the COTS device. Check power save status: ``` iw dev wlan0 get power_save ``` If it returns "Power save: on", turn it off by: ``` sudo iw dev wlan0 set power_save off ``` ## No space left on device It might be due to too many dmesg/log/journal, disk becomes full. ``` systemd-journald[5694]: Failed to open system journal: No space left on device ``` You can try following operations. ``` systemd-tmpfiles --clean sudo systemd-tmpfiles --remove rm /var/log/* -rf apt --autoremove purge rsyslog ``` Add followings into `/etc/systemd/journald.conf` ``` SystemMaxUse=64M Storage=volatile RuntimeMaxUse=64M ForwardToConsole=no ForwardToWall=no ``` ## Ping issue due to hostname resolving issue caused by DNS server change You might need to change nameserver to 8.8.8.8 in /etc/resolv.conf on board. ## FMCOMMS board eeprom issue causes Linux crash Some FMCOMMS2/3/4/x boards shipped with wrong/empty eeprom, so that on some platform (like ZCU102) it causes issues like Linux crash. You can follow https://github.com/analogdevicesinc/fru_tools to reprogram the eeprom. - Insert the FMCOMMS board on a platform (such as 32bit zed/zc706/zc702/etc) that can boot and boot into Linux - On board Linux: ``` git clone https://github.com/analogdevicesinc/fru_tools.git cd fru_tools/ make find /sys -name eeprom (It might return like: /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom) fru-dump -i /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom -b ``` - If there is issue, you will see some "mismatch" warning like: ``` read 256 bytes from /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom fru_dump 0.8.1.7, built 04Aug2022 FRU Version number mismatch 0xff should be 0x01 ``` - To reprogram the eeprom (FMCOMMS4 as an example): ``` fru-dump -i ./masterfiles/AD-FMCOMMS4-EBZ-FRU.bin -o /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom ``` - Reboot the board, and try to read eeprom again, correct information should be shown like: ``` fru-dump -i /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom -b read 256 bytes from /sys/devices/soc0/fpga-axi@0/41620000.i2c/i2c-0/0-0050/eeprom Date of Man : Mon Jul 22 20:23:00 2013 Manufacturer : Analog Devices Product Name : AD9364 RF Eval/Software Dev Kit Serial Number : 00045 Part Number : AD-FMCOMMS4-EBZ FRU File ID : Empty Field PCB Rev : C PCB ID : FMCOMMSFMC04A BOM Rev : 1 ``` ## Not booting due to SPI flash Before loading content on SD card, the on board SPI flash controls some configurations, such as the kernel file and AD9361 crystal frequency (ad9361_ext_refclk=0x2625a8b). When (suspect) there is an issue, the SPI flash can be restored to default by interrupting booting (hitting enter before Linux loading in the UART console), then ``` Zynq> env default -a ## Resetting to default environment Zynq> saveenv Saving Environment to SPI Flash... SF: Detected n25q256a with page size 256 Bytes, erase size 4 KiB, total 32 MiB Erasing SPI flash...Writing to SPI flash...done ``` ## Kernel compiling issue like GCC plugins Sometimes after the GNU/GCC tool chain update in the host PC or the slightly kernel update (such as 5.15.0 --> 5.15.36), it might prompt user to select among some new options while compiling kernel like: ``` ... Xen guest support on ARM (XEN) [N/y/?] n Use a unique stack canary value for each task (STACKPROTECTOR_PER_TASK) [Y/n/?] (NEW) n * * GCC plugins * GCC plugins (GCC_PLUGINS) [Y/n/?] (NEW) n ... ``` In these cases, the best/safest way is to chose **n** and **weakest** options. Otherwise the compiling might fail or potential issues might happen. ## Missing libidn.so.11 while run boot_bin_gen.sh You might need to prepare/fake libidn.so.11 by ``` sudo ln -s /usr/lib/x86_64-linux-gnu/libidn.so.12.6.3 /usr/lib/x86_64-linux-gnu/libidn.so.11 ``` Please check/confirm what is the exact **libidn.so.12.6.3** in your system. ## Zcu102 booting kernel panic due to RTC https://github.com/open-sdr/openwifi/issues/366 ## Kernel panic due to hardware capacitor and current load https://github.com/open-sdr/openwifi/issues/457 ## lightdm memory leakage leads to issue after long run Better to disable lightdm via systemctl ## Wrong memory size on adrv9361z7035 SoM https://github.com/open-sdr/openwifi/issues/404 reports that Linux only sees half memory size than the actual DDR memory size in the hardware. The root cause is the old/wrong u-boot.elf hard coded the memory size as 512MB. It is already fixed to the correct 1GB (0x40000000): https://github.com/analogdevicesinc/u-boot-xlnx/blob/master/arch/arm/dts/zynq-adrv9361.dts The solution is re-building u-boot.elf from https://github.com/analogdevicesinc/u-boot-xlnx and re-generating BOOT.BIN for adrv9361z7035 SoM. Steps to re-build u-boot.elf for adrv9361z7035 SoM: ``` git clone https://github.com/analogdevicesinc/u-boot-xlnx.git cd u-boot-xlnx source environment_setting.sh (could be XILINX_DIR/Vitis/2022.2/settings64.sh or directory of your tool chain) export ARCH=arm export CROSS_COMPILE=arm-linux-gnueabihf- make zynq_adrv9361_defconfig make -j8 make u-boot.elf ``` ## Unsupported PRODUCT_ID 0xFF https://ez.analog.com/microcontroller-no-os-drivers/f/q-a/101813/ad9361-spi32766-0-ad9361_probe-unsupported-product_id-0xff/303302 https://wiki.analog.com/resources/tools-software/linux-software/fru_dump # Known issues specific to OpenWrt ## No UART output on zcu102 ### Step 1: Check board Support was only validated with the ZCU102 HW Revision 1.1. Ensure that your board is the correct hardware revision. Straightforward differences can be consulted [here](https://github.com/torvalds/linux). The device trees for these boards build upon the previous generation, making it easy to track the differences: zynqmp.dtsi -> zynqmp-zcu102-revA.dts -> zynqmp-zcu102-revB.dts -> zynqmp-zcu102-rev1.0.dts -> zynqmp-zcu102-rev1.1.dts ### Step 2: Check SODIMM modules We discovered that SODIMM modules may differ between ZCU102 HW Revision 1.1 boards. The 4GB SODIMM module is found underneath the metal casing on the board (carefully pry a screwdriver between the casing and lips holding it in place to release it). Known working modules: - MTA8ATF51264HZ-2G6B1 Known failing modules: - MTA4ATF51264HZ-2G6E1 Solutions - Use Zynq FSBL instead of U-Boot SPL, see [below](#solution-use-zynq-fsbl-instead-of-u-boot-spl). - Buy known working module - Fix U-Boot SPL yourself (if you do, let me know how), the information [below](#the-problem) should get you started. ### Solution: Use Zynq FSBL instead of U-Boot SPL The OpenWrt boot partition has the following relevant files: - boot.bin - u-boot.itb Both can be removed as Xilinx bootflow uses a single boot.bin instead (u-boot.itb is basically packed into boot.bin). There are two options to generate this single, large, boot.bin. #### Option 1: Use build_zynqmp_boot_bin.sh The easiest way it to use the build_zynqmp_boot_bin.sh file in this repository. The downside to this option is that older versions of Zynq FSBL, U-Boot Proper, ... will be used. #### Option 2: Generate boot.bin yourself with OpenWrt generated components Building boot.bin requires the components listed in the .bif file as generated by build_zynqmp_boot_bin.sh. You can find recent versions of the necessary (note that openwrt does not expect the bitstream to be packed into boot.bin) files in the openwrt build directory (including fsbl.elf despite it not ending up in the default openwrt image). Copy the files to a new folder, edit the .bif and use Xilinx bootgen tool to generate boot.bin. ### The problem In the OpenWrt work, we preferred to use the U-Boot community bootflow over the Xilinx bootflow. Mainly because this allows us to avoid the use of a Vivado installation and Xilinx tools in general. In the community bootflow, U-Boot Secondary Program Loader (SPL) replaces the Zynq FSBL generated by Xilinx. U-Boot SPL starts execution in OCM. Here, UART output is not available by default, only a debug uart can be enabled for use. To allow U-Boot SPL to use the driver model, it needs to relocate itself to DDR memory. Before it can do this, it should configure the DDR Controller and DDR Phy. To configure these, U-Boot SPL relies on the psu_init_gpl.c file generated by Zynq FSBL. The function within psu_init_gpl.c where the DDR registers are set is found [here](https://github.com/robgar2001/u-boot/blob/c1604001a7bc23475166e5eea08ebfcee2da9555/board/xilinx/zynqmp/zynqmp-zcu102-rev1.1/psu_init_gpl.c#L1848). So, these settings only work for the known working modules, not for the failing modules. How does Zynq FSBL tackle this? Well, they use the Serial Presence Detect (SPD) of the DDR module (small EEPROM chip that advertises valid settings). More precisely, in [this](https://github.com/Xilinx/embeddedsw/blob/02ac3f3fa487be2d5f3a6ab19d0c35b06cc94bdb/lib/sw_apps/zynqmp_fsbl/src/xfsbl_ddr_init.c#L6834) function, DDR memory is initialized. Here, it first writes the default values from the psu_init_gpl.c file, after which it calls [this](https://github.com/Xilinx/embeddedsw/blob/02ac3f3fa487be2d5f3a6ab19d0c35b06cc94bdb/lib/sw_apps/zynqmp_fsbl/src/xfsbl_ddr_init.c#L6886C2-L6886C40) and [this](https://github.com/Xilinx/embeddedsw/blob/02ac3f3fa487be2d5f3a6ab19d0c35b06cc94bdb/lib/sw_apps/zynqmp_fsbl/src/xfsbl_ddr_init.c#L6898) function to read SPD, calculate the values for this specific SODIMM module and update the registers. Fixing U-Boot SPL can be done in two ways: - To start, it would be nice to update the psu_init_gpl.c files to boot with the failing module. One approach (I tested this but it failed) is to modify Zynq FSBL to print out the newly calculated register values based on SPD and update psu_init_gpl.c. - A more permanent solution is to add or enable SPD support in U-Boot SPL (you can use Zynq FSBL code as a guideline).