RAK7391 - SATA M.2 drive not detected

Greg · December 30, 2022, 8:00am

Hi,
I’m having trouble trying to get a SATA SSD to work in the M.2 interface on the RAK7391.
The only SATA drive that I could find locally with B-Key was an ADATA Ultimate SU650 with 512GB storage.
I’m running RAKPiOS and have checked and applied the latest system updates.

When I run the lspci command I don’t see any SATA controller listed.

Are you able to confirm:

whether RAKPiOS is using the latest PiOS kernel with native SATA support;
whether the SATA SSD above is compatible with RAK7391 M.2 interface; if so
provide instructions on how to detect and mount the drive?

Any help would be very much appreciated.

Kind Regards
Greg

xoseperez · December 30, 2022, 11:12am

Hi Greg!
Unfortunately the RAK7391 does not have a SATA controller, that’s why it does not show up. The M.2 slot is originally conceived to host a 5G modem and only provides PCIe and USB3 signals.
It is possible to use it with an SSD but it’s got to be an M.2 B-key NVMe drive (not SATA).

Greg · December 30, 2022, 12:21pm

Thanks for the clarification.
I’ll try and source an NVMe SSD with B-Key.

Greg · January 6, 2023, 9:53pm

Hi Xose Perez,
I managed to locate an 128GB NVMe M.2 SSD with M+B key and installed this in the RAK7391 M.2 B-key slot, but have been unable to get this to work either. When I run the lspci command, I can see the KIOXIA controller for what I presume it the NVMe SSD. However, when I run the dmesg command it looks like there is some kind of issue with the NVMe controller and the devices is disabled. See below details.

Kind Regards
Greg

Do you think this is an incompatability with the NVMe SSD or driver issue?

rak@rakpios:/ $ lspci
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
04:00.0 Non-Volatile memory controller: KIOXIA Corporation Device 0001
05:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
rak@rakpios:/ $

rak@rakpios:/ $ dmesg | grep nvme
[ 1.293228] nvme nvme0: pci function 0000:04:00.0
[ 1.293380] nvme 0000:04:00.0: enabling device (0000 → 0002)
[ 66.527069] nvme nvme0: I/O 13 QID 0 timeout, disable controller
[ 66.635037] nvme nvme0: Device shutdown incomplete; abort shutdown
[ 66.635447] nvme nvme0: Removing after probe failure status: -4
rak@rakpios:/ $

Greg · January 11, 2023, 7:20pm

Hi Xose Perez,
I purchased a KingSpec 256GB M.2 NVMe PCIe SSD off Aliexpress and this is recognized by the RAK7391.

Cheers

xoseperez · January 18, 2023, 7:25am

I’m glad you found the proper drive.
We know there are few SSD drives out there that are compatible with the RAK7391 (I have a KingSpec with me too).
As I mentioned the reason is that the main use for the M.2 slot is connectivity so we designed it to be compatible with LTE and 5G modules out there primarily.

barryjump · January 23, 2023, 11:32pm

Hi @Greg would you mind sharing the steps to getting your working NVME mounted?

I’ve got a Kingspec 128GB as well and it’s visible:

rak@rakpios:~ $ sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
[sudo] password for rak:
UUID                                 NAME         FSTYPE   SIZE MOUNTPOINT LABEL  MODEL
                                     mmcblk0              29.1G
E830-23B2                            ├─mmcblk0p1  vfat     256M /boot      boot
7d51fe61-54a4-4db2-bc4b-05c7beb9cdff └─mmcblk0p2  ext4    28.9G /          rootfs
                                     mmcblk0boot0            4M
                                     mmcblk0boot1            4M
                                     nvme0n1             119.2G                   NE-128

But I’m not clear on how to get it mounted properly for use.
Thanks!

Greg · January 27, 2023, 10:28pm

Hi Barry,
Looking at your output from the lsblk command, you will need to create a primary partition using fdisk and format it. Once you’ve don this you should see something like “nvme0n1p1” .

Note:
To format the partition using ext4 flle system I ran “sudo mkfs -t ext4 /dev/nvme0n1p1”

Once done, you can then mount the partition by doing the following:

Create a folder to mount the partition in the /mnt folder e.g. “sudo mkdir /mnt/mynvmessd”.
Mount the drive nvme drive partition to the new folder e.g. “sudo mount /dev/nvme0n1p1 /mnt/mynvmessd”

If you then run “df -h” you should then be able to see your partition and the mountpoint, in this example /mnt/mynvmessd.

However, please note that this method is not persistent and you’ll have to remount the drive when you power cycle or reboot the device.

If you want the drive to be mounted automatically on reboot you’ll need to edit the file system table “/etc/fstab”.

However, what I’ve done is copied the RAKOS operating system from the EMMC drive on the CM4 to my NVMe drive and modified/re-flashed the CM4 bootloader so that the CM4 boots from the NVMe drive. To do this you’ll need to use the raspberry pi usbboot tool.

Let me know if you get stuck or would like more information on how to boot from the NVME drive.

Cheers
Greg

barryjump · February 4, 2023, 5:51pm

Terrific thanks for the assistance @Greg I will try those steps today.

Meantime, there don’t seem to be many other RAK7391 users yet, so I’m struggling to find an answer if others are experiencing the same issue as me:

[  737.575932] usb usb2-port4: attempt power cycle
[  741.983757] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[  746.071756] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[  746.072267] usb usb2-port4: unable to enumerate USB device
[  750.143766] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[  754.375754] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[  758.463708] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?

I’ve stripped all the peripherals while running dmesg -w and monitored the output while adding them back in, but even stripped this issue persists. Do you see the same output from dmesg?
I’m trying to figure out what is on usb2-port4 but it appears to be an issue on the board, considering it persists with no peripherals.

Greg · February 4, 2023, 8:36pm

@barryjump

With regards to the usb2-port4 message, I get the same messages when I running the dmesg command with no peripherals connected (see output below).

rak@rakpios:~ $ dmesg | grep usb2
[ 1.385990] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.15
[ 1.386026] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.386053] usb usb2: Product: xHCI Host Controller
[ 1.386074] usb usb2: Manufacturer: Linux 5.15.74-v8+ xhci-hcd
[ 1.386096] usb usb2: SerialNumber: 0000:05:00.0
[ 5.876347] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 9.964251] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 9.964490] usb usb2-port4: attempt power cycle
[ 14.368198] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 18.460271] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 18.460507] usb usb2-port4: unable to enumerate USB device
[ 22.532304] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 26.772221] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 30.860215] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?
[ 30.860432] usb usb2-port4: attempt power cycle
…message repeats…

When I run the dmesg -w I can see that it looks like it detects the USB2.0 Hub, but has and issue with usb2-port 4 for some reason

[ 5.801295] usb 3-1.4: Product: USB 2.0 Hub
[ 5.852037] hub 3-1.4:1.0: USB hub found
[ 5.852422] hub 3-1.4:1.0: 4 ports detected
[ 5.876347] usb usb2-port4: Cannot enable. Maybe the USB cable is bad?

Running the lsusb command, it looks like both USB2.0 root hubs are detected.

rak@rakpios:~ $ lsusb
Bus 003 Device 005: ID 1a86:7523 QinHeng Electronics CH340 serial converter
Bus 003 Device 004: ID 1a86:7523 QinHeng Electronics CH340 serial converter
Bus 003 Device 003: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 003 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Looking at the Rakwireless RAK7391 datasheet (RAK7391 WisGate Connect Datasheet | RAKwireless Documentation Center) it mentions that the RAK7391 has two onboard USB 2.0 root hubs that expand one USB 2.o port on the CM4 to seven ports; one is available on the USB2 connector on the board, two are connected to mPCIe slots #1 and #2, and two more are connected to WisBlock slots and the other two are converted to UART and connected to the WisBlock interfaces as well.

In summary, I was not aware of the use with port4 on the usb2 root hub until you raised it. Like you. I’m not 100% sure what usb2-port four is mapped to, but suspect that it is one of the functions that I referred to above. Also, on the RAK7391 datasheet there is a block diagram (figure 2) showing how the USB2 root hubs are connected. Unfortunately, it doesn’t indicate what ports are used to connect to the interfaces. I’ll do some digging and would be interested in any of your findings.

Regards
Greg

Note:
I just noticed that at the bottom of the dmesg -w output there are a number of over-current messages relating to usb ports, one if which is related to usb2-port4. Note sure whether this is related, but I was interested to see whether you have the messages?
[ 894.590350] usb 1-1-port1: over-current change #1
[ 894.725345] usb usb2-port4: over-current change #1
[ 894.809016] usb 1-1-port1: over-current condition
[ 894.811199] usb 1-1-port2: over-current change #1
[ 894.940845] usb usb2-port4: over-current condition
[ 895.033030] usb 1-1-port2: over-current condition
[ 895.035221] usb 1-1-port3: over-current change #1
[ 895.256966] usb 1-1-port3: over-current condition
[ 895.259155] usb 1-1-port4: over-current change #1
[ 895.476956] usb 1-1-port4: over-current condition

Interestingly, if I disable USB2 on the /boot/config.txt, the message “usb usb2-port4: Cannot enable. Maybe the USB cable is bad?” is replaced with “usb usb2-port2: Cannot enable. Maybe the USB cable is bad?”. Also, all the other usb over-current messages disappear with exception to the on relating to usb2-port4. Note sure whether this is related and more of an observation.

UPDATE:
After enabling usb 2 in the /boot/config.txt file and rebooting the device, the “usb usb2-port4: Cannot enable. Maybe the USB cable is bad?” messages did not return when running the dmesg command. The only message that I had relating to usb2-port4 was the over-current message (see below).

Can you try:

commenting out the “dtoverlay=dwc2,dr_mode=host” in the /boot/config.txt file to disable usb2;
reboot the device;
uncomment “dtoverlay=dwc2,dr_mode=host” in the /boot/config.txt file to enable usb2;
reboot the device;
run dmesg command to see if messages relating to usb2-port4 persist?

rak@rakpios:~ $ dmesg | grep usb2
[ 1.682242] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.15
[ 1.684949] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.687614] usb usb2: Product: xHCI Host Controller
[ 1.690151] usb usb2: Manufacturer: Linux 5.15.74-v8+ xhci-hcd
[ 1.692745] usb usb2: SerialNumber: 0000:05:00.0
[ 1.805673] usb usb2-port4: over-current change #1
[ 2.021419] usb usb2-port4: over-current condition

barryjump · February 5, 2023, 12:04am

Thanks for confirming @Greg. I havent seen the “over-current change #1” error message at all. I just checked dmesg and followed the output to confirm, but no message like that yet.

I wonder if it may be related to the issues @AdrianDE is having over here: RAK 7391 - No USB Connection on VIA VL805 - #6 by AdrianDE

In one of the posts he mentions that he tried reseting one of the USB2 hubs:

rak@rakpios:/sys/bus/pci/devices $ sudo usbreset 001/002
Resetting USB2.0 Hub ... ok

But that it had no effect.

Perhaps someone from support could step in? I’m eager to start using all (3) mPCIE ports and M.2 reliably.

Greg · February 5, 2023, 9:33am

@barry.Thanks I’ll have a look at the link that you sent through.

Note:
I think I’ve seen you name on a couple of forums discussing configuration of the Basic Station with ChirpStack. Have you had any experience configuring Basic Server with ChirpStack?
I can get it to work using secured connection to the TTN server, but cannot get it to work with ChirpStack even when I try to configure it without authentication and it’s driving me nuts… I’m presently using UDP Packet Forwarder, but it would be nice to get Basic Station working.

Regards
Greg

barryjump · February 8, 2023, 2:04am

@Greg sorry for the late reply. Yes, I’ve been using Basicstation for a while - though never with Chipstack. I use AWS IoT as my LNS.
You might want to check out @xoseperez’s implementation of Basicstation: GitHub - xoseperez/basicstation: Basics™ Station Packet Forward protocol using Docker

Though it might look like it’s more specific to TTN, there is a Chipstack section:

Let me know how it works out.