Scarlet: No Camera Images

[lcamero]

Hi nerian Support Team,

I am working with a Scarlet Stereo Camera mounted on an Unmanned Ground Vehicle (UGV) running ROS and am running into an issue that I've found no comparable examples of in these forums.

At first, everything on the web-interface worked correctly. I had issues getting the ROS driver to connect to the camera; I changed the camera's IP to the sub-net we use (192.168.131.x) and lowered settings to the 1024x768 resolution at 32 fps; 256 pixels disparity range; disparity map and left image preset.

As I worked on other devices on this UGV, I abruptly cut power to the camera multiple times.

Now, I am not able to obtain any images from the Preview tab in the web-interface, nor am I able to run any calibration after changing presets.

Taking a look at the System Status it seems that the cameras are stuck in some sort of crash-reboot loop, which I saw before in these forums: Process stereod stalled for 10.2366s! Restarting!:

00:28:55 daemon.info stereod: Daemon started
00:28:55 daemon.info stereod: FPGA: IP build date was 2023-06-07 22:20
00:28:55 daemon.info stereod: TemperatureMonitor: Access to system management wizard
00:28:55 daemon.info stereod: CameraManager::subsystemInitialize(): initializing CameraManager
00:28:55 daemon.info stereod: Resetting image sensors
00:28:55 daemon.info stereod: Using configured ROI of 2432x2048
00:28:55 daemon.info stereod: Resetting VDMA 0
00:28:55 daemon.info stereod: Resetting VDMA 1
00:28:56 daemon.info stereod: ServerSideDataChannelService::subsystemInitialize() - registering and starting data channels
00:28:56 daemon.info stereod: IMU: Attempting GPIO reset and I2C setup
00:28:56 daemon.info stereod: Initialized IMU BNO080 channel
00:28:56 daemon.info stereod: RectificationCalc::subsystemInitialize(): initializing RectificationCalc
00:28:56 daemon.info stereod: Camera projection matrix factors are: scaleX 2 scaleY 2
00:28:56 daemon.info stereod: Relative scaling with respect to calibrated resolution is: relX 2 relY 2
00:28:56 daemon.info stereod: Revised calibration with selected ROI of 1216x1024 @ 2,1
00:28:56 daemon.info stereod: IMU: Initial reset of BNO080
00:28:56 daemon.info stereod: IMU: Polling for GPIO interrupt assertion / SHTP transfer every 100us
00:28:56 daemon.info stereod: IMU: Updating config from parameters
00:28:56 daemon.info stereod: IMU: Resetting and configuring BNO080
00:28:57 daemon.info stereod: Rectification adapted for currently active ROI.
00:28:57 daemon.info stereod: Activating image masking
00:28:57 daemon.info stereod: TemperatureMonitor::subsystemInitialize
00:28:57 daemon.info stereod: Resetting FPGA
00:28:57 daemon.info stereod: All 10 subsystems initialized and nominal, ready to go live.
00:28:57 daemon.info stereod: IMU: Setting BNO080 orientation with quaternion -0.707107 0 0 0.707107
00:28:57 daemon.info stereod: IMU: Setting BNO080 sensor channel report frequencies
00:28:57 daemon.info stereod: IMU: Polling sensor data ring buffer every 40000us
00:29:02 daemon.warn stereod: No new sensor frame from sensor 1 for 5s
00:29:02 daemon.warn stereod: Sensor 0 last frame: 4.986s
00:29:02 daemon.warn stereod: Sensor 1 last frame: 5s
00:29:02 daemon.info stereod: -------------------------------------------------------------
00:29:02 daemon.info stereod: VDMA0:
00:29:02 daemon.info stereod: Status register: 00000000000000000000000000001111
00:29:02 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:29:02 daemon.info stereod: Bytes available 0: 3735552
00:29:02 daemon.info stereod: Bytes available 1: 3735552
00:29:02 daemon.info stereod: FIFO overruns 0: 0
00:29:02 daemon.info stereod: FIFO overruns 1: 0
00:29:02 daemon.info stereod: Short frame errors: 0
00:29:02 daemon.info stereod: FIFO overrun errors: 0
00:29:02 daemon.info stereod:
00:29:02 daemon.info stereod: SENSOR a0070000:
00:29:02 daemon.info stereod: Status register: 00000000000000000000000000000010
00:29:02 daemon.info stereod: Flags: 00000000000000010000000000000000
00:29:02 daemon.info stereod: Frames: 121
00:29:02 daemon.info stereod: Errors: 1467
00:29:02 daemon.info stereod: Fifo max: 2
00:29:02 daemon.info stereod: Phase shift: 175
00:29:02 daemon.info stereod: Phase done: 0
00:29:02 daemon.info stereod: Phase sync: 1
00:29:02 daemon.info stereod: PHFSM(0): 0
00:29:02 daemon.info stereod: CLKINVERT: 0
00:29:02 daemon.info stereod: PH_SGN_CNT: 24
00:29:02 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:29:02 daemon.info stereod: -------------------------------------------------------------
00:29:02 daemon.info stereod: -------------------------------------------------------------
00:29:02 daemon.info stereod: VDMA1:
00:29:02 daemon.info stereod: Status register: 00000000000000000000000000000000
00:29:02 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:29:02 daemon.info stereod: Bytes available 0: 0
00:29:02 daemon.info stereod: Bytes available 1: 0
00:29:02 daemon.info stereod: FIFO overruns 0: 0
00:29:02 daemon.info stereod: FIFO overruns 1: 0
00:29:02 daemon.info stereod: Short frame errors: 0
00:29:02 daemon.info stereod: FIFO overrun errors: 0
00:29:02 daemon.info stereod:
00:29:02 daemon.info stereod: SENSOR a0080000:
00:29:02 daemon.info stereod: Status register: 00000000000000000000000000000000
00:29:02 daemon.info stereod: Flags: 00000000000000010000000000000000
00:29:02 daemon.info stereod: Frames: 0
00:29:02 daemon.info stereod: Errors: 44722
00:29:02 daemon.info stereod: Fifo max: 0
00:29:02 daemon.info stereod: Phase shift: 118
00:29:02 daemon.info stereod: Phase done: 0
00:29:02 daemon.info stereod: Phase sync: 1
00:29:02 daemon.info stereod: PHFSM(0): 0
00:29:02 daemon.info stereod: CLKINVERT: 0
00:29:02 daemon.info stereod: PH_SGN_CNT: 0
00:29:02 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:29:02 daemon.info stereod: -------------------------------------------------------------
00:29:07 daemon.err watchdogd: Process stereod stalled for 10.2366s! Restarting!
00:29:07 daemon.info stereod: Caught signal 15
00:29:09 daemon.info nvparamd: D-Bus client name com.nerian.stereod was removed
00:29:09 daemon.info nvparamd: D-Bus client name com.nerian.stereod was added
00:29:10 daemon.info nvparamd: Universal fallback governor set to D-Bus client :1.273

The output above repeats and I am unable to get any images.

I am using the latest firmware: 3.3.1.

I have attached my configuration settings.

Is there something in the configuration that is causing this? Could it be my UGV's network? Are there any other logs that I could provide?

Thanks,
Luis Camero, Clearpath Robotics Inc.

[k.schauwecker]

Hi Luis,
we are testing your configuration and will get back to you shortly.

Regards,
Konstantin

[lcamero]

Hi Konstantin,

I appreciate the quick reply.

Let me know if there's anything else you need from me.

Thanks,
Luis Camero

[k.schauwecker]

Hi Luis,

we have checked your configuration logs. Judging from the data, the most likely explanation is an electrical defect inside the camera. You would need to mail the camera to our office in Germany for it to be repaired.

Before doing so, there is one last thing that you can try, which is to perform a full firmware reset. To do so, unscrew the sealing screw on the back to give access to the reset button. Hold this button pressed while the power supply is being connected. Then wait until the device is detected again by NVCom on a connected computer.

This procedure will bring the device back to the firmware state at which it has left our factory. This means that all firmware updates, configuration changes and calibration data will be lost. If the camera still doesn't deliver image data afterwards, please mail it to us and we will get it fixed.

Best regards,
Konstantin

[lcamero]

Hi Konstantin,

Unfortunately, resetting the camera did not resolve the issue.

It continues having the same issues, see the new logs:

00:22:47 daemon.err watchdogd: Process stereod stalled for 10.2842s! Restarting!
00:22:47 daemon.info stereod: Caught signal 15
00:22:49 daemon.info nvparamd: D-Bus client name com.nerian.stereod was removed
00:22:49 daemon.info nvparamd: D-Bus client name com.nerian.stereod was added
00:22:49 daemon.info nvparamd: Universal fallback governor set to D-Bus client :1.114
00:22:49 daemon.info stereod: Daemon started
00:22:49 daemon.info stereod: FPGA: IP build date was 2022-12-09 00:10
00:22:49 daemon.info stereod: TemperatureMonitor: Access to system management wizard
00:22:49 daemon.info stereod: CameraManager::subsystemInitialize(): initializing CameraManager
00:22:49 daemon.info stereod: Resetting image sensors
00:22:50 daemon.info stereod: Using configured ROI of 2432x2048
00:22:50 daemon.info stereod: Resetting VDMA 0
00:22:50 daemon.info stereod: Resetting VDMA 1
00:22:50 daemon.info stereod: ServerSideDataChannelService::subsystemInitialize() - registering and starting data channels
00:22:50 daemon.info stereod: IMU: Attempting GPIO reset and I2C setup
00:22:50 daemon.info stereod: Initialized IMU BNO080 channel
00:22:50 daemon.info stereod: RectificationCalc::subsystemInitialize(): initializing RectificationCalc
00:22:50 daemon.info stereod: Camera projection matrix factors are: scaleX 1 scaleY 1
00:22:50 daemon.info stereod: Relative scaling with respect to calibrated resolution is: relX 1 relY 1
00:22:50 daemon.info stereod: Revised calibration with selected ROI of 2432x2048 @ 4,1
00:22:50 daemon.info stereod: IMU: Initial reset of BNO080
00:22:50 daemon.info stereod: IMU: Polling for GPIO interrupt assertion / SHTP transfer every 100us
00:22:50 daemon.info stereod: IMU: Updating config from parameters
00:22:50 daemon.info stereod: IMU: Resetting and configuring BNO080
00:22:51 daemon.info stereod: IMU: Setting BNO080 orientation with quaternion -0.707107 0 0 0.707107
00:22:51 daemon.info stereod: IMU: Setting BNO080 sensor channel report frequencies
00:22:51 daemon.info stereod: IMU: Polling sensor data ring buffer every 40000us
00:22:54 daemon.info stereod: Rectification adapted for currently active ROI.
00:22:54 daemon.info stereod: Activating image masking
00:22:54 daemon.info stereod: TemperatureMonitor::subsystemInitialize
00:22:54 daemon.info stereod: Resetting FPGA
00:22:54 daemon.info stereod: All 10 subsystems initialized and nominal, ready to go live.
00:22:59 daemon.warn stereod: No new sensor frame from sensor 1 for 5s
00:22:59 daemon.warn stereod: Sensor 0 last frame: 5s
00:22:59 daemon.warn stereod: Sensor 1 last frame: 5s
00:22:59 daemon.info stereod: -------------------------------------------------------------
00:22:59 daemon.info stereod: VDMA0:
00:22:59 daemon.info stereod: Status register: 00000000000000000000000000001111
00:22:59 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:22:59 daemon.info stereod: Bytes available 0: 14942208
00:22:59 daemon.info stereod: Bytes available 1: 14942208
00:22:59 daemon.info stereod: FIFO overruns 0: 0
00:22:59 daemon.info stereod: FIFO overruns 1: 0
00:22:59 daemon.info stereod: Short frame errors: 0
00:22:59 daemon.info stereod: FIFO overrun errors: 0
00:22:59 daemon.info stereod:
00:22:59 daemon.info stereod: SENSOR a0070000:
00:22:59 daemon.info stereod: Status register: 00000000000000000000000000000010
00:22:59 daemon.info stereod: Flags: 00000000000000010000000000000000
00:22:59 daemon.info stereod: Frames: 33
00:22:59 daemon.info stereod: Errors: 711
00:22:59 daemon.info stereod: Fifo max: 2
00:22:59 daemon.info stereod: Phase shift: 250
00:22:59 daemon.info stereod: Phase done: 0
00:22:59 daemon.info stereod: Phase sync: 1
00:22:59 daemon.info stereod: PHFSM(0): 0
00:22:59 daemon.info stereod: CLKINVERT: 0
00:22:59 daemon.info stereod: PH_SGN_CNT: 0
00:22:59 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:22:59 daemon.info stereod: -------------------------------------------------------------
00:22:59 daemon.info stereod: -------------------------------------------------------------
00:22:59 daemon.info stereod: VDMA1:
00:22:59 daemon.info stereod: Status register: 00000000000000000000000000000000
00:22:59 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:22:59 daemon.info stereod: Bytes available 0: 0
00:22:59 daemon.info stereod: Bytes available 1: 0
00:22:59 daemon.info stereod: FIFO overruns 0: 0
00:22:59 daemon.info stereod: FIFO overruns 1: 0
00:22:59 daemon.info stereod: Short frame errors: 0
00:22:59 daemon.info stereod: FIFO overrun errors: 0
00:22:59 daemon.info stereod:
00:22:59 daemon.info stereod: SENSOR a0080000:
00:22:59 daemon.info stereod: Status register: 00000000000000000000000000000000
00:22:59 daemon.info stereod: Flags: 00000000000000010000000000000000
00:22:59 daemon.info stereod: Frames: 0
00:22:59 daemon.info stereod: Errors: 6915
00:22:59 daemon.info stereod: Fifo max: 0
00:22:59 daemon.info stereod: Phase shift: 35
00:22:59 daemon.info stereod: Phase done: 0
00:22:59 daemon.info stereod: Phase sync: 1
00:22:59 daemon.info stereod: PHFSM(0): 0
00:22:59 daemon.info stereod: CLKINVERT: 0
00:22:59 daemon.info stereod: PH_SGN_CNT: 0
00:22:59 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:22:59 daemon.info stereod: -------------------------------------------------------------
00:23:04 daemon.err watchdogd: Process stereod stalled for 10.2867s! Restarting!
00:23:04 daemon.info stereod: Caught signal 15
00:23:06 daemon.info nvparamd: D-Bus client name com.nerian.stereod was removed
00:23:07 daemon.info nvparamd: D-Bus client name com.nerian.stereod was added
00:23:07 daemon.info nvparamd: Universal fallback governor set to D-Bus client :1.118
00:23:07 daemon.info stereod: Daemon started
00:23:07 daemon.info stereod: FPGA: IP build date was 2022-12-09 00:10
00:23:07 daemon.info stereod: TemperatureMonitor: Access to system management wizard
00:23:07 daemon.info stereod: CameraManager::subsystemInitialize(): initializing CameraManager
00:23:07 daemon.info stereod: Resetting image sensors
00:23:07 daemon.info stereod: Using configured ROI of 2432x2048
00:23:07 daemon.info stereod: Resetting VDMA 0
00:23:07 daemon.info stereod: Resetting VDMA 1
00:23:07 daemon.info stereod: ServerSideDataChannelService::subsystemInitialize() - registering and starting data channels
00:23:07 daemon.info stereod: IMU: Attempting GPIO reset and I2C setup
00:23:07 daemon.info stereod: Initialized IMU BNO080 channel
00:23:07 daemon.info stereod: RectificationCalc::subsystemInitialize(): initializing RectificationCalc
00:23:07 daemon.info stereod: Camera projection matrix factors are: scaleX 1 scaleY 1
00:23:07 daemon.info stereod: Relative scaling with respect to calibrated resolution is: relX 1 relY 1
00:23:07 daemon.info stereod: Revised calibration with selected ROI of 2432x2048 @ 4,1
00:23:07 daemon.info stereod: IMU: Initial reset of BNO080
00:23:07 daemon.info stereod: IMU: Polling for GPIO interrupt assertion / SHTP transfer every 100us
00:23:08 daemon.info stereod: IMU: Updating config from parameters
00:23:08 daemon.info stereod: IMU: Resetting and configuring BNO080
00:23:09 daemon.info stereod: IMU: Setting BNO080 orientation with quaternion -0.707107 0 0 0.707107
00:23:09 daemon.info stereod: IMU: Setting BNO080 sensor channel report frequencies
00:23:09 daemon.info stereod: IMU: Polling sensor data ring buffer every 40000us
00:23:11 daemon.info stereod: Rectification adapted for currently active ROI.
00:23:11 daemon.info stereod: Activating image masking
00:23:11 daemon.info stereod: TemperatureMonitor::subsystemInitialize
00:23:11 daemon.info stereod: Resetting FPGA
00:23:11 daemon.info stereod: All 10 subsystems initialized and nominal, ready to go live.
00:23:16 daemon.warn stereod: No new sensor frame from sensor 1 for 5s
00:23:16 daemon.warn stereod: Sensor 0 last frame: 5s
00:23:16 daemon.warn stereod: Sensor 1 last frame: 5s
00:23:16 daemon.info stereod: -------------------------------------------------------------
00:23:16 daemon.info stereod: VDMA0:
00:23:16 daemon.info stereod: Status register: 00000000000000000000000000001111
00:23:16 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:23:16 daemon.info stereod: Bytes available 0: 14942208
00:23:16 daemon.info stereod: Bytes available 1: 14942208
00:23:16 daemon.info stereod: FIFO overruns 0: 0
00:23:16 daemon.info stereod: FIFO overruns 1: 0
00:23:16 daemon.info stereod: Short frame errors: 0
00:23:16 daemon.info stereod: FIFO overrun errors: 0
00:23:16 daemon.info stereod:
00:23:16 daemon.info stereod: SENSOR a0070000:
00:23:16 daemon.info stereod: Status register: 00000000000000000000000000000010
00:23:16 daemon.info stereod: Flags: 00000000000000010000000000000000
00:23:16 daemon.info stereod: Frames: 33
00:23:16 daemon.info stereod: Errors: 801
00:23:16 daemon.info stereod: Fifo max: 2
00:23:16 daemon.info stereod: Phase shift: 250
00:23:16 daemon.info stereod: Phase done: 0
00:23:16 daemon.info stereod: Phase sync: 1
00:23:16 daemon.info stereod: PHFSM(0): 0
00:23:16 daemon.info stereod: CLKINVERT: 0
00:23:16 daemon.info stereod: PH_SGN_CNT: 0
00:23:16 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:23:16 daemon.info stereod: -------------------------------------------------------------
00:23:16 daemon.info stereod: -------------------------------------------------------------
00:23:16 daemon.info stereod: VDMA1:
00:23:16 daemon.info stereod: Status register: 00000000000000000000000000000000
00:23:16 daemon.info stereod: ROI: (16, 4) to (2456, 2054)
00:23:16 daemon.info stereod: Bytes available 0: 0
00:23:16 daemon.info stereod: Bytes available 1: 0
00:23:16 daemon.info stereod: FIFO overruns 0: 0
00:23:16 daemon.info stereod: FIFO overruns 1: 0
00:23:16 daemon.info stereod: Short frame errors: 0
00:23:16 daemon.info stereod: FIFO overrun errors: 0
00:23:16 daemon.info stereod:
00:23:16 daemon.info stereod: SENSOR a0080000:
00:23:16 daemon.info stereod: Status register: 00000000000000000000000000000000
00:23:16 daemon.info stereod: Flags: 00000000000000010000000000000000
00:23:16 daemon.info stereod: Frames: 0
00:23:16 daemon.info stereod: Errors: 27806
00:23:16 daemon.info stereod: Fifo max: 0
00:23:16 daemon.info stereod: Phase shift: 259
00:23:16 daemon.info stereod: Phase done: 0
00:23:16 daemon.info stereod: Phase sync: 1
00:23:16 daemon.info stereod: PHFSM(0): 0
00:23:16 daemon.info stereod: CLKINVERT: 0
00:23:16 daemon.info stereod: PH_SGN_CNT: 24
00:23:16 daemon.info stereod: PH_SGN_CNT_DIFF: 1
00:23:16 daemon.info stereod: -------------------------------------------------------------
00:23:22 daemon.err watchdogd: Process stereod stalled for 10.2802s! Restarting!

Would you help me start the process of sending the Scarlet back to you?

Thanks,
Luis Camero

[k.schauwecker]

Hi Luis,

I just sent you an email with return instructions.

Best regards,
Konstantin