Skip to content

Capability 004: CAN Adapter Link

Status: Accepted

Context

The Superordinate Controller reaches the CAN bus only through the CAN Adapter. A write to its USB serial device succeeds as long as the device exists, even when the adapter behind it has stopped forwarding; and the device disappears and reappears whenever the adapter restarts.

Following the principle Safe when silent of the mission statement, silence must never pass for health. With this capability, the Superordinate Controller reads the replies of the CAN Adapter, judges its availability from them, and reconnects to it by itself.

References

Details

Availability of the CAN Adapter

The CAN Adapter is available while all of the following hold:

Condition Violated when Holds again when
The serial device is open A read or write fails The device is reopened
Every CAN Frame Line is confirmed A line has no confirmation 100 ms after it was written, or the adapter reports a full queue The next line is confirmed in time
The adapter reports its status No Status Report for 10 s since the device was opened or since the last one The next Status Report arrives

A confirmation is the reply # CAN frame received via USB. Now forwarding on CAN.

Replies of the CAN Adapter

Reply Handling
Confirmation Counts for availability, not logged
# CAN frame received via CAN. Now forwarding on USB. Ignored
> line Handed on as a CAN frame received from the bus
Status Report Counts for availability; the first one after opening the device is logged
Any other # line, e.g. # No other node acknowledged the CAN frame. Logged, at most once per second

Environment Variables

Variable Meaning Default Valid values
CAN_ADAPTER_RECONNECT_DELAY Seconds between two attempts to reopen the serial device 1 Integer from 1 to 60
CAN_ADAPTER_RECONNECT_RETRIES Attempts to reopen before giving up; 0 means unlimited 0 Integer from 0

Functional Requirements

  1. The Superordinate Controller opens the serial device of the CAN Adapter for reading and writing, and for itself alone: no other process may open it while the controller holds it, except one that the operating system exempts, such as one with CAP_SYS_ADMIN on Linux.
  2. It reads every reply of the CAN Adapter and handles it as given under Replies of the CAN Adapter.
  3. It judges the availability of the CAN Adapter as given under Availability of the CAN Adapter. It is the availability that /healthz/ready of capability 002 reports: 503 while the CAN Adapter is unavailable.
  4. While the CAN Adapter is unavailable, a Remote Drive Command is answered with 503 Service Unavailable and yields no Drive Command. A Stop Command still enters the Safe Mode, and is answered with 503 Service Unavailable.
  5. The own cycle of capability 001 goes on while the CAN Adapter is unavailable, so that the Drive gets the Neutral Drive Command as soon as the adapter forwards again.
  6. When a read or write fails, the Superordinate Controller closes the serial device, logs the loss, and tries to reopen it every CAN_ADAPTER_RECONNECT_DELAY seconds. It logs every attempt. After CAN_ADAPTER_RECONNECT_RETRIES failed attempts, it exits with a non-zero status.

Non-Functional Requirements

  1. A CAN Adapter that stops confirming is reported unavailable by /healthz/ready at most 350 ms after it stopped: one cycle time plus the 100 ms of the confirmation. Measured by polling /healthz/ready every 25 ms, from the first unconfirmed CAN Frame Line on a pseudo-terminal.
  2. A serial device that reappears is open again at most CAN_ADAPTER_RECONNECT_DELAY + 0.5 s after it reappeared. Measured by the receive time of the first CAN Frame Line on the new pseudo-terminal.

Technical Requirements

  1. The serial device path is resolved anew on every attempt to open it, so that a link such as /dev/serial/by-id/… follows the adapter when the operating system assigns it another device after a restart.
  2. A device that is no terminal, such as /dev/null, is opened for writing only and counts as always available, so that the controller runs without hardware.

Error Handling

  1. If the serial device is in use by another process at the start, the Superordinate Controller logs it and exits with a non-zero status, as for a device that does not exist (capability 001, Error Handling 3).
  2. An invalid value of CAN_ADAPTER_RECONNECT_DELAY or CAN_ADAPTER_RECONNECT_RETRIES is logged, naming the variable, and the Superordinate Controller exits with a non-zero status before it opens the device.
  3. A line from the CAN Adapter that is no reply of can_adapter.md is logged like an error line.
  4. A change of availability is logged once, with its cause.

Out of Scope

  • Evaluating the counters of the Status Report beyond logging them
  • Resending a line that the CAN Adapter did not confirm; the next cycle follows within 250 ms
  • Reconnecting at the start: a missing or busy device at the start ends the process
  • Showing the availability in the Web App

Acceptance Criteria

All criteria except the last run against a simulated CAN Adapter on a pseudo-terminal.

  1. The simulated adapter confirms every line and sends a Status Report every 5 s: /healthz/ready answers 200 OK, and a Remote Drive Command is answered with 202 Accepted.
  2. The simulated adapter stops confirming: within 350 ms, /healthz/ready answers 503, and a Remote Drive Command is answered with 503 and yields no Drive Command. It confirms again: /healthz/ready answers 200 OK.
  3. The simulated adapter answers a line with # Incoming CAN Frame Line queue is full.: /healthz/ready answers 503 until the next line is confirmed.
  4. The simulated adapter confirms lines but sends no Status Report for 10 s: /healthz/ready answers 503 until the next Status Report.
  5. The simulated adapter sends # No other node acknowledged the CAN frame. 10 times within 1 s: the log contains it once.
  6. While the controller runs, a second process tries to open the serial device: it fails; an exempt process, such as root in a privileged container of the CI, opens it and finds it held exclusively. Started while another process holds the device: exits with a non-zero status and a log message naming the device; not verifiable while the controller itself is exempt.
  7. The pseudo-terminal is closed and a new one is linked in its place 2 s later: /healthz/ready answers 503 meanwhile, the controller logs the loss and each attempt, and within CAN_ADAPTER_RECONNECT_DELAY + 0.5 s the new pseudo-terminal receives CAN Frame Lines and /healthz/ready answers 200 OK.
  8. Started with CAN_ADAPTER_RECONNECT_RETRIES=2 and CAN_ADAPTER_RECONNECT_DELAY=1, the pseudo-terminal is closed and not replaced: the controller exits with a non-zero status after 2 failed attempts.
  9. Started with an invalid value, e.g. CAN_ADAPTER_RECONNECT_DELAY=0, =abc, or CAN_ADAPTER_RECONNECT_RETRIES=-1: exits with a non-zero status and a log message naming the variable.
  10. On the robot: the USB cable of the CAN Adapter is unplugged and plugged in again; the controller reconnects by itself, and a Stop Command from the Web App is answered with 503 while unplugged and with 202 afterwards (manual test test/manual/robot.md).

Decisions

  1. Availability is judged from the replies, not from the success of a write. A write to a USB serial device succeeds as long as the device exists, even when the adapter behind it has stopped. Consequence: the controller depends on the reply format of the CAN Adapter release named in can_adapter.md.
  2. The controller reconnects by itself, a bounded number of times if so configured. The CAN Adapter restarts itself by its watchdog, which makes its USB device disappear and reappear; a reconnect follows it without restarting the controller. Giving up after a configured number of attempts hands recovery to a supervisor such as systemd. Consequence: with the default of unlimited attempts, a permanently lost adapter shows only in /healthz/ready and the log.
  3. The heartbeat goes on while the CAN Adapter is unavailable. Withholding it does not make the robot safer, since the Drive stops by its own timeout when nothing arrives, and it would delay the recovery. Consequence: the controller keeps writing to an adapter that does not confirm.
  4. A Stop Command is executed even while the CAN Adapter is unavailable. Entering the Safe Mode is never wrong; the 503 tells the Robot Operator that it may not have reached the robot. Consequence: a Stop Command answered with 503 still ends the active Remote Drive Command.