VT Server implementation questions

[dlips]

Hi everyone,
I want to start a VT server implementation but I am still new to the Isobus world. Thanks to the great work of @ad3154 we have already a good base for the object tree parser in this branch. I would like to contribute to the VT server but need some guidance. I need to brush up my C++ and gain some deeper knowledge of the 17833 standard, so in the beginning my code and questions will be really basic.

Looking at the implementation of the VT client, I would create a VirtualTerminalServer class that is basically the counter-part of the VirtualTerminalClient class. So VirtualTerminalServer would abstract the communication and present an higher-level interface (with send_* methods and the possibility to register event listeners). Is this a valid approach? Also I would like to hear any other advice you might have.

Thanks to everyone contributing to this library. I am happy to see an open source Isobus library with such a high code quality :)

[ad3154]

Yes, I think that sounds good. The approach @tomstix is going for makes a base class for a VT interface and derives from that a server and client, which is also a fine approach if there's a lot of common code. If not using that approach, it might be good to still re-use some of the enumerations in the client if possible just to minimize duplication, perhaps by putting those in their own header file and nested namespace?

I think we'll want some send_* methods to be private (or protected), and for the class to automatically respond to some client messages based on settings that can be configured in the public interface. For example:

public:
    void set_number_supported_soft_keys(std::uint8_t numberSoftKeys);
    std::uint8_t get_number_supported_soft_keys() const;

private:
   bool  send_get_number_of_soft_keys_response() const;

But many things will need to be public, such as a method that can be called when a button is pressed, held, or released so that the interface can send the appropriate messages, and ways to register for events from a client, such as when they change an attribute, change a number value, change a string value, etc.

[dlips]

Ok, I'm going with the base class approach for now. In case somebody is interested, you can see the current state at my forked repo. Although I just realized that I'm working in the vt_server_managed_working_set branch. Should have made a new branch for this. At the moment, there is only the base class with enums moved from VTClient (including refactor of the vt_client_tests to match the move enums) and an empty scaffold of a server class and corresponding test file). I still need to understand more of the code structure and inner workings of isobus++, but the VTClient class is a good example

[dlips]

I am making a little progress, working towards getting the initialization with a working set master to work. I was thinking about a test setup and wondered how complete the vt client implementation is. For example, would it be feasible to connect the vt client example with my vt server implementation over a virtual can socket on linux? (This goes more in the direction of manual testing. I saw how the unit testing is done for the client, where handcrafted messages are processed. For unit testing I want to implement something similar for the vt server)

[ad3154]

It should be feasible to connect with with vcan, yes. That's how I was doing my testing as well. Just make sure you give them different ISO NAMEs (make sure the function code of the server is the VT function code) and ideally give the VT server the standard preferred address of 38 (0x26). There is a script in our scripts folder as well to set up vcan0 for convenience.

[dlips]

Ok, I am working on the initialization at the moment ("hand-shake" between working set master and VT). I am a bit confused about the PartneredControlFunction. For the VT client example and in the vt client test "VTStatusMessage", the partnered control function is created with a NAME filter for the NAME function VirtualTerminal. Looking at the code of the VirtualTerminalClient, when receiving a VTStatusMessage the contained values for the activeWorkingSet etc are read to the member variables of the client. The point that confuses me is that there seems to be no check if the VTStatusMessage is actually from the VT that was initialized before. In principle there could be multiple terminal, whose message would all match the NAME filter.

In turn, thinking about the implementation for the server, there is the same problem that one could have multiple working set masters on the bus. Maybe the client is meant to be used as a per terminal basis, where one would have additional code, e.g., in the main program that listens for VTStatusMessages and creates a new client if there is a status message from a VT that has not been seen before.
On the server side, this logic needs to be kept in the server, as the VT is required to able to switch between working sets. So to present the UI for switching between working sets, the server needs to know about all connected working sets.

In summary, what I would do is to implement code that listens for working set master PNGs in the main server class. From there a new ManageWorkingSet class is instantiated with a new PartneredControlFunction that has a NAME filter specific to this working set master, whenever a new working set is discovered. Accordingly, the ManageWorkingSet class needs to be extended to include a state machine for managing the connection state of the specific working set. Does this sound to go in the right direction?

[ad3154]

For the client: A name filter for a partner will only ever match one control function on the bus, so if there are 2 terminals on the bus, you'd need a second, unique partner to receive messages from it. If 2 CFs on the bus match a filter, and you only have 1 partner with that filter, you will get which ever is the first to claim its address and matches that filter. So, if you wanted to create an application that tolerates multiple VT servers, you'd need multiple PartneredControlFunction classes, and you could discriminate on which one you want to talk to by using additional name fields, or any other logic you want before instantiating a VT client. Typically you'd use the function instance to do this, which will tell you the VT number you want to use.

For the server: You will want to do an "Any Control Function" callback, not a PartneredControlFunction, which will match all control functions sending a specific PGN. isobus::CANNetworkManager::CANNetwork.add_any_control_function_parameter_group_number_callback.

Then, using that callback, you will want to keep track of all senders of the working set master message as you described, and have some state machine(s) for communication. You can save the regular ControlFunction * for each sender of that message to track your clients reliably without needing any name filters.

It might make sense to have some wrapper class manage the state machine(s) and who is connected, and use the managed working set objects to store state and object pool information for each working set master?

Consider this following code, where when I was trying to do something similar with the status message. Basically I see if the sender of the message is a VT client I know about, and if not, I add a new managed working set.

bool VtServer::check_if_source_is_managed(isobus::CANMessage *message)
{
	// Check if we're managing this CF
	bool foundManagedWorkingSetMatch = false;

	// This is the static callback for the instance.
	// See if we need to set up a new managed working set.
	for (auto cf : managedWorkingSetList)
	{
		if (cf->get_control_function() == message->get_source_control_function())
		{
			// Found a match
			foundManagedWorkingSetMatch = true;
			break;
		}
	}

	if (!foundManagedWorkingSetMatch)
	{
		if ((message->get_data()[0] == static_cast<std::uint8_t>(Function::WorkingSetMaintenanceMessage)) &&
		    (message->get_data()[1] & 0x01)) // Init bit is set
		{
			// This CF is probably trying to initiate communication with us.
			auto tempManagedWorkingSet = new ManagedWorkingSet(message->get_source_control_function());
			auto &data = message->get_data();
			managedWorkingSetList.push_back(tempManagedWorkingSet);

			spdlog::info("Client {} initiated working set maintenance messages with version {}", tempManagedWorkingSet->get_control_function()->get_address(), data[2]);
			if (data[2] > 6)
			{
				spdlog::warn("Client {} version {} is not supported", tempManagedWorkingSet->get_control_function()->get_address(), data[2]);
			}
			tempManagedWorkingSet->set_working_set_maintenance_message_timestamp_ms(isobus::SystemTiming::get_timestamp_ms());
		}
		else
		{
			// Whomever this is has probably timed out. Send them a NACK
			spdlog::warn("Received a non-status message from a client at address {}, but they are not connected to this VT.", message->get_identifier().get_source_address());
			send_acknowledgement(isobus::AcknowledgementType::Negative, static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::ECUtoVirtualTerminal), vtControlFunction, message->get_source_control_function());
		}
	}
	return foundManagedWorkingSetMatch;
}

[ad3154]

Here's the added context of what my receive callback looked like (very prototype, haha):

void VtServer::process_rx_message(isobus::CANMessage *message, void *parentPointer)
{
	if ((nullptr != message) &&
	    (nullptr != message->get_source_control_function()) &&
	    (nullptr != parentPointer) &&
	    (isobus::CAN_DATA_LENGTH <= message->get_data_length()))
	{
		VtServer *targetInstance = reinterpret_cast<VtServer *>(parentPointer);

		if (targetInstance->check_if_source_is_managed(message))
		{
			for (auto cf : targetInstance->managedWorkingSetList)
			{
				if (cf->get_control_function() == message->get_source_control_function())
				{
					const std::string OBJECT_POOLS_PATH = "object_pool_data";
					// Managed working set
					ManagedWorkingSet *workingSet = reinterpret_cast<ManagedWorkingSet *>(parentPointer);
					std::vector<std::uint8_t> &data = message->get_data();
					switch (message->get_identifier().get_parameter_group_number())
					{
						case static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::ECUtoVirtualTerminal):
						{
							switch (data[0])
							{
								case static_cast<std::uint32_t>(Function::ObjectPoolTransferMessage):
								{
									auto tempPool = data; // Make a copy of the data (ouch)
									tempPool.erase(tempPool.begin()); // Strip off the mux byte (double ouch)
									spdlog::info("An ecu at address {} transferred {} bytes of object pool data to us.", message->get_identifier().get_source_address(), tempPool.size());
									cf->add_iop_raw_data(tempPool);
								}
								break;

								case static_cast<std::uint32_t>(Function::GetMemoryMessage):
								{
									std::uint32_t requiredMemory = (data[2] | (static_cast<std::uint32_t>(data[3]) << 8) | (static_cast<std::uint32_t>(data[4]) << 16) | (static_cast<std::uint32_t>(data[5]) << 24));
									spdlog::info("An ecu requested {} bytes of memory.", requiredMemory);
									// We got so much memory in this bad boy, just always reply that there's enough.
									std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
									buffer[0] = static_cast<std::uint8_t>(Function::GetMemoryMessage);
									buffer[1] = 5; // IS Version ISO11783 - 6 : 2014(E), Third Edition
									buffer[2] = 0; // There is enough memory
									buffer[3] = 0xFF; // Reserved
									buffer[4] = 0xFF; // Reserved
									buffer[5] = 0xFF; // Reserved
									buffer[6] = 0xFF; // Reserved
									buffer[7] = 0xFF; // Reserved
									isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
									                                                       buffer.data(),
									                                                       isobus::CAN_DATA_LENGTH,
									                                                       targetInstance->vtControlFunction,
									                                                       message->get_source_control_function(),
									                                                       isobus::CANIdentifier::PriorityLowest7);
								}
								break;

								case static_cast<std::uint32_t>(Function::GetNumberOfSoftKeysMessage):
								{
									std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
									buffer[0] = static_cast<std::uint8_t>(Function::GetNumberOfSoftKeysMessage);
									buffer[1] = 0; // No navigation softkeys
									buffer[2] = 0xFF; // Reserved
									buffer[3] = 0xFF; // Reserved
									buffer[4] = 120; // Pixel width of X softkey descriptor
									buffer[5] = 64; // Pixel width of Y softkey descriptor
									buffer[6] = 64; // Number of possible virtual Soft Keys in a Soft Key Mask
									buffer[7] = 0; // No physical softkeys

									isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
									                                                       buffer.data(),
									                                                       isobus::CAN_DATA_LENGTH,
									                                                       targetInstance->vtControlFunction,
									                                                       message->get_source_control_function(),
									                                                       isobus::CANIdentifier::PriorityLowest7);
								}
								break;

								case static_cast<std::uint32_t>(Function::GetTextFontDataMessage):
								{
									std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
									buffer[0] = static_cast<std::uint8_t>(Function::GetTextFontDataMessage);
									buffer[1] = 0xFF; // Reserved
									buffer[2] = 0xFF; // Reserved
									buffer[3] = 0xFF; // Reserved
									buffer[4] = 0xFF; // Reserved
									buffer[5] = 0xFF; // Say we support all small fonts
									buffer[6] = 0xFF; // Say we support all large fonts
									buffer[7] = 0x8F; // Support normal, bold, italic, proportional
									isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
									                                                       buffer.data(),
									                                                       isobus::CAN_DATA_LENGTH,
									                                                       targetInstance->vtControlFunction,
									                                                       message->get_source_control_function(),
									                                                       isobus::CANIdentifier::PriorityLowest7);
								}
								break;

								case static_cast<std::uint32_t>(Function::GetHardwareMessage):
								{
									std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
									buffer[0] = static_cast<std::uint8_t>(Function::GetHardwareMessage);
									buffer[1] = 0xFF; // Power-up time
									buffer[2] = 2; // 256 Colour Mode
									buffer[3] = 0x0F; // Support pointing event message
									buffer[4] = (DATA_MASK_AREA_SIZE_PX & 0xFF); // X Pixels LSB
									buffer[5] = (DATA_MASK_AREA_SIZE_PX >> 8); // X Pixels MSB
									buffer[6] = (DATA_MASK_AREA_SIZE_PX & 0xFF); // Y Pixels LSB
									buffer[7] = (DATA_MASK_AREA_SIZE_PX >> 8); // Y Pixels MSB
									isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
									                                                       buffer.data(),
									                                                       isobus::CAN_DATA_LENGTH,
									                                                       targetInstance->vtControlFunction,
									                                                       message->get_source_control_function(),
									                                                       isobus::CANIdentifier::PriorityLowest7);
								}
								break;

								case static_cast<std::uint32_t>(Function::GetSupportedWidecharsMessage):
								{
								}
								break;

								case static_cast<std::uint32_t>(Function::LoadVersionCommand):
								{
									std::uint8_t errorCodes = 0x01; // Version label incorrect
									if (std::filesystem::is_directory(OBJECT_POOLS_PATH))
									{
										constexpr std::uint8_t VERSION_LABEL_LENGTH = 7;
										std::string versionLabel;
										versionLabel.reserve(VERSION_LABEL_LENGTH);

										for (std::uint_fast8_t i = 0; i < VERSION_LABEL_LENGTH; i++)
										{
											versionLabel.push_back(static_cast<char>(data[i + 1]));
										}

										for (const auto &entry : std::filesystem::directory_iterator(OBJECT_POOLS_PATH))
										{
											if (std::string::npos != entry.path().filename().string().find(cf->get_control_function()->get_NAME().get_full_name()))
											{
												bool anyVersionMatched = false;

												// Check version label
												if (std::string::npos != entry.path().filename().string().find(versionLabel))
												{
													// Version label matched
													anyVersionMatched = true;
													spdlog::debug("Attempting to load {}", entry.path().filename().string());
													std::ifstream iopFile(entry.path(), std::ios::binary);

													if (iopFile)
													{
														auto fileData = isobus::IOPFileInterface::read_iop_file(entry.path().string());

														if (0 != fileData.size())
														{
															spdlog::info("Loaded {}", entry.path().filename().string());
															cf->add_iop_raw_data(fileData);
															errorCodes = 0;
														}
														else
														{
															spdlog::warn("Failed to load {} after opening it", entry.path().string());
															errorCodes |= 0x01;
														}
													}
													else
													{
														spdlog::warn("Failed to load {}", entry.path().filename().string());
														errorCodes |= 0x01;
													}
												}
												else
												{
													// This one didn't match.
													spdlog::debug("Ignoring {} because the version label did not match.", entry.path().filename().string());
												}
											}
											else
											{
												spdlog::warn("Could not load object pool for ECU {} because no pools matched the NAME", message->get_identifier().get_source_address());
											}
										}
									}
									else
									{
										spdlog::warn("Could not load object pool for ECU {} because no pools exist", message->get_identifier().get_source_address());
									}

									if (cf->get_any_object_pools())
									{
										cf->start_parsing_thread();
										spdlog::debug("Starting parsing thread for loaded pool data.");
									}
									std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
									buffer[0] = static_cast<std::uint8_t>(Function::LoadVersionCommand);
									buffer[1] = 0xFF; // Reserved
									buffer[2] = 0xFF; // Reserved
									buffer[3] = 0xFF; // Reserved
									buffer[4] = 0xFF; // Reserved
									buffer[5] = errorCodes;
									buffer[6] = 0xFF; // Reserved
									buffer[7] = 0xFF; // Reserved
									isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
									                                                       buffer.data(),
									                                                       isobus::CAN_DATA_LENGTH,
									                                                       targetInstance->vtControlFunction,
									                                                       message->get_source_control_function(),
									                                                       isobus::CANIdentifier::PriorityLowest7);
								}
								break;

								case static_cast<std::uint32_t>(Function::StoreVersionCommand):
								{
									if (cf->get_any_object_pools())
									{
										constexpr std::uint8_t VERSION_LABEL_LENGTH = 7;
										std::string cfName = std::to_string(cf->get_control_function()->get_NAME().get_full_name());
										std::string versionLabel;
										bool allPoolsSaved = true;
										versionLabel.reserve(VERSION_LABEL_LENGTH);
										std::filesystem::create_directory(OBJECT_POOLS_PATH);

										for (std::uint_fast8_t i = 0; i < VERSION_LABEL_LENGTH; i++)
										{
											versionLabel.push_back(static_cast<char>(data[i + 1]));
										}

										for (std::size_t i = 0; i < cf->get_number_iop_files(); i++)
										{
											std::ofstream iopFile(OBJECT_POOLS_PATH + "/" + cfName + "_" + std::to_string(i) + "_" + versionLabel + "_" + ".iop", std::ios::out | std::ios::binary);

											if (iopFile)
											{
												auto &iop = cf->get_iop_raw_data(i);
												iopFile.write(reinterpret_cast<char *>(&iop[0]), iop.size() * sizeof(std::uint8_t));
												iopFile.close();
												spdlog::info("Object pool {} for NAME {} was stored with version label: {}", i, cfName, versionLabel);
											}
											else
											{
												spdlog::warn("Object pool {} for NAME {} could not be stored.", i, cfName);
												allPoolsSaved = false;
												break;
											}
										}

										std::array<std::uint8_t, isobus::CAN_DATA_LENGTH> buffer = { 0 };
										buffer[0] = static_cast<std::uint8_t>(Function::StoreVersionCommand);
										buffer[1] = 0xFF; // Reserved
										buffer[2] = 0xFF; // Reserved
										buffer[3] = 0xFF; // Reserved
										buffer[4] = 0xFF; // Reserved
										if (allPoolsSaved)
										{
											buffer[5] = 0; // No error
										}
										else
										{
											buffer[5] = 0x04; // Any other error
										}
										buffer[6] = 0xFF; // Reserved
										buffer[7] = 0xFF; // Reserved
										isobus::CANNetworkManager::CANNetwork.send_can_message(static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::VirtualTerminalToECU),
										                                                       buffer.data(),
										                                                       isobus::CAN_DATA_LENGTH,
										                                                       targetInstance->vtControlFunction,
										                                                       message->get_source_control_function(),
										                                                       isobus::CANIdentifier::PriorityLowest7);
									}
									else
									{
										// Whomever this is is being bad, send them a NACK
										targetInstance->send_acknowledgement(isobus::AcknowledgementType::Negative, static_cast<std::uint32_t>(isobus::CANLibParameterGroupNumber::ECUtoVirtualTerminal), targetInstance->vtControlFunction, cf->get_control_function());
									}
								}
								break;

								case static_cast<std::uint32_t>(Function::EndOfObjectPoolMessage):
								{
									if (cf->get_any_object_pools())
									{
										cf->start_parsing_thread();
									}
									else
									{
										spdlog::warn("End of object pool message ignored - no object pools are loaded for the source control function");
									}
								}
								break;

								case static_cast<std::uint32_t>(Function::WorkingSetMaintenanceMessage):
								{
									if (0 != cf->get_working_set_maintenance_message_timestamp_ms())
									{
										cf->set_working_set_maintenance_message_timestamp_ms(isobus::SystemTiming::get_timestamp_ms());
									}
								}
								break;

								default:
								{
								}
								break;
							}
						}
						break;

						default:
						{
						}
						break;
					}
					break;
				}
			}
		}
	}
	else
	{
		spdlog::warn("VT received a garbage message, ignoring. Consider opening a ticket on github for this issue with an associated CAN trace.");
	}
}

[dlips]

Thank you, I think I got it now. The thing that confused me was the list of partnered control functions in the PartneredControlFunction class because I missed that it is static. I agree that it is good idea to separate some functionalities of the state management to a wrapper class. Maybe this could be the ManagedWorkingSet class and we can have a separate class VTObjectPoolParser that is just responsible for parsing an object tree.

[ad3154]

Minor update to this thread... I am still trying to create a VT server, and I've made a great deal of progress. I now have the majority of objects rendering properly, all the handshaking, and button/soft-key events handled.

This project is taking a MASSIVE amount of effort, but the work goes on.

So, of course some things are still not 100%, but the end is in sight at this point.

Stay tuned, I guess, haha 🙂

[ad3154]

I would like to know about that if the "VT to ECU Message" rate is a standard value?

Only the status message multiplexer has a standard interval. The amount of time the VT takes to complete a command and send a reply is not defined. The standard even acknowledges that "operations can take an indefinite amount of time" and the "response message can be delayed accordingly" in ISO11783-6 Annex E. So unfortunately it does happen where some screens take longer than others to do some tasks. 100ms is pretty brutal

[wynrbwa]

Thank you very much!

[cuixiaopi]

Hello, is AgIsoUniversalTerminal open now? I want to perform localization testing.

[ad3154]

Soon I think. I just sent off some messages about this today. I am hoping to get clarity here early next week, as it's becoming a bit crazy how long it's been ready but unavailable... believe me, as soon as it's up, I'll post about it everywhere.

[arcostasi]

Hello @ad3154

I've been following the discussion and appreciate for the updates shared thus far. Similarly to @cuixiaopi, I am interested in utilizing and conducting tests with the AgIsoUniversalTerminal.

Could you please provide any updates on the availability of the AgIsoUniversalTerminal for testing? I am eager to start working with it as soon as it becomes available and would appreciate any information on availability and access.

Thank you!

[DrMoster]

hi, dear friends, I have encountered some problems in the process of developing VT.
When I have all the object pools, the ECU will send me Command and macro messages. However, I found that the Command and macro messages were not sent completely through the CAN analyzer. May I ask what might be the cause of this.

[ad3154]

Hello!

Can you give us some more information so we can see if we can help you?

Are you using our library, and if so, for which side - client or server or both? Which branch/commit?
If you are using our library, can you provide all logging output from the stack?
Can you include a candump, trace, or vector format log?
What kind of hardware are you using and are you using our drivers? If so, which one(s)?