MDS CTS Buffer overflow error when trying to add crate or assign slot

[zaphd]

Affiliation
University of Washington / ZaP-HD

Version(s) Affected
The MDSplus Version(s) affected: 7.153.3

Platform(s)
Ubuntu 24.04.4

Installation Method(s)
mdsplus.org repo

Describe the bug
When attempting to add a CAMAC crate (with "mdscts addcrate" or "mdscts assign") or assign a digitizer to a slot, it returns with

*** buffer overflow detected ***: terminated

I had this issue about a year ago and solved it by recreating the crate.db file, but this time that is not solving the issue.

The only lead I have right now is gdb says the dumped core file was created by mdsdcl. My thought is there is some permissions issues with the mds directory (/usr/local/mdsplus/bin) not being allowed to write to etc/cts, but all permission when looking at "ls -l" look correct.

[mwinkel-dev]

Hi @zaphd -- Thanks for the bug report. I'll look into the code now and then later today post a status update to this issue.

[mwinkel-dev]

Hi @zaphd -- Is there any chance that the CTS_DB_DIR environment variable is undefined or contains garbage? If so, it would be attempting to access the crate.db file in a different directory (that might have the wrong permissions).

Also check the permissions on the cts.db file.

(I'm assuming that you have already checked those obvious problems, so will now find out if we have a spare CAMAC crate that can be used for testing the software.)

[zaphd]

Hi @mwinkel-dev, thanks for the help. Yes, I have checked the CTS_DB_DIR with echo and it returns as etc/cts and the crate.db file has read-write permissions.

[mwinkel-dev]

Hi @zaphd -- Thanks for confirming that CTS_DB_DIR is set correctly for your system.

Have you enabled logging by setting the CTS_DEBUG environment variable? If not, set it to a positive integer. For example, setting it to 5 will log all messages of that severity and lower. Here are the possible levels. (Note that IMPORTANT = 2.)

// MSGLVL() values
enum
{
  ALWAYS = 1, // ... well, almost always -- for debug printout
  IMPORTANT,  // not fatal
  FUNCTION_NAME = 5,
  DETAILS = 9
};

My hope is that enabling logging will shed some clues on where the buffer overflow is occurring.

However, if that doesn't pinpoint the problem, then we will likely have to arrange a Zoom session with you.

[mwinkel-dev]

Hi @zaphd -- In addition to the crate.db file, the source code mentions a cts.db file. If the cts.db exists on your system, check its permissions too.

Also note that your CTS_DB_DIR specifies a relative path and not an absolute path. Thus if you run mdscts from a different directory than you usually do, that it probably won't find your <basedir>/etc/cts/crate.db file.

[mwinkel-dev]

Hi @zaphd -- The mdscts command is based on mdsdcl. The latter contains a lex parser to process the command lines. Perhaps what is occurring is that the lex parser is going bonkers and exceeding its buffer.

Are there any unusual characters on the command lines that you are typing? For example, does the crate name contain any special characters that might trip up the lexical analyzer?

[mwinkel-dev]

Hi @zaphd -- The *** buffer overflow detected ***: terminated message is coming from the operating system (or perhaps the SCSI device driver). Apparently, Ubuntu 24.04 has increased security checks.

Do you have a way to determine if your computer's SCSI card is working OK? And that the CAMAC crate isn't returning garbage?

[zaphd]

Hi @mwinkel-dev, the crate.db, cts.db and buffer.db files all have the same level of read-write permissions. I set the CTS_DEBUG to 5 but the dump files are still just blank. I've also tried running mdscts in the cts folder and outside of it, both result in the overflow.

There shouldn't be any characters that are causing that. The crates are gkj5xx:.

I was wondering if it was the increased security checks of Ubutntu 24.04. The same day this issues cropped up, I was able to communicate to the crates using mdsccl just fine so I'm assuming the SCSI card and CAMACs are working. Now that the crate.db file is broken CCL won't work anymore of course.

If I manually make the crate.db file (editing with nano) The crates will communicate, but I cannot assign any modules to them. For some reason even with I manually edit the cts.db file, nothing seems to take effect.

[mwinkel-dev]

Hi @zaphd -- Thanks for the additional details and for doing the additional tests.

We don't compile mdscts with gcc's enhanced security option, -D_FORTIFY_SOURCE. Thus the error message is being thrown by something lower in the software stack that was compiled with enhanced security. The likely possibilities are the SCSI driver or the file system.

It is encouraging news that the mdsccl command was initially able to communicate with the SCSI card and CAMACs.

Try setting the CTS_DEBUG level to 9 and then running mdscts. It probably won't reveal any clues, but is an easy test to do.

Also, have there been any recent changes at your site?

• Installed a new CAMAC crate in a rack?
• Inserted new cards in a CAMAC crate?
• Applied Ubuntu updates?
• Anything else that coincided with the onset of this mdscts problem?

Unfortunately, we do not have any CAMAC crates and cards. Thus, we'll have to troubleshoot this problem via a Zoom session with you. I suggest this plan of action:

1. I'll compile a debug version of mdsdcl and mdscts (plus associated libraries) for Ubuntu 24.04 and provide it to you.
2. You can then temporarily install the debug version, run mdscts and capture the core dump file for us.
3. We'll analyze the core dump to pinpoint the root cause of the buffer overflow.

[zaphd]

Hi @mwinkel-dev -- Setting debug level to 9 did in fact spit something out finally. Not sure how helpful it is but here is the output of trying to add a crate. One not, I changed the location of the cts folder to the home directory in case being in the /etc folder was causing the issue. The CTS_DB_DIR was updated accordingly.

zaphd@aa-zappa:~$ mdscts
CTS> addcrate
_Physical name:: gkj501
library version: $MDSPLUS/lib/libCamShr.so.1.8
build date: 2025-06-30
check_for_file('crate.db')
get_file_name('/home/zaphd/cts/crate.db')
check_for_file('crate.db'): SUCCESS !!!
map_data_file('crate.db')
check_for_file('crate.db')
get_file_name('/home/zaphd/cts/crate.db')
check_for_file('crate.db'): SUCCESS !!!
get_db_file_size('crate.db')
get_file_name('/home/zaphd/cts/crate.db')
gdbfs('crate.db') size 1
get_file_name('/home/zaphd/cts/crate.db')
map_data_file('crate.db'): SUCCESS !!!
get_file_count()
get_db_file_size('crate.db')
get_file_name('/home/zaphd/cts/crate.db')
gdbfs('crate.db') size 1
get_file_count(0):
get_db_file_size('crate.db')
get_file_name('/home/zaphd/cts/crate.db')
gdbfs('crate.db') size 1
get_file_name('/home/zaphd/cts/mdscts_temp_file_XXXXXX')
expand_db()
create_tmp_file()
*** buffer overflow detected ***: terminated
Aborted

This issue with the buffer overflow happened when I replaced our old operations server running ubuntu 18 with a more modern platform and install ubuntu 24. So a lot has changed on that front. No crates have changed though.

Sounds like a plan. Thank you so much for your help.

[mwinkel-dev]

Hi @zaphd -- Thanks for the CTS_DEBUG=9 log. It is good to see that mdscts is doing a fair amount before it crashes. I'll scrutinize that section of code to see if I see any spots where it can derail.

The debug version will be uploaded to the cloud and I'll then arrange to get the link to you.

[mwinkel-dev]

Hi @zaphd -- Note that the paths in your CTS_DEBUG=9 log are in /home/zaphd/cts not in <basedir>/etc/cts.

Clarification: I see that your previous post explained that you changed the directory. Apologies for overlooking that note.

[mwinkel-dev]

Hi @zaphd -- The root cause has been found. Turns out that Ubuntu 24.04 has tightened up the security of the mkstemp() function. Which of course is what the create_tmp_file() function of mdscts is using. I wrote a dinky program that calls mkstemp() the way mdscts does and it crashes on Ubuntu 24.04. Changing it to conform to the new requirements for using mkstemp() works fine.

I will compile a new mdscts for you that you can temporarily install and test. If it works OK for you, then I'll submit a PR to have that fix made to our alpha branch.

ADDEMDNUM: This post was likely wrong. The investigation continues.