Rule 17 no subdirectories?

[SirWumpus]

Your entry may NOT have subdirectories. However, you can provide files that are in different directories as mkiocccentry(1) will copy them to the work directory (see the guidelines for more details).

I think this is mistake to disallow subdirectories.

• Hibachi used subdirectories in order to arrange a viewable site with images and CGI-BIN scripts.

• This will make a MESS of any test subdirectory and 10's or 100's a bloody lot of data files that could end up being promoted to the current working directory and breaking a test suite structure.

As a work around, maybe suggest that any such subdirectories (tests, data, etc.) be submitted with the entry as a tarball (.tgz or .txz) with a Makefile rule to unpack the support files.

[xexyl]

Your entry may NOT have subdirectories. However, you can provide files that are in different directories as mkiocccentry(1) will copy them to the work directory (see the guidelines for more details).

I think this is mistake to disallow subdirectories.

• Hibachi used subdirectories in order to arrange a viewable site with images and CGI-BIN scripts.

• This will make a MESS of any test subdirectory and 10's or 100's a bloody lot of data files that could end up being promoted to the current working directory and breaking a test suite structure.

As a work around, maybe suggest that any such subdirectories (tests, data, etc.) be submitted with the entry as a tarball (.tgz or .txz) with a Makefile rule to unpack the support files.

Those are old entries. But the idea is that the tarball must only have one directory.

One could include another archive in it if they wished. See the source code to txzchk for examples of how these are done. It uses fnamchk and mkiocccentry calls txzchk.

Hopefully that clarifies it?

[SirWumpus]

Those are old entries

Huh? Hibachi might be old, but it did have supporting file tree.

But the idea is that the tarball must only have one directory.

That does not help. I kind expect something like:

2024-entry-1234/
    .auth.json
    .info.json
    Makefile
    prog.c
    remarks.md
    test/
        Makefile
        input/
        output/
        ...

My understanding is that mkioccentry will promote all the contents of test/ into the top level, which will utterly destroy a test tree. So maybe allow:

2024-entry-1234/
    .auth.json
    .info.json
    Makefile
    prog.c
    remarks.md
    test-tree.tgz
    support-tree.tgz

[xexyl]

Those are old entries

Huh? Hibachi might be old, but it did have supporting file tree.

But the idea is that the tarball must only have one directory.

That does not help. I kind expect something like:

2024-entry-1234/
    .auth.json
    .info.json
    Makefile
    prog.c
    remarks.md
    test/
        Makefile
        input/
        output/
        ...

My understanding is that mkioccentry will promote all the contents of test/ into the top level, which will utterly destroy a test tree. So maybe allow:

Nope. You can't specify a subdirectory. You would have to specify the files in that subdirectory and it would then put it in the same single directory.

One of the many tests I put in txzchk is testing the single directory.

2024-entry-1234/
    .auth.json
    .info.json
    Makefile
    prog.c
    remarks.md
    test-tree.tgz
    support-tree.tgz

Yes this is what you would have to do.

[lcn2]

Nope. You can't specify a subdirectory. You would have to specify the files in that subdirectory and it would then put it in the same single directory.

One of the many tests I put in txzchk is testing the single directory.

2024-entry-1234/
.auth.json
.info.json
Makefile
prog.c
remarks.md
test-tree.tgz
support-tree.tgz
Yes this is what you would have to do.

Moreover you can have a Makefile rule that builds the sub-directory as part of some action (say triggered by make all) as long your make clobber does the cleanup.

The judges might, if it is warranted, when releasing a winning entry, leave the sub-directories in place. But as @xexyl pointed out in GH-discussioncomment-11691184, the submission compressed tarball cannot contain sub-directories.

SUGGESTION

Perhaps, @xexyl, we need a new FAQ about this? And if so, perhaps rule 17 should reference such an FAQ where it talks about no-subdirectory?

The FAQ might be "Q: How may I use sub-directories in my submission if Rule 17 disallows them?".

What do you think?

UPDATE 0

P.s., this is why the contest is going to be in pending status for about a month. We knew there would be good suggestions, such as what @SirWumpus raised, to consider before the contest into open status and the rules and guidelines became official.

The new repo process is to allow for this sort of feedback. Thanks @SirWumpus ‼️

[xexyl]

Nope. You can't specify a subdirectory. You would have to specify the files in that subdirectory and it would then put it in the same single directory.
One of the many tests I put in txzchk is testing the single directory.
2024-entry-1234/
.auth.json
.info.json
Makefile
prog.c
remarks.md
test-tree.tgz
support-tree.tgz
Yes this is what you would have to do.

Moreover you can have a Makefile rule that builds the sub-directory as part of some action (say triggered by make all) as long your make clobber does the cleanup.

The judges might, if it is warranted, when releasing a winning entry, leave the sub-directories in place. But as @xexyl pointed out in GH-discussioncomment-11691184, the submission compressed tarball cannot contain sub-directories.

SUGGESTION

Perhaps, @xexyl, we need a new FAQ about this? And if so, perhaps rule 17 should reference such an FAQ where it talks about no-subdirectory?

The FAQ might be "Q: How may I use sub-directories in my submission if Rule 17 disallows them?".

What do you think?

Hmm yes that sounds like a good idea given that we already had a comment about it.

I can't fork this repo yet. It seems that I'm not able to when I have the old temp repo forked: somehow it's tied together. I will I guess have to delete that fork (sadly) and fork this one. I don't see that happening today.

However do you wish me to do that? But of course I opened another discussion about issues with the guidelines so maybe that should be discussed first?

[lcn2]

Status update

As we understand things, the impact of the above is being coded in "the other repo" ..

"the other repo"

BTW: "the other repo" is a traditional phrase borrowed the phrase "the other place" as used on the Westminster parliament when we refer to related repos that are not part of this repo, such as the mkiocccentry repo.

And while the phrase has its roots in antiquity based on rivalry, we use in jest and good fun.

back to the previous sentence :-)

.. by @xexyl as part of a version 1.2 release of "the other repo".

We are sure that when @xexyl completes the work, it will be commented on here.

[SirWumpus]

I thought the final solution was to bundle the subdirectories into an archive when submitting the entry and setup the makefile targets to unpack and clean as necessary?

[xexyl]

I thought the final solution was to bundle the subdirectories into an archive when submitting the entry and setup the makefile targets to unpack and clean as necessary?

'Final solution': bad @SirWumpus, bad!

But I thought that too. I'm confused here.

[xexyl]

Status update

As we understand things, the impact of the above is being coded in "the other repo" ..

"the other repo"

BTW: "the other repo" is a traditional phrase borrowed the phrase "the other place" as used on the Westminster parliament when we refer to related repos that are not part of this repo, such as the mkiocccentry repo.

And while the phrase has its roots in antiquity based on rivalry, we use in jest and good fun.

back to the previous sentence :-)

.. by @xexyl as part of a version 1.2 release of "the other repo".

We are sure that when @xexyl completes the work, it will be commented on here.

Please explain what you mean. Isn't it just a tarball that the submitter can include? Or did you modify it to say that entries can have subdirectories? If so that'll require a discussion with txzchk and also chkentry (I think chkentry) plus other things.

[xexyl]

Status update

As we understand things, the impact of the above is being coded in "the other repo" ..

"the other repo"

BTW: "the other repo" is a traditional phrase borrowed the phrase "the other place" as used on the Westminster parliament when we refer to related repos that are not part of this repo, such as the mkiocccentry repo.
And while the phrase has its roots in antiquity based on rivalry, we use in jest and good fun.

back to the previous sentence :-)

.. by @xexyl as part of a version 1.2 release of "the other repo".
We are sure that when @xexyl completes the work, it will be commented on here.

Please explain what you mean. Isn't it just a tarball that the submitter can include? Or did you modify it to say that entries can have subdirectories? If so that'll require a discussion with txzchk and also chkentry (I think chkentry) plus other things.

Oh! You maybe mean updating the FAQ?

[xexyl]

I am updating the FAQ now .. or rather it's been updated but checking the html file looks okay. I also added the stats of the great fork merge to one of the questions (the one about the date that you added). I thought that would be fun to see.

[xexyl]

Okay FAQ committed .. once merged if it works for you, @SirWumpus, then please let me know. If you think it could be improved then please let me know how and I'll be happy to take care of it!

[lcn2]

Even though @xexyl did a good job of updating the FAQ about Rule 17, we are beginning to doubt the wisdom of no subdirectories.

Well the purpose of this "pending state" is, in part, to try and improve the Rules and Guidelines. Maybe this is one such improvement is to allow a submission to use sub-directories to some extent?

[xexyl]

Symlinks ARE disallowed by txzchk!

Many other things are. I believe the only thing I will have to do is make an update to the posix safe function to have an option to allow / and then modify the function that checks for directories.

Of course mkiocccentry also needs to use -r for cp but that shouldn't be a problem.

The checks I put in txzchk are pretty exhaustive.

The ACL stuff I think would not be copied over due to the tarball format version but maybe I am wrong.

[vog]

Are hard links an issue too?

Yes, the tar format does have support for hard links and e.g. GNU "tar" even has several options on how to deal with them (recognize on archive creation or not, convert to symlinks on archive extraction or not, etc.).

So I'd propose to make extra sure that txzchk disallows hard links in addition to symlinks. Although you can't hard link to system files (you can only hard-link to another file in the same archive), many strange things (and perhaps even opportunities to cheat) can happen if we allow for hard links.

[lcn2]

Symlinks ARE disallowed by txzchk!

Correct.

Many other things are. I believe the only thing I will have to do is make an update to the posix safe function to have an option to allow / and then modify the function that checks for directories.

Correct.

Of course mkiocccentry also needs to use -r for cp but that shouldn't be a problem.

Correct.

The checks I put in txzchk are pretty exhaustive.

The ACL stuff I think would not be copied over due to the tarball format version but maybe I am wrong.

Is there a test that attempts to use a different tar format that tries to "sneak" them in?

[xexyl]

Are hard links an issue too?

Yes, the tar format does have support for hard links and e.g. GNU "tar" even has several options on how to deal with them (recognize on archive creation or not, convert to symlinks on archive extraction or not, etc.).

I don't think it does with the format we use, however. And not using that format would show user names and that would be flagged. Also how would I check for it? Iirc it doesn't show any extra flag and we don't extract the tarball.

So I'd propose to make extra sure that txzchk disallows hard links in addition to symlinks. Although you can't hard link to system files (you can only hard-link to another file in the same archive), many strange things (and perhaps even opportunities to cheat) can happen if we allow for hard links.

That's true but see above point.

[xexyl]

Symlinks ARE disallowed by txzchk!

Correct.

Right. I deliberately added many many checks. This ironically makes the updates take a bit more looking into but it should be doable.

Many other things are. I believe the only thing I will have to do is make an update to the posix safe function to have an option to allow / and then modify the function that checks for directories.

Correct.

Yup although this was not actually necessary as we put it in the first place!

Of course mkiocccentry also needs to use -r for cp but that shouldn't be a problem.

Correct.

This is, as said elsewhere, already done!

The checks I put in txzchk are pretty exhaustive.
The ACL stuff I think would not be copied over due to the tarball format version but maybe I am wrong.

Is there a test that attempts to use a different tar format that tries to "sneak" them in?

Well: it checks for user names and without the format we use (v7) I think that would flag it, right?

[xexyl]

I'll be back later .. will look at finishing it then. Got distracted (deliberately) and now I have to go for a w while ... but at least the mkiocccentry part is done! It is not certain how long I'll be but it could be a while.

[xexyl]

I just thought of something. We need a new check if we're going to do this. Please advise (or put it in the new issue).

If we're going to allow subdirectories this will mean we need a maximum number of subdirectories. We don't want to descend down too many. Although the txzchk would just be parsing text if a person submitted something that had 100+ subdirectories that would obviously be a problem.

Personally I think that a good max count is 5 or 7 at most. But maybe you have a different number in mind.

There are multiple ways I could check this. The simplest is count the number of slashes found, after the initial directory. (So if we want no more than 7 we might do MAX_TARBALL_SUBDIRS as 7 + 1 to account for the first directory + 7).

[xexyl]

And apparently v7 does allow for links. Checking for symlinks is easy and it's already done. Not sure about hard links. Maybe it's already in there ... not sure. Anyway going afk now.

[xexyl]

Oh one more thing that's nice. In v7 the maximum filename length is 99: exactly what we want (and checked as of the most recent version).

Of course how to verify its v7 format is another matter entirely. Checking for user names and group names (as the tool does) might be one way. But if someone messes with the tarball manually without corrupting it ...

[xexyl]

Just checked. It should be blocked already as it checks for it not being - or (now) d. So that's fine ... really afk now!

[lcn2]

REQUEST

We have just created issue 1060 in mkiocccentry repo.

PLEASE CONTINUE THE IMPLEMENTATION DISCUSSION AND DETAILS OVER THERE.

Once that issue 1060 in mkiocccentry repo is resolved, then Rule 17 can be adjusted as per issue #40.

Given the above, we thank @SirWumpus for starting this discuss and will now close it so that the respective work can be carried out on those two locations.

[xexyl]

I can update rules, guidelines, FAQ and other things tomorrow.

[lcn2]

I can update rules, guidelines, FAQ and other things tomorrow.

Please do.

Meanwhile we are making good progress on the submit server rate limiter.

UPDATE 0

FYI: See GH-issue #40