Include Org QL as a part of Org mode

[yantar92]

As discussed in https://old.reddit.com/r/emacs/comments/18frlq4/packages_that_you_would_like_to_be_in_emacs_core/kd3pran/, let's see how we merge Org QL (org-ql.el) into Org mode upstream.

Approximate checklist:

• Get rid of mandatory third-party dependencies (optional dependencies are ok)
• Switch core Org mode libraries to org-ql
• Integrate tests into Org test suite
• Integrate documentation into the Org manual

[alphapapa]

Thanks, Ihor.

• Get rid of mandatory third-party dependencies (optional dependencies are ok)

Sure, most of these are easily replaced: dash, f, s, and ov.

However, two tricky ones:

1. ts: It makes timestamp-related code much easier in org-ql. Emacs's timestamp-related libraries have improved somewhat since I first released ts, but it still leaves a lot to be done by the developer. I would hate to have to rewrite all of the code that uses it in other terms. It's also a matter of convenience for users who want to write their own functions to query with certain timestamps, because manipulating timestamps without ts is much more awkward. Any ideas?

2. org-super-agenda: Obviously, the support for it is a convenience, and basically just binds a variable, so that could be made optional. But I'd like to keep the API clean and simple, and the UI the same. For example, when I'm setting up a new org-ql-search, I almost always choose an auto group from the completion; I wouldn't want that to require an extra step.

   Would it almost make sense to keep org-ql-search and org-ql-view outside of Org.git, at least for a while? That is, to only merge org-ql.el, the search backend, and org-ql-find.el, the completing-read UI, into Org. What do you think?

[yantar92]

Adam Porter ***@***.***> writes:

> * Get rid of mandatory third-party dependencies (optional dependencies are ok) Sure, most of these are easily replaced: `dash`, `f`, `s`, and `ov`. However, two tricky ones: 1. `ts`: It makes timestamp-related code much easier in `org-ql`. Emacs's timestamp-related libraries have improved somewhat since I first released `ts`, but it still leaves a lot to be done by the developer. I would hate to have to rewrite all of the code that uses it in other terms. It's also a matter of convenience for users who want to write their own functions to query with certain timestamps, because manipulating timestamps without `ts` is much more awkward. Any ideas?

May you summarize which functions in ts.el are absent in the Emacs' timestamp library?

2. `org-super-agenda`: Obviously, the support for it is a convenience, and basically just binds a variable, so that could be made optional. But I'd like to keep the API clean and simple, and the UI the same. For example, when I'm setting up a new `org-ql-search`, I almost always choose an auto group from the completion; I wouldn't want that to require an extra step. Would it almost make sense to keep `org-ql-search` and `org-ql-view` outside of Org.git, at least for a while? That is, to only merge `org-ql.el`, the search backend, into Org. What do you think?

Yes, that's actually what I had in mind. Your rewrite of org-agenda is entirely different beast and we certainly do not want to parallel (and, more importantly, incompatible) implementations of org-agenda in the core. Is `org-super-agenda` support necessary if we do not merge `org-ql-search`? `org-ql-find` would be nice to have though. As a replacement for `org-refile` completion interface, for example.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[alphapapa]

May you summarize which functions in ts.el are absent in the Emacs' timestamp library?

I feel like it's more a question of API. Everything that ts does could be done without it, but many operations were very awkward and confusing without it. Giving that up would mean not only making org-ql's code more awkward and complicated, but making it harder for the user to pass in timestamps having been adjusted to their needs.

Yes, that's actually what I had in mind. Your rewrite of org-agenda is entirely different beast and we certainly do not want to parallel (and, more importantly, incompatible) implementations of org-agenda in the core.

FTR, org-ql-view is mostly compatible with org-agenda, e.g. in terms of agenda commands that act on items in a buffer. But, yes, of course, I wouldn't suggest supplanting org-agenda with it until it were much more mature and feature-complete.

Is org-super-agenda support necessary if we do not merge org-ql-search?

I don't think so, no. org-ql.el itself has no references to that support, other than the package-requires header.

org-ql-find would be nice to have though. As a replacement for org-refile completion interface, for example.

Yes, I would like to upstream that as well.

[yantar92]

Adam Porter ***@***.***> writes:

> May you summarize which functions in ts.el are absent in the Emacs' timestamp library? I feel like it's more a question of API. Everything that `ts` does *could* be done without it, but many operations were very awkward and confusing without it. Giving that up would mean not only making `org-ql`'s code more awkward and complicated, but making it harder for the user to pass in timestamps having been adjusted to their needs.

AFAIK, the following ts.el functions are used in org-ql: - ts-now - ts-apply - ts-adjust - ts-parse-fill - ts-in (trivial) - ts-parse-org If not using ts structs, ts-apply, ts-adjust, and ts-parse-fill can be easily implemented with the same API but returning internal Emacs time representations. What am I missing?

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[alphapapa]

Of course, the API could be reimplemented in various ways. The ts library could also be refactored to use Emacs internal time values by default, which has been suggested before, although since the struct already has a slot for Emacs's internal time values, that wouldn't be a significant benefit in and of itself.

One purpose of the ts struct is to provide convenient accessors for elements of the timestamp without having to decode and format the time value repeatedly; whether that's an important enough benefit to keep may be debatable, but I did measure a significant performance benefit when I was writing it (and whether user code typically uses that pattern is, again, another matter).

The main purpose of ts is to provide a unified API. Emacs's default time/date functions are spread across various libraries and symbol prefixes, and various functions take different arguments and types, which becomes bewildering. The API's using a single data type is a significant benefit, as all of the functions can be composed, chained, etc.

Not that I would refuse to, but I would strongly prefer not to discard this beneficial API for use with Org QL. I would almost rather try to upstream some form of ts itself (if not the data structure, then the API, at least). That might not be an enjoyable process, though...

[yantar92]

Adam Porter ***@***.***> writes:

Not that I would refuse to, but I would strongly prefer not to discard this beneficial API for use with Org QL. I would almost rather try to upstream some form of `ts` itself (if not the data structure, then the API, at least). That might not be an enjoyable process, though...

We cannot use non-optional libraries that are not a part of Emacs in Org mode. The best we can do is include necessary libraries as a part of Org, but that should be justified and should only be the last resort.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[alphapapa]

Yes, that's why I mentioned trying to upstream ts into Emacs.

[yantar92]

Adam Porter ***@***.***> writes:

Yes, that's why I mentioned trying to upstream `ts` into Emacs.

Do you need any help?

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[alphapapa]

Yes, that's why I mentioned trying to upstream ts into Emacs.

Do you need any help?

If you're willing to help with that idea, I'd gladly accept it. I hadn't given it a lot of thought recently, until now, but I think I would like to at least try to pursue that before taking it out of org-ql, because resorting to the existing Emacs date/time code for simple things like "timestamp for next Tuesday at noon" is awkward enough for developers, not to mention users.

[alphapapa]

@yantar92 FYI, I've added this section to the readme: e41fe90

[alphapapa]

@yantar92 Also, regarding ts: Would it seem reasonable to you to try to merge a version of Org QL that supported both Emacs native time values and ts structs, i.e. one that has optional support for ts? Obviously it could not depend on or require ts to be installed, but providing the support optionally would allow the process to continue without depending on the idea of adding some form of ts to Emacs.

[yantar92]

Adam Porter ***@***.***> writes:

@yantar92 FYI, I've added this section to the readme: e41fe90

e41fe90#diff-ecbc1aa90e9ff97a00b0b2aab1551bceee0c4d21993146bdcb1af4de31c9cac6R959 line is not accurate. Authors should not just contact ***@***.***, but they should send a filled email form. You may instead refer to https://orgmode.org/worg/org-contribute.html#copyright

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[yantar92]

Adam Porter ***@***.***> writes:

@yantar92 Also, regarding `ts`: Would it seem reasonable to you to try to merge a version of Org QL that supported both Emacs native time values and `ts` structs, i.e. one that has optional support for `ts`? Obviously it could not depend on or require `ts` to be installed, but providing the support optionally would allow the process to continue without depending on the idea of adding some form of `ts` to Emacs.

Even if there is optional support for ts.el, we still need to implement a fallback when ts is not available. That will involve adding the equivalent functions right inside Org sources. But if we have such functions inside Org source, what is the point using ts library?

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>