HyRolo - add hyrolo-file-list support for directories and wildcards (#…

…403)

* HyRolo - add hyrolo-file-list support for directories and wildcards

Introduce hyrolo--expanded-file-list to hold full list of hyrolo
files after expansion.

Remove all previously obsoleted rolo-* variables in favor of hyrolo-*.

Use hyperbole-hyrolo group for all hyrolo customizable variables.

* Fix hyrolo-tests.el sorting tests and hyrolo-outline-level handling

ChangeLog

@@ -1,12 +1,109 @@
+2023-11-17  Bob Weiner  <[email protected]>
+
+* test/hyrolo-tests.el (hyrolo-sort-records-at-different-levels): Add
+    'hyrolo-find-file' call to ensure current buffer is the first hyrolo file.
+                       (hyrolo-sort-test): Change to use 'hyrolo-find-file'.
+
+* hyrolo.el (hyrolo-mode-outline-level): Fix to handle all file formats that
+    HyRolo now supports but ignore Emacs outline entry matches without trailing
+    whitespace or of formfeeds, as these can cause a hang in HyRolo search.
+            (hyrolo-outline-level): Add to handle 'outline-level' calculation
+    unique to standard HyRolo entries (asterisk with a following whitespace
+    character).
+            (hyrolo-find-file): Improve 'outline-regexp' handling and set
+    'outline-level' function if an Emacs outline.
+	    (hyrolo-add): Compare entry-regexp against default-value of
+    'hyrolo-entry-regexp' rather than a literal string.
+
+2023-11-14  Bob Weiner  <[email protected]>
+
+* hyrolo.el (hyrolo-sort): Fix bug where 'hyrolo-sort-level' had assumed
+    the file to sort was in the current buffer but 'hyrolo-sort' had not
+    read in that file.
+            (hyrolo-map-level, hyrolo-sort-level): Remove 'hyrolo-file' arg
+    since had already been using current buffer instead.
+
+* test/hpath-tests.el (hpath:find-report-lisp-variable-path-name-when-not-exists):
+    Update test and doc to simply verify that 'hpath:find' errors when called
+    with a non-existent path.
+
+2023-11-13  Bob Weiner  <[email protected]>
+
+* hbut.el (hbut:source): Add save-excursion so does not move point
+    to the beginning of the source name.
+
+2023-11-12  Bob Weiner  <[email protected]>
+
+* hpath.el (hpath:file-position-to-line-and-column): Add to translate
+    character positions to an implicit link to line and column.
+  hbut.el (ibut:insert-text): Simplify using the above function.
+
+* hui-mouse.el (hkey-alist): Comment out unreleased hynote-mode clause.
+
+* hyrolo.el (hyrolo-buffer-exists-p): Fix for improved buffer-name matching
+    by using 'get-buffer' instead of 'get-file-buffer'.
+            (hyrolo-next-match-function): Update to require return of the
+    position of the beginning of the match.
+            (hyrolo-next-regexp-match): Change to return (match-beginning 0)
+    rather than the end.
+            (hyrolo-grep-file): Fix to handle when no entry delimiters, then use
+    the current line as the entry.
+            (hyrolo-name-at): Rename to 'hyrolo-name-at-p'; change return value
+    to a cons of (<entry-name> . <entry-source>).  If no entry delimiters,
+    use the current, non-empty line as <entry-name> and add a 'hyrolo-line-entry
+    text property to it.
+            (hyrolo-to): Add support for non-delimited single line entries.
+            (hyolo-edit): Fix to search only over a specified file.
+            (hyrolo-helm-org-rifle): Fix to filter out any non-file buffers.
+            (hyrolo-kill): Change to not switch buffers since shows whether the
+    record was killed or not.
+
+* hpath.el (hpath:expand, hpath:expand-list): Add optional exists-flag to return
+    nil if expanded path does not exist.
+
+* hyrolo.el (hyrolo--expanded-file-list, hyrolo-set-file-list,
+             hyrolo-file-list-changed): Finish support for recursively
+    searching dirs and expanding file wildcards in 'hyrolo-file-list' by
+    adding the above and caching the expanded list in 'hyrolo--expanded-file-list'.
+  HY-NEWS (HYROLO): Document the above new feature.
+            (hyrolo-edit): Improve 'completing-read' prompt.
+	    (rolo-display-buffer, rolo-file-list, rolo-entry-regexp): Remove all
+  of these variables that have been obsolete since V6.  Use hyrolo- prefix instead.
+            (hyrolo-grep): Rewrote for compatibility with new caching scheme.
+
+2023-11-11  Bob Weiner  <[email protected]>
+
+* hyrolo.el (hyrolo-initialize-file-list): Completely remove this function,
+    obsolete since V8.0.1.  Use `hyrolo--initialize-file-list' instead.
+            (:group 'hyperbole-hyrolo): Change from 'hyperbole and 'hyperbole-hyrolo.
+	    (hyrolo-get-file-list): Add and change refs to `hyrolo-file-list' to use
+    this instead, so can get the expanded version of the file list.
+            (hyrolo-expand-path-list): Add to expand a list of paths containing env
+    or lisp variables, directories and file patterns with wildcards into a final list
+    of files.
+
+2023-11-08  Bob Weiner  <[email protected]>
+
+* hpath.el (hpath:expand): Add doc for ${variable} expansion.
+           (hpath:expand-list): Add to recursively expand matching files within a
+    list of directories.
+
+2023-11-07  Bob Weiner  <[email protected]>
+
+* test/hui-tests.el (hui--ibut-link-directly-to-dired): Fix to handle
+    alternative dired formats with text after the last colon on the first
+    line.
+
+* hbut.el (hbut:at-p): Document interactive call help display.
+
 2023-11-06  Bob Weiner  <[email protected]>
 
-* hui.el (hui:link-possible-types): Fix CI/CD build error where
-    ilink was inserted when pathname was expected due to batch
-    diff when referent is in dired-mode.  Fix to skip button type
-    clauses it this case.
-  test/hui-tests.el (hui--ibut-link-directly-to-dired): Check that
-    looking at a string before trying to (read (current-buffer));
-    otherwise, 'read' will fail.
+* hui.el (hui:link-possible-types): Fix CI/CD build error where ilink was
+    inserted when pathname was expected due to batch diff when referent is
+    in dired-mode.  Fix to skip button type clauses in this case.
+  test/hui-tests.el (hui--ibut-link-directly-to-dired): Check that looking
+    at a string before trying to (read (current-buffer)); otherwise,
+    'read' will fail.
 
 2023-11-05  Bob Weiner  <[email protected]>
 
@@ -2596,7 +2693,9 @@ name with 'name' and 'name-key'.
 2022-08-20  Bob Weiner  <[email protected]>
 
 * hyrolo.el (hyrolo-grep): Add support for 'outline-regexp' and
-    'outline-level' variables from multiple file types.
+    'outline-level' variables from multiple file types.  This includes
+    adding 'hyrolo-hdr-regexp' to both 'hyrolo-entry-regexp' and
+   'outline-regexp' within the *HyRolo* match buffer.
             (hyrolo-mode): Set local value of 'outline-level' to
     'hyrolo-mode-outline-level' and define that function to support
     star outlines and Koutlines.  Also, make 'hyrolo-entry-regexp'

HY-NEWS

@@ -153,6 +153,25 @@
 
 ** HYROLO  (See "(hyperbole)HyRolo").
 
+  *** Koutline and Markdown File Support: The `hyrolo-file-list' can now
+      contain Koutline or Markdown files and will search their outline entries
+      just like it does for Org and Emacs outline files.  See
+      "(hyperbole)HyRolo Concepts".
+
+  *** hyrolo-file-list Support for Dirs, File Wildcards and Variables:
+
+      If you include an existing directory (invalid ones are ignored) in your
+      hyrolo list, hyrolo will search recursively across all of its files
+      that match `hyrolo-file-suffix-regexp'.  See `hpath.el#hpath:expand-list'.
+
+      If you include file wildcards in pathnames, they will be expanded into
+      matching files internally by HyRolo.  See "(emacs)Wildcards".
+
+      If you include an Environment variable or Emacs Lisp variable with the
+      ${var} format in a path, it will be resolved prior to searching the path.
+      Variables with values that include multiple paths, e.g. PATH, are resolved
+      to the first existing entry that matches.  See "(hyperbole)pathname".
+
   *** HyRolo Match Buffer Key Bindings: Improved movement keys:
 
         - {b} Move backward at the same outline level
@@ -165,11 +184,6 @@
 
       See "(hyperbole)HyRolo Keys".
 
-  *** Koutline and Markdown File Support: The `hyrolo-file-list' can now
-      contain Koutline or Markdown files and will search their outline entries
-      just like it does for Org and Emacs outline files.  See
-      "(hyperbole)HyRolo Concepts".
-
   *** Koutline Fast Search: Koutlines may now be included in
       `hyrolo-file-list' and individual cells extracted properly with any
       HyRolo query.  Then the interactive commands:
@@ -489,9 +503,69 @@
       beginning of a cell.  Must end with a colon or a newline.
 
   *** (hui:link-possible-types): At end of any line in a dired buffer,
-    return (link-to-directory default-directory), so when using any
-    link-directly command, always have a way to link to the dired buffer
-    itself.
+      return (link-to-directory default-directory), so when using any
+      link-directly command, always have a way to link to the dired buffer
+      itself.
+
+  *** (hyrolo-expand-list): Added to expand `hyrolo-file-list', filtering
+      to matches of `hyrolo-file-suffix-regexp'.  See documentation for
+      `hpath:expand-list' for the kinds of expansions performed.
+
+  *** (hpath:expand-list): Added to expand paths, filtering final list
+      to files with match-regexp.  Paths expansion substitutes up to one
+      ${variable} per path, expands file wildcards and recursively walks
+      directory trees for files with match-regexp.
+
+  *** (hyrolo-map-level, hyrolo-sort-level): Remove 'hyrolo-file' arg
+      since had already been using current buffer instead.
+
+  *** (hpath:file-position-to-line-and-column): Add to translate
+      character positions to an implicit link to line and column.
+
+  *** (hyrolo-next-match-function, hyrolo-next-regexp-match): Change to return
+      position of the beginning of the match rather than the end.
+
+  *** (hyrolo-grep-file): Handle when no entry delimiters, then use
+      the current line as the entry.
+
+  *** (hyrolo-name-at): Rename to 'hyrolo-name-at-p'; change return value
+      to a cons of (<entry-name> . <entry-source>).  If no entry delimiters,
+      use the current, non-empty line as <entry-name> and add a 'hyrolo-line-entry
+      text property to it.
+
+  *** (hyrolo-to): Add support for non-delimited single line entries.
+
+  *** (hyrolo-helm-org-rifle): Filter out any non-file buffers.
+
+  *** (hpath:expand, hpath:expand-list): Add optional exists-flag to return
+      nil if expanded path does not exist.
+
+  *** (hyrolo--expanded-file-list, hyrolo-set-file-list,
+       hyrolo-file-list-changed): Finish support for recursively searching dirs
+       and expanding file wildcards in 'hyrolo-file-list' by adding the above
+       and caching the expanded list in 'hyrolo--expanded-file-list'.
+
+  *** (hyrolo-edit): Improve 'completing-read' prompt.
+
+  *** (rolo-display-buffer, rolo-file-list, rolo-entry-regexp): Remove all of
+      these variables that have been obsolete since V6.  Use hyrolo- prefix instead.
+
+  *** (hyrolo-initialize-file-list): Completely remove this function, obsolete since
+      V8.0.1.  Use `hyrolo--initialize-file-list' instead.
+
+  *** (defgroup 'hyperbole-hyrolo): Change from 'hyperbole and 'hyperbole-hyrolo.
+
+  *** (hyrolo-get-file-list): Add and change refs to `hyrolo-file-list' to use
+      this instead, so can get the expanded version of the file list.
+
+  *** (hyrolo-expand-path-list): Add to expand a list of paths containing env or lisp
+      variables, directories and file patterns with wildcards into a final list of files.
+
+  *** (hpath:expand-list): Add to recursively expand matching files within a list of
+      directories.
+
+  *** (hui--ibut-link-directly-to-dired): Handle alternative dired formats with text
+      after the last colon on the first
 
 
 ** ORG MODE INTEGRATION (See "(hyperbole)Smart Key - Org Mode").

MANIFEST

@@ -59,7 +59,7 @@ hpath.el             - GNU Hyperbole support routines for handling UNIX paths
 htz.el               - Timezone-based time and date support for GNU Hyperbole
 hypb.el              - Miscellaneous GNU Hyperbole support features
 hypb-ert.el          - Hyperbole test runner action button types
-hyrolo.el            - Hierarchical, multi-file, easy-to-use contact management system
+hyrolo.el            - Hierarchical, multi-file, easy-to-use record management system
 hyrolo-logic.el      - Logic functions for GNU Hyperbole Rolo files
 hywconfig.el         - Save ring of window configurations
 set.el               - General mathematical operators for unordered sets