Emacs CODE implementation

CODE implementation

https://renatgalimov.github.io/org-basb-code/

How to work with this guide?

• Copy-paste the config source block at the end of this document into your .emacs file or into a file in your emacs directory.
• In case of putting it into a separate file, add the following code to your emacs config file (load-file "~/path/to/basb.el")
• If you want to try this setup on a fresh Emacs do the following:
  • Create a new directory inside of your HOME: mkdir ~/.emacs.code.d
  • Put there a new init.el:

    (setq user-init-file (or load-file-name (buffer-file-name)))
    (setq user-emacs-directory (file-name-directory user-init-file))
    (load-file "~/.emacs.code.d/CODE.el")
            

  • Copy-paste the config into a new "~/.emacs.code.d/CODE.el"
  • Load a fresh Emacs with emacs -q --load ~/.emacs.code.d/init.el
  • Emacs will download and install all the packages defined in this guide. You might need depelopment tools installed to compile some packages. File a bug with errors you get and I will extend this guide.
• I will be using C-c n prefix for keybindings in this package.

Prerequisites

Pandoc

sudo apt install pandoc

brew install pandoc

Ensure that Pandoc is in your path.

pandoc --version

pandoc 2.14.0.2
Compiled with pandoc-types 1.22, texmath 0.12.3, skylighting 0.10.5.1,
citeproc 0.4.0.1, ipynb 0.1.0.1
User data directory: /Users/renat.galimov/.local/share/pandoc
Copyright (C) 2006-2021 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

melpa and melpa-stable

We will be downloading packages from these repos

(unless (boundp 'package-archives)
  (setq package-archives '()))

(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
                    (not (gnutls-available-p))))
       (proto (if no-ssl "http" "https")))
  (add-to-list 'package-archives (cons "gnu" (concat proto "://elpa.gnu.org/packages/")) t)
  ;; (add-to-list 'package-archives (cons "nongnu" (concat proto "://elpa.nongnu.org/packages/")) t)
  (add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t)
  (add-to-list 'package-archives (cons "melpa-stable" (concat proto "://stable.melpa.org/packages/")) t))

(package-refresh-contents)

(unless (bound-and-true-p package--initialized)
  (package-initialize))

straight.el

For this guide, we will be using straight.el as your package manager.

(defvar bootstrap-version)
(if (package-installed-p 'use-package)
    (eval-when-compile
      (require 'use-package))
  (package-refresh-contents)
  (package-install 'use-package))

(unless (boundp 'straight-use-package)
  (message "Bootstrapping straight.el")
  (let ((bootstrap-file
         (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
        (bootstrap-version 5))
    (unless (file-exists-p bootstrap-file)
      (with-current-buffer
          (url-retrieve-synchronously
           "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
           'silent 'inhibit-cookies)
        (goto-char (point-max))
        (eval-print-last-sexp)))
    (load bootstrap-file nil 'nomessage))
  (message "Bootstrapping straight.el done"))

Org-roam

Obviously, we need org-roam itself

(setq org-roam-v2-ack t)
(use-package org-roam
  :after org
  :ensure t
  :pin "melpa-stable"
  :bind (("C-c n l" . #'org-roam-buffer-toggle)
         ("C-c n f" . #'org-roam-node-find)
         ("C-c n v" . #'org-roam-node-visit)
         ("C-c n i" . #'org-roam-node-insert)
         (("C-c q" . #'org-roam-tag-add)))

  :config
  ;; We want org-id to work with our roam directory
  (setq org-roam-file-extensions '("org" "org_archive")
        org-id-extra-files (org-roam-list-files)))

Capture

Emacs is a text editor.

Capture documents

• Capture targets:
  • E-books
  • Documents
  • Videos
  • Audios
  • Images

    To capture we use Pandoc and org-pandoc-import

    (straight-use-package
     '(org-pandoc-import
       :host github
       :repo "tecosaur/org-pandoc-import"
       :files ("*.el" "filters" "preprocessors")))
    
    (global-set-key (kbd "C-c n o") #'org-pandoc-import-as-org)
            

    Pandoc can convert almost any text format to org-mode representation. One of the current drawbacks - it cannot import online web pages.

Capture web-pages

org-web-tools

org-web-tools offers org-web-tools-read-url-as-org function, which can download an URL to an org buffer.

(use-package org-web-tools
  :ensure t
  :pin "melpa-stable"
  :bind (("C-c n u" . org-web-tools-read-url-as-org)))

Often, downloaded files need manual cleanup.

Firefox web-clipper and Pandoc

An alternative approach is to use a web-clipper and Pandoc.

• Open the web article in your browser and activate a web-clipper. I use Firefoxes built-in one.
• Save entire HTML into a file.
• Convert the HTML with Pandoc

  pandoc -f html -t org <source-file>.html -o <target-file>.org
      

  The resulting file might require some cleanup, but the quality of the output is the best among other methods.

Capture templates

<<Please, contribute your capture templates>>

Finding your own records

Crawling over your own notes is a key part of the project workflow.

Below I summarized information about all full-text search engines I found for org-mode.

Depending on your own need you might select one or multiple of them.

Unfortunately, I didn’t find any ideal solution for a full-text search yet. Packages that make better sorting are slower than packages that give results fast but in a random order.

Org full-text search

• Requested features:
  • helm or counsel integration.
  • live search.
  • result previewing support.
  • a key-binding to capture results into currently clocked org file.
  • results ordering support
    • Headlines
    • Summary
    • Hightlight
  • large file-sets support

    ITEM	FRONTEND	SPEED	SORT
    Org-roam Full-text search
    \_ ripgrep (helm-rg)	helm ivy	fast	nil
    \_ helm-org-rifle	helm	slow	t
    \_ deft	helm	fast	nil
    \_ org-ql	helm	slow
    \_ recoll	helm ivy	fast	nil
    \_ org-fts	ivy	fast
    \_ org-agenda search

ripgrep (helm-rg)

GitHub - cosmicexplorer/helm-rg: ripgrep is nice Now I use it as a default text search engine.

brew install ripgrep

(use-package helm-rg
  :ensure t
  :after org-roam
  :pin "melpa-stable"
  :config
  (defun helm-rg-roam-directory (&optional query)
    "Search with rg in your roam directory, QUERY."
    (interactive)
    (let ((helm-rg-default-directory org-roam-directory)
          (helm-rg--current-dir org-roam-directory))
      (helm-rg query nil)))
  :bind (("C-c n R" . helm-rg-roam-directory)))

helm-org-rifle

GitHub - alphapapa/org-rifle: Rifle through your Org-mode buffers and acquire your target

This one is good. It gives you an idea about the context. But it’s not ordering the data by the highlights.

I find org-rifle too slow at the moment. But its output is exacly what I want.

(use-package helm-org-rifle :ensure t
  :after org-roam
  :pin "melpa-stable"
  :config
  (defun org-rifle-roam-directory ()
    (interactive)
    (helm-org-rifle-directories org-roam-directory))
  :bind (("C-c n s" . org-rifle-roam-directory)))

deftGitHub - dfeich/helm-deft: A helm based Emacs module to help search in a predetermined list of directories. Inspired by the deft module.

Helm implementation didn’t work for me. So I set a default version here.

One of the drawbacks here is that you can’t see the text you matched. From my point of view - helm-rg gives more precise information about the context.

(use-package deft :ensure t
  :after org-roam
  :config (setq deft-directory org-roam-directory
                deft-recursive t)
  :bind (("C-c n d" . deft)))
;; (use-package helm-deft
;;   :ensure t
;;   :straight (:host github
;;                    :repo "dfeich/helm-deft"
;;                    :files ("*.el"))
;;   :config
;;   (setq helm-deft-dir-list `(,org-roam-directory)
;;         helm-deft-extension '("org"))
;;   :bind (("C-c n d" . helm-deft)))

org-qlGitHub - alphapapa/org-ql: An Org-mode query language, including search commands and saved views

Doesn’t look suitable for large filesets, but helm implementation is good for medium-sized collections.

(use-package org-ql :ensure t
  :after org
  :config
  (progn
    (message "Loading org-ql")
    (setq org-ql-search-directories-files-recursive t
          org-ql-search-directories-files-regexp ".org\\(_archive\\)?$"))
  )

(use-package helm-org-ql :ensure t
  :after org-ql
  :config
  (setq helm-org-ql-recursive-paths t)
  :bind (("C-c n q" . helm-org-ql-org-directory)))

recollGitHub - emacs-helm/helm-recoll: helm interface for the recoll desktop search tool. I found recoll being to hard to set up. I wasn’t able to get it working on MacOS.org-ftsmicrofts/elisp at main · zot/microfts · GitHub

• It didn’t work on MacOS from scratch.
• I tried to compile its binary manually but that didn’t work either.
• This one looks promising. Let’s keep an eye on it.

org-agenda search

• Not trying this for now because agenda wants to open all its files for search.

Organize

Org-mode tags

This is a default method. You just tag your projects with a project tag, areas with an area tag and so on.

You can search projects with helm or with org-ql:

Org-roam links

An alternative to use Org-roam references as tags. You create notebooks called: projects, areas, resources and insert a link to the target record according to where it belongs.

If you go to the areas note and call org-roam-buffer-toggle, you will see a backlink to you note in the side buffer.

Org-roam-ui

A nice second-brain visualizator: org-roam-ui.

(use-package org-roam-ui
  :ensure t
  :after org-roam
  ;;         normally we'd recommend hooking orui after org-roam, but since org-roam does not have
  ;;         a hookable mode anymore, you're advised to pick something yourself
  ;;         if you don't care about startup time, use
  :hook (after-init . org-roam-ui-mode)

  :config
  (setq org-roam-ui-sync-theme t
        org-roam-ui-follow t
        org-roam-ui-update-on-save t
        org-roam-ui-open-on-start t))

Distill

Progressive summarization

Make org-emphasize multi-linear

To bypass the limit of two lines for org-emphasize marks enable the code below.

;; Make org-emphasis to work on up to 10 lines selection.
(setcar (nthcdr 4 org-emphasis-regexp-components) 10)
(org-set-emph-re 'org-emphasis-regexp-components org-emphasis-regexp-components)

Highlighting

Highlighting is a key part of progressive summarization. Here I will be highlighting with highlight.el and enriched mode. If you don’t want to put your text file into the enriched text mode, you can use org-emphasize instead of functions provided here.

highlight.el

When working with plain text buffers, like org-mode or markdown, you can use enriched text mode with the highlight library to mark the text.

;; If you get errors saying somethign about facemenu, try
;; uncommenting this.
;; (setq facemenu-menu nil)

(use-package highlight :ensure t
  :config
  (defun hlt-general()
    (interactive)
    (unless (bound-and-true-p enriched-mode)
      (enriched-mode t))
    (hlt-highlight-region (region-beginning) (region-end) 'highlight))

  ;; Without this setq highlights won't be saved even in enriched mode
  (setq hlt-use-overlays-flag nil)

  :bind (("C-c n h" . hlt-general)
         ("C-c n H" . hlt-unhighlight-region)))


;; If you cannot save your enriched files because of the :inherit
;; error, try uncommenting this function.

;; (defun enriched-face-ans (face)
;;   "Return annotations specifying FACE.
;; FACE may be a list of faces instead of a single face;
;; it can also be anything allowed as an element of a list
;; which can be the value of the `face' text property."
;;   (cond ((and (consp face) (eq (car face) 'foreground-color))
;;          (list (list "x-color" (cdr face))))
;;         ((and (consp face) (eq (car face) 'background-color))
;;          (list (list "x-bg-color" (cdr face))))
;;         ((and (listp face) (eq (car face) :foreground))
;;          (list (list "x-color" (cadr face))))
;;         ((and (listp face) (eq (car face) :background))
;;          (list (list "x-bg-color" (cadr face))))
;;         ((and (listp face) (eq (car face) :inherit))
;;          (enriched-face-ans (cdr face)))
;;         ((listp face)
;;          (apply 'append (mapcar 'enriched-face-ans face)))
;;         ((let* ((fg (face-attribute face :foreground))
;;                 (bg (face-attribute face :background))
;;                 (props (face-font face t))
;;                 (ans (cdr (format-annotate-single-property-change
;;                            'face nil props enriched-translations))))
;;            (unless (eq fg 'unspecified)
;;              (setq ans (cons (list "x-color" fg) ans)))
;;            (unless (eq bg 'unspecified)
;;              (setq ans (cons (list "x-bg-color" bg) ans)))
;;            ans))))

org-capture a region

To keep track of highlighted notes we will use org-capture.

;; Use =org-capture f= to put a link to the text you selected.into an
;; org entry with the current timer enabled.

(defun r/org-capture-get-selected-text ()
  (with-current-buffer (org-capture-get :original-buffer)
    (string-trim
     (replace-regexp-in-string
      "\n" " "
      (cond ((eq major-mode 'pdf-view-mode)
             (pdf-info-gettext (pdf-view-current-page) (car (pdf-view-active-region))))
            (t (buffer-substring-no-properties (region-beginning) (region-end))))))))
(defun r/org-capture-get-link (path)
  (with-current-buffer (org-capture-get :original-buffer)
    (cond ((eq major-mode 'pdf-view-mode) (switch-to-buffer (org-capture-get :original-buffer)) (org-pdftools-get-link))
          (t (concat path "::" (r/org-capture-get-selected-text))))))

(with-eval-after-load "org-capture"
  (add-to-list
   'org-capture-templates
   '("f" "Curently watched" item (clock)
     "%(r/org-capture-get-selected-text) [[%(r/org-capture-get-link \"%F\")][↗]]%?" :unnarrowed t)))

;; The code below automatically highlights the region we captured
(defun do-highlight-on-capture ()
  "Highlight selected region of the buffer you were in at capture."
  (save-excursion
    (with-current-buffer (plist-get org-capture-plist :original-buffer)
      (cond ((eq major-mode 'pdf-view-mode) (switch-to-buffer (org-capture-get :original-buffer)) (pdf-annot-add-highlight-markup-annotation (car (pdf-view-active-region))))
            (t (hlt-general))))))
(defun highlight-on-capture ()
  (when (equal (plist-get org-capture-plist :key) "f")
    (do-highlight-on-capture)))

(add-hook 'org-capture-after-finalize-hook #'highlight-on-capture)

This is my basic marking mechanism. Whenever I’m reading an article in Emacs (transformed to an org-mode or markdown file), I click C-c f to insert an entry to the notebook I’m currently on.

Working with PDF files

Don’t forget to install pdf-tools dependencies.

brew install glib

(use-package pdf-tools
  :ensure t
  :mode ("\\.pdf\\'" . pdf-view-mode)
  :init
  (require 'pdf-info)
  (condition-case nil
      (pdf-info-check-epdfinfo)
    (error (let ((default-directory (file-name-directory pdf-info-epdfinfo-program)))
               (pdf-tools-install t t)))))

(use-package org-pdftools
  :ensure t
  :hook (org-mode . org-pdftools-setup-link))

Express

This area is too large to cover it in this guide. If you’re new to org-mode consult

Exporting and publishing sections of the org manual.

Emacs config

<<set-up-melpa>>
<<install-straight-el>>
<<install-org-roam>>
<<install-org-roam-ui>>
<<install-org-pandoc-import>>
<<install-org-web-tools>>
<<install-helm-rg>>
<<install-org-rifle>>
<<install-deft>>
<<install-org-ql>>
<<increase-org-emphasis-limit>>
<<highlight-capture>>
<<setup-highlight-el>>
<<install-pdf-tools>>

Useful links

• GTD/BASB Templates for Emacs and Org-Mode · GitHub
• yt:AyhPmypHDEw
• GitHub - weirdNox/org-noter: Emacs document annotator, using Org-mode

Improving this guide

• [ ] per-org-document style
• [ ] org-pdf-import videos for
  • [ ] epub
  • [ ] doc files
• [ ] org web read url
• [ ] fix invalid face error.
• [ ] Faces not saving in enriched mode in vanilla emacs setup.
• [ ] Links made by capture are broken.
• [X] Use normal emacs
• [ ] Org pdftools link
• [ ] Capturing emails
• [ ] Multi-line emphasis are not working on MacOS Emacs
• [ ] Org pdftools: fix links for Linux and MacOS
• [ ] Videos:

  Be slow when doing videos. Give more details?

  • [ ] Capturing different formats:
    • [ ] HTML
    • [ ] PDF
  • [ ] Organize: