#+TITLE: Org QL Notes * [#A] Contents :PROPERTIES: :TOC: :include siblings :depth 1 :ignore this :force depth :END: :CONTENTS: - [[#overview][Overview]] - [[#tasks][Tasks]] - [[#checklists][Checklists]] - [[#examples--testing][Examples / testing]] - [[#in-the-wild][In the wild]] - [[#profiling][Profiling]] - [[#references][References]] - [[#testing][Testing]] :END: * [#A] Overview ** Bugs ([[org-ql-search:todo%253A%2520tags%253Abug?super-groups=%2528%2528%253Aauto-property%2520%2522Milestone%2522%2529%2529&sort=%2528todo%2529&title=%2522Bugs%2522][view]]) #+BEGIN: org-ql :query "todo: tags:bug" :columns ((priority "P") ((property "milestone") "M") (todo "Keyword") heading) :sort (priority todo date) :ts-format "%Y-%m-%d %H:%M" | P | M | Keyword | Heading | |---+-----+---------+------------------------------------------------| | | | NEXT | [[Check Org 9.4 source code for second argument][Check Org 9.4 source code for second argument]] | | A | 0.6 | PROJECT | [[Compatibility with Org 9.4 custom link changes][Compatibility with Org 9.4 custom link changes]] | #+END: ** Milestones ([[org-ql-search:todo%253A?super-groups=%2528%2528%253Aauto-property%2520%2522milestone%2522%2529%2529&sort=%2528todo%2529&title=%2522Milestones%2522][view]]) #+BEGIN: org-ql :query "todo: property:milestone" :columns (((property "milestone") "M") (priority "P") todo heading) :sort (priority date) :take 9 | M | P | Todo | Heading | |--------+---+---------+--------------------------------------------------------------------------------| | 0.6 | A | PROJECT | [[Compatibility with Org 9.4 custom link changes][Compatibility with Org 9.4 custom link changes]] | | 0.6 | A | PROJECT | [[Reverse sorting][Reverse sorting]] | | future | B | TODO | [[Sorter for property values][Sorter for property values]] | | 0.7 | B | TODO | [[Bookmarks save list of files instead of e.g. ~org-agenda-files~ when appropriate][Bookmarks save list of files instead of e.g. ~org-agenda-files~ when appropriate]] | | 0.7 | B | TODO | [[Change ~clocked~ and ~closed~'s behavior with just-number args][Change ~clocked~ and ~closed~'s behavior with just-number args]] | | 0.6 | B | TODO | [[Review all normalizers for potential loops][Review all normalizers for potential loops]] | | 0.6 | B | TODO | [[Normalize query while preserving body][Normalize query while preserving body]] | | 0.6 | B | TODO | [[Use string queries in view headers when possible][Use string queries in view headers when possible]] | | 0.7 | B | PROJECT | [[Optimized, date-specific timestamp regexps][Optimized, date-specific timestamp regexps]] | #+END: ** Underway ([[org-ql-search:todo%253AUNDERWAY?sort=%2528priority%2529&title=%2522Underway%2522][view]]) #+BEGIN: org-ql :query (or (todo "UNDERWAY") (and (todo "PROJECT") (descendants (todo "UNDERWAY")))) :columns (((property "milestone") "M") (priority "P") (todo "Keyword") heading) :sort (priority date) :ts-format "%Y-%m-%d %H:%M" | M | P | Keyword | Heading | |-----+---+----------+--------------------------------------------------------------------| | 0.7 | B | PROJECT | [[Optimized, date-specific timestamp regexps][Optimized, date-specific timestamp regexps]] | | 0.7 | B | PROJECT | [[Group tag support][Group tag support]] | | | | UNDERWAY | [[~clocked~][~clocked~]] | | | | UNDERWAY | [[~closed~][~closed~]] | | | | UNDERWAY | [[~deadline~][~deadline~]] | | | | UNDERWAY | [[~planning~][~planning~]] | | | | UNDERWAY | [[~scheduled~][~scheduled~]] | | | | UNDERWAY | [[~ts~][~ts~]] | | | | UNDERWAY | [[Benchmarking tags searches without and with new group-tags support][Benchmarking tags searches without and with new group-tags support]] | #+END: ** To-do ([[org-ql-search:todo:?super-groups=((:todo%20"NEXT")%20(:todo%20"PROJECT")%20(:auto-priority))&sort=(todo)][view]]) #+BEGIN: org-ql :query "todo: priority:A" :columns ((priority "P") ((property "milestone") "M") todo heading) :sort (priority date) :take 7 | P | M | Todo | Heading | |---+-----+---------+------------------------------------------------| | A | 0.6 | PROJECT | [[Compatibility with Org 9.4 custom link changes][Compatibility with Org 9.4 custom link changes]] | | A | | PROJECT | [[Convert simple sexp queries to non-sexp][Convert simple sexp queries to non-sexp]] | | A | 0.6 | PROJECT | [[Reverse sorting][Reverse sorting]] | | A | | PROJECT | [[Org%20link%20types%20%5B2/3%5D][Org link types {2/3}]] | #+END: ** Stuck projects ([[org-ql-search:%2528and%2520%2528todo%2520%2522PROJECT%2522%2529%2520%2528not%2520%2528descendants%2520%2528todo%2520%2522NEXT%2522%2520%2522UNDERWAY%2522%2529%2529%2529%2529?super-groups=%2528%2528%253Aauto-property%2520%2522milestone%2522%2529%2529&sort=%2528priority%2529&title=%2522Stuck%2520Projects%2522][view]]) #+BEGIN: org-ql :query (and (todo "PROJECT") (not (descendants (todo "NEXT" "UNDERWAY")))) :columns (((property "milestone") "M") (priority "P") heading) :sort (priority date) :take 7 | M | P | Heading | |--------+---+-----------------------------------------| | | A | [[Convert simple sexp queries to non-sexp][Convert simple sexp queries to non-sexp]] | | 0.6 | A | [[Reverse sorting][Reverse sorting]] | | | A | [[Org%20link%20types%20%5B2/3%5D][Org link types {2/3}]] | | | B | [[Outline path predicate][Outline path predicate]] | | | B | [[Document the sorting functions][Document the sorting functions]] | | future | B | [[Recursive queries][Recursive queries]] | | future | B | [[Timeline view][Timeline view]] | #+END: * [#A] Tasks :PROPERTIES: :TOC: :include descendants :depth 1 :END: :CONTENTS: - [[#sorter-for-property-values][Sorter for property values]] - [[#add-auto-keyword-to-planning-predicate][Add :auto keyword to (planning) predicate]] - [[#bookmarks-save-list-of-files-instead-of-eg-org-agenda-files-when-appropriate][Bookmarks save list of files instead of e.g. org-agenda-files when appropriate]] - [[#change-deadlines-auto-argument-to-auto-andor-auto-t][Change (deadline)'s auto argument to :auto and/or :auto t]] - [[#outline-path-in-buffers-files-arg][Outline path in buffers-files arg]] - [[#add-more-sorters][Add more sorters?]] - [[#default-sort][Default sort]] - [[#partial-match-for-property-queries][Partial match for property queries]] - [[#remove-org-ql-macro][Remove org-ql macro]] - [[#change-clocked-and-closeds-behavior-with-just-number-args][Change clocked and closed's behavior with just-number args]] - [[#review-all-normalizers-for-potential-loops][Review all normalizers for potential loops]] - [[#normalize-query-while-preserving-body][Normalize query while preserving body]] - [[#example-next-upcoming-event][Example: Next upcoming event]] - [[#org-agenda-skip-function][org-agenda-skip-function]] - [[#update-commentary][Update commentary]] - [[#org-ql-block-let-org-agenda-format-its-output][org-ql-block: Let org-agenda format its output]] - [[#omnifocus-like-screencasts][OmniFocus-like screencasts]] - [[#use-ripgrep-to-search-unopened-files][Use ripgrep to search unopened files]] - [[#key-cache-results-by-action-function][Key cache results by action function]] - [[#use-session-async-to-have-a-persistent-search-process][Use session-async to have a persistent search process]] - [[#overlay-based-caching-inspired-by-org-num-mode][Overlay-based caching inspired by org-num-mode]] - [[#fancier-searching-for-inherited-tags][Fancier searching for inherited tags]] - [[#compatibility-with-org-94-custom-link-changes][Compatibility with Org 9.4 custom link changes]] - [[#convert-simple-sexp-queries-to-non-sexp][Convert simple sexp queries to non-sexp]] - [[#reverse-sorting][Reverse sorting]] - [[#recursive-sub-queries-in-non-sexp-format][Recursive sub-queries in non-sexp format]] - [[#optimized-date-specific-timestamp-regexps][Optimized, date-specific timestamp regexps]] - [[#multi-pass-query-normalization][Multi-pass query normalization]] - [[#optimization-for-olp-predicate][Optimization for olp predicate]] - [[#outline-path-predicate][Outline path predicate]] - [[#tools-for-saving-queries-and-accessing-them-34][Tools for saving queries and accessing them {3/4}]] - [[#group-tag-support][Group tag support]] - [[#document-the-sorting-functions][Document the sorting functions]] - [[#recursive-queries][Recursive queries]] - [[#timeline-view][Timeline view]] - [[#implement-view-with-tabulated-list-mode-or-magit-section][Implement view with tabulated-list-mode or magit-section]] - [[#dynamic-blocks][Dynamic blocks]] - [[#predicate-helper-functions][Predicate helper functions]] - [[#new-transient-transient-lisp-variable-class][New Transient transient-lisp-variable class]] - [[#normalize-queries][Normalize queries]] - [[#update-view-screenshots][Update view screenshots]] - [[#test-caching][Test caching]] - [[#alternative-parsing-libraries][Alternative parsing libraries]] - [[#timestamp-predicates-using-relative-dates-break-caching][Timestamp predicates using relative dates break caching]] - [[#link-target-doesnt-work][~(link :target)~ doesn't work]] - [[#link-regexp-p-doesnt-work][~(link :regexp-p)~ doesn't work]] - [[#fix-custom-sorter-breaks-cached-results][Fix: Custom sorter breaks cached results]] - [[#update-dash-dependencies][Update dash dependencies]] - [[#checking-links-for-unsafe-parameters][Checking links for unsafe parameters]] - [[#views-multiple-sorters-are-not-preserved][Views: Multiple sorters are not preserved]] - [[#make-dynamic-blocks-warn-about-sexp-queries][Make dynamic blocks warn about sexp queries]] - [[#add-emacs-271-to-testyml][Add Emacs 27.1 to test.yml]] - [[#fix-org-ql-view--link-open-on-org-93][Fix org-ql-view--link-open on Org 9.3+]] - [[#fix-query-sexp-to-string-functions-handling-of-eg-descendants][Fix query-sexp-to-string function's handling of, e.g. descendants]] - [[#helm-command][Helm command]] - [[#add-a-with-time-argument-to-timestamp-predicates][Add a :with-time argument to timestamp predicates]] - [[#node-caching]["Node" caching]] - [[#define-predicates-with-a-macro][Define predicates with a macro]] - [[#move-this-notes-file-into-an-orphan-metanotes-branch][Move this notes file into an orphan meta/notes branch]] - [[#quickly-change-sortinggrouping-in-search-views][Quickly change sorting/grouping in search views]] - [[#byte-compile-lambdas][Byte-compile lambdas]] - [[#documentfigure-out-tag-inheritance][Document/figure out tag inheritance]] - [[#dual-matching-with-regexp-and-predicates][Dual matching with regexp and predicates]] - [[#operate-on-list-of-heading-positions][Operate on list of heading positions]] - [[#use-macros-for-date][Use macros for date]] :END: ** TODO [#B] Sorter for property values :enhancement: :PROPERTIES: :milestone: future :END: [2021-06-18 Fri 02:26] e.g. to sort entries in this file by the =milestone= property. Something like: #+BEGIN_SRC elisp (org-ql-search (current-buffer) '(todo) :sort '((property "milestone") priority)) #+END_SRC ** TODO [#B] Add ~:auto~ keyword to ~(planning)~ predicate It should act like ~(or (deadline auto) (scheduled :to today))~. ** TODO [#B] Bookmarks save list of files instead of e.g. ~org-agenda-files~ when appropriate :PROPERTIES: :milestone: 0.7 :END: [2020-12-05 Sat 01:24] The bookmarked plist should use the "contracted" form rather than the list of files so that, if the list of files changes before the bookmark is loaded, it will use the new list of files. [2021-06-18 Fri 03:12] This has the potential to cause bugs, so I'm going to defer it until at least 0.7. ** TODO [#B] Change ~(deadline)~'s ~auto~ argument to ~:auto~ and/or ~:auto t~ For consistency, because plain ~auto~ looks like a variable, and even though it's in a quoted form, it could be confusing. ** TODO [#B] Outline path in buffers-files arg :PROPERTIES: :ID: 6935361a-9e1d-48ec-8d17-876a90b90f50 :END: e.g. #+BEGIN_SRC elisp (org-ql (olp "~/org/inbox.org" "Emacs" "Ideas") (todo "NEXT")) #+END_SRC Also, should support an ~id~ one. ** TODO [#B] Add more sorters? + [ ] =category= + [ ] Any date :: e.g. it would search for timestamps (active/inactive?) anywhere in an entry ** TODO [#B] Default sort Would probably be useful to have a default sort option. ** TODO [#B] Partial match for property queries e.g. something like [[https://200ok.ch/posts/2020-02-09_creating_org_mode_sparse_trees_in_emacs_and_organice.html][Organice has now]]. [2020-02-13 Thu 00:42] Something is needed to help search property values by partial matches. For example: #+BEGIN_SRC org ,* [[https://github.com/fniessen/org-html-themes][org-html-themes: Framework including two themes, Bigblow and ReadTheOrg]] ,:PROPERTIES: ,:author: Fabrice Niessen ,:END: #+END_SRC Searching that with a query like =property:author=Fabrice= returns nothing; the full value must be used, like ~property:author="Fabrice Niessen"~. It should be possible to do something like ~property:author=~Fabrice~ to search for partial matches. ** TODO Remove ~org-ql~ macro :PROPERTIES: :milestone: 0.7 :END: 0.6 will be the last stable release to have it. ** TODO [#B] Change ~clocked~ and ~closed~'s behavior with just-number args :PROPERTIES: :milestone: 0.7 :END: + [[file:~/src/emacs/org-ql/tests/test-org-ql.el::;;%20TODO:%20While%20it%20seems%20helpful%20for%20(clocked)%20and%20(close)%20to][To-do item]] #+BEGIN_SRC elisp ;; TODO: While it seems helpful for (clocked) and (close) to ;; implicitly look into the past (because entries can't be ;; clocked or closed in the future), it makes the API ;; inconsistent. It would be better to be consistent and ;; require the user to pass these predicates a negative number. #+END_SRC ** TODO [#B] Review all normalizers for potential loops :PROPERTIES: :milestone: 0.6 :END: Since I found (and fixed) one after I made normalizers apply repeatedly, I should check the rest, because there might be a few more that could do it. ** TODO [#B] Normalize query while preserving body :enhancement: :PROPERTIES: :milestone: 0.6 :END: Sometimes it would be helpful to define a predicate that normalizes to another predicate while having a different body. [[https://github.com/plundaahl/dotfiles/blob/ef4d25498ab895c3c22b588ae2506fd69bdd8755/emacs/package-defs/org-ql/pred-created.el#L19][For example]]: #+BEGIN_SRC elisp (org-ql-defpred created (&key from to on) "Search for entries with \"CREATED\" property in range or on date" :body (if (and on (or from to)) (error "Either specify FROM and/or TO, or ON") (let ((heading-time (pcl/as-ts (org-entry-get (point) "CREATED"))) (on (pcl/as-ts on)) (from (pcl/as-ts (or on from))) (to (pcl/as-ts (or on to)))) (and t (if from (ts<= from heading-time) t) (if to (ts< heading-time (ts-adjust 'day +1 to)) t))))) #+END_SRC That predicate would best be normalized to ~(property "CREATED")~, but its body would still need to be evaluated to compare the property value as a timestamp. There are two obvious possibilities: 1. Normalize it to ~(and (property "CREATED") (...form that compares (org-entry-get (point) "CREATED") as a timestamp...))~, without a defined body. 2. Allow ~-defpred~ to normalize the query to ~(property "CREATED")~ while preserving a separate body. Maybe something like normalizing it to ~(and (property "CREATED") (created ...))~, which would still call the body; however, the normalizer must avoid looping on ~(created ...)~, so maybe a sentinel value is needed. ** TODO [#C] Example: Next upcoming event :docs: [2020-11-26 Thu 18:17] Inspired by [[https://github.com/unhammer/org-upcoming-modeline][GitHub - unhammer/org-upcoming-modeline: put upcoming org event in modeline]] ([[https://www.reddit.com/r/emacs/comments/k1e4eu/show_next_orgappointment_in_modeline/gdp6fqv/][Reddit thread]]). #+BEGIN_SRC elisp (pcase-let* ((now (ts-now)) (items (org-ql-select (org-agenda-files) '(ts-active :from 0 :to 1) :action '(cons (save-excursion (car (sort (cl-loop while (re-search-forward org-tsr-regexp nil t) collect (ts-parse-org (match-string 1))) #'ts<))) (point-marker)))) (`(,time . ,marker) (car (seq-sort-by #'car #'ts< items))) (heading (org-with-point-at marker (org-link-display-format (nth 4 (org-heading-components))))) (seconds-until (ts-difference time now)) ;; NOTE: Using day of year to avoid end-of-month turnover in day number. (days-until (- (ts-doy time) (ts-doy now))) (time-string (cond ((<= seconds-until org-upcoming-modeline-duration-threshold) (ts-human-format-duration seconds-until 'abbreviate)) ((= 0 days-until) (ts-format "%H:%M" time)) ((= 1 days-until) (ts-format "tomorrow %H:%M" time)) (t ; > 1 days-until (ts-format "%a %H:%M" time))))) (propertize (format " ⏰ %s: %s" time-string heading) 'face 'org-level-4 'help-echo (format "%s at <%s>" heading (ts-format "%Y-%m-%d %H:%M" time)))) #+END_SRC ** TODO [#C] ~org-agenda-skip-function~ As discussed [[https://www.reddit.com/r/emacs/comments/cnrt2d/orgqlblock_integrates_orgql_into_org_agenda/ewi1q36/][here]], this is a cool feature that allows further integration into existing custom agenda commands. Example: #+BEGIN_SRC elisp ;;; lima-0ac22.el --- -*- lexical-binding: t; -*- (defun org-ql-skip-function (query) "Return a function for `org-agenda-skip-function' for QUERY. Compared to using QUERY in `org-ql', this effectively turns QUERY into (not QUERY)." (let* ((predicate (org-ql--query-predicate '(regexp "ryo-modal")))) (lambda () ;; This duplicates the functionality of `org-ql--select'. (let (orig-fns) (--each org-ql-predicates ;; Save original function mappings. (let ((name (plist-get it :name))) (push (list :name name :fn (symbol-function name)) orig-fns))) (unwind-protect (progn (--each org-ql-predicates ;; Set predicate functions. (fset (plist-get it :name) (plist-get it :fn))) ;; Run query. ;; FIXME: "If this function returns nil, the current match should not be skipped. ;; Otherwise, the function must return a position from where the search ;; should be continued." (funcall predicate)) (--each orig-fns ;; Restore original function mappings. (fset (plist-get it :name) (plist-get it :fn)))))))) (let ((org-agenda-custom-commands '(("z" "Z" ((tags-todo "PRIORITY=\"A\"+Emacs/!SOMEDAY")) ((org-agenda-skip-function (org-ql-skip-function '(regexp "ryo-modal"))))) ((org-agenda-files ("~/org/inbox.org")))))) (org-agenda nil "z")) #+END_SRC I should benchmark it to see how much difference it makes, because all those ~fset~ calls on each heading isn't free. But if a macro were used to rewrite the built-in predicates to their full versions, all of that could be avoided... ** TODO [#C] Update commentary ** MAYBE org-ql-block: Let org-agenda format its output [2020-11-16 Mon 22:12] As suggested by Kevin J. Foley. See [[https://github.com/alphapapa/org-ql/pull/113#issuecomment-728674220][#113]]. It might actually be simple to do and might work very well. ** MAYBE OmniFocus-like screencasts [2020-11-18 Wed 03:03] Looking at OmniFocus's web site now, their short videos showing features demonstrate a lot of features that exist in org-ql already. After implementing section-based views, it would be cool to make some short demos showing similar features. ** MAYBE Use ripgrep to search unopened files [2020-11-26 Thu 02:20] It [[https://github.com/BurntSushi/ripgrep/issues/176][supports multiline search now]], so it might be suitable now. ** MAYBE Key cache results by action function Could allow reusing results for different queries. Mentioned in [[https://www.reddit.com/r/emacs/comments/kewl3a/how_are_folks_using_orgagenda_and_orgroam_together/gg8iytf/][Reddit discussion]]. ** MAYBE Use [[https://codeberg.org/FelipeLema/session-async.el][session-async]] to have a persistent search process That could run queries in the other process and then send results to the main Emacs process. ** MAYBE [#C] Overlay-based caching inspired by org-num-mode [2019-12-30 Mon 22:42] Newer versions of Org have =org-num-mode=, which uses =font-lock= and =after-change-functions= to update overlays in the buffer with outline numbering. Maybe a similar approach could be used to cache arbitrary values for headings in a buffer without having to discard the whole buffer's cache when the buffer changes. [2020-11-09 Mon 01:51] I feel like that's probably unlikely to work well. I imagine it would require storing the query at every heading, which would be very wasteful. As well, adding more overlays to an Org buffer is probably not a good idea, because there are already enough of those. However, there might still be a useful idea here somewhere... ** MAYBE [#C] Fancier searching for inherited tags When tag inheritance is enabled, and the given tags aren't file-level tags, we could search directly to headings containing the matching tags, and then only do per-heading matching on the subtrees. Sometimes that would be much faster. However, that might make the logic special-cased and complicated. Might need a redesign of the whole matching/predicate system to do cleanly. ** PROJECT [#A] Compatibility with Org 9.4 custom link changes :bug: :PROPERTIES: :milestone: 0.6 :END: [2020-11-13 Fri 22:36] From [[https://www.orgmode.org/Changes.html][the changelog]]: #+BEGIN_QUOTE Calling conventions changes when opening or exporting custom links This changes affects export back-ends, and libraries providing new link types. Function used in :follow link parameter is required to accept a second argument. Likewise, function used in :export parameter needs to accept a fourth argument. See org-link-set-parameters for details. Eventually, the function org-export-custom-protocol-maybe is now called with a fourth argument. Even though the 3-arguments definition is still supported, at least for now, we encourage back-end developers to switch to the new signature. #+END_QUOTE Unfortunately it does not say what the new, required second argument is. [2020-11-22 Sun 17:22] For now, I'll add an optional, ignored second argument to the follow function; if I'm lucky, it will work anyway. *** NEXT Check Org 9.4 source code for second argument ** PROJECT [#A] Convert simple sexp queries to non-sexp :PROPERTIES: :END: [2020-11-11 Wed 00:28] This will be very helpful for storing links. Surely simple ones won't be too hard... #+BEGIN_SRC elisp (defun org-ql--query-sexp-to-plain (query) "Return a plain query string for sexp QUERY. If QUERY can't be converted to a plain one, return nil." ;; This started out pretty simple...but at least it's not just one long function, right? (cl-labels ((complex-p (query) (or (contains-p 'or query))) (contains-p (symbol list) (cl-loop for element in list thereis (or (eq symbol element) (and (listp element) (contains-p symbol element))))) (format-args (args) (let (non-paired paired next-keyword) (cl-loop for arg in args do (cond (next-keyword (push (cons next-keyword arg) paired) (setf next-keyword nil)) ((keywordp arg) (setf next-keyword (substring (symbol-name arg) 1))) (t (push arg non-paired)))) (string-join (append (mapcar #'format-atom non-paired) (nreverse (--map (format "%s=%s" (car it) (cdr it)) paired))) ","))) (format-atom (atom) (cl-typecase atom (string (if (string-match (rx space) atom) (format "%S" atom) (format "%s" atom))) (t (format "%s" atom)))) (format-form (form) (pcase form (`(not . (,rest)) (concat "!" (format-form rest))) (`(priority . ,_) (format-priority form)) ;; FIXME: Convert (src) queries to non-sexp form...someday... (`(src . ,_) (user-error "Converting (src ...) queries to non-sexp form is not implemented")) (_ (pcase-let* ((`(,pred . ,args) form) (args-string (pcase args ('() "") ((guard (= 1 (length args))) (format "%s" (car args))) (_ (format-args args))))) (format "%s:%s" pred args-string))))) (format-and (form) (pcase-let* ((`(and . ,rest) form)) (string-join (mapcar #'format-form rest) " "))) (format-priority (form) (pcase-let* ((`(priority . ,rest) form) (args (pcase rest (`(,(and comparator (or < <= > >= =)) ,letter) (priority-letters comparator letter)) (_ rest)))) (concat "priority:" (string-join args ",")))) (priority-letters (comparator letter) (let* ((char (string-to-char (upcase (symbol-name letter)))) (numeric-priorities '(?A ?B ?C)) ;; NOTE: The comparator inversion is intentional. (others (pcase comparator ('< (--select (> it char) numeric-priorities)) ('<= (--select (>= it char) numeric-priorities)) ('> (--select (< it char) numeric-priorities)) ('>= (--select (<= it char) numeric-priorities)) ('= (--select (= it char) numeric-priorities))))) (mapcar #'char-to-string others)))) (unless (complex-p query) (pcase query (`(and . ,_) (format-and query)) (_ (format-form query)))))) (--map (cons it (org-ql--query-sexp-to-plain it)) '((priority >= B) (priority > B) (priority < B) (priority < A) (priority = A) (todo) (todo "TODO") (todo "TODO" "NEXT") (ts :from -1 :to 1) (ts :on today) (ts-active :from "2017-01-01" :to "2018-01-01") (heading "quoted phrase" "word") (and (tags "book" "books") (priority "A")) (and (tags "space") (not (regexp "moon"))) (src :lang "elisp" :regexps ("defun"))) ) #+END_SRC [2020-11-11 Wed 01:45] Seems to work well. Now to integrate that into link-saving... [2020-11-11 Wed 02:41] Seems to work. Will [[https://github.com/alphapapa/org-ql/issues/147#issuecomment-725287074][wait for feedback]] before merging. [2020-11-11 Wed 19:13] Seems to be working properly. One more thing to do though, I think: *** TODO [#B] Use string queries in view headers when possible :PROPERTIES: :milestone: 0.6 :END: Maybe make it an option to automatically convert them when possible, because if a user wanted to add complexity to a string query, he'd have to rewrite it as a sexp. ** PROJECT [#A] Reverse sorting :PROPERTIES: :milestone: 0.6 :END: + [[https://github.com/alphapapa/org-ql/issues/143][Reverse sort order? · Issue #143 · alphapapa/org-ql · GitHub]] *** TODO Update docs Need to mention incompatible change (though that isn't the best way to describe it) in the changelog. *** WAITING [[https://github.com/alphapapa/org-ql/issues/143#issuecomment-863638651][Get feedback]] ** PROJECT Recursive sub-queries in non-sexp format [2021-09-08 Wed 15:41] Eric Abrahamsen showed me this in =#emacs:matrix.org=: #+BEGIN_SRC elisp (defvar test-pexs '((query (+ (or compound-term term))) (term (or subquery prefixed-term kv-term value) term-end) (subquery "(" query ")" `(query -- (if (= 1 (length query)) query (list query)))) (prefixed-term (or negated-term near-term)) (negated-term (or "not " "-") term `(term -- (list 'not term))) (near-term "near " term `(term -- (list 'near term))) (compound-term (or or-terms and-terms)) (or-terms (or subquery prefixed-term term) "or " (or subquery prefixed-term term) `(t1 t2 -- (list 'or t1 t2))) (and-terms (or subquery prefixed-term term) "and " (or subquery prefixed-term term) `(t1 t2 -- (list 'and t1 t2))) (value (or quoted-value plain-value)) (plain-value (substring (+ [word]))) (quoted-value "\"" (substring (+ (not "\"") (any))) "\"") (kv-term plain-value ":" value `(k v -- (cons (intern k) v))) (term-end (opt (+ [space]))))) #+END_SRC #+BEGIN_QUOTE The INFINITE RECURSIVE SUBQUERIES is particularly nice ​BTW, I see what you mean about the peg macros, that's a real pain. I have a small patch that should fix that ​This was half-cribbed from org-ql to begin with, so it's only right that you get something in return :) ​The subtlety is all in the ordering of the or statements #+END_QUOTE It should be pretty easy to add some of that to our query parsing. ** PROJECT [#B] Optimized, date-specific timestamp regexps :enhancement: :PROPERTIES: :milestone: 0.7 :ID: fc8ccf6e-5311-4121-a0b7-58482dbd2e85 :END: + In branch =wip/ts-optimized-regexps=. [2020-11-28 Sat 19:12] For example, a regexp like this would make timestamp searches very fast: #+BEGIN_SRC elisp (rx "<" "2020" "-" (or "10-31" "11-01" "11-02") ">") ;;=> "<2020-\\(?:\\(?:1\\(?:0-31\\|1-0[12]\\)\\)\\)>" #+END_SRC It shouldn't be hard to generate a list of date strings that =rx-to-string= could optimize, and that would avoid parsing and testing many timestamps which wouldn't match when specific date ranges are given. *** TODO Limit or optimize [2020-12-22 Tue 04:55] Noticed that the tests affected by this change are now slower in the test suite, presumably because they span a range of years and spend more time incrementing ~ts~ structs in the regexp-building function than running the search on the small amount of test data. For smaller date ranges, and for searching larger sets of data, the performance is improved. So there needs to be some kind of heuristic to handle this. Or maybe the regexp-building function could be smarter rather than "brute-forcing" its way through every date in the range. (This also shows how valuable Buttercup's showing of each test's duration is, otherwise I might not have noticed this issue.) #+BEGIN_EXAMPLE inactive without arguments (preamble) (2.43ms) without arguments (no preamble) (2.58ms) :from a timestamp (preamble) (421.88ms) :from a timestamp (no preamble) (8.65ms) :from a number of days (preamble) (20.77ms) :from a number of days (no preamble) (3.06ms) :to a timestamp (preamble) (424.83ms) :to a timestamp (no preamble) (8.48ms) :to a number of days (preamble) (220.44ms) :to a number of days (no preamble) (2.68ms) :on a timestamp (preamble) (9.51ms) :on a timestamp (no preamble) (8.03ms) :on a number of days (preamble) (67.11ms) :on a number of days (no preamble) (3.25ms) #+END_EXAMPLE [2020-12-22 Tue 19:16] It should be simple to do something like this: + If the range spans at least a year, include every day and month number, and each year number. + If the range spans less than a year but at least a month, include every day number, each month number, and the year number(s). + If the range spans less than a month, probably just increment and collect the numbers, like the code currently does (which will ensure the proper result in case the range spans the end of a month and/or year). If necessary, another step could be to divide the range into three parts: the middle being the part that spans whole months/years, and then the ends being the parts that span partial months/years. Then a regexp could be made for each part and combined into a single regexp which would match each part separately (rather than combining all numbers into one potential timestamp). *** UNDERWAY ~clocked~ **** TODO Tests **** DONE Use optimized regexp in predicate **** DONE Benchmark #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("unoptimized" (progn (org-ql-defpred clocked (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry was clocked in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) ;; (clocked) and (closed) implicitly look into the past. (let ((from (->> (ts-now) (ts-adjust 'day (* -1 num-days)) (ts-apply :hour 0 :minute 0 :second 0)))) `(clocked :from ,from)))) :preambles ((`(,predicate-names ,(pred numberp)) (list :regexp org-ql-clock-regexp :query t)) (`(,predicate-names) (list :regexp org-ql-clock-regexp :query t))) :body (org-ql--predicate-ts :from from :to to :regexp org-ql-clock-regexp :match-group 1)) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(clocked :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))) ("optimized" (progn (org-ql-defpred clocked (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry was clocked in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) ;; (clocked) and (closed) implicitly look into the past. (let ((from (->> (ts-now) (ts-adjust 'day (* -1 num-days)) (ts-apply :hour 0 :minute 0 :second 0)))) `(clocked :from ,from)))) :preambles ((`(,predicate-names ,(pred numberp)) (list :regexp org-ql-clock-regexp :query t)) (`(,predicate-names . ,(and rest (guard (or (plist-get rest :from) (plist-get rest :to) (plist-get rest :on))))) ;; Use date-optimized timestamp regexp. (-let (((&plist :from :to :on :type) rest)) (org-ql--from-to-on) (list :regexp (-let* ((from (or from (ts-adjust 'day (- org-ql-ts-days-from-default) (ts-now)))) (to (or to (ts-adjust 'day org-ql-ts-days-to-default (ts-now)))) (ts-regexp (org-ql--ts-range-to-regexp from to :type 'inactive))) (rx-to-string `(seq bol (0+ blank) "CLOCK:" (1+ blank) (0+ not-newline) (regexp ,ts-regexp)))) :query query))) (`(,predicate-names) (list :regexp org-ql-clock-regexp :query t))) :body (org-ql--predicate-ts :from from :to to :regexp org-ql-clock-regexp :match-group 1)) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(clocked :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------+--------------------+---------------+----------+------------------| | optimized | 1.85 | 1.746526 | 0 | 0 | | unoptimized | slowest | 3.239420 | 0 | 0 | *** UNDERWAY ~closed~ **** TODO Tests **** DONE Use optimized regexp in predicate **** DONE Benchmark #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("unoptimized" (progn (org-ql-defpred closed (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry was closed in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) ;; (clocked) and (closed) implicitly look into the past. (let ((from (->> (ts-now) (ts-adjust 'day (* -1 num-days)) (ts-apply :hour 0 :minute 0 :second 0)))) `(closed :from ,from)))) :preambles ((`(,predicate-names . ,_) ;; Predicate still needs testing. (list :regexp org-closed-time-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-closed-time-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(closed :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))) ("optimized" (progn (org-ql-defpred closed (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry was closed in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) ;; (clocked) and (closed) implicitly look into the past. (let ((from (->> (ts-now) (ts-adjust 'day (* -1 num-days)) (ts-apply :hour 0 :minute 0 :second 0)))) `(closed :from ,from)))) :preambles ((`(,predicate-names . ,(and rest (guard (or (plist-get rest :from) (plist-get rest :to) (plist-get rest :on))))) (-let (((&plist :from :to :on :type) rest)) (org-ql--from-to-on) (list :regexp (-let* ((from (or from (ts-adjust 'day (- org-ql-ts-days-from-default) (ts-now)))) (to (or to (ts-adjust 'day org-ql-ts-days-to-default (ts-now)))) (ts-regexp (org-ql--ts-range-to-regexp from to :type 'inactive))) (rx-to-string `(seq bow (0+ blank) "CLOSED:" (1+ blank) (regexp ,ts-regexp)))) :query query))) (`(,predicate-names . ,_) ;; Predicate still needs testing. (list :regexp org-closed-time-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-closed-time-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(closed :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------+--------------------+---------------+----------+------------------| | unoptimized | 1.04 | 0.382831 | 0 | 0 | | optimized | slowest | 0.396930 | 0 | 0 | *** UNDERWAY ~deadline~ **** TODO Tests **** DONE Use optimized regexp in predicate **** DONE Benchmark #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("unoptimized" (progn (org-ql-defpred deadline (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry has deadline in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names auto) ;; Use `org-deadline-warning-days' as the :to arg. (let ((to (->> (ts-now) (ts-adjust 'day org-deadline-warning-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(deadline-warning :to ,to))) (`(,predicate-names ,(and num-days (pred numberp))) (let ((to (->> (ts-now) (ts-adjust 'day num-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(deadline :to ,to)))) ;; NOTE: Does this normalizer cause the preamble to not be used? (Adding one to the deadline-warning definition to be sure.) :preambles ((`(,predicate-names . ,_) (list :regexp org-deadline-time-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-deadline-time-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(deadline :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))) ("optimized" (progn (org-ql-defpred deadline (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry has deadline in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names auto) ;; Use `org-deadline-warning-days' as the :to arg. (let ((to (->> (ts-now) (ts-adjust 'day org-deadline-warning-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(deadline-warning :to ,to))) (`(,predicate-names ,(and num-days (pred numberp))) (let ((to (->> (ts-now) (ts-adjust 'day num-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(deadline :to ,to)))) ;; NOTE: Does this normalizer cause the preamble to not be used? (Adding one to the deadline-warning definition to be sure.) :preambles ((`(,predicate-names . ,(and rest (guard (or (plist-get rest :from) (plist-get rest :to) (plist-get rest :on))))) ;; Use date-optimized timestamp regexp. (-let (((&plist :from :to :on :type) rest)) (org-ql--from-to-on) (list :regexp (-let* ((from (or from (ts-adjust 'day (- org-ql-ts-days-from-default) (ts-now)))) (to (or to (ts-adjust 'day org-ql-ts-days-to-default (ts-now)))) (ts-regexp (org-ql--ts-range-to-regexp from to :type 'active))) (rx-to-string `(seq bow (0+ blank) "DEADLINE:" (1+ blank) (regexp ,ts-regexp)))) :query query))) (`(,predicate-names . ,_) (list :regexp org-deadline-time-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-deadline-time-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(deadline :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------+--------------------+---------------+----------+------------------| | unoptimized | 1.54 | 0.258749 | 0 | 0 | | optimized | slowest | 0.397937 | 0 | 0 | *** UNDERWAY ~planning~ **** TODO Tests **** DONE Use optimized regexp in predicate **** DONE Benchmark #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("unoptimized" (progn (org-ql-defpred planning (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry has planning timestamp in given period (i.e. its deadline, scheduled, or closed timestamp). If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) (let ((to (->> (ts-now) (ts-adjust 'day num-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(planning :to ,to)))) :preambles ((`(,predicate-names . ,_) (list :regexp org-ql-planning-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-ql-planning-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(planning :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))) ("optimized" (progn (org-ql-defpred planning (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" ;; warnings, because we pre-process that argument in a macro before ;; this function is called. "Return non-nil if current entry has planning timestamp in given period (i.e. its deadline, scheduled, or closed timestamp). If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value." :normalizers ((`(,predicate-names ,(and num-days (pred numberp))) (let ((to (->> (ts-now) (ts-adjust 'day num-days) (ts-apply :hour 23 :minute 59 :second 59)))) `(planning :to ,to)))) :preambles ((`(,predicate-names . ,(and rest (guard (or (plist-get rest :from) (plist-get rest :to) (plist-get rest :on))))) (-let (((&plist :from :to :on :type) rest)) (org-ql--from-to-on) (list :regexp (-let* ((from (or from (ts-adjust 'day (- org-ql-ts-days-from-default) (ts-now)))) (to (or to (ts-adjust 'day org-ql-ts-days-to-default (ts-now)))) (ts-regexp (org-ql--ts-range-to-regexp from to))) (rx-to-string `(seq bow (0+ blank) (or "CLOSED" "DEADLINE" "SCHEDULED") ":" (1+ blank) (regexp ,ts-regexp)))) :query query))) (`(,predicate-names . ,_) (list :regexp org-ql-planning-regexp :query query))) :body (org-ql--predicate-ts :from from :to to :regexp org-ql-planning-regexp :match-group 1 :limit (line-end-position 2))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(planning :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------+--------------------+---------------+----------+------------------| | optimized | 1.28 | 0.450906 | 0 | 0 | | unoptimized | slowest | 0.576033 | 0 | 0 | *** UNDERWAY ~scheduled~ :LOGBOOK: CLOCK: [2020-12-20 Sun 07:01]--[2020-12-20 Sun 07:58] => 0:57 :END: **** TODO Benchmark **** TODO Tests **** DONE Use optimized regexp in predicate *** UNDERWAY ~ts~ [2020-12-19 Sat 05:55] Seems to be working well. Not sure how best to integrate with other timestamp-related predicates. The code isn't that much, so maybe copying it into each predicate would be best. **** TODO Tests **** DONE Use optimized regexp in predicate **** DONE Benchmark [2020-12-20 Sun 07:51] This is a great improvement. I tested it on the ~scheduled~ predicate too, but it doesn't make nearly as big a difference, because just searching for the ~SCHEDULED:~ prefix avoids testing most timestamps. #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("unoptimized" (progn (org-ql-defpred (ts ts-active ts-a ts-inactive ts-i) (&key from to _on regexp (match-group 0) (limit (org-entry-end-position))) ;; NOTE: Arguments to this predicate are pre-processed in `org-ql--normalize-query'. ;; The underscore before `on' prevents "unused lexical variable" warnings due to the ;; pre-processing converting that argument to FROM and TO. The `regexp' argument is ;; also provided by the pre-processing and is not to be given by the user. FROM and ;; TO are actually expected to be `ts' structs. The docstring is written for users. "Return non-nil if current entry has a timestamp in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value. TYPE may be `active' to match active timestamps, `inactive' to match inactive ones, or `both' / nil to match both types. LIMIT bounds the search for the timestamp REGEXP. It defaults to the end of the entry, i.e. the position returned by `org-entry-end-position', but for certain searches it should be bound to a different positiion, e.g. for planning lines, the end of the line after the heading." ;; MAYBE: Define active/inactive ones separately? :normalizers ((`(,(or 'ts-active 'ts-a) . ,rest) `(ts :type active ,@rest)) (`(,(or 'ts-inactive 'ts-i) . ,rest) `(ts :type inactive ,@rest))) :preambles ((`(,predicate-names . ,rest) (list :regexp (pcase (plist-get rest :type) ((or 'nil 'both) org-tsr-regexp-both) ('active org-tsr-regexp) ('inactive org-ql-tsr-regexp-inactive)) ;; Predicate needs testing only when args are present. :query (-let (((&keys :from :to :on) rest)) ;; FIXME: This used to be (when (or from to on) query), but that doesn't seem right, so I ;; changed it to this if, and the tests pass either way. Might deserve a little scrutiny. (if (or from to on) query t))))) ;; TODO: DRY this with the clocked predicate. :body (cl-macrolet ((next-timestamp () `(when (re-search-forward regexp limit t) (ts-parse-org (match-string match-group)))) (test-timestamps (pred-form) `(cl-loop for next-ts = (next-timestamp) while next-ts thereis ,pred-form))) (save-excursion (cond ((not (or from to)) (re-search-forward regexp limit t)) ((and from to) (test-timestamps (ts-in from to next-ts))) (from (test-timestamps (ts<= from next-ts))) (to (test-timestamps (ts<= next-ts to))))))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(ts :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))) ("optimized" (progn (org-ql-defpred (ts ts-active ts-a ts-inactive ts-i) (&key from to _on regexp (match-group 0) (limit (org-entry-end-position))) ;; NOTE: Arguments to this predicate are pre-processed in `org-ql--normalize-query'. ;; The underscore before `on' prevents "unused lexical variable" warnings due to the ;; pre-processing converting that argument to FROM and TO. The `regexp' argument is ;; also provided by the pre-processing and is not to be given by the user. FROM and ;; TO are actually expected to be `ts' structs. The docstring is written for users. "Return non-nil if current entry has a timestamp in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be either `ts' structs, or strings parseable by `parse-time-string' which may omit the time value. TYPE may be `active' to match active timestamps, `inactive' to match inactive ones, or `both' / nil to match both types. LIMIT bounds the search for the timestamp REGEXP. It defaults to the end of the entry, i.e. the position returned by `org-entry-end-position', but for certain searches it should be bound to a different positiion, e.g. for planning lines, the end of the line after the heading." ;; MAYBE: Define active/inactive ones separately? :normalizers ((`(,(or 'ts-active 'ts-a) . ,rest) `(ts :type active ,@rest)) (`(,(or 'ts-inactive 'ts-i) . ,rest) `(ts :type inactive ,@rest))) :preambles ((`(,predicate-names . ,(and rest (guard (or (plist-get rest :from) (plist-get rest :to) (plist-get rest :on))))) (-let (((&plist :from :to :on :type) rest)) (org-ql--from-to-on) (list :regexp (-let* ((from (or from (ts-adjust 'day (- org-ql-ts-days-from-default) (ts-now)))) (to (or to (ts-adjust 'day org-ql-ts-days-to-default (ts-now))))) (org-ql--ts-range-to-regexp from to)) :query query))) (`(,predicate-names . ,rest) (list :regexp (pcase (plist-get rest :type) ((or 'nil 'both) org-tsr-regexp-both) ('active org-tsr-regexp) ('inactive org-ql-tsr-regexp-inactive)) ;; Predicate needs testing only when args are present. :query (-let (((&keys :from :to :on) rest)) ;; FIXME: This used to be (when (or from to on) query), but that doesn't seem right, so I ;; changed it to this if, and the tests pass either way. Might deserve a little scrutiny. (if (or from to on) query t))))) ;; TODO: DRY this with the clocked predicate. :body (cl-macrolet ((next-timestamp () `(when (re-search-forward regexp limit t) (ts-parse-org (match-string match-group)))) (test-timestamps (pred-form) `(cl-loop for next-ts = (next-timestamp) while next-ts thereis ,pred-form))) (save-excursion (cond ((not (or from to)) (re-search-forward regexp limit t)) ((and from to) (test-timestamps (ts-in from to next-ts))) (from (test-timestamps (ts<= from next-ts))) (to (test-timestamps (ts<= next-ts to))))))) (setf org-ql-cache (make-hash-table :weakness 'key)) (org-ql-select (org-agenda-files) '(ts :from "2020-01-01" :to "2020-12-31") :action '(substring-no-properties (org-get-heading t t))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------+--------------------+---------------+----------+------------------| | optimized | 3.78 | 1.243185 | 0 | 0 | | unoptimized | slowest | 4.700824 | 0 | 0 | ** PROJECT Multi-pass query normalization It would allow, e.g. one query to be normalized into another, and then into another. It would be helpful for timestamp-related ones, I think. ** PROJECT [#B] Optimization for ~olp~ predicate I noticed that, of course, a search like =h:bar olp:foo= is much faster than =olp:foo,bar=. Then I realized that the last argument to =olp= could be removed and replaced with a =heading= predicate with that argument, which is much faster. [2020-12-05 Sat 01:27] However, this does not always work correctly, as [[https://github.com/alphapapa/org-ql/issues/160#issuecomment-738524404][yantar92 reported]]: #+BEGIN_QUOTE Consider the following example: * No deadline ** Learn *** Research **** Plasticity ***** TODO Schwaiger [MRS-Fall] (2017) Characterizing the mechanical properties of individual phases in nanostructured composites I may try to match the last heading like the following "olp:dead phase". It will not match. #+END_QUOTE *** TODO Devise solution to desired ~olp~ optimization *** NEXT Write test for ~olp~ that catches bug that attempted optimization caused ** PROJECT [#B] Outline path predicate [2019-10-07 Mon 11:15] There are two potential types of matching on outline paths: matching on any part of the outline path, and matching a specific path. For example, with this file: #+BEGIN_SRC org ,* Food ,** Fruits ,*** Blueberries ,*** Grapes ,** Vegetables ,*** Carrots ,*** Potatoes #+END_SRC Matching could work like this: + ~(outline "Food")~ :: Would return all nodes. + ~(outline "Fruits")~ :: Would return all fruits. Matching at a specific path would be something like: + ~(outline-path "Food" "Fruits")~ :: Would return all fruits. But if there were another =Fruits= heading somewhere in the file, under a different outline path, it would not return its nodes. I'm not sure the second type of matching belongs in predicates, but rather in [[id:6935361a-9e1d-48ec-8d17-876a90b90f50][this]]. To implement this with good performance probably needs an outline-path cache. I can probably repurpose the tags caching, but maybe it should be generalized. [2019-10-07 Mon 13:09] This is basically done with =be2bf6df316b96b3ed56851b8ffe0e227796b621= and =be2bf6df316b96b3ed56851b8ffe0e227796b621=, but not the specific-path matching. I left a =MAYBE= in the code about "anchored" path matching, which would accomplish that. ** PROJECT Tools for saving queries and accessing them [3/4] + Added example to =examples.org=. *** PROJECT [#A] Org link types [2/3] :PROPERTIES: :ID: 4db73c1c-a4ed-425e-9e38-8d334ed03e1e :END: This would be useful for having a menu of saved queries as Org links, or even bookmarking saved queries. **** TODO For saved queries **** DONE For searches [2020-11-08 Sun 22:59] Let's try a very simple implementation so I could write a link like this to search the current buffer: #+BEGIN_SRC org [[org-ql-search:property:author="AUTHOR"]] #+END_SRC [2020-11-08 Sun 23:22] Seems to work! #+BEGIN_SRC elisp :results silent ;;;; Org link type ;; This section adds a custom link type to Org. See info:org#Adding hyperlink types. (org-link-set-parameters "org-ql-search" :follow #'org-ql-search--link-open :store #'org-ql-search--link-store) (defun org-ql-search--link-open (query) "Open Org QL QUERY for current buffer." (org-ql-search (current-buffer) query)) (defun org-ql-search--link-store () "Store a link to current Org QL query." ;; TODO: When we have an org-ql-view-mode, test it here instead of org-ql-view-query. (when org-ql-view-query (org-store-link-props :type "org-ql-search" :link (concat "org-ql-search:" (org-ql-view--format-query org-ql-view-query)) :description org-ql-view-title) t)) #+END_SRC Tested on these queries: #+BEGIN_SRC org + [[org-ql-search:(property%20:author%20"Chris%20Wellons")][org-ql-search:(property :author "Chris Wellons")]] + [[org-ql-search:(link%20"nullprogram")][org-ql-search:(link "nullprogram")]] + [[org-ql-search:link:nullprogram]] #+END_SRC [2020-11-10 Tue 00:35] I'd like to support other parameters to the search, like grouping and sorting, so: #+BEGIN_SRC elisp :results silent ;;;; Org link type ;; This section adds a custom link type to Org. See info:org#Adding hyperlink types. (org-link-set-parameters "org-ql-search" :follow #'org-ql-search--link-open :store #'org-ql-search--link-store) (defun org-ql-search--link-open (query) "Open Org QL QUERY for current buffer." (require 'url-parse) (pcase-let* ((`(,query . ,params) (url-path-and-query (url-parse-make-urlobj "org-ql-search" nil nil nil nil query))) (params (url-parse-query-string params)) ;; Hacky or elegant? (_ (mapc (lambda (pair) (cl-callf (lambda (it) (intern (concat ":" it))) (car pair)) (cl-callf read (cdr pair))) params)) (params (cl-loop for (key . value) in params append (list key value)))) (apply #'org-ql-search (current-buffer) query params))) (defun org-ql-search--link-store () "Store a link to current Org QL query." (when org-ql-view-query (org-store-link-props :type "org-ql-search" :link (concat "org-ql-search:" (org-ql-view--format-query org-ql-view-query)) :description org-ql-view-title) t)) #+END_SRC That seems to work, like: #+BEGIN_SRC org [[org-ql-search:property:author="Chris%20Wellons"?super-groups=((:auto-outline-path%20t))]] #+END_SRC [2020-11-10 Tue 01:34] Okay, this seems to take care of all parameters: #+BEGIN_SRC elisp (defun org-ql-search--link-open (path) "Open Org QL query for current buffer at PATH. PATH should be the part of an \"org-ql-search:\" URL after the protocol. See, e.g. `org-ql-search--link-store'." (require 'url-parse) (require 'url-util) (pcase-let* ((`(,query . ,params) (url-path-and-query (url-parse-make-urlobj "org-ql-search" nil nil nil nil path))) (query (url-unhex-string query)) (params (when params (url-parse-query-string params))) ;; `url-parse-query-string' returns "improper" alists, which makes this awkward. (sort (when (alist-get "sort" params nil nil #'string=) (read (alist-get "sort" params nil nil #'string=)))) (groups (when (alist-get "super-groups" params nil nil #'string=) (read (alist-get "super-groups" params nil nil #'string=)))) (title (when (alist-get "title" params nil nil #'string=) (read (alist-get "title" params nil nil #'string=))))) (org-ql-search (current-buffer) query :sort sort :super-groups groups :title title))) (defun org-ql-search--link-store () "Store a link to the current Org QL view. Only views that search a single buffer may be linked to." (require 'url-parse) (require 'url-util) (unless (or (bufferp org-ql-view-buffers-files) (= 1 (length org-ql-view-buffers-files))) (user-error "Only views searching a single buffer may be linked")) (when org-ql-view-query (let* ((params (list (when org-ql-view-super-groups (list "super-groups" (prin1-to-string org-ql-view-super-groups))) (when org-ql-view-sort (list "sort" (prin1-to-string org-ql-view-sort))) (when org-ql-view-title (list "title" (prin1-to-string org-ql-view-title))))) (filename (concat (url-hexify-string (org-ql-view--format-query org-ql-view-query)) "?" (url-build-query-string (delete nil params)))) (url (url-recreate-url (url-parse-make-urlobj "org-ql-search" nil nil nil nil filename)))) (org-store-link-props :type "org-ql-search" :link url :description (concat "org-ql-search: " org-ql-view-title))) t)) #+END_SRC **** DONE For all parameters *** DONE Bookmarks [2020-11-08 Sun 23:25] Already done in =e5b4cd106558790563af26a8e32ec9508f904855=. *** DONE Access saved query from saved query list *** DONE Save query from ql-agenda buffer ** PROJECT [#B] Group tag support :PROPERTIES: :milestone: 0.7 :END: + [[https://github.com/alphapapa/org-ql/issues/145][Tag hierarchy support · Issue #145 · alphapapa/org-ql · GitHub]] + [[https://github.com/alphapapa/org-ql/pull/146][Change: (org-ql--tags-at) Support tag hierarchies by maxchaos · Pull Request #146 · alphapapa/org-ql · GitHub]] *** UNDERWAY Benchmarking tags searches without and with new group-tags support #+BEGIN_SRC elisp (bench-multi-lexical :times 10 :ensure-equal t :forms (("without group-tags support" (org-ql-select (org-ql-search-directories-files) '(tags "Emacs") :action #'point)) )) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------------+--------------------+---------------+----------+------------------| | without group-tags support | slowest | 5.512271 | 0 | 0 | #+BEGIN_SRC elisp (bench-multi-lexical :times 10 :ensure-equal t :forms (("with group-tags support" (org-ql-select (org-ql-search-directories-files) '(tags "Emacs") :action #'point)) )) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------------+--------------------+---------------+----------+------------------| | with group-tags support | slowest | 5.154639 | 0 | 0 | [2020-11-09 Mon 17:43] I think I need to enhance the benchmarking macros to make this easier. But that might require copying much of =benchmark-run-compiled=, so let me try something else: This is messy, but it ought to be fair enough (the only difference being the minor change in =org-ql--tags-at=. #+BEGIN_SRC elisp (bench-multi-lexical :times 10 :ensure-equal t :forms (("without group-tags support" (progn (setf org-ql-cache (make-hash-table :weakness 'key) org-ql-tags-cache (make-hash-table :weakness 'key) org-ql-node-value-cache (make-hash-table :weakness 'key)) (defun org-ql--expand-tag-hierarchy (tags &optional excluded) "Return TAGS along with their associated group tags. This function recursively searches for groups that each given tag belongs to, directly or indirectly, and includes the corresponding group tags to the result. TAGS should be a list of tags (i.e., strings). If non-nil, EXCLUDED should be a list of group tags that will not be automatically added to the results unless they are already in TAGS." (let ((groups (org-tag-alist-to-groups org-current-tag-alist)) (excluded (append tags excluded))) (let (group-tags) (dolist (tag tags) (pcase-dolist (`(,group-tag . ,group-members) groups) (when (and (not (member group-tag excluded)) ;; Check if one of the members in the group matches tag. ;; Notice that each member may be a plain string or ;; a regexp pattern (enclosed between curly brackets). (--some (if (string-match-p "^[{].+[}]$" it) ;; If pattern (it) is a regexp, remove the brackets and ;; make sure that it either matches the whole tag or not. (string-match-p (concat "^" (substring it 1 -1) "$") tag) ;; Check if member (it) is identical to tag. (string= it tag)) group-members)) (push group-tag group-tags)))) ;; If group tags not already included have been found, ;; then recursively expand them as well. ;; Notice that by passing (group-tags excluded) to the next call ;; instead of ((append tags group-tags)) ensures that we do not ;; unnecessarily loop over the elements of TAGS more than once. (if group-tags (append tags (org-ql--expand-tag-hierarchy group-tags excluded)) tags)))) (defun org-ql--tags-at (position) "Return tags for POSITION in current buffer. Returns cons (INHERITED-TAGS . LOCAL-TAGS)." ;; I'd like to use `-if-let*', but it doesn't leave non-nil variables ;; bound in the else clause, so destructured variables that are non-nil, ;; like found caches, are not available in the else clause. (if-let* ((buffer-cache (gethash (current-buffer) org-ql-tags-cache)) (modified-tick (car buffer-cache)) (tags-cache (cdr buffer-cache)) (buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (cached-result (gethash position tags-cache))) ;; Found in cache: return them. (pcase cached-result ('org-ql-nil nil) (_ cached-result)) ;; Not found in cache: get tags and cache them. (let* ((local-tags (or (when (looking-at org-ql-tag-line-re) (split-string (match-string-no-properties 2) ":" t)) 'org-ql-nil)) (inherited-tags (or (when org-use-tag-inheritance (save-excursion (if (org-up-heading-safe) ;; Return parent heading's tags. (-let* (((inherited local) (org-ql--tags-at (point))) (tags (when (or inherited local) (cond ((and (listp inherited) (listp local)) (->> (append inherited local) -non-nil -uniq)) ((listp inherited) inherited) ((listp local) local))))) (cl-typecase org-use-tag-inheritance (list (setf tags (-intersection tags org-use-tag-inheritance))) (string (setf tags (--select (string-match org-use-tag-inheritance it) tags)))) (pcase org-tags-exclude-from-inheritance ('nil tags) (_ (-difference tags org-tags-exclude-from-inheritance)))) ;; Top-level heading: use file tags. org-file-tags))) 'org-ql-nil)) (all-tags (list inherited-tags local-tags))) ;; Check caches again, because they may have been set now. ;; TODO: Is there a clever way we could avoid doing this, or is it inherently necessary? (setf buffer-cache (gethash (current-buffer) org-ql-tags-cache) modified-tick (car buffer-cache) tags-cache (cdr buffer-cache) buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (unless (and buffer-cache buffer-unmodified-p) ;; Buffer-local tags cache empty or invalid: make new one. (setf tags-cache (make-hash-table)) (puthash (current-buffer) (cons (buffer-modified-tick) tags-cache) org-ql-tags-cache)) (puthash position all-tags tags-cache)))) (org-ql-select (org-ql-search-directories-files) '(tags "Emacs") :action #'point))) ("with group-tags support" (progn (setf org-ql-cache (make-hash-table :weakness 'key) org-ql-tags-cache (make-hash-table :weakness 'key) org-ql-node-value-cache (make-hash-table :weakness 'key)) (defun org-ql--expand-tag-hierarchy (tags &optional excluded) "Return TAGS along with their associated group tags. This function recursively searches for groups that each given tag belongs to, directly or indirectly, and includes the corresponding group tags to the result. TAGS should be a list of tags (i.e., strings). If non-nil, EXCLUDED should be a list of group tags that will not be automatically added to the results unless they are already in TAGS." (let ((groups (org-tag-alist-to-groups org-current-tag-alist)) (excluded (append tags excluded))) (let (group-tags) (dolist (tag tags) (pcase-dolist (`(,group-tag . ,group-members) groups) (when (and (not (member group-tag excluded)) ;; Check if one of the members in the group matches tag. ;; Notice that each member may be a plain string or ;; a regexp pattern (enclosed between curly brackets). (--some (if (string-match-p "^[{].+[}]$" it) ;; If pattern (it) is a regexp, remove the brackets and ;; make sure that it either matches the whole tag or not. (string-match-p (concat "^" (substring it 1 -1) "$") tag) ;; Check if member (it) is identical to tag. (string= it tag)) group-members)) (push group-tag group-tags)))) ;; If group tags not already included have been found, ;; then recursively expand them as well. ;; Notice that by passing (group-tags excluded) to the next call ;; instead of ((append tags group-tags)) ensures that we do not ;; unnecessarily loop over the elements of TAGS more than once. (if group-tags (append tags (org-ql--expand-tag-hierarchy group-tags excluded)) tags)))) (defun org-ql--tags-at (position) "Return tags for POSITION in current buffer. Returns cons (INHERITED-TAGS . LOCAL-TAGS)." ;; I'd like to use `-if-let*', but it doesn't leave non-nil variables ;; bound in the else clause, so destructured variables that are non-nil, ;; like found caches, are not available in the else clause. (if-let* ((buffer-cache (gethash (current-buffer) org-ql-tags-cache)) (modified-tick (car buffer-cache)) (tags-cache (cdr buffer-cache)) (buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (cached-result (gethash position tags-cache))) ;; Found in cache: return them. (pcase cached-result ('org-ql-nil nil) (_ cached-result)) ;; Not found in cache: get tags and cache them. (let* ((local-tags (or (when (looking-at org-ql-tag-line-re) (split-string (match-string-no-properties 2) ":" t)) 'org-ql-nil)) (inherited-tags (or (when org-use-tag-inheritance (save-excursion (if (org-up-heading-safe) ;; Return parent heading's tags. (-let* (((inherited local) (org-ql--tags-at (point))) (tags (when (or inherited local) (cond ((and (listp inherited) (listp local)) (->> (append inherited local) -non-nil -uniq)) ((listp inherited) inherited) ((listp local) local))))) (cl-typecase org-use-tag-inheritance (list (setf tags (-intersection tags org-use-tag-inheritance))) (string (setf tags (--select (string-match org-use-tag-inheritance it) tags)))) (pcase org-tags-exclude-from-inheritance ('nil tags) (_ (-difference tags org-tags-exclude-from-inheritance)))) ;; Top-level heading: use file tags. org-file-tags))) 'org-ql-nil)) all-tags) (when org-group-tags (unless (eq local-tags 'org-ql-nil) (setq local-tags (org-ql--expand-tag-hierarchy local-tags))) (unless (eq inherited-tags 'org-ql-nil) (setq inherited-tags (org-ql--expand-tag-hierarchy inherited-tags)))) (setq all-tags (list inherited-tags local-tags)) ;; Check caches again, because they may have been set now. ;; TODO: Is there a clever way we could avoid doing this, or is it inherently necessary? (setf buffer-cache (gethash (current-buffer) org-ql-tags-cache) modified-tick (car buffer-cache) tags-cache (cdr buffer-cache) buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (unless (and buffer-cache buffer-unmodified-p) ;; Buffer-local tags cache empty or invalid: make new one. (setf tags-cache (make-hash-table)) (puthash (current-buffer) (cons (buffer-modified-tick) tags-cache) org-ql-tags-cache)) (puthash position all-tags tags-cache)))) (org-ql-select (org-ql-search-directories-files) '(tags "Emacs") :action #'point))) )) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------------+--------------------+---------------+----------+------------------| | without group-tags support | 1.01 | 52.832562 | 4 | 1.989522 | | with group-tags support | slowest | 53.425342 | 5 | 2.479128 | [2020-11-09 Mon 17:57] Well, the performance difference seems smaller than I expected. For single iterations, it ought to be unnoticeable. Although I'm still a bit skeptical about this benchmark: I feel like it ought to have more of an impact than that, but maybe I'm wrong--and that would be great! Next steps: + [X] Post benchmark code on PR and ask Panagiotis to verify + [X] Also ask him to run benchmark actually using group tags (since I don't actually have any, even though the boolean is t) + [X] Discuss caching of group tag expansion. It seems like we ought to cache the expansions as well, because sibling headings (especially at level 1) ought to get their group tags re-expanded individually, even when we've already expanded them for another heading. + [X] Remove unused =result= variable ** PROJECT [#B] Document the sorting functions Note that the built-in sorting only works on Org elements, which is the default ~:action~. So if a different action is used, sorting will not work. In that case, the action should be mapped across the Org element results from outside the ~org-ql~ form. ** PROJECT [#B] Recursive queries :PROPERTIES: :milestone: future :END: For lack of a better term. A way to query for certain headings, and then gather results of a different query at each result of the first query, displaying all results in a single view. This works pretty well. It needs polishing, and some refactoring so items can be indented completely (rather than leaving the keyword unindented, as it is now). #+BEGIN_SRC elisp (cl-defun org-ql-agenda-recursive (buffers-or-files queries &key action narrow sort) (cl-labels ((rec (queries element indent) (org-with-point-at (org-element-property :org-marker element) (when-let* ((results (progn (org-narrow-to-subtree) (org-ql-select (current-buffer) (car queries) :action 'element-with-markers :narrow t :sort sort)))) ;; Indent entry for each level (setf results (--map (org-element-put-property it :raw-value (concat (s-repeat (* 5 indent) " ") (org-element-property :raw-value it))) results)) (cons it (if (cdr queries) (--map (rec (cdr queries) it) results) results)))))) (when-let* ((indent 0) (results (org-ql-select buffers-or-files (car queries) :action 'element-with-markers :narrow narrow :sort sort))) (->> (if (cdr queries) (--map (rec (cdr queries) it (1+ indent)) results) results) (-flatten-n (1- (length queries))) -non-nil (org-ql-agenda--agenda nil nil :entries))))) (cl-defun org-ql-select-recursive (buffers-or-files queries &key action narrow sort) (cl-labels ((rec (queries element indent) (org-with-point-at (org-element-property :org-marker element) (when-let* ((results (progn (org-narrow-to-subtree) (org-ql-select (current-buffer) (car queries) :action 'element-with-markers :narrow t :sort sort)))) ;; Indent entry for each level (setf results (--map (org-element-put-property it :raw-value (concat (s-repeat (* 5 indent) " ") (org-element-property :raw-value it))) results)) (cons it (if (cdr queries) (--map (rec (cdr queries) it) results) results)))))) (when-let* ((indent 0) (results (org-ql-select buffers-or-files (car queries) :action 'element-with-markers :narrow narrow :sort sort))) (->> (if (cdr queries) (--map (rec (cdr queries) it (1+ indent)) results) results) (-flatten-n (1- (length queries))) -non-nil)))) #+END_SRC ** PROJECT [#B] Timeline view :PROPERTIES: :ID: 00573552-ffe9-4608-8904-7f6c73246b6d :milestone: future :END: e.g. as mentioned by Samuel Wales at https://lists.gnu.org/archive/html/emacs-orgmode/2019-08/msg00330.html. Prototype code: #+BEGIN_SRC elisp (cl-defun org-ql-timeline (buffers-files query) (let ((results (org-ql-select buffers-files query :action (lambda () (let* ((heading-string (->> (org-element-headline-parser (line-end-position)) org-ql--add-markers org-ql-agenda--format-element)) (timestamps (cl-loop with limit = (org-entry-end-position) while (re-search-forward org-ts-regexp-both limit t) collect (ts-parse-org (match-string 0)))) (timestamp-strings (->> timestamps (-sort #'ts<) (--map (concat " " (ts-format it)))))) (s-join "\n" (cons heading-string timestamp-strings)))) :sort '(date)))) (org-ql-agenda--agenda nil nil :strings results))) (org-ql-timeline (org-agenda-files) '(and "Emacs" (ts))) ;; More timeline-like version, organized by date rather than task. (cl-defun org-ql-timeline* (buffers-files query &key filter-ts) (let* ((ts-ht (ht)) (results (org-ql-select buffers-files query :action (lambda () (let* ((heading-string (->> (org-element-headline-parser (line-end-position)) org-ql--add-markers org-ql-agenda--format-element)) (date-timestamps ;; Each one set to 00:00:00. (cl-loop with limit = (org-entry-end-position) while (re-search-forward org-ts-regexp-both limit t) collect (->> (match-string 0) ts-parse-org (ts-apply :hour 0 :minute 0 :second 0))))) (setf date-timestamps (delete-dups date-timestamps)) (when filter-ts (setf date-timestamps (cl-remove-if-not filter-ts date-timestamps))) (--each date-timestamps (push heading-string (gethash it ts-ht))))))) (tss-sorted (-sort #'ts< (ht-keys ts-ht))) (strings (cl-loop for ts in tss-sorted collect (concat "\n" (propertize (ts-format "%Y-%m-%d" ts) 'face 'org-agenda-structure)) append (ht-get ts-ht ts)))) (org-ql-agenda--agenda nil nil :strings strings))) (org-ql-timeline* (org-agenda-files) '(ts :from -14) :filter-ts `(lambda (ts) (ts<= ,(ts-adjust 'day -14 (ts-now)) ts))) #+END_SRC Another, more up-to-date implementation: #+BEGIN_SRC elisp ;; NOTE: ts structs don't (sometimes? or always?) compare properly ;; with default hash tables, e.g. this code: ;; (let* ((ts-a #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1572670800.0)) ;; (ts-b #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1572584400.0))) ;; (list :equal (equal ts-a ts-b) ;; :sxhash-equal (equal (sxhash ts-a) (sxhash ts-b)))) ;;=> (:equal nil :sxhash-equal t) ;; So we must use the "contents-hash" table as described in the Elisp manual. (define-hash-table-test 'contents-hash 'equal 'sxhash-equal) (cl-defun org-ql-view-timeline (buffers-files &key from to on) "FIXME: DOcstring" (cl-flet ((parse-ts-arg (arg type) ;; Parse ARG as a string or TS struct and adjust it to the beginning ;; or end of its day, depending on whether TYPE is `:begin' or `:end'. (-let (((hour minute second) (cl-ecase type (:begin '(0 0 0)) (:end '(23 59 59))))) (->> (cl-typecase arg (string (ts-parse arg)) (ts arg)) (ts-apply :hour hour :minute minute :second second))))) (let* ((ts-predicate `(lambda (ts) ,(cond (on `(ts-in ,(parse-ts-arg on :begin) ts ,(parse-ts-arg on :end))) ((and from to) `(ts-in ,(parse-ts-arg from :begin) ts ,(parse-ts-arg to :end))) (from `(ts<= ,(parse-ts-arg from :begin) ts)) (to `(ts<= ts ,(parse-ts-arg to :end))) (t (user-error "Huh?"))))) (query (cond (on `(ts :from ,(parse-ts-arg on :begin) :to ,(parse-ts-arg on :end))) (t (append (list 'ts) (when from `(:from ,(parse-ts-arg from :begin))) (when to `(:to ,(parse-ts-arg to :end))))))) (date-ts-table (make-hash-table :test 'contents-hash)) (_results (org-ql-select buffers-files query :action (lambda () (let* ((string (->> (org-element-headline-parser (line-end-position)) org-ql--add-markers org-ql-view--format-element))) (cl-loop with limit = (org-entry-end-position) while (re-search-forward org-ts-regexp-both limit t) for ts = (->> (match-string 0) ts-parse-org) when (funcall ts-predicate ts) do (cl-pushnew (cons ts (concat (ts-format " %H:%M" ts) string)) (gethash (ts-apply :hour 0 :minute 0 :second 0 ts) date-ts-table) :test #'equal)))))) (date-tss-sorted (->> date-ts-table hash-table-keys (-sort #'ts<))) (string (cl-loop for date-ts in date-tss-sorted for date-string = (propertize (ts-format "%Y-%m-%d" date-ts) 'face 'org-agenda-structure) concat (concat "\n" date-string) concat (cl-loop for (ts . entry) in (->> (gethash date-ts date-ts-table) (-sort (-on #'ts< #'car))) concat (concat "\n" entry))))) (org-ql-view--display :buffer "Timeline" :header "Timeline" :string string)))) ;; Used like: ;; (org-ql-view-timeline "~/org/main.org" :from "2019-11-01") #+END_SRC [2019-09-26 Thu 21:28] Would probably make sense to implement this using the view-sections someday. ** PROJECT [#B] Implement view with tabulated-list-mode or magit-section [2019-09-02 Mon 05:20] Especially with some of the new packages that make =tabulated-list-mode= easier to use, like =navigel=. However, it would probably break grouping, or require some kind of adapter or extension to do grouping, so I don't know if that would work. Something like =magit-section= would be more flexible, and could be recursively grouped, like in =magit-todos=. [2019-09-08 Sun 10:06] Came up with a prototype yesterday, in branch =wip/view-section=. Seems to work pretty well. One of the things in that branch is =org-ql-item=, which is a struct used to carry data for query results. It seems to work well. Another idea for it is to simply store the element from =org-element-headline-parser= in one of its slots, and populate all of the other slots lazily, like =ts=. It already does that for a couple of slots, but I think it makes sense to do it for all of them, to reduce the overhead of making the struct for every query result. *** MAYBE [#C] Experiment with =widget= :PROPERTIES: :milestone: future :END: The code that powers the customization UI. Has collapsible and customizable widgets. Might be perfect. Might even enable editing items in the list, with functions to make the changes in the source buffers. *** Code idea Inserting items into a view could look something like this: #+BEGIN_SRC elisp (org-ql-view--insert-items :header (ts-format "%Y-%m-%d" (ts-now)) :items (org-ql-query :select #'org-ql-current-item :from (org-agenda-files) :where '(or (deadline auto) (scheduled :on today) (ts-active :on today))) :group-by '(org-ql-item-priority org-ql-item-todo)) #+END_SRC Items would be structs, and the =group-by= argument would be a list of accessors, like how =magit-todos= works. Arbitrary functions could also be passed to =group-by=, as whatever value the function returns is used to group them. =org-ql-current-item= would be a function that turns the result of =org-element-headline-parser= into the struct. Not sure if it should automatically add the number of items to the header, or if that should be done manually. *** Prior art **** [[https://github.com/m2ym/direx-el][GitHub - m2ym/direx-el: Directory Explorer for GNU Emacs]] Appears to be another implementation of magit-section-like expandable sections. Not sure which came first. Its code seems like it may be helpful. **** magit-section ** PROJECT Dynamic blocks *** PROJECT [#B] Save view to dynamic block :PROPERTIES: :milestone: 0.7 :END: [2020-11-10 Tue 04:31] A command would save users from having to write out the dynamic block manually. [2020-11-12 Thu 03:23] A command to do this would be very helpful. (Yes, I entered this idea twice. I should use my own systems better, apparently. But that's what this package is all about, right?) **** NEXT Lookup Org function to create dynamic blocks **** NEXT Write function to save view to dynamic block *** DONE [#A] Implement dynamic blocks + *Tasks* - [X] Merge code - [X] Document the feature For example, [[https://egli.dev/posts/using-org-mode-for-meeting-minutes/][this blog article]] shows a way that Org's existing dynamic =columnview= blocks can be very useful. =org-ql= queries could be useful in them as well. [2020-11-09 Mon 22:00] I just realized that this is probably much easier than I realized. + [[info:org#Dynamic%20blocks][info:org#Dynamic blocks]] #+BEGIN_SRC elisp (cl-defun org-dblock-write:org-ql (params) "FIXME: Docstring" (pcase-let* (((map :query :columns :sort :ts-format) params) (format-fns (list (cons 'heading (lambda (element) (org-make-link-string (org-element-property :raw-value element) (org-element-property :raw-value element)))) (cons 'todo (lambda (element) (or (org-element-property :todo-keyword element) ""))) (cons 'priority (lambda (element) (--if-let (org-element-property :priority element) (char-to-string it) ""))) (cons 'deadline (lambda (element) (--if-let (org-element-property :deadline element) (ts-format ts-format (ts-parse-org-element it )) ""))) (cons 'scheduled (lambda (element) (--if-let (org-element-property :scheduled element) (ts-format ts-format (ts-parse-org-element it )) ""))))) (elements (org-ql-query :from (current-buffer) :where query :select '(org-element-headline-parser (line-end-position)) :order-by sort))) (cl-labels ((format-element (element) (string-join (cl-loop for column in columns for fn = (alist-get column format-fns) collect (funcall fn element)) " | "))) (insert "| " (string-join (--map (capitalize (symbol-name it)) columns) " | ") " |" "\n") (insert "|- \n") (dolist (element elements) (insert "| " (format-element element) " |" "\n")) (org-table-align)))) #+END_SRC [2020-11-09 Mon 22:35] This works pretty well! For example: #+BEGIN_SRC org ,#+BEGIN: org-ql :query (todo) :format (priority todo heading deadline scheduled) :sort (priority date) :ts-format "%Y-%m-%d %H:%M" | Priority | Todo | Heading | Deadline | Scheduled | |----------+-------+------------+------------------+------------------| | A | TODAY | Heading 1 | 2020-11-11 00:00 | | | B | TODO | Heading 2 | | 2020-11-09 00:00 | ,#+END: ,#+BEGIN: columnview :id global :hlines t :indent t | ITEM | TODO | PRIORITY | TAGS | |----------------+-------+----------+------| | Test heading 1 | TODAY | B | | |----------------+-------+----------+------| | Heading 2 | TODO | B | | ,#+END: ,* TODAY [#A] Heading 1 DEADLINE: <2020-11-11 Wed> ,* TODO [#B] Heading 2 SCHEDULED: <2020-11-09 Mon> #+END_SRC [2020-11-10 Tue 00:03] I think it's ready to merge now. *** DONE [#A] Org block to insert results of queries with links to entries :PROPERTIES: :ID: 422754bb-3a7c-4dbf-b303-4056d3cafb7e :END: [2020-01-16 Thu 06:20] This idea just came to me when I was thinking about using the search-based paradigm vs. outline-based. This would allow both, e.g. some kind of =#+BEGIN_QUERY= block that would update when =C-c C-c= is pressed on it. [2020-11-13 Fri 22:57] I keep rediscovering ideas that I've had previously. This is now done as the dynamic block feature. I guess I should actually use these tools I've made. ** PROJECT [#B] Predicate helper functions :PROPERTIES: :milestone: future :END: [2020-11-24 Tue 16:50] This idea started off by writing a =week= predicate: #+BEGIN_SRC elisp (org-ql-defpred week (&optional relative) "Match entries with a timestamp in the calendar week RELATIVE. RELATIVE is a number relative to the current week (i.e. 0 or nil is this week, -1 is last week, 1 is next week)." :normalizers ((`(,predicate-names . ,(or `(,(and (pred numberp) relative)) `nil)) (let* ((relative (or relative 0)) (now (ts-now)) (week-start (->> now (ts-adjust 'day (- 0 (ts-dow now))) (ts-adjust 'day (* 7 relative)) (ts-apply :hour 0 :minute 0 :second 0))) (week-end (->> now (ts-adjust 'day (- 7 (ts-dow now))) (ts-adjust 'day (* 7 relative)) (ts-apply :hour 23 :minute 59 :second 59)))) `(ts :from ,week-start :to ,week-end))) (`(,predicate-names ,(and (pred stringp) date)) (let* ((then (ts-parse date)) (week-start (->> then (ts-adjust 'day (- 0 (ts-dow then))) (ts-apply :hour 0 :minute 0 :second 0))) (week-end (->> then (ts-adjust 'day (- 7 (ts-dow then))) (ts-apply :hour 23 :minute 59 :second 59)))) `(ts :from ,week-start :to ,week-end))))) #+END_SRC That works pretty well: #+BEGIN_SRC elisp :results code (list (org-ql--normalize-query '(week)) (org-ql--normalize-query '(week -1)) (org-ql--normalize-query '(week "2020-11-01"))) #+END_SRC #+RESULTS: #+BEGIN_SRC elisp ((ts :from #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1606024800.0) :to #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1606715999.0)) (ts :from #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1605420000.0) :to #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1606111199.0)) (ts :from #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1604206800.0) :to #s(ts nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 1604901599.0))) #+END_SRC (Hmm, the patterns in those beginning-of-week and end-of-week timestamps are interesting...) And that's pretty useful. But what if someone wanted to write a query like ~(closed :on (week -1))~ to match entries closed in the last week? It wouldn't help at all. One idea would be to add a =type= argument to the =week= predicate. But that would be awkward and a bit ugly. And it wouldn't solve the problem, anyway, because while the =ts= predicate can take a type argument for =active=, =inactive=, or =both=, it can't do, e.g. =clocked=, =closed=, =deadline=, etc. So what's really needed is a way to insert the week-based arguments into other selectors. But that would require there to be a ~(week)~ function defined, which would mean polluting the global namespace and potential conflicts. So maybe another macro could define "helper" functions which would be available in the normalizers. Or maybe the =defpred= macro could take another argument for "helpers", although that would probably require binding them around each =pcase= expression--not hard, but not especially elegant. The next question is, how would those work in string queries? I'm not sure there's a good way to translate them. ** PROJECT [#C] [[https://github.com/magit/transient/issues/76][New Transient transient-lisp-variable class]] :compatibility: :PROPERTIES: :milestone: 0.7 :END: [2020-10-19 Mon 00:23] Should try to use this instead of whatever bespoke code is currently used. ** PROJECT [#C] Normalize queries :PROPERTIES: :milestone: future :END: [2019-07-16 Tue 11:49] This serves two purposes: 1. Equivalent queries will return the same results from the cache. 2. The selectors that can be converted to the fastest preamble regexps will be sorted first, so the fastest preamble will be used. Although this may not always be straightforward. For example, in a file with only a few =TODO= items, the ~(todo "TODO")~ selector would convert to a preamble that would quickly search through the file. But if there were a thousand =TODO= items, it wouldn't be as much of a benefit, and a ~(regexp "something")~ selector's preamble might be much faster, depending on how many times =something= appears in the file. So the second purpose might actually be a drawback, because it would prevent users from optimizing their queries with knowledge of their data. Maybe there should be an option to not normalize queries, so advanced users can order their selectors manually. ** PROJECT [#C] Update view screenshots e.g. doesn't currently show the =View= header. ** PROJECT [#C] Test caching See notes on 1dce9467f25428b5289d3665cd840820969ed65a. It would be good to test the caching explicitly, at least for some queries, because if I were to completely break it again, in such a way that results were stored but retrieval always failed, the tests wouldn't catch it. ** CANCELED [#C] Alternative parsing libraries + e.g. Bovine and Wisent come with Emacs, which would allow us to drop the =peg= dependency +(which doesn't use lexical binding)+ + [[https://github.com/cute-jumper/parsec.el][parsec]] is third-party, but looks powerful [2021-09-08 Wed 15:40] (=peg= does use lexical binding now, and it's in ELPA and getting a bit of attention, so no need to drop it, I think.) ** DONE [#A] Timestamp predicates using relative dates break caching :bug: :PROPERTIES: :milestone: 0.6 :branch: wip/fix-223 :issue-link: [[https://github.com/alphapapa/org-ql/issues/223][Cache is not invalidated for relative date/time queries · Issue #223 · alphapapa/org-ql · GitHub]] :END: Could fix this in a 0.5.3 release, but I think too much code changes are involved for a bugfix release. *** DONE Fix/commit checklists + Closing issue: 223 *** DONE Move from-to-on argument processing to query normalizers **** DONE Update defpred docs + [X] Mention that normalizers are run until they return the same result; need to guard against infinite loops. - [X] Update docstring - [X] Update readme - [X] defpred tutorial **** DONE Remove old ~--from-to-on~ macro completely **** DONE closed + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results **** DONE clocked + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results **** DONE ts + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results **** DONE planning + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results **** DONE deadline + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results **** DONE scheduled + [X] Use new ~--normalize-from-to-on~ macro + [X] Remove old ~--from-to-on~ from ~--query-predicate~ + [X] Add tests - [X] Query normalization - [X] Results ** DONE [#A] [[https://github.com/alphapapa/org-ql/pull/220][~(link :target)~ doesn't work]] :bug: :PROPERTIES: :milestone: 0.5.2 :END: :LOGBOOK: CLOCK: [2021-06-16 Wed 17:02]--[2021-06-16 Wed 17:41] => 0:39 :END: ** DONE [#A] [[https://github.com/alphapapa/org-ql/pull/220][~(link :regexp-p)~ doesn't work]] :bug: :PROPERTIES: :milestone: 0.5.2 :END: ** DONE [#A] Fix: Custom sorter breaks cached results :bug: :PROPERTIES: :milestone: 0.5.1 :END: + [[https://github.com/alphapapa/org-ql/issues/186][custom sort functions pollute cache · Issue #186 · alphapapa/org-ql · GitHub]] + [[https://github.com/alphapapa/org-ql/pull/187][replace sort with -sort. by natask · Pull Request #187 · alphapapa/org-ql · GitHub]] ** DONE [#A] Update =dash= dependencies :PROPERTIES: :milestone: 0.6 :END: + [[https://github.com/alphapapa/org-ql/issues/209][Out-of-date dependences (dash-functional) · Issue #209 · alphapapa/org-ql · GitHub]] + [[https://github.com/alphapapa/org-ql/pull/197][Bump dash.el to fix helm-org-ql by landakram · Pull Request #197 · alphapapa/org-ql · GitHub]] ** DONE [#A] Checking links for unsafe parameters :PROPERTIES: :ID: ba70e375-eddb-40df-8892-fb418c1f70d1 :milestone: 0.5 :END: :LOGBOOK: - State "PROJECT" from "UNDERWAY" [2020-11-12 Thu 00:26] - State "UNDERWAY" from [2020-11-11 Wed 23:09] :END: Theoretically one could put a sexp-based query into a link that would run arbitrary code to do something evil. Like: [[org-ql-search:(message "AHA")]] That's very unlikely to be abused, but it would be good to protect against it. Two possibilities: 1. For sexp-based queries in links and dynamic blocks, prompt for confirmation before running. 2. Use a special variable to control whether lambdas and arbitrary sexps are allowed in queries, and disable it for links and dynamic blocks. (That might be difficult to do, since they could be buried in an ~and~ or something. A whitelist approach might be needed.) *** DONE [#A] Tag org-super-agenda 1.2 and bump required version in org-ql :PROPERTIES: :milestone: 0.5 :END: That /should/ force the version of org-super-agenda with the fix to be installed when org-ql is upgraded. [2020-11-22 Sun 17:19] org-super-agenda 1.2 is tagged and released, so now we can depend on it. *** DONE Add automated tests :LOGBOOK: - State "UNDERWAY" from "TODO" [2020-11-12 Thu 00:24] - State "TODO" from "MAYBE" [2020-11-11 Wed 23:16] - State "MAYBE" from [2020-11-11 Wed 23:15] :END: Maybe impractical, but maybe we could at least test that potentially unsafe ones signal errors. [2020-11-12 Thu 00:24] Works better than I expected. All the tests seem to correctly pass, signaling the correct errors for the correct reasons--except for the tests specific to org-super-agenda. For that, I'm currently waiting for MELPA to build the version of org-super-agenda that has the fix applied, so I can install that into the test sandbox, and then those two tests should pass also. *** DONE Enumerate and test parameters and potentially unsafe types CLOSED: [2020-11-11 Wed 23:26] :LOGBOOK: - State "DONE" from "UNDERWAY" [2020-11-11 Wed 23:26] - State "UNDERWAY" from [2020-11-11 Wed 23:15] :END: #+CAPTION: Template for making testable links #+BEGIN_SRC org [[org-ql-search:todo:?]] #+END_SRC #+CAPTION: Expression to insert encoded values into template (after the =?=) #+BEGIN_SRC elisp (insert (url-hexify-string (concat "buffers-files=" (prin1-to-string '((lambda () (message "AHA"))))))) #+END_SRC + [X] Buffers-Files: Expanded by =org-ql-view--expand-buffers-files=: - [X] Quoted lambda: (safe) [[org-ql-search:todo:?buffers-files%3D%28lambda%20nil%20%28message%20%22AHA%22%29%29]] - [X] Unquoted lambda: (safe) [[org-ql-search:todo:?buffers-files%3D%28lambda%20nil%20%28message%20%22AHA%22%29%29]] - [X] Quoted lambda in list (safe): [[org-ql-search:todo:?buffers-files%3D%28%28quote%20%28lambda%20nil%20%28message%20%22AHA%22%29%29%29%29]] - [X] Unquoted lambda in list: (safe) [[org-ql-search:todo:?buffers-files%3D%28%28lambda%20nil%20%28message%20%22AHA%22%29%29%29]] + [X] Groups - [X] Quoted lambda (safe): [[org-ql-search:todo:?super-groups%3D%28lambda%20nil%20%28message%20%22AHA%22%29%29]] - [X] Unquoted lambda (safe): [[org-ql-search:todo:?super-groups%3D%28lambda%20nil%20%28message%20%22AHA%22%29%29]] - [X] Quoted expression (safe): [[org-ql-search:todo:?super-groups%3D%28message%20%22AHA%22%29]] - [X] Unquoted expression (safe): [[org-ql-search:todo:?super-groups%3D%22AHA%22]] - [X] ~:pred~ selector (UNSAFE, but caught with new org-super-agenda variable): [[org-ql-search:todo:?super-groups%3D%28%28%3Apred%20%28lambda%20%28_%29%20%28message%20%22AHA%22%29%29%29%29]] - [X] =:auto-map= selector (UNSAFE, but caught with new org-super-agenda variable): [[org-ql-search:todo:?super-groups%3D%28%28%3Aauto-map%20%28lambda%20%28_%29%20%28message%20%22AHA%22%29%29%29%29]] + [X] Title - [X] Quoted lambda (produces the same encoded value as unquoted lambda): (safe) [[org-ql-search:todo:?title%3D%28lambda%20%28_%20_%29%20%28message%20%22AHA%22%29%29]] - [X] Unquoted lambda: (safe) [[org-ql-search:todo:?title%3D%28lambda%20%28_%20_%29%20%28message%20%22AHA%22%29%29]] - [X] Expression: (safe) [[org-ql-search:todo:?title%3D%28message%20%22AHA%22%29]] + [X] Sort - [X] Bare, quoted lambda: (maybe unsafe, but caught now): [[org-ql-search:todo:?sort%3D%28lambda%20%28_%20_%29%20%28message%20%22AHA%22%29%29]] - [X] Bare, unquoted lambda: (UNSAFE, but caught now): [[org-ql-search:todo:?sort%3D%28lambda%20%28_%20_%29%20%28message%20%22AHA%22%29%29]] - [X] Quoted lambda in list: (maybe unsafe, but caught now): [[org-ql-search:todo:?sort%3D%28%28quote%20%28lambda%20%28_%20_%29%20%28message%20%22AHA%22%29%29%29%29]] - [X] Unquoted lambda in a list: (UNSAFE, but caught now): [[org-ql-search:todo:?sort=((lambda%20nil%20(message%20"AHA")))]] For the query expression: 1. String queries are parsed by the PEG parsing function (which I will probably rename soon), which should only allow known Org QL predicates, not arbitrary functions. For example: #+BEGIN_SRC elisp (org-ql--plain-query "message:AHA") ;;=> (regexp "message:AHA") (org-ql--plain-query '(message "AHA")) ;;=> (wrong-type-argument stringp (message "AHA")) (org-ql--plain-query "(message \"AHA\"") ;;=> (and (regexp "(message") (regexp "AHA")) #+END_SRC 2. Sexp queries already prompt for confirmation, unless the user has set =org-ql-view-ask-unsafe-links= to nil. [2020-11-11 Wed 23:27] That's all the parameters and all the types that I can think to test. ** DONE [#A] Views: Multiple sorters are not preserved :bug: :PROPERTIES: :milestone: 0.5 :url: https://github.com/alphapapa/org-ql/issues/136 :END: ** DONE [#A] Make dynamic blocks warn about sexp queries :PROPERTIES: :milestone: 0.5 :END: :LOGBOOK: CLOCK: [2020-11-16 Mon 22:56]--[2020-11-17 Tue 00:25] => 1:29 :END: [2020-11-12 Thu 05:22] I guess to be super-extra careful, just in case someone had =org-update-all-dblocks= in the =before-save-hook= or something. [2020-11-17 Tue 00:25] It warns and the warning is tested. ** DONE [#A] Add Emacs 27.1 to =test.yml= :PROPERTIES: :milestone: 0.5 :END: [2020-11-16 Mon 05:22] Also releasing =makem.sh= 0.2 with this change. ** DONE [#A] Fix =org-ql-view--link-open= on Org 9.3+ :compatibility:bug: :PROPERTIES: :milestone: 0.5 :url: https://github.com/alphapapa/org-ql/issues/147 :END: The version of Org in my personal that passes a URL-decoded string (i.e. as if run through =url-unhex-string=) as the argument to =org-ql-view--link-open=. But Org 9.3 in Emacs 27.1 passes a non-URL-decoded string, so =org-ql-view--link-open= needs to pass it through =url-unhex-string= itself. But I don't know which version of Org that changed in. I'm comparing the function =org-open-at-point=, but it's a 114-line function, and in neither version does it call =url-unhex-string=, so whatever code decodes the string must be elsewhere. I do recall something about links changing in Org 9.3 (or thereabouts), so that was probably part of it. Maybe I can find it in the release notes. I just need to know basically which version it happened in. I noticed because the CI tests on GitHub show the link-safety tests failing on the Emacs snapshot version. However, I think they're not currently vulnerable on that Org version, because the link parameters fail to be parsed correctly, so all the arguments to =org-ql-search= should end up being nil. [2020-11-14 Sat 20:41] I should probably do something like this in =org-ql-view--link-open=: #+BEGIN_SRC elisp (when (version<= "9.3" (org-version)) ;; Org 9.3+ makes a backward-incompatible change to link escaping. ;; I don't think it would be a good idea to try to guess whether ;; the string received by this function was made with or without ;; that change, so we'll just test the current version of Org. ;; Any links created with older Org versions and then opened with ;; newer ones will have to be recreated. (setf path (url-unhex-string path))) #+END_SRC But, first, I should write tests for saving and opening links, so it can actually be tested on different versions automatically. [2020-11-16 Mon 05:12] Finally, all of the tests pass on my Org version and on 9.3. And I tested for all the combinations of link and bookmark saving/opening I could think of. I hope they work and are safe. *** DONE [#A] Write tests for storing/opening links :PROPERTIES: :milestone: 0.5 :END: [2020-11-16 Mon 05:11] Took way longer than I expected. I hope it was worth it. *** DONE [#A] Check Org release notes for link changes [2020-11-13 Fri 22:44] From [[https://www.orgmode.org/Changes_old.html][the changelog]]: #+BEGIN_QUOTE Change bracket link escaping syntax Org used to percent-encode sensitive characters in the URI part of the bracket links. Now, escaping mechanism uses the usual backslash character, according to the following rules, applied in order: - All consecutive \ characters at the end of the link must be escaped; - Any ] character at the very end of the link must be escaped; - All consecutive \ characters preceding ][ or ]] patterns must be escaped; - Any ] character followed by either [ or ] must be escaped; - Others ] and \ characters need not be escaped. When in doubt, use the function org-link-escape in order to turn a link string into its properly escaped form. The following function will help switching your links to the new syntax: (defun org-update-link-syntax (&optional no-query) "Update syntax for links in current buffer. Query before replacing a link, unless optional argument NO-QUERY is non-nil." (interactive "P") (org-with-point-at 1 (let ((case-fold-search t)) (while (re-search-forward "\\[\\[[^]]*?%\\(?:2[05]\\|5[BD]\\)" nil t) (let ((object (save-match-data (org-element-context)))) (when (and (eq 'link (org-element-type object)) (= (match-beginning 0) (org-element-property :begin object))) (goto-char (org-element-property :end object)) (let* ((uri-start (+ 2 (match-beginning 0))) (uri-end (save-excursion (goto-char uri-start) (re-search-forward "\\][][]" nil t) (match-beginning 0))) (uri (buffer-substring-no-properties uri-start uri-end))) (when (or no-query (y-or-n-p (format "Possibly obsolete URI syntax: %S. Fix? " uri))) (setf (buffer-substring uri-start uri-end) (org-link-escape (org-link-decode uri))))))))))) The old org-link-escape and org-link-unescape functions have been renamed into org-link-encode and org-link-decode. #+END_QUOTE This is exactly the kind of breaking, backwards-incompatible change that I've said should mandate a major-version increment. It's not only a change in Org's code, and a change that affects third-party packages, but it's a change in the file format! Is it even possible to support both Org 9.3+ and earlier versions at the same time? This change doesn't even seem to make sense to me. Percent-encoding solves so many problems in a simple way: pass a string to the encoding function on the way in, and to the decoding function on the way out. Now, instead of a simple, standard way of encoding links, there's a list of Org-specific rules and Org-specific encoding/decoding functions. What is gained this way? ** DONE [#A] Fix query-sexp-to-string function's handling of, e.g. =descendants= :bug: :PROPERTIES: :milestone: 0.5 :END: [2020-11-14 Sat 20:45] Fixed in =89ff02a1501b53b4e20cdc66a8fcaa37e7d15734=. ** DONE [#A] Helm command In branch =wip/helm-org-ql=. Works really well, should add it and demonstrate it. *** DONE Add *** DONE Demonstrate *** DONE Parsing non-Lisp queries [2019-09-12 Thu 12:56] Lisp is so much easier to deal with, but some people don't like parentheses. So I'm trying to add a non-Lisp-style query syntax. It gets complicated. The =peg= library helps, but its documentation is sparse and incomplete. This seems to work fairly well for single-token queries, but I'm not sure if I can or should cram it all into one parser, or use separate ones for certain keywords. #+BEGIN_SRC elisp (-let* ((input "todo:check|someday") (input "tags:universe+space") (input "heading:\"spaced phrase\"") (input "") (input "heading:\"spaced phrase\"+another") combinator (parsed (peg-parse-string ((predicate (substring keyword) ":" (opt args)) (keyword (or "heading" "tags" "todo" "property")) (args (+ (and (or quoted-arg unquoted-arg) (opt separator)))) (quoted-arg "\"" unquoted-arg "\"") (unquoted-arg (substring (+ (not (or separator "\"")) (any)))) (separator (or (and "|" (action (setf combinator 'or))) (and "+" (action (setf combinator 'and))) (and ":" (action (setf combinator 'arg)))))) input 'noerror)) ((predicate . args) (nreverse parsed))) (when predicate (list :predicate predicate :args args :combinator combinator))) ;;=> (:predicate "heading" :args ("spaced phrase" "another" t) :combinator and) #+END_SRC I don't know where the =t= is coming from. The next step is to make it work with multi-token queries. It needs to handle all of the tokens in one parser so it can handle quoted phrases (if we split on spaces, it would split quoted phrases). But that makes getting the arguments out of it more difficult. Probably need to do something like this: #+BEGIN_SRC elisp (-let* ((input "todo:check|someday") (input "tags:universe+space") (input "heading:\"spaced phrase\"") (input "") (input "heading:\"spaced phrase\"+another") combinator (parsed (peg-parse-string ((query (+ (or (and predicate `(pred args -- (list :predicate pred :args args))) (and plain-string `(s -- (list :predicate 'regexp :args s)))) (opt (syntax-class whitespace)))) (plain-string (substring (+ (not (syntax-class whitespace)) (any)))) (predicate (substring keyword) ":" (opt args)) (keyword (or "heading" "tags" "todo" "property")) (args (+ (and (or quoted-arg unquoted-arg) (opt separator)))) (quoted-arg "\"" unquoted-arg "\"") (unquoted-arg (substring (+ (not (or separator "\"")) (any)))) (separator (or (and "|" (action (setf combinator 'or))) (and "+" (action (setf combinator 'and))) (and ":" (action (setf combinator 'arg)))))) input 'noerror))) parsed) #+END_SRC In which lists are pushed onto the stack and returned, rather than strings. But I don't understand yet exactly how to use the =var= forms to consume input from the "value stack"; I need to study the examples more. I'm also not sure if that will even work with a variable number of arguments. This seems to work, but we'll have to parse the args again in a separate step: #+BEGIN_SRC elisp (-let* ((input "todo:check|someday") (input "tags:universe+space") (input "heading:\"spaced phrase\"") (input "") (input "heading:\"spaced phrase\"+another") (input "heading:\"spaced phrase\"+another todo:check") combinator (parsed (peg-parse-string ((query (+ (or (and predicate `(pred args -- (list :predicate pred :args args))) (and plain-string `(s -- (list :predicate 'regexp :args s)))) (opt (+ (syntax-class whitespace) (any))))) (plain-string (substring (+ (not (syntax-class whitespace)) (any)))) (predicate (substring keyword) ":" (opt args)) (keyword (or "heading" "tags" "todo" "property")) (args (substring (+ (and (or quoted-arg unquoted-arg) (opt separator))))) (quoted-arg "\"" (+ (not (or separator "\"")) (any)) "\"") (unquoted-arg (+ (not (or separator "\"" (syntax-class whitespace))) (any))) (separator (or (and "|" (action (setf combinator 'or))) (and "+" (action (setf combinator 'and))) (and ":" (action (setf combinator 'arg)))))) input 'noerror))) parsed) ;;=> (t (:predicate "todo" :args "check") (:predicate "heading" :args "\"spaced phrase\"+another")) #+END_SRC Well, a bit of fiddling (lots of trial-and-error required) produced this: #+BEGIN_SRC elisp (-let* ((input "todo:check|someday") (input "tags:universe+space") (input "heading:\"spaced phrase\"") (input "") (input "heading:\"spaced phrase\"+another") (input "heading:\"spaced phrase\"+another todo:check") combinator (parsed (peg-parse-string ((query (+ (or (and predicate `(pred args -- (list :predicate pred :args args))) (and plain-string `(s -- (list :predicate 'regexp :args s)))) (opt (+ (syntax-class whitespace) (any))))) (plain-string (substring (+ (not (syntax-class whitespace)) (any)))) (predicate (substring keyword) ":" (opt args)) (keyword (or "heading" "tags" "todo" "property")) (args (list (+ (and (substring (or quoted-arg unquoted-arg)) (opt separator))))) (quoted-arg "\"" (+ (not (or separator "\"")) (any)) "\"") (unquoted-arg (+ (not (or separator "\"" (syntax-class whitespace))) (any))) (separator (or (and "|" (action (setf combinator 'or))) (and "+" (action (setf combinator 'and))) (and ":" (action (setf combinator 'arg)))))) input 'noerror))) parsed) ;;=> (t (:predicate "todo" :args ("check")) (:predicate "heading" :args ("\"spaced phrase\"" "another"))) #+END_SRC That seems pretty usable! ** DONE [#B] Add a ~:with-time~ argument to timestamp predicates :PROPERTIES: :milestone: 0.6 :ID: eada7d28-2580-4553-824e-ddab991a30d5 :END: + In branch =wip/with-time=. Which would obviate the need to [[https://gist.github.com/mskorzhinskiy/fac0662a39a7b1bf5b306f175943bc97#file-org-ql-rasmi-el-L59][use the ~org-scheduled-time-hour-regexp~]] to find entries scheduled for a certain time of day. Probably should do this after [[id:fc8ccf6e-5311-4121-a0b7-58482dbd2e85][Optimized, date-specific timestamp regexps]]. Also relevant to [[https://github.com/alphapapa/org-ql/pull/73][examples: add two new examples by mskorzhinskiy · Pull Request #73 · alphapapa/org-ql · GitHub]]. [2021-06-17 Thu 20:01] Implemented for ~scheduled~ in branch =wip/with-time2= (I should check for my own prior work before starting, haha). It's fairly simple, and since I haven't finished the date-specific timestamp regexps branch yet, I might as well do this first, because it's useful. The other branch has some needed work too, so I should fold it into the second branch. [2021-06-17 Thu 20:11] Merged second branch into first and deleted second. [2021-06-19 Sat 03:30] Everything seems to work. Merged and pushed to master. *** DONE Merge to master + [X] Probably squash merge. + [X] Push *** DONE Update docs **** DONE Readme ***** DONE deadline ***** DONE ts ***** DONE scheduled ***** DONE planning ***** DONE General date/time predicate info **** DONE Docstrings ***** DONE deadline ***** DONE ts ***** DONE scheduled ***** DONE planning *** DONE Ensure changes to these predicates are consistent :LOGBOOK: CLOCK: [2021-06-19 Sat 00:12]--[2021-06-19 Sat 00:36] => 0:24 :END: *** DONE ~deadline~ **** DONE Tests *** DONE ~scheduled~ + [X] Like ~planning~, select the regexp in ~org-ql--query-predicate~. + [X] Use new regexps. **** DONE Tests *** DONE ~planning~ **** DONE Tests **** DONE regexp defvar *** DONE ~ts~ [2021-06-18 Fri 00:00] Had to add a bunch of extra regexps, but probably needed to do that anyway. Seems to work... **** DONE Tests [2021-06-17 Thu 23:59] The tests pass and seem to work correctly. ** DONE [#B] "Node" caching [2019-09-05 Thu 12:30] At each node checked by a predicate, make a struct that stores attributes we can query for, as well as parent node position. This would let us speed up ancestor-based queries, like =(ancestor (todo "WAITING"))=. Ideally it would also serve as the tag hierarchy cache. It would probably be an all-encompassing system, because predicates would need to refer to the cached node when available. So maybe the struct should be like =ts-defstruct=, with lazy, caching accessors, which would move some of the predicates' code into the accessors. Maybe a good improvement to make later, after the project is more developed. [2019-10-07 Mon 13:08] This has basically been implemented in =be2bf6df316b96b3ed56851b8ffe0e227796b621=, but as functions and values rather than with structs. It remains to be seen how this works with =ancestor= queries, but I suspect it will help a lot. *** Struct PoC code This works okay (except the priority accessor needs to be fixed, because Org priorities are awkward to get). I'm guessing all the extra function calls would make it undesirable in cases of returning many results, but it's a flexible concept that makes sorting easy. #+BEGIN_SRC elisp (ts-defstruct org-ql-node file position marker (level nil :accessor-init (org-with-point-at (org-ql-node-marker struct) (org-outline-level))) (heading nil :accessor-init (org-with-point-at (org-ql-node-marker struct) ;; TODO: Org 9.2+ adds 2 more args to `org-get-heading'. (org-get-heading t t))) (priority nil :accessor-init (org-with-point-at (org-ql-node-marker struct) (org-get-priority ))) (tags nil :accessor-init (org-with-point-at (org-ql-node-marker struct) (->> (org-ql--tags-at (point)) -flatten (delq 'org-ql-nil)))) (todo nil :accessor-init (org-with-point-at (org-ql-node-marker struct) (org-get-todo-state))) (outline-path nil :accessor-init (org-with-point-at (org-ql-node-marker struct) (org-split-string (org-format-outline-path (org-get-outline-path) nil nil "") "")))) (defcustom helm-org-ql-sort '(org-ql-node-priority org-ql-node-todo) "FIXME" ) (cl-defun helm-org-ql (buffers-files &optional no-and) "Display results in BUFFERS-FILES for an `org-ql' query using Helm. Interactively, search the current buffer. NOTE: Atoms in the query are turned into strings where appropriate, which makes it unnecessary to type quotation marks around words that are intended to be searched for as indepenent strings. Also, unless NO-AND is non-nil (interactively, with prefix), all query tokens are wrapped in an implied (and) form. This is because a query must be a sexp, so when typing multiple clauses, either (and) or (or) would be required around them, and (and) is typically more useful, because it narrows down results. For example, this raw input: Emacs git Is transformed into this query: (and \"Emacs\" \"git\") However, quoted strings remain quoted, so this input: \"something else\" (tags \"funny\") Is transformed into this query: (and \"something else\" (tags \"funny\"))" (interactive (list (current-buffer) current-prefix-arg)) (let ((helm-input-idle-delay helm-org-ql-input-idle-delay)) (helm :sources (helm-build-sync-source "helm-org-ql-agenda-files" :candidates (lambda () (let* ((query (helm-org-ql--input-to-query helm-pattern no-and)) (window-width (window-width (helm-window)))) (when query (let ((results (org-ql-select buffers-files query :action '(make-org-ql-node :marker (point-marker))))) (when helm-org-ql-sort (dolist (sorter (reverse helm-org-ql-sort)) (setf results (sort results sorter)))) (cl-loop for it in-ref results do (setf it (concat (buffer-name (org-ql-node-file it)) ":" (or (org-ql-node-todo it) "") (or (org-ql-node-priority it) "") (org-ql-node-heading it) "\\" (org-ql-node-outline-path it)))) results)))) :match #'identity :fuzzy-match nil :multimatch nil :volatile t :action #'helm-org-goto-marker)))) #+END_SRC ** DONE [#B] Define predicates with a macro :PROPERTIES: :milestone: 0.6 :END: [2020-11-21 Sat 16:40] i.e. the macro defines the predicate, preamble, and normalizer in one form. WIP on the =wip/define-predicate= branch. Seems to be working well so far. [2020-11-22 Sun 17:17] Everything is converted, everything works, and all the tests pass. Worked out just as I hoped. Will merge for 0.6. *** DONE [#A] Merge new defpred into master *** DONE Test defining custom predicates in user config *** DONE Show how to define custom predicates Tutorial is published. *** DONE Run tests *** DONE Convert all predicates to new macro ** DONE [#B] Move this notes file into an orphan =meta/notes= branch [2020-11-12 Thu 03:17] Will probably have to merge or delete some WIP branches first, otherwise they'll probably get conflicts. [2020-11-22 Sun 19:40] Did this a day or two ago. Didn't rebase all the WIP branches, but they shouldn't be any trouble. ** DONE [#B] Quickly change sorting/grouping in search views With caching, the search doesn't need to be repeated, so resorting/regrouping can be very fast. ** DONE Byte-compile lambdas CLOSED: [2018-05-09 Wed 17:30] :LOGBOOK: - State "DONE" from [2018-05-09 Wed 17:30] :END: =elfeed-search--update-list= byte-compiles lambdas returned by =elfeed-search-compile-filter=. Maybe I could do something like this too. If I can get this working, I should profile it to see what difference it makes. *** Profiling Going to try byte-compiling the predicate function: #+BEGIN_SRC elisp (elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (and (or (date :date '= (org-today)) (date :deadline '<= (+ org-deadline-warning-days (org-today))) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda))))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------------------+--------------+--------------+--------------| | org-agenda-ng--agenda | 10 | 0.8370581039 | 0.0837058104 | | org-agenda-finalize-entries | 10 | 0.652886608 | 0.0652886608 | | org-super-agenda--filter-finalize-entries | 10 | 0.641794501 | 0.0641794501 | | org-super-agenda--group-items | 10 | 0.636057006 | 0.0636057006 | | org-super-agenda--group-dispatch | 130 | 0.631911849 | 0.0048608603 | | org-super-agenda--group-tag | 50 | 0.592883869 | 0.0118576773 | | list | 2720 | 0.5792795169 | 0.0002129704 | | mapcar | 331 | 0.2333591920 | 0.0007050126 | | org-agenda-ng--filter-buffer | 10 | 0.09247626 | 0.009247626 | | org-agenda-ng--format-element | 150 | 0.0649320479 | 0.0004328803 | | org-entry-get | 860 | 0.0408285349 | 4.747...e-05 | | org-agenda-ng--date-p | 910 | 0.0385646249 | 4.237...e-05 | | org-element-headline-parser | 150 | 0.0374417470 | 0.0002496116 | | org-is-habit-p | 270 | 0.0290107389 | 0.0001074471 | | org--property-local-values | 270 | 0.0268615979 | 9.948...e-05 | | org-get-property-block | 270 | 0.0244613309 | 9.059...e-05 | | org-get-tags-at | 150 | 0.017875864 | 0.0001191724 | | org-super-agenda--group-habit | 10 | 0.015910656 | 0.0015910655 | | mapc | 2540 | 0.0158616290 | 6.244...e-06 | | org-agenda-ng--add-faces | 150 | 0.0143329670 | 9.555...e-05 | Now the same thing without byte-compiling: #+BEGIN_SRC elisp (elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (and (or (date :date '= (org-today)) (date :deadline '<= (+ org-deadline-warning-days (org-today))) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda))))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------------------+--------------+--------------+--------------| | org-agenda-ng--agenda | 10 | 0.846645537 | 0.0846645537 | | org-agenda-finalize-entries | 10 | 0.662896805 | 0.0662896805 | | sort | 40 | 0.591123256 | 0.0147780814 | | org-entries-lessp | 400 | 0.5901201620 | 0.0014753004 | | mapcar | 201 | 0.2318270599 | 0.0011533684 | | org-agenda-ng--filter-buffer | 10 | 0.092519787 | 0.0092519787 | | org-super-agenda--filter-finalize-entries | 10 | 0.0664278040 | 0.0066427804 | | org-agenda-ng--format-element | 150 | 0.064658994 | 0.0004310599 | | org-super-agenda--group-items | 10 | 0.0602504089 | 0.0060250408 | | org-super-agenda--group-dispatch | 130 | 0.0561904470 | 0.0004322342 | | org-entry-get | 860 | 0.0437458889 | 5.086...e-05 | | org-agenda-ng--date-p | 910 | 0.0382623409 | 4.204...e-05 | | org-element-headline-parser | 150 | 0.0374662920 | 0.0002497752 | | org-is-habit-p | 270 | 0.0320861079 | 0.0001188374 | | org--property-local-values | 270 | 0.0298690430 | 0.0001106260 | | org-get-property-block | 270 | 0.0274716649 | 0.0001017469 | | org-super-agenda--group-habit | 10 | 0.019117901 | 0.0019117901 | | org-get-tags-at | 150 | 0.0178958930 | 0.0001193059 | | mapc | 2470 | 0.0150361130 | 6.087...e-06 | | org-agenda-ng--add-faces | 150 | 0.0143092169 | 9.539...e-05 | Virtually indistinguishable. Going to try moving the =byte-compile= call from the =org-agenda-ng= macro to other places... #+BEGIN_SRC elisp (elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (and (or (date :date '= (org-today)) (date :deadline '<= (+ org-deadline-warning-days (org-today))) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda))))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------------------+--------------+--------------+--------------| | org-agenda-ng--agenda | 10 | 0.8476316779 | 0.0847631678 | | mapcar | 331 | 0.8159452220 | 0.0024650913 | | org-agenda-ng--filter-buffer | 10 | 0.674217912 | 0.0674217912 | | org-element-headline-parser | 150 | 0.6171195889 | 0.0041141305 | | line-beginning-position | 620 | 0.5802579680 | 0.0009358999 | | org-agenda-finalize-entries | 10 | 0.082065157 | 0.0082065157 | | org-super-agenda--filter-finalize-entries | 10 | 0.0708772279 | 0.0070877227 | | org-super-agenda--group-items | 10 | 0.065523103 | 0.0065523103 | | org-agenda-ng--format-element | 150 | 0.0652783740 | 0.0004351891 | | org-super-agenda--group-dispatch | 130 | 0.0614253589 | 0.0004725027 | | org-entry-get | 860 | 0.0494023029 | 5.744...e-05 | | org-agenda-ng--date-p | 910 | 0.0388435519 | 4.268...e-05 | | org-is-habit-p | 270 | 0.0375687549 | 0.0001391435 | | org--property-local-values | 270 | 0.0353892929 | 0.0001310714 | | org-get-property-block | 270 | 0.0329700440 | 0.0001221112 | | org-super-agenda--group-habit | 10 | 0.024468601 | 0.0024468601 | | re-search-backward | 1500 | 0.0186344089 | 1.242...e-05 | | org-get-tags-at | 150 | 0.0180038809 | 0.0001200258 | | mapc | 2540 | 0.0156518099 | 6.162...e-06 | | org-agenda-ng--add-faces | 150 | 0.0144141080 | 9.609...e-05 | Doesn't seem to make any difference. ** DONE Document/figure out tag inheritance I think it should probably be enabled in most cases, to avoid missing results that users would expect to find, but it will reduce performance in some cases, so users should be able to turn it off when they don't need it. [2018-06-12 Tue 14:32] The docstring for ~org-map-entries~ says: #+BEGIN_QUOTE If your function needs to retrieve the tags including inherited tags at the *current* entry, you can use the value of the variable ‘org-scanner-tags’ which will be much faster than getting the value with ‘org-get-tags-at’. If your function gets properties with ‘org-entry-properties’ at the *current* entry, bind ‘org-trust-scanner-tags’ to t around the call to ‘org-entry-properties’ to get the same speedup. Note that if your function moves around to retrieve tags and properties at a *different* entry, you cannot use these techniques. #+END_QUOTE [2019-09-26 Thu 21:31] Handled with the tag caching recently implemented. ** DONE [#B] Dual matching with regexp and predicates :PROPERTIES: :ID: 39972bb5-fdd0-4754-93ba-c85796a67ccf :END: /Note: This is underway in the =preamble-re= branch./ Searching and matching could be sped up by constructing a regexp that searches directly to the next possible match, and then matching with predicate functions. For example, a search like: #+BEGIN_SRC elisp (org-ql (org-agenda-files) (and (regexp "lisp") (scheduled < today))) #+END_SRC Only entries that contain the word =lisp= can be matches, and searching each entry for that word is wasteful. Instead, we could search the buffer for the next occurrence of =lisp=, then check the scheduled date for that entry. This would require processing the predicate to pull out matchers that can be done as buffer-wide regexps, e.g. =regexp=, =heading-regexp=, =todo=, and possibly =tags=. Org has some regexp-building functions that might make this fairly easy, and then we could probably use ~rx~ to make an optimized version of the regexp. It would also require some refactoring to the searching that would go directly to regexp matches when possible, rather than checking every entry with the predicate. [2019-07-16 Tue 11:14] Made new branch =preamble-re-new= based on current =master=. Seems to work well. Here's some code for testing and comparing performance (~bench-multi-lets~ is from [[https://github.com/alphapapa/emacs-package-dev-handbook#bench-multi-lets][here]]). [2019-07-16 Tue 11:56] Going to merge to =master= as 0.2, so marking this as done, even though there's a bit more that can be done from here. *** Benchmark code #+BEGIN_SRC elisp (cl-defmacro org-ql-preamble-bench (&key query (file "tests/data.org") (times 10)) `(bench-multi-lets :times ,times :ensure-equal t :lets (("preamble" ((org-ql-use-preamble t))) ("no preamble" ((org-ql-use-preamble nil)))) :forms ((,(prin1-to-string query) (org-ql-select,file ',query :action (lambda () (org-get-heading t t))))))) #+END_SRC #+BEGIN_SRC elisp (org-ql-preamble-bench :query (regexp "Emacs") :times 100) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------------------+--------------------+---------------+----------+------------------| | preamble: (regexp "Emacs") | 1.22 | 0.141767 | 0 | 0 | | no preamble: (regexp "Emacs") | slowest | 0.172398 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :file "~/org/inbox.org" :query (regexp "Emacs") :times 5) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------------------+--------------------+---------------+----------+------------------| | preamble: (regexp "Emacs") | 1.59 | 2.011043 | 0 | 0 | | no preamble: (regexp "Emacs") | slowest | 3.206370 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :file "~/org/inbox.org" :query (and (regexp "Emacs") (todo)) :times 5) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------------+--------------------+---------------+----------+------------------| | preamble: (and (regexp "Emacs") (todo)) | 1.59 | 2.211503 | 0 | 0 | | no preamble: (and (regexp "Emacs") (todo)) | slowest | 3.512741 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :file "~/org/inbox.org" :query (and (regexp "Emacs") (todo) (scheduled)) :times 5) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------------------------+--------------------+---------------+----------+------------------| | preamble: (and (regexp "Emacs") (todo) (scheduled)) | 1.69 | 2.042456 | 0 | 0 | | no preamble: (and (regexp "Emacs") (todo) (scheduled)) | slowest | 3.453756 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :file "~/org/inbox.org" :query (todo "WAITING") :times 2) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------------------+--------------------+---------------+----------+------------------| | preamble: (todo "WAITING") | 15.60 | 0.070684 | 0 | 0 | | no preamble: (todo "WAITING") | slowest | 1.102722 | 0 | 0 | Wow, that's a huge improvement! ** DONE Operate on list of heading positions CLOSED: [2018-05-10 Thu 15:02] :LOGBOOK: - State "DONE" from [2018-05-10 Thu 15:02] :END: [2017-12-31 Sun 17:54] I wonder if, instead of parsing the whole buffer with =org-element-parse-buffer=, we could simply work on a list of heading positions, e.g. a loop would search forward to the next heading position, then call whatever predicates it needed at the heading's position, using =save-excursion= around each function call. The predicates would need to be updated to get their data from the buffer, instead of using =org-element-property=, but that wouldn't be hard. [2018-05-10 Thu 15:01] I already changed to using buffer-parsing predicates instead of =org-element-parse-buffer=. ** DONE Use macros for =date= CLOSED: [2018-05-10 Thu 14:59] :LOGBOOK: - State "DONE" from [2018-05-10 Thu 14:59] :END: If I made the =date= selector a macro, I could avoid the need to quote the comparator. Also, maybe instead of having a single =date= selector, I should have =scheduled=, =deadline=, etc. * Checklists ** Commits :PROPERTIES: :ID: d8d7b88e-5737-437e-af76-2253f8340de3 :END: To complete before and after pushing any commit. *** Additions When committing an additional feature: + [ ] Make WIP branch + [ ] Check repo for closing issues + [ ] Check magit-todos list of items branched from master + [ ] Lint + [ ] Test - [ ] Locally - [ ] On different Emacs versions with GitHub CI (very important) + [ ] Update docs + [ ] Update changelog - [ ] Mention closing issues (optionally) + [ ] Commit - [ ] Mention closing issues in commit message + [ ] Merge to master + [ ] Push master + [ ] Close related tasks in this file + [ ] Delete WIP branch *** Fixes When committing a fix: + [ ] Check repo for closing issues + [ ] Check magit-todos list of items branched from master + [ ] Lint + [ ] Test - [ ] Locally - [ ] On different Emacs versions with GitHub CI (very important) + [ ] Update changelog - [ ] Mention changes - [ ] Mention closing issues + [ ] Commit - [ ] Mention closing issues in commit message + [ ] Push + [ ] Close related tasks in this file ** Release template + [ ] Make WIP branch + [ ] =Meta= pre-release commit - [ ] Update version numbers + [ ] =org-ql.el= + [ ] =helm-org-ql.el= + [ ] =README.org= + [ ] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [ ] Changelog entry + [ ] Merge to stable branch - [ ] Non-fast-forward merge WIP branch into stable branch - [ ] Tag and sign merge commit + [ ] Push stable branch + [ ] Push tags + [ ] Merge to master or make stable branch + [ ] Push master/stable + [ ] Delete WIP branch + [ ] Make GitHub release + [ ] Announce - [ ] Reddit - [ ] org-mode mailing list ** Archive *** DONE Release 0.4.9 + [X] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [X] Changelog entry + [X] Update version numbers - [X] =org-ql.el= - [X] =helm-org-ql.el= (N/A) - [X] =README.org= + [X] Tag and sign + [X] Push + [X] Merge to master *** DONE Release 0.5 + [X] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [ ] Changelog entry + [X] Update version numbers - [X] =org-ql.el= - [X] =helm-org-ql.el= (N/A yet) - [X] =README.org= + [X] Tag and sign + [X] Push + [X] Merge to master or make stable branch + [X] Push master/stable *** DONE Release 0.5.1 + [X] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [X] Changelog entry + [X] Update version numbers - [X] =org-ql.el= - [X] =helm-org-ql.el= - [X] =README.org= + [X] Tag and sign + [X] Push + [X] Merge to master or make stable branch + [X] Push master/stable *** DONE Release 0.5.2 :PROPERTIES: :ID: 5b9fd52c-dae2-473c-8f00-0fd840327b75 :END: + [X] Make WIP branch + [X] =Meta= pre-release commit - [X] Update version numbers + [X] =org-ql.el= + [X] =helm-org-ql.el= + [X] =README.org= + [X] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [X] Changelog entry + [X] Merge to stable branch - [X] Non-fast-forward merge WIP branch into stable branch - [X] Tag and sign merge commit + [X] Push stable branch + [X] Merge to master or make stable branch + [X] Push master/stable + [X] Delete WIP branch *** DONE Release: 0.6 :PROPERTIES: :ID: db2ed038-b8b4-45c1-8b12-d450a95d69b1 :END: + [X] Make WIP branch + [X] =Meta= pre-release commit - [X] Update version numbers + [X] =org-ql.el= + [X] =helm-org-ql.el= + [X] =README.org= + [X] Complete [[id:d8d7b88e-5737-437e-af76-2253f8340de3][commit checklist]] + [X] Changelog entry + [ ] Merge to stable branch - [ ] Non-fast-forward merge WIP branch into stable branch - [ ] Tag and sign merge commit + [X] Push stable branch + [X] Push tags + [X] Merge to master or make stable branch + [X] Push master/stable + [X] Delete WIP branch + [X] Make GitHub release + [X] Announce - [X] Reddit - [X] org-mode mailing list * Examples / testing :PROPERTIES: :TOC: :include descendants :depth 1 :END: :CONTENTS: - [[#property-matching][Property matching]] - [[#regexp-matching][Regexp matching]] - [[#screenshot-code][Screenshot code]] - [[#sorting][Sorting]] :END: #+BEGIN_SRC elisp (org-agenda-ng org-agenda-files (and (or (date :deadline '<= (org-today)) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda))) ((group (tags "bills")) (group (todo "SOMEDAY")))) (org-agenda-ng org-agenda-files (and (or (date :deadline '<= (org-today)) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda)))) (org-agenda-ng "~/org/main.org" (and (or (date :deadline '<= (org-today)) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda)))) (org-ql org-agenda-files (and (todo "SOMEDAY") (tags "Emacs"))) (org-ql org-agenda-files (and (todo "SOMEDAY") (tags "Emacs") (priority >= "B"))) (org-ql "~/org/main.org" (and (or (tags "Emacs") (priority >= "B")) (not (done)))) (org-ql "~/org/main.org" (and (or (tags "Emacs") (priority >= "B")) (done))) #+END_SRC ** Property matching #+BEGIN_SRC elisp (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (property "agenda-group")) (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (property "agenda-group" "plans")) #+END_SRC ** Regexp matching #+BEGIN_SRC elisp (org-ql "~/src/emacs/org-super-agenda/test/test.org" (regexp "over")) (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" (regexp "over")) #+END_SRC ** Screenshot code #+BEGIN_SRC elisp (org-super-agenda--test-with-org-today-date "2017-07-08 00:00" (org-ql "~/src/emacs/org-super-agenda/test/test.org" (and (or (date = today) (deadline <=) (scheduled <= today)) (not (done))))) #+END_SRC ** Sorting #+BEGIN_SRC elisp (org-ql "~/src/emacs/org-super-agenda/test/test.org" (regexp "over") :sort (priority deadline scheduled)) (org-ql "~/src/emacs/org-super-agenda/test/test.org" (regexp "over") :sort (date)) (org-ql "~/src/emacs/org-super-agenda/test/test.org" (todo) :sort (todo)) #+END_SRC * In the wild :PROPERTIES: :TOC: :include descendants :depth 1 :END: :CONTENTS: - [[#alois-janicek][Alois Janicek]] - [[#andrea-giugliano][Andrea Giugliano]] - [[#benson-chu][Benson Chu]] - [[#kevin-brubeck-unhammer][Kevin Brubeck Unhammer]] - [[#mikhail-skorzhinskiy][Mikhail Skorzhinskiy]] - [[#trey-peacock][Trey Peacock]] :END: ** [[https://github.com/AloisJanicek/.doom.d-2nd][Alois Janicek]] *** [[https://github.com/AloisJanicek/.doom.d-2nd/commit/41ed1080f6f90463fc1f1d7e47cef9864756867c][further tweaking org-ql views · AloisJanicek/.doom.d-2nd@41ed108 · GitHub]] #+BEGIN_SRC elisp :eval never ;; Hydras (defhydra gtd-agenda (:color blue :body-pre (if (aj/has-heading-p +INBOX) (org-ql-search `(,+INBOX) "*" :sort '(date)) (org-ql-search (org-agenda-files) '(todo "NEXT") :sort '(date priority todo) :groups '((:auto-category t)))) ) "agenda" ("a" (org-ql-agenda (org-agenda-files) (and (or (ts-active :on today) (deadline auto) (scheduled :to today)) (not (done)))) "agenda") ("t" (org-ql-search (org-agenda-files) '(todo "TODO") :sort '(date priority todo) :groups '((:auto-category t))) "todo") ("n" (org-ql-search (org-agenda-files) '(todo "NEXT") :sort '(date priority todo) :groups '((:auto-category t))) "next") ("h" (org-ql-search (org-agenda-files) '(todo "HOLD") :sort '(date priority todo) :groups '((:auto-category t))) "hold") ("s" (org-ql-search (org-agenda-files) '(tags "someday") :sort '(date priority todo) :groups '((:auto-category t))) "someday") ("r" (org-ql-search (org-agenda-files) '(ts :from -7 :to today) :sort '(date priority todo) :groups '((:auto-ts t))) "recent") ("i" (org-ql-search `(,+INBOX) "*" :sort '(date)) "inbox") ) #+END_SRC ** [[https://ag91.github.io/][Andrea Giugliano]] *** [[https://ag91.github.io/blog/2020/09/27/org-agenda-and-your-future-or-how-to-keep-score-of-your-long-term-goals-with-org-mode/][Org Agenda and Your Future, or how to keep score of your long term goals with Org Mode - Where parallels cross]] ** [[https://github.com/pestctrl/emacs-config][Benson Chu]] *** [[https://github.com/pestctrl/emacs-config/commit/fa3068003373a0c93e23c728b5dbad2d7c11e2e1][Experimental agenda view with org-ql · pestctrl/emacs-config@fa30680 · GitHub]] #+BEGIN_SRC elisp :eval never ("f" "fastdev?" ((org-ql-block '(tags "refile") ((org-agenda-overriding-header "Refile tasks"))) (tags-todo ,(concat dev-tag "/!" (mapconcat #'identity my/active-projects-and-tasks "|")) ((org-agenda-overriding-header "Stuck Projects") (org-agenda-skip-function 'my/dev-show-stuck-projects) (org-tags-match-list-sublevels 'indented) (org-agenda-sorting-strategy '((agenda category-keep))))) (tags-todo ,(concat dev-tag "-short" "/!" (mapconcat #'identity my/active-projects-and-tasks "|")) ((org-agenda-overriding-header "Active Projects") (org-agenda-skip-function 'my/dev-show-active-projects) (org-tags-match-list-sublevels 'indented) (org-agenda-sorting-strategy '((agenda category-keep))))) (org-ql-block '(and (tags "dev") (todo "WAIT")) ((org-agenda-overriding-header "Waiting tasks"))) (org-ql-block '(and (tags "dev") (todo "NEXT")) ((org-agenda-overriding-header "Things to do"))) (agenda "" ((org-agenda-skip-function 'my/agenda-custom-skip) (org-agenda-span 'day) (org-agenda-tag-filter-preset (quote (,dev-tag))) (org-agenda-skip-deadline-if-done t) (org-agenda-skip-scheduled-if-done t) (org-super-agenda-groups '((:name "Overdue" :and (:deadline past :log nil)) (:name "Upcoming" :deadline future) (:name "Should do" :and (:scheduled past :log nil)) (:name "Today" :time-grid t :and (:not (:and (:not (:scheduled today) :not (:deadline today))))))))))) #+END_SRC ** [[https://github.com/unhammer/org-upcoming-modeline][Kevin Brubeck Unhammer]] *** [[https://github.com/unhammer/org-upcoming-modeline][org-upcoming-modeline: put upcoming org event in modeline]] + [[https://www.reddit.com/r/emacs/comments/k1e4eu/show_next_orgappointment_in_modeline/][Show next (org-)appointment in modeline? : r/emacs]] ** [[https://gist.github.com/mskorzhinskiy/fac0662a39a7b1bf5b306f175943bc97][Mikhail Skorzhinskiy]] [2020-12-18 Fri 03:41] Shows a number of complex custom queries and grouping, including his "dashboard" view. Shared at [[https://www.reddit.com/r/emacs/comments/kewl3a/how_are_folks_using_orgagenda_and_orgroam_together/gg8j251/][How are folks using org-agenda and org-roam TOGETHER? : emacs]]. #+BEGIN_SRC elisp (setq org-ql-views '(("stuck" lambda nil (interactive) (org-ql-search (org-agenda-files) '(and (tags "story") (not (tags "ignore")) (not (done)) ;; Finished stories should be excluded (not (descendants (todo "NEXT"))) ;; If there are already ;; something in progress ;; it will shown (and (not (descendants (done))) ;; There are not scheduled not finished items (not (descendants (scheduled))))) :narrow nil :super-groups '((:name "Waiting" :order 8 :todo "WAIT") (:name "Important" :order 1 :deadline t :priority>= "B") (:name "Work" :order 2 :tag "work") (:name "Study" :order 2 :tag "study") (:name "Stucked" :order 3 :tag "story")) :title "stuck-projects")) ("reports" lambda nil (interactive) (org-ql-search (org-agenda-files) '(and (or (tags-local "weekly") (tags-local "monthly")) (not (tags "ignore"))) :narrow nil :super-groups '((:name "Weekly reports" :tag "weekly") (:name "Monthly reports" :tag "monthly")) :title "Introspection reports")) ("next" lambda nil (interactive) (org-ql-search (org-agenda-files) '(and (or (tags-local "refile") (todo "PROG") (todo "WAIT") (todo "NEXT")) (not (tags "ignore")) (not (property "linked")) (not (done))) :sort '(date) :narrow nil :super-groups `((:name "In progress" :order 1 :todo "PROG") (:name "Daily" :order 2 :regexp ,org-repeat-re) (:name "Waiting" :order 3 :todo "WAIT") (:name "Refile" :order 3 :tag "refile") (:name "Important" :order 3 :priority>= "B") (:auto-tags t :order 5)) :title "Next actions")) ("calendar" lambda nil (interactive) (org-ql-search (org-agenda-files) `(and (ts-active) (regexp ,org-scheduled-time-hour-regexp) (not (done))) :sort '(date) :narrow nil :super-groups '((:auto-planning t)) :title "Calendar")) ("dashboard" lambda nil (interactive) (org-ql-search (org-agenda-files) '(and (or (ts-active :to today) (deadline auto) (todo "PROG") (and (tags "journal") (not (tags "weekly")) (not (tags "monthly")) (not (tags "yearly")) (todo))) (not (todo "WAIT")) (not (tags "ignore")) (not (property "linked")) (not (done))) :sort '(date) :narrow nil :super-groups `((:name "In progress" :order 1 :tag "monthly" :tag "weekly" :todo "PROG") (:name "Agenda" :order 2 :deadline t :regexp ,org-scheduled-time-hour-regexp) (:name "Daily" :order 2 :and (:todo nil :regexp ,org-repeat-re)) (:name "Today" :order 3 :tag "journal") (:name "Important" :order 3 :priority>= "B") (:auto-tags t :order 5)) :title "Dashboard")))) #+END_SRC ** [[https://github.com/tpeacock19/org-ql-config][Trey Peacock]] [2021-07-08 Thu 23:29] #+BEGIN_QUOTE A recent post on reddit asked the question, Why does the recent zettelkasten craze use one file per note rather than one headline per note? Naturally, it brought many differing perspectives and approaches to org-mode usage. I wanted to show my own configuration that largely leverages alphapapa’s wonderful package org-ql. Along with some built-in functionality of org-mode I’m able to search headlines across all of my org files and visit them in indirect buffers automatically. Additionally, you can search backlinks with similar functionality. This is not meant to be a standalone package by any means, but simply an example of what can be achieved using org-ql. My personal emacs configuration is named baal, so you can safely ignore or rename any references to it. #+END_QUOTE *** TODO Add to examples in docs Should probably ask him first. * Profiling :PROPERTIES: :TOC: :include descendants :depth 1 :END: :CONTENTS: - [[#2019-08-29-thu-0624--benchmarking-org-ql-compared-to-re-search-forward-for-getting-headings-in-buffer][{2019-08-29 Thu 06:24} Benchmarking org-ql compared to re-search-forward for getting headings in buffer]] - [[#caching-of-inherited-tags][Caching of inherited tags]] - [[#intersecting-query-results][Intersecting query results]] - [[#more-profiling][More profiling]] - [[#preambles][Preambles]] - [[#profiling-flet-across-all-agenda-files][Profiling flet across all agenda files]] - [[#profiling-flet-on-a-single-file][Profiling flet on a single file]] - [[#profiling-org-trust-scanner-tags][Profiling org-trust-scanner-tags]] - [[#profiling-position-based][Profiling position-based]] - [[#profiling-tags-matching][Profiling tags matching]] - [[#using-org-element-parse-buffer][Using org-element-parse-buffer]] - [[#withwithout-tsel][with/without ts.el]] :END: ** [2019-08-29 Thu 06:24] Benchmarking org-ql compared to re-search-forward for getting headings in buffer :PROPERTIES: :ID: fcc09229-ed24-42eb-a1fd-31d8f7d4c8d5 :END: Minimal difference, and that's a very large file, too. On smaller files it's thousandths of a second. #+BEGIN_SRC elisp (with-current-buffer (find-buffer-visiting "~/org/inbox.org") (bench-multi-lexical :times 1 :ensure-equal t :forms (("org-ql" (org-ql-select (current-buffer) '(level 1) :action '(progn (font-lock-ensure (line-beginning-position) (line-end-position)) (cons (org-get-heading t) (point))))) ("re-search-forward" (org-with-wide-buffer (goto-char (point-min)) (when (org-before-first-heading-p) (outline-next-heading)) (cl-loop while (re-search-forward (rx bol "*" (1+ blank)) nil t) do (font-lock-ensure (line-beginning-position) (line-end-position)) collect (cons (org-get-heading t) (match-beginning 0)))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------+--------------------+---------------+----------+------------------| | re-search-forward | 1.17 | 0.520375 | 0 | 0 | | org-ql | slowest | 0.608281 | 0 | 0 | ** Caching of inherited tags [2019-09-05 Thu 07:59] Implemented a per-buffer tags cache that seems to significantly speed up tags queries that use tag inheritance. It persists as long as the buffer remains unmodified, and it's usable from any code as a single function that automatically uses caching. It also returns inherited tags and local tags separately, which could be useful for having separate selectors, one for inherited tags, one for local tags, and one for both. #+BEGIN_SRC elisp (defvar org-ql-tags-cache (ht) "Per-buffer tags cache. Keyed by buffer. Each value is a cons of the buffer's modified tick, and another hash table keyed on buffer position, whose values are a list of two lists, inherited tags and local tags, as strings.") (defun org-ql--tags-at (position) "Return tags for POSITION in current buffer. Returns cons (INHERITED-TAGS . LOCAL-TAGS)." ;; I'd like to use `-if-let*', but it doesn't leave non-nil variables ;; bound in the else clause, so destructured variables that are non-nil, ;; like found caches, are not available in the else clause. (if-let* ((buffer-cache (gethash (current-buffer) org-ql-tags-cache)) (modified-tick (car buffer-cache)) (tags-cache (cdr buffer-cache)) (buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (cached-result (gethash position tags-cache))) ;; Found in cache: return them. (pcase cached-result ('org-ql-nil nil) (_ cached-result)) ;; Not found in cache: get tags and cache them. (let* ((local-tags (or (org-get-tags position 'local) 'org-ql-nil)) (inherited-tags (or (save-excursion (when (org-up-heading-safe) (-let* (((inherited local) (org-ql--tags-at (point))) (inherited-tags (when (or inherited local) (cond ((and (listp inherited) (listp local)) (append inherited local)) ((cond ((listp inherited) inherited) ((listp local) local))))))) (when inherited-tags (->> inherited-tags -non-nil -uniq))))) 'org-ql-nil)) (all-tags (list inherited-tags local-tags))) ;; Check caches again, because they may have been set now. ;; TODO: Is there a clever way we could avoid doing this, or is it inherently necessary? (setf buffer-cache (gethash (current-buffer) org-ql-tags-cache) modified-tick (car buffer-cache) tags-cache (cdr buffer-cache) buffer-unmodified-p (eq (buffer-modified-tick) modified-tick)) (cond ((or (not buffer-cache) (not buffer-unmodified-p)) ;; Buffer-local tags cache empty or invalid: make new one. (puthash (current-buffer) (cons (buffer-modified-tick) (let ((table (make-hash-table))) (puthash position all-tags table) table)) org-ql-tags-cache) ;; Return tags, not the cons put on the buffer-cache. all-tags) ;; Buffer-local tags cache found, but no result: store this one. (t (puthash position all-tags tags-cache)))))) (org-ql--defpred tags-cached (&rest tags) "Return non-nil if current heading has one or more of TAGS (a list of strings)." ;; TODO: Try to use `org-make-tags-matcher' to improve performance. It would be nice to not have ;; to run `org-get-tags' for every heading, especially with inheritance. (cl-macrolet ((tags-p (tags) `(and ,tags (not (eq 'org-ql-nil ,tags))))) (-let* (((inherited local) (org-ql--tags-at (point)))) (cl-typecase tags (null (or (tags-p inherited) (tags-p local))) (otherwise (or (when (tags-p inherited) (seq-intersection tags inherited)) (when (tags-p local) (seq-intersection tags local)))))))) #+END_SRC Benchmark results: #+BEGIN_SRC elisp (let* ((buffers '("~/org/main.org")) (tags '("Emacs"))) (bench-multi-lexical :times 1 :ensure-equal t :forms (("uncached" (let ((org-ql-cache (ht))) (org-ql-select buffers `(tags ,@tags)))) ("cached" (let ((org-ql-cache (ht)) (org-ql-tags-cache (ht))) (org-ql-select buffers `(tags-cached ,@tags))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | cached | 6.51 | 0.519871 | 0 | 0 | | uncached | slowest | 3.386679 | 0 | 0 | ** Intersecting query results An idea that /might/ be helpful for performance in /some/ cases, depending on the query, the data, and whether the query has a preamble. But it looks like it would very rarely be helpful. #+BEGIN_SRC elisp (cl-defun org-ql-agenda-intersection (buffers-files queries &key entries sort buffer narrow super-groups) "Like `org-ql-agenda', but intersects multiple queries." (declare (indent defun)) (let* ((org-ql-cache (ht)) (entries (->> queries (--map (org-ql-select buffers-files it :action 'element-with-markers :narrow narrow :sort sort)) (-reduce #'-intersection)))) (org-ql-agenda--agenda buffers-files queries :entries entries :super-groups super-groups))) (bench-multi-lexical :times 1 :forms (("intersection" (let ((org-use-tag-inheritance nil)) (org-ql-agenda-intersection (org-agenda-files) '((todo "TODO") (tags "Emacs")) :sort '(priority deadline) :super-groups org-super-agenda-groups))) ("normal" (let ((org-use-tag-inheritance nil)) (org-ql-agenda (org-agenda-files) (and (todo "TODO") (tags "Emacs")) :sort (priority deadline) :super-groups org-super-agenda-groups))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------+--------------------+---------------+----------+------------------| | normal | 3.70 | 0.233147 | 0 | 0 | | intersection | slowest | 0.862512 | 0 | 0 | *** Alternative approach [2019-09-01 Sun 08:17] This is very experimental, but the results are surprising. When the action function returns a fairly simple list, the intersection is very slightly faster. When returning full elements, the intersection is much slower, so that it more than doubles the runtime. I wonder if the element list comparison is short-circuiting, or if it looks at the whole lists, because it seems like it shouldn't take more than 4-5 list elements before it realizes that two lists don't match. Anyway, looks like this approach isn't viable, at least not without a much more complicated implementation, which probably wouldn't be worth it. #+BEGIN_SRC elisp (let* ((action-fn (lambda () (list (current-buffer) (point) (substring-no-properties (org-get-heading t t))))) (files '("~/org/main.org"))) ;; NOTE: Careful to use the same files and action in each one. I duplicated ;; the variable in each form to make individual testing easier. (bench-multi-lexical :times 1 :ensure-equal t :forms (("normal" (->> (let ((org-ql-cache (ht)) (action-fn (lambda () (list (current-buffer) (point) (substring-no-properties (org-get-heading t t))))) (files '("~/org/main.org"))) (org-ql-select files '(and (not (done)) (or (habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) :action action-fn)))) ("Testing" (let* ((org-ql-cache (ht)) (files '("~/org/main.org")) (action-fn (lambda () (list (current-buffer) (point) (substring-no-properties (org-get-heading t t))))) (and-queries '(not (done))) (or-queries '((habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) (and-results (org-ql-select files and-queries :action action-fn)) (or-results (cl-loop for query in or-queries append (org-ql-select files query :action action-fn)))) (seq-intersection and-results (->> or-results -uniq))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------+--------------------+---------------+----------+------------------| | Testing | 1.15 | 0.248376 | 0 | 0 | | normal | slowest | 0.284897 | 0 | 0 | #+BEGIN_SRC elisp ;; With caching enabled (let* ((action-fn (lambda () (list (current-buffer) (point) (substring-no-properties (org-get-heading t t))))) (files '("~/org/main.org"))) (bench-multi-lexical :times 1 :ensure-equal t :forms (("normal" (->> (org-ql-select files '(and (not (done)) (or (habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) :action action-fn))) ("Testing" (let* ((files '("~/org/main.org")) (and-queries '(not (done))) (or-queries '((habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) (and-results (org-ql-select files and-queries :action action-fn)) (or-results (cl-loop for query in or-queries append (org-ql-select files query :action action-fn)))) (seq-intersection and-results (->> or-results -uniq))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------+--------------------+---------------+----------+------------------| | normal | 13.72 | 0.002311 | 0 | 0 | | Testing | slowest | 0.031707 | 0 | 0 | Using full views: #+BEGIN_SRC elisp (let* ((action-fn (lambda () (list (current-buffer) (point) (substring-no-properties (org-get-heading t t))))) (files '("~/org/main.org"))) (bench-multi-lexical :times 1 :forms (("normal" (->> (let ((org-ql-cache (ht)) (files '("~/org/main.org"))) (org-ql-search files '(and (not (done)) (or (habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))))))) ("Testing" (let* ((org-ql-cache (ht)) (files '("~/org/main.org")) (and-queries '(not (done))) (or-queries '((habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) (and-results (org-ql-select files and-queries :action 'element-with-markers)) (or-results (cl-loop for query in or-queries append (org-ql-select files query :action 'element-with-markers))) (final-results (seq-intersection and-results (->> or-results -uniq)))) (org-ql-agenda--agenda nil nil :entries final-results) ))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------+--------------------+---------------+----------+------------------| | normal | 1.74 | 0.534742 | 0 | 0 | | Testing | slowest | 0.931897 | 0 | 0 | Just gathering results, but using elements: #+BEGIN_SRC elisp (let* ((action-fn 'element-with-markers) (files '("~/org/main.org"))) ;; NOTE: Careful to use the same files and action in each one. I duplicated ;; the variable in each form to make individual testing easier. (bench-multi-lexical :times 1 :ensure-equal t :forms (("normal" (->> (let ((org-ql-cache (ht))) (org-ql-select files '(and (not (done)) (or (habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) :action action-fn)))) ("Testing" (let* ((org-ql-cache (ht)) (and-queries '(not (done))) (or-queries '((habit) (deadline auto) (scheduled :to today) (ts-active :on today) (closed :on today))) (and-results (org-ql-select files and-queries :action action-fn)) (or-results (cl-loop for query in or-queries append (org-ql-select files query :action action-fn)))) (seq-intersection and-results (->> or-results -uniq))))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------+--------------------+---------------+----------+------------------| | normal | 2.27 | 0.314218 | 0 | 0 | | Testing | slowest | 0.714587 | 0 | 0 | ** More profiling [2018-05-10 Thu 15:02] I think these are decent improvements. #+BEGIN_SRC elisp (elp-profile 1 nil (org-agenda-ng "~/org/main.org" (or (habit) (and (or (date '= (org-today)) (deadline '<=) (scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda))) (and (todo "DONE" "CANCELLED") (closed '= (org-today)))))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------+--------------+--------------+--------------| | mapcar | 164 | 1.5004585290 | 0.0091491373 | | org-agenda-ng--agenda | 1 | 1.348231247 | 1.348231247 | | org-agenda-ng--filter-buffer | 1 | 1.1391189879 | 1.1391189879 | | org-agenda-ng--date-plain-p | 1267 | 0.6198571040 | 0.0004892321 | | org-entry-get | 3983 | 0.2979337370 | 7.480...e-05 | | org-is-habit-p | 1365 | 0.2049101109 | 0.0001501172 | | org--property-local-values | 1365 | 0.1940614150 | 0.0001421695 | | org-agenda-ng--habit-p | 1272 | 0.1911009179 | 0.0001502365 | | org-agenda-ng--format-element | 52 | 0.177965411 | 0.0034224117 | | org-get-property-block | 1365 | 0.1760004519 | 0.0001289380 | | org-get-tags-at | 52 | 0.1362824969 | 0.0026208172 | | org-agenda-ng--date-p | 3880 | 0.1351176629 | 3.482...e-05 | | org-up-heading-safe | 226 | 0.1276747609 | 0.0005649325 | | re-search-backward | 2028 | 0.1144211070 | 5.642...e-05 | | org-entry-properties | 2618 | 0.0848660999 | 3.241...e-05 | | org-agenda-ng--todo-p | 1319 | 0.081952653 | 6.213...e-05 | | org-get-todo-state | 1319 | 0.0796836810 | 6.041...e-05 | | re-search-forward | 3754 | 0.0739803739 | 1.970...e-05 | | org-inlinetask-in-task-p | 1365 | 0.0657829330 | 4.819...e-05 | | org-agenda-ng--scheduled-p | 1247 | 0.0619497850 | 4.967...e-05 | ** Preambles Not sure if clearing the cache is necessary here, because it seemed to make nearly no difference in the results, but I don't know why. #+BEGIN_SRC elisp :results silent (cl-defmacro org-ql-preamble-bench (&key query (file "tests/data.org") (times 10)) `(bench-multi-lets :times ,times :ensure-equal t :lets (("preamble" ((org-ql-use-preamble t) (org-ql-cache (ht)))) ("no preamble" ((org-ql-use-preamble nil) (org-ql-cache (ht))))) :forms ((,(prin1-to-string query) (org-ql-select ,file ',query :action '(org-get-heading t t)))))) #+END_SRC *** =closed= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (closed)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-----------------------+--------------------+---------------+----------+------------------| | preamble: (closed) | 4.80 | 0.086553 | 0 | 0 | | no preamble: (closed) | slowest | 0.415165 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (closed <= "2019-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------------------------------------+--------------------+---------------+----------+------------------| | preamble: (closed <= "2019-01-01") | 4.21 | 0.105782 | 0 | 0 | | no preamble: (closed <= "2019-01-01") | slowest | 0.445374 | 0 | 0 | *** =deadline= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (deadline)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------------+--------------------+---------------+----------+------------------| | preamble: (deadline) | 27.63 | 0.014656 | 0 | 0 | | no preamble: (deadline) | slowest | 0.404952 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (deadline <= "2019-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-----------------------------------------+--------------------+---------------+----------+------------------| | preamble: (deadline <= "2019-01-01") | 27.91 | 0.014606 | 0 | 0 | | no preamble: (deadline <= "2019-01-01") | slowest | 0.407682 | 0 | 0 | *** =habit= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (habit)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------+--------------------+---------------+----------+------------------| | preamble: (habit) | 70.09 | 0.016489 | 0 | 0 | | no preamble: (habit) | slowest | 1.155649 | 0 | 0 | *** =level= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (level 1)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |------------------------+--------------------+---------------+----------+------------------| | preamble: (level 1) | 1.34 | 0.562950 | 0 | 0 | | no preamble: (level 1) | slowest | 0.754050 | 0 | 0 | *** =property= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (property "agenda-group")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------------------------+--------------------+---------------+----------+------------------| | preamble: (property "agenda-group") | 70.44 | 0.016571 | 0 | 0 | | no preamble: (property "agenda-group") | slowest | 1.167203 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (property "ID")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |------------------------------+--------------------+---------------+----------+------------------| | preamble: (property "ID") | 3.51 | 0.369830 | 0 | 0 | | no preamble: (property "ID") | slowest | 1.299684 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (property "agenda-group" "plans")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |------------------------------------------------+--------------------+---------------+----------+------------------| | preamble: (property "agenda-group" "plans") | 72.54 | 0.016862 | 0 | 0 | | no preamble: (property "agenda-group" "plans") | slowest | 1.223197 | 0 | 0 | *** =scheduled= #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (scheduled)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------+--------------------+---------------+----------+------------------| | preamble: (scheduled) | 4.45 | 0.100968 | 0 | 0 | | no preamble: (scheduled) | slowest | 0.449321 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (scheduled <= "2019-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |------------------------------------------+--------------------+---------------+----------+------------------| | preamble: (scheduled <= "2019-01-01") | 4.13 | 0.111067 | 0 | 0 | | no preamble: (scheduled <= "2019-01-01") | slowest | 0.458726 | 0 | 0 | *** =tags= If tag inheritance is enabled, we have to check tags on every heading. When it's disabled, we can search directly to headings with the given tags. #+BEGIN_SRC elisp (let ((org-use-tag-inheritance t)) (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (tags "Emacs"))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-----------------------------+--------------------+---------------+----------+------------------| | no preamble: (tags "Emacs") | 1.01 | 1.899647 | 0 | 0 | | preamble: (tags "Emacs") | slowest | 1.921799 | 0 | 0 | #+BEGIN_SRC elisp (let ((org-use-tag-inheritance nil)) (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (tags "Emacs"))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-----------------------------+--------------------+---------------+----------+------------------| | preamble: (tags "Emacs") | 2.08 | 0.274555 | 0 | 0 | | no preamble: (tags "Emacs") | slowest | 0.570116 | 0 | 0 | *** ~ts~ #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |-------------------+--------------------+---------------+----------+------------------| | preamble: (ts) | 1.13 | 0.475646 | 0 | 0 | | no preamble: (ts) | slowest | 0.535950 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts :from "2019-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts :from "2019-01-01") | 1.11 | 0.537445 | 0 | 0 | | preamble: (ts :from "2019-01-01") | slowest | 0.594534 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts :from "2017-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts :from "2017-01-01") | 1.13 | 0.526891 | 0 | 0 | | preamble: (ts :from "2017-01-01") | slowest | 0.594360 | 0 | 0 | Not sure why that one is slower with preamble. #+BEGIN_SRC elisp (org-ql-preamble-bench :times 10 :query (ts :from "2017-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts :from "2017-01-01") | 1.04 | 0.025688 | 0 | 0 | | preamble: (ts :from "2017-01-01") | slowest | 0.026642 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts :to "2010-01-01")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts :to "2010-01-01") | 1.10 | 0.538603 | 0 | 0 | | preamble: (ts :to "2010-01-01") | slowest | 0.593466 | 0 | 0 | *** ~ts-active~ #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-a)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------------------+--------------------+---------------+----------+------------------| | preamble: (ts-a) | 4.77 | 0.071489 | 0 | 0 | | no preamble: (ts-a) | slowest | 0.340896 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-a :from "2017-07-06")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------------------------+--------------------+---------------+----------+------------------| | preamble: (ts-a :from "2017-07-06") | 1.78 | 0.188369 | 0 | 0 | | no preamble: (ts-a :from "2017-07-06") | slowest | 0.335975 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-a :to "2017-07-06")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | preamble: (ts-a :to "2017-07-06") | 4.64 | 0.075307 | 0 | 0 | | no preamble: (ts-a :to "2017-07-06") | slowest | 0.349445 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-a :on "2017-07-06")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | preamble: (ts-a :on "2017-07-06") | 4.33 | 0.076075 | 0 | 0 | | no preamble: (ts-a :on "2017-07-06") | slowest | 0.329106 | 0 | 0 | *** ~ts-inactive~ #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-i)) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |---------------------+--------------------+---------------+----------+------------------| | preamble: (ts-i) | 1.21 | 0.459152 | 0 | 0 | | no preamble: (ts-i) | slowest | 0.555632 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-i :from "2019-07-06")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts-i :from "2019-07-06") | 1.09 | 0.531976 | 0 | 0 | | preamble: (ts-i :from "2019-07-06") | slowest | 0.579745 | 0 | 0 | #+BEGIN_SRC elisp (org-ql-preamble-bench :times 1 :file "~/org/inbox.org" :query (ts-i :to "2019-07-06")) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |--------------------------------------+--------------------+---------------+----------+------------------| | no preamble: (ts-i :to "2019-07-06") | 1.34 | 0.553428 | 0 | 0 | | preamble: (ts-i :to "2019-07-06") | slowest | 0.743881 | 0 | 0 | ** Profiling flet across all agenda files *** With flet #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--agenda :files org-agenda-files :pred (lambda () (and (todo) (or (date :deadline '<= (org-today)) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda)))))) #+END_SRC #+RESULTS: #+begin_example mapcar 711 25.608392569 0.0360174297 org-agenda-ng--agenda 5 24.019318793 4.8038637586 org-agenda-ng--filter-buffer 40 14.160293256 0.3540073313 org-agenda-ng--date-p 21595 4.2111783960 0.0001950071 org-agenda-finalize-entries 5 4.0930243110 0.8186048622 org-super-agenda--filter-finalize-entries 5 3.937522006 0.7875044012 org-agenda-ng--todo-p 37080 3.5687476730 9.624...e-05 org-get-todo-state 37080 3.4737076600 9.368...e-05 outline-next-heading 34625 3.4689080650 0.0001001850 re-search-forward 42280 3.0743315830 7.271...e-05 org-agenda-ng--format-element 1180 2.9511605820 0.0025009835 org-element-headline-parser 1180 2.6757063699 0.0022675477 org-super-agenda--group-items 5 2.187362092 0.4374724183 org-super-agenda--group-dispatch 70 2.184685662 0.0312097951 org-entry-get 22730 2.0711872869 9.112...e-05 org-entry-properties 21595 1.8958912070 8.779...e-05 org-super-agenda--group-tag 25 1.8498977799 0.0739959111 org-element-timestamp-parser 3785 1.8234333229 0.0004817525 org-parse-time-string 7560 1.7121709579 0.0002264776 org-element--get-time-properties 1180 1.1814058020 0.0010011913 #+end_example *** Without flet #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--agenda :files org-agenda-files :pred (lambda () (and (org-agenda-ng--todo-p) (or (org-agenda-ng--date-p :deadline '<= (org-today)) (org-agenda-ng--date-p :scheduled '<= (org-today))) (not (apply #'org-agenda-ng--todo-p org-done-keywords-for-agenda)))))) #+END_SRC #+RESULTS: #+begin_example mapcar 711 26.910164986 0.0378483333 org-agenda-ng--agenda 5 21.012501837 4.2025003674 org-agenda-ng--filter-buffer 40 13.751964650 0.3437991162 org-agenda-ng--todo-p 37080 5.8788306440 0.0001585445 org-agenda-ng--format-element 1180 4.5712275970 0.0038739216 org-get-todo-state 37080 4.1661659069 0.0001123561 org-agenda-ng--date-p 21595 4.1442710769 0.0001919088 org-entry-get 22730 2.8275069239 0.0001243953 org-entry-properties 21595 2.6558403739 0.0001229840 outline-next-heading 34625 2.0894695999 6.034...e-05 org-element-headline-parser 1180 1.9110445780 0.0016195293 re-search-forward 42280 1.6994989150 4.019...e-05 org-agenda-ng--add-faces 1180 1.6172592580 0.0013705586 org-agenda-ng--add-scheduled-face 1180 1.607386145 0.0013621916 org-get-tags-at 1180 1.1521010509 0.0009763568 org-back-to-heading 64530 1.1005834200 1.705...e-05 org-up-heading-safe 2360 1.0182265390 0.0004314519 outline-back-to-heading 64530 1.0086056729 1.563...e-05 org-parse-time-string 7560 0.8314918499 0.0001099856 org-time-string-to-absolute 3780 0.8277485280 0.0002189810 #+end_example ** Profiling flet on a single file This shows that the difference between them, if any, is so small as to be irrelevant. The convenience and clarity are a big win. *** With flet #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--agenda :files "~/org/main.org" :pred (lambda () (and (todo) (or (date :deadline '<= (org-today)) (date :scheduled '<= (org-today))) (not (apply #'todo org-done-keywords-for-agenda)))))) #+END_SRC #+RESULTS: #+begin_example mapcar 526 3.7898506779 0.0072050393 org-agenda-ng--agenda 5 2.7695176850 0.5539035370 org-agenda-ng--filter-buffer 5 1.414347774 0.2828695548 org-agenda-ng--format-element 265 0.8871611419 0.0033477778 org-get-tags-at 265 0.7891641319 0.0029779778 org-up-heading-safe 1150 0.7581951110 0.0006593000 re-search-backward 3700 0.5948686769 0.0001607753 org-agenda-ng--todo-p 6690 0.5840980579 8.730...e-05 org-get-todo-state 6690 0.5666448919 8.470...e-05 org-agenda-ng--date-p 5940 0.5196037069 8.747...e-05 org-entry-get 6195 0.4144106150 6.689...e-05 org-entry-properties 5940 0.3640680380 6.129...e-05 org-element-headline-parser 265 0.2810144920 0.0010604320 outline-next-heading 6195 0.2495287770 4.027...e-05 org-back-to-heading 14565 0.1959557380 1.345...e-05 re-search-forward 7850 0.1933439489 2.462...e-05 outline-back-to-heading 14565 0.1753121230 1.203...e-05 org-outline-level 2300 0.1676228200 7.287...e-05 org-agenda-finalize-entries 5 0.1607656930 0.0321531386 org-super-agenda--filter-finalize-entries 5 0.1316961509 0.0263392301 #+end_example *** Without flet #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--agenda :files "~/org/main.org" :pred (lambda () (and (org-agenda-ng--todo-p) (or (org-agenda-ng--date-p :deadline '<= (org-today)) (org-agenda-ng--date-p :scheduled '<= (org-today))) (not (apply #'org-agenda-ng--todo-p org-done-keywords-for-agenda)))))) #+END_SRC #+RESULTS: #+begin_example mapcar 526 3.7766218089 0.0071798893 org-agenda-ng--agenda 5 2.75718831 0.551437662 org-agenda-ng--filter-buffer 5 1.402551392 0.2805102784 org-agenda-ng--format-element 265 0.8864161399 0.0033449665 org-get-tags-at 265 0.7896260759 0.0029797210 org-up-heading-safe 1150 0.7589292910 0.0006599385 re-search-backward 3700 0.5956338739 0.0001609821 org-agenda-ng--todo-p 6690 0.5781650060 8.642...e-05 org-get-todo-state 6690 0.5603983020 8.376...e-05 org-agenda-ng--date-p 5940 0.5209897369 8.770...e-05 org-entry-get 6195 0.4158440950 6.712...e-05 org-entry-properties 5940 0.3640524090 6.128...e-05 org-element-headline-parser 265 0.2810144710 0.0010604319 outline-next-heading 6195 0.2485497380 4.012...e-05 org-back-to-heading 14565 0.1957209180 1.343...e-05 re-search-forward 7850 0.1927130979 2.454...e-05 outline-back-to-heading 14565 0.1751091780 1.202...e-05 org-outline-level 2300 0.1680958539 7.308...e-05 org-agenda-finalize-entries 5 0.1610422239 0.0322084448 org-super-agenda--filter-finalize-entries 5 0.132423043 0.0264846085 #+end_example ** Profiling =org-trust-scanner-tags= [2018-05-10 Thu 12:59] Turned on =org-trust-scanner-tags=, going to try profiling again: #+BEGIN_SRC elisp ;; (elp-profile 1 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" ;; (tags "world"))) (elp-profile 10 nil (org-agenda-ng org-agenda-files (tags "Emacs"))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------------------+--------------+--------------+--------------| | org-agenda-ng--agenda | 10 | 44.092598282 | 4.4092598282 | | mapcar | 282 | 40.234516707 | 0.1426755911 | | org-agenda-ng--filter-buffer | 80 | 26.895492471 | 0.3361936558 | | org-element-headline-parser | 3980 | 10.387614362 | 0.0026099533 | | org-agenda-finalize-entries | 10 | 9.194458252 | 0.9194458252 | | org-agenda-ng--tags-p | 70250 | 8.1897379849 | 0.0001165799 | | org-agenda-ng--format-element | 3980 | 6.5944325679 | 0.0016568926 | | outline-next-heading | 70320 | 6.1190180490 | 8.701...e-05 | | re-search-forward | 97050 | 5.8706467829 | 6.049...e-05 | | org-get-tags-at | 74230 | 5.4078158059 | 7.285...e-05 | | org-super-agenda--filter-finalize-entries | 10 | 5.2320123400 | 0.5232012340 | | org-super-agenda--group-items | 10 | 5.1260959210 | 0.5126095921 | | org-super-agenda--group-dispatch | 130 | 5.119333624 | 0.0393794894 | | sort | 20 | 3.8204368569 | 0.1910218428 | | org-element--parse-objects | 6180 | 3.5386578929 | 0.0005725983 | | org-is-habit-p | 5970 | 3.2497755920 | 0.0005443510 | | org-entry-get | 5970 | 3.2347964049 | 0.0005418419 | | org--property-local-values | 5970 | 3.1796357319 | 0.0005326023 | | org-get-property-block | 5970 | 3.0767919680 | 0.0005153755 | | org-entries-lessp | 20020 | 2.6563960079 | 0.0001326871 | Now trying again without it: #+BEGIN_SRC elisp ;; (elp-profile 1 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org" ;; (tags "world"))) (elp-profile 10 nil (org-agenda-ng org-agenda-files (tags "Emacs"))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-------------------------------------------+--------------+--------------+--------------| | mapcar | 1791 | 57.096304538 | 0.0318795670 | | org-agenda-ng--agenda | 10 | 54.232133506 | 5.4232133505 | | org-agenda-ng--filter-buffer | 80 | 30.065167040 | 0.3758145880 | | org-get-tags-at | 74230 | 13.840202495 | 0.0001864502 | | org-agenda-ng--format-element | 3980 | 13.429297797 | 0.0033741954 | | org-element-headline-parser | 3980 | 12.771776652 | 0.0032089891 | | org-agenda-finalize-entries | 10 | 9.1439433990 | 0.9143943399 | | org-agenda-ng--tags-p | 70250 | 9.0249653730 | 0.0001284692 | | org-super-agenda--filter-finalize-entries | 10 | 7.300515859 | 0.7300515859 | | outline-next-heading | 70320 | 7.2384435649 | 0.0001029357 | | org-super-agenda--group-items | 10 | 4.918585855 | 0.4918585855 | | org-super-agenda--group-dispatch | 130 | 4.9125893509 | 0.0377891488 | | re-search-forward | 101020 | 4.6294823850 | 4.582...e-05 | | org-up-heading-safe | 7370 | 4.4629885620 | 0.0006055615 | | org-is-habit-p | 5960 | 4.2772351910 | 0.0007176569 | | org-entry-get | 5960 | 4.2595350800 | 0.0007146870 | | org-super-agenda--group-tag | 50 | 3.8942044929 | 0.0778840898 | | re-search-backward | 26150 | 3.3660083490 | 0.0001287192 | | org--property-local-values | 5960 | 3.1793476329 | 0.0005334475 | | org-get-property-block | 5960 | 3.0662425979 | 0.0005144702 | Wow, using =org-trust-scanner-tags= saves a /lot/ of time. ** Profiling position-based *** Macro #+BEGIN_SRC elisp (defmacro elp-profile (times &rest body) (declare (indent defun)) `(let ((prefixes '("org-" "string-" "s-" "buffer-" "append" "delq" "map" "list" "car" "save-" "outline-" "delete-dups" "sort" "line-" "nth" "concat" "char-to-string" "rx-" "goto-" "when" "search-" "re-")) output) (dolist (prefix prefixes) (elp-instrument-package prefix)) (dotimes (x ,times) ,@body) (elp-results) (elp-restore-all) (point-min) (forward-line 20) (delete-region (point) (point-max)) (setq output (buffer-substring-no-properties (point-min) (point-max))) (kill-buffer) (delete-window) output)) #+END_SRC *** ng-flet #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--test-agenda-today)) #+END_SRC #+RESULTS: #+begin_example mapcar 121 0.1292609089 0.0010682719 org-agenda-ng--test-agenda-today 5 0.0860146149 0.017202923 org-agenda-ng--agenda 5 0.0858901769 0.0171780353 org-agenda-ng--format-element 75 0.0308815090 0.0004117534 org-agenda-ng--filter-buffer 5 0.026709027 0.0053418054 org-agenda-ng--date-p 455 0.0210552310 4.627...e-05 org-element-headline-parser 75 0.016209908 0.0002161321 org-get-tags-at 75 0.008953666 0.0001193822 org-agenda-ng--add-faces 75 0.0072834109 9.711...e-05 org-element-timestamp-interpreter 150 0.0068781430 4.585...e-05 org-entry-get 290 0.0060815609 2.097...e-05 org-up-heading-safe 210 0.005708647 2.718...e-05 org-agenda-finalize-entries 5 0.005201221 0.0010402442 org-element-timestamp-parser 150 0.005191617 3.461078e-05 org-entry-properties 290 0.0048787450 1.682...e-05 org-element--get-time-properties 75 0.004112675 5.483...e-05 org-agenda-ng--add-deadline-face 75 0.0039314910 5.241...e-05 org-back-to-heading 725 0.0037559990 5.180...e-06 org-agenda-ng--add-scheduled-face 75 0.0031766149 4.235...e-05 org-parse-time-string 300 0.0031740200 1.058...e-05 #+end_example *** ng-funcall #+BEGIN_SRC elisp (elp-profile 5 (org-agenda-ng--test-agenda-today)) #+END_SRC #+RESULTS: #+begin_example mapcar 121 0.1296645480 0.0010716078 org-agenda-ng--test-agenda-today 5 0.086714029 0.0173428058 org-agenda-ng--agenda 5 0.086584611 0.0173169222 org-agenda-ng--format-element 75 0.0307461019 0.0004099480 org-agenda-ng--filter-buffer 5 0.027136826 0.0054273652 org-agenda-ng--date-p 455 0.0213037090 4.682...e-05 org-element-headline-parser 75 0.016251755 0.0002166900 org-get-tags-at 75 0.008959605 0.0001194614 org-agenda-ng--add-faces 75 0.0072381410 9.650...e-05 org-element-timestamp-interpreter 150 0.0069832960 4.655...e-05 org-entry-get 290 0.0061220340 2.111...e-05 org-up-heading-safe 210 0.0057036860 2.716...e-05 org-agenda-finalize-entries 5 0.005372899 0.0010745798 org-element-timestamp-parser 150 0.0050518689 3.367...e-05 org-entry-properties 290 0.0049517209 1.707...e-05 org-agenda-ng--add-deadline-face 75 0.0039273909 5.236...e-05 org-element--get-time-properties 75 0.0039059429 5.207...e-05 org-back-to-heading 725 0.0037793259 5.212...e-06 org-parse-time-string 300 0.0032196259 1.073...e-05 org-agenda-ng--add-scheduled-face 75 0.0031410580 4.188...e-05 #+end_example *** orig Make sure to kill any existing agenda buffers first. #+BEGIN_SRC elisp (elp-profile 1 (org-agenda-list nil nil 'week)) #+END_SRC #+RESULTS: #+begin_example org-agenda-list 1 9.693596196 9.693596196 org-agenda-get-day-entries 56 8.630330659 0.1541130474 org-agenda-get-scheduled 56 6.6207980570 0.1182285367 org-is-habit-p 2792 2.2907458449 0.0008204677 org-entry-get 2798 2.287390186 0.0008175090 org-agenda--timestamp-to-absolute 7708 2.0970420100 0.0002720604 org-agenda-get-deadlines 56 1.6941886389 0.0302533685 org-at-planning-p 4399 1.3993312159 0.0003181021 org--property-local-values 2793 1.2699226760 0.0004546805 org-get-property-block 2794 1.2182695930 0.0004360306 org-time-string-to-absolute 7708 1.1513844880 0.0001493752 org-inlinetask-in-task-p 6969 1.139932302 0.0001635718 org-parse-time-string 7880 1.0635759220 0.0001349715 org-closest-date 3864 1.0383435800 0.0002687224 re-search-forward 15199 0.9607921779 6.321...e-05 org-back-to-heading 12667 0.8564486210 6.761...e-05 outline-back-to-heading 12667 0.8362207570 6.601...e-05 line-beginning-position 10333 0.7998346869 7.740...e-05 org-agenda-format-item 279 0.7552402350 0.0027069542 re-search-backward 16726 0.5694224969 3.404...e-05 #+end_example ** Profiling tags matching *** ng #+BEGIN_SRC elisp (elp-profile 1 nil (org-agenda-ng "~/org/main.org" (tags "computer"))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |--------------------------------+--------------+--------------+--------------| | mapcar | 4217 | 12.612716455 | 0.0029909216 | | org-agenda-ng--agenda | 1 | 9.721410651 | 9.721410651 | | org-get-tags-at | 1845 | 7.4793860389 | 0.0040538677 | | org-up-heading-safe | 9361 | 6.4622674019 | 0.0006903394 | | re-search-backward | 25001 | 5.3399866239 | 0.0002135909 | | org-agenda-ng--filter-buffer | 1 | 4.874598854 | 4.874598854 | | org-agenda-ng--tags-p | 1238 | 4.8067623430 | 0.0038826836 | | org-agenda-ng--format-element | 607 | 3.6325626609 | 0.0059844524 | | org-outline-level | 17484 | 1.0298924459 | 5.890...e-05 | | org-add-props | 2074 | 0.8305549259 | 0.0004004604 | | org-element-headline-parser | 607 | 0.2092664829 | 0.0003447553 | | org-back-to-heading | 11813 | 0.1252112960 | 1.059...e-05 | | outline-back-to-heading | 11813 | 0.1100693780 | 9.317...e-06 | | org-end-of-subtree | 607 | 0.0721986340 | 0.0001189433 | | outline-on-heading-p | 11813 | 0.0675261030 | 5.716...e-06 | | outline-next-heading | 1239 | 0.0627980999 | 5.068...e-05 | | re-search-forward | 3273 | 0.0612446620 | 1.871...e-05 | | org-agenda-finalize-entries | 1 | 0.041846274 | 0.041846274 | | buffer-substring-no-properties | 6329 | 0.0308716979 | 4.877...e-06 | | line-end-position | 903 | 0.0280484950 | 3.106...e-05 | *** ng without inheritance #+BEGIN_SRC elisp (elp-profile 1 nil (org-agenda-ng "~/org/main.org" (tags "computer"))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |--------------------------------+--------------+--------------+--------------| | mapcar | 4217 | 12.580246839 | 0.0029832219 | | org-agenda-ng--agenda | 1 | 8.777776059 | 8.777776059 | | org-get-tags-at | 1845 | 8.2853503299 | 0.0044907047 | | org-up-heading-safe | 9361 | 7.2710981889 | 0.0007767437 | | re-search-backward | 25001 | 5.3360082060 | 0.0002134317 | | org-agenda-ng--filter-buffer | 1 | 4.865602689 | 4.865602689 | | org-agenda-ng--tags-p | 1238 | 4.7983754310 | 0.0038759090 | | org-agenda-ng--format-element | 607 | 3.6273825100 | 0.0059759184 | | org-outline-level | 17484 | 1.0284417919 | 5.882...e-05 | | org-back-to-heading | 11813 | 0.9390534479 | 7.949...e-05 | | org-split-string | 4940 | 0.833825087 | 0.0001687905 | | string-match | 9102 | 0.8231629100 | 9.043...e-05 | | org-element-headline-parser | 607 | 0.2034305819 | 0.0003351409 | | outline-back-to-heading | 11813 | 0.1096120189 | 9.278...e-06 | | org-end-of-subtree | 607 | 0.0710802559 | 0.0001171009 | | outline-on-heading-p | 11813 | 0.0670029359 | 5.671...e-06 | | outline-next-heading | 1239 | 0.0622323519 | 5.022...e-05 | | re-search-forward | 3273 | 0.0603102519 | 1.842...e-05 | | org-agenda-finalize-entries | 1 | 0.037286496 | 0.037286496 | | buffer-substring-no-properties | 6329 | 0.0285818689 | 4.516...e-06 | *** original #+BEGIN_SRC elisp (elp-profile 1 nil (with-current-buffer "main.org" (org-tags-view nil "computer"))) #+END_SRC #+RESULTS: | Function | Times called | Total time | Average time | |-----------------------------+--------------+--------------+--------------| | org-tags-view | 1 | 2.620578129 | 2.620578129 | | org-scan-tags | 1 | 1.448883817 | 1.448883817 | | org-agenda-format-item | 607 | 0.9273893060 | 0.0015278242 | | org-add-props | 2042 | 0.8877267209 | 0.0004347339 | | org-agenda-finalize | 1 | 0.144506782 | 0.144506782 | | re-search-forward | 2154 | 0.1367046650 | 6.346...e-05 | | string-match | 8742 | 0.1002517259 | 1.146...e-05 | | org-get-priority | 607 | 0.0961996220 | 0.0001584837 | | org-agenda-align-tags | 1 | 0.095166495 | 0.095166495 | | org-agenda-prepare | 1 | 0.081724472 | 0.081724472 | | org-outline-level | 1246 | 0.0771033170 | 6.188...e-05 | | org-agenda-finalize-entries | 1 | 0.071707404 | 0.071707404 | | org-agenda-prepare-buffers | 1 | 0.057903921 | 0.057903921 | | org-get-heading | 607 | 0.0517784369 | 8.530...e-05 | | mapcar | 3738 | 0.0418641110 | 1.119...e-05 | | org-agenda-highlight-todo | 607 | 0.0273123070 | 4.499...e-05 | | mapconcat | 609 | 0.024743305 | 4.062...e-05 | | sort | 2 | 0.02117069 | 0.010585345 | | org-activate-plain-links | 132 | 0.0203558980 | 0.0001542113 | | org-activate-bracket-links | 78 | 0.0198589680 | 0.0002546021 | ** Using =org-element-parse-buffer= This basically works, as a very basic kind of agenda view, but we can already see that it's much slower (at least, for single-day views) because =org-element-parse-buffer= is slow compared to the agenda code. [2018-05-10 Thu 15:03] *Note:* This is the old, much slower code that used =org-element-parse-buffer=. *** Macro #+BEGIN_SRC elisp (defmacro elp-profile (times prefixes &rest body) (declare (indent defun)) (let ((prefixes (append '(org- string- s- buffer- append delq map list car save- outline- delete-dups sort line- nth concat char-to-string rx- goto- when search- re-) prefixes))) `(let (output) (dolist (prefix ',prefixes) (elp-instrument-package (symbol-name prefix))) (dotimes (x ,times) ,@body) (elp-results) (elp-restore-all) (point-min) (forward-line 20) (delete-region (point) (point-max)) (setq output (buffer-substring-no-properties (point-min) (point-max))) (kill-buffer) (delete-window) (let ((rows (s-lines output))) (append (list (list "Function" "Times called" "Total time" "Average time") 'hline) (cl-loop for row in rows collect (s-split (rx (1+ space)) row 'omit-nulls))))))) #+END_SRC [2018-05-09 Wed 17:31] *Note*: I seem to have misplaced the =org-agenda-ng--test-agenda-today= function I used in these tests. *** ng #+BEGIN_SRC elisp (elp-profile 1 (org-agenda-ng--test-agenda-today)) #+END_SRC #+RESULTS: #+begin_example org-element--parse-elements 5832 18.501891926 0.0031724780 mapcar 98 6.3412930759 0.0647070722 org-agenda-ng--test-agenda-today 1 6.30112088 6.30112088 org-agenda-ng--agenda-multi 1 6.301086333 6.301086333 org-agenda-ng--get-entries 8 6.249823971 0.7812279963 mapc 2803 5.9078545849 0.0021076898 org-element-parse-buffer 8 4.796204625 0.5995255781 org-element--current-element 6557 3.7164850469 0.0005667965 org-element-headline-parser 6557 3.5548915590 0.0005421521 org-end-of-subtree 6557 1.3663438270 0.0002083794 org-agenda-ng--filter-tree 8 1.3661297829 0.1707662228 org-element-map 8 1.365995685 0.1707494606 line-end-position 9503 0.4900104040 5.156...e-05 org-at-heading-p 12389 0.4574876539 3.692...e-05 re-search-forward 20193 0.4566419319 2.261...e-05 outline-on-heading-p 19286 0.4559736580 2.364...e-05 org-outline-level 6775 0.3970647569 5.860...e-05 org-back-to-heading 6897 0.3689033409 5.348...e-05 outline-back-to-heading 6897 0.3587580889 5.201...e-05 line-beginning-position 11702 0.2887473860 2.467...e-05 #+end_example *** orig Make sure to kill any existing agenda buffers first. #+BEGIN_SRC elisp (elp-profile 1 (org-agenda-list nil nil 'week)) #+END_SRC #+RESULTS: #+begin_example org-agenda-list 1 1.717467214 1.717467214 org-agenda-get-day-entries 7 1.124906724 0.1607009605 org-agenda-get-scheduled 7 0.9354116170 0.1336302310 org-get-tags-at 62 0.5598817790 0.0090303512 org-up-heading-safe 262 0.5531687240 0.0021113310 org-back-to-heading 2646 0.5316139889 0.0002009123 org-agenda-finalize 1 0.485185818 0.485185818 org-agenda-skip 749 0.4059732339 0.0005420203 org-at-planning-p 921 0.2137141959 0.0002320458 org-is-habit-p 574 0.1953095319 0.0003402605 org-entry-get 579 0.195275877 0.0003372640 org--property-local-values 574 0.1899641870 0.0003309480 org-get-property-block 574 0.1789142650 0.0003116973 re-search-backward 3586 0.1769497870 4.934...e-05 org-inlinetask-in-task-p 1495 0.1639709820 0.0001096795 outline-back-to-heading 2646 0.1387423440 5.243...e-05 re-search-forward 3177 0.1353034909 4.258...e-05 org-agenda-get-deadlines 7 0.1341101260 0.0191585894 line-beginning-position 2041 0.1000089939 4.899...e-05 org-get-todo-state 749 0.070545971 9.418...e-05 org-agenda-prepare 1 0.052788647 0.052788647 org-agenda-prepare-buffers 1 0.050402675 0.050402675 org-agenda-get-timestamps 7 0.0428353419 0.0061193345 org-agenda--timestamp-to-absolute 1498 0.0414448099 2.766...e-05 org-time-string-to-absolute 1498 0.0385546620 2.573...e-05 org-agenda-finalize-entries 7 0.03400573 0.0048579614 org-super-agenda--finalize-entries 7 0.0339819269 0.0048545609 org-outline-level 505 0.0335176070 6.637...e-05 org-super-agenda--group-items 7 0.0268096709 0.0038299529 org-super-agenda--group-dispatch 84 0.024216379 0.0002882902 org-parse-time-string 1572 0.0211881319 1.347...e-05 org-closest-date 749 0.0178231690 2.379...e-05 org-before-first-heading-p 578 0.0142076310 2.458...e-05 org-refresh-category-properties 1 0.013815263 0.013815263 org-in-src-block-p 753 0.0135235599 1.795...e-05 org-refresh-stats-properties 1 0.012230309 0.012230309 org-habit-parse-todo 5 0.0121474539 0.0024294907 org-get-limited-outline-regexp 1502 0.0100960829 6.721...e-06 mapcar 309 0.009495786 3.073...e-05 org-super-agenda--group-habit 7 0.009318616 0.0013312308 string-match 6253 0.0090207890 1.442...e-06 org-super-agenda--group-dispatch-and 7 0.0067444 0.0009634857 org-agenda-get-blocks 7 0.0060974240 0.0008710605 outline-on-heading-p 2655 0.0060741090 2.287...e-06 org-agenda-get-sexps 7 0.0057924309 0.0008274901 org-super-agenda--group-regexp 7 0.005592509 0.0007989298 org-refresh-properties 2 0.005461118 0.002730559 org-super-agenda--get-item-entry 31 0.004812809 0.0001552519 org-agenda-align-tags 1 0.004677699 0.004677699 org-set-regexps-and-options 1 0.004533983 0.004533983 org--setup-collect-keywords 2 0.004499168 0.002249584 org-agenda-format-item 31 0.0039898020 0.0001287032 org-date-to-gregorian 420 0.0039648049 9.440...e-06 org-agenda-highlight-todo 31 0.0039479179 0.0001273521 string-to-number 8202 0.0031924329 3.892...e-07 org-end-of-subtree 12 0.003047285 0.0002539404 org-super-agenda--group-tag 35 0.002941374 8.403...e-05 org-inlinetask-outline-regexp 1495 0.002887697 1.931...e-06 org-get-wdays 749 0.0027946290 3.731...e-06 org-refresh-effort-properties 1 0.002750279 0.002750279 org-entry-beginning-position 31 0.002345909 7.567...e-05 sort 25 0.0021000059 8.400...e-05 mapc 445 0.0018132390 4.074...e-06 line-end-position 72 0.00177188 2.460...e-05 mapconcat 341 0.0016574119 4.860...e-06 org-entries-lessp 60 0.0016203450 2.700...e-05 concat 336 0.0016188640 4.818...e-06 org-habit-insert-consistency-graphs 1 0.001368346 0.001368346 org-add-props 137 0.001291149 9.424...e-06 org-element-at-point 1 0.001266403 0.001266403 org-agenda-new-marker 62 0.001253987 2.022...e-05 outline-next-heading 42 0.001123596 2.675...e-05 org-split-string 148 0.0011049120 7.465...e-06 org-element--parse-to 1 0.001074227 0.001074227 org-get-scheduled-time 5 0.0010200300 0.000204006 org-super-agenda--group-todo 21 0.001003225 4.777...e-05 org-time-string-to-time 69 0.0009947770 1.441...e-05 org-element--current-element 4 0.0009597410 0.0002399352 org-entry-properties 5 0.0008807149 0.0001761429 org-at-date-range-p 172 0.0008412679 4.891...e-06 org-agenda-mode 1 0.000787997 0.000787997 org-super-agenda--group-log 14 0.0007874939 5.624...e-05 buffer-substring 31 0.0007650360 2.467...e-05 org-heading-components 9 0.00071644 7.960...e-05 org-entry-end-position 31 0.0007046580 2.273...e-05 string-prefix-p 789 0.0006732279 8.532...e-07 org-agenda-skip-eval 1498 0.0006586869 4.397...e-07 org-get-repeat 5 0.000656577 0.0001313153 org-activate-bracket-links 3 0.000648713 0.0002162376 org-agenda-fix-displayed-tags 31 0.000636446 2.053...e-05 s-join 252 0.0006324150 2.509...e-06 org-element-keyword-parser 4 0.000530311 0.0001325777 org-in-commented-heading-p 4 0.00051077 0.0001276925 buffer-substring-no-properties 308 0.0004763350 1.546...e-06 org-habit-build-graph 5 0.0004689199 9.3784e-05 car 1626 0.0004469320 2.748...e-07 org-activate-plain-links 3 0.000382191 0.000127397 org-super-agenda--transform-groups 7 0.0003747289 5.353...e-05 org-get-priority 26 0.0003598600 1.384...e-05 org-agenda-fontify-priorities 1 0.000357076 0.000357076 org-super-agenda--group-priority 14 0.00035224 2.516e-05 org-agenda-prepare-window 1 0.00033704 0.00033704 org-find-text-property-in-string 316 0.0003191880 1.010...e-06 org-not-nil 821 0.0003016530 3.674...e-07 list 940 0.0002995150 3.186...e-07 org-super-agenda--group-time-grid 7 0.0002954999 4.221...e-05 org-agenda-format-date-aligned 7 0.0002794120 3.991...e-05 org-super-agenda--get-tags 103 0.0002654610 2.577...e-06 delq 513 0.0002634709 5.135...e-07 org-replace-escapes 5 0.0002382050 4.7641e-05 car-safe 911 0.0002314199 2.540...e-07 org-today 48 0.0002232539 4.651...e-06 org-get-time-of-day 19 0.000219044 1.152...e-05 org-days-to-iso-week 9 0.0002125040 2.361...e-05 append 433 0.0002083290 4.811...e-07 org-get-category 31 0.000198604 6.406...e-06 string-match-p 31 0.0001787880 5.767...e-06 org-agenda-today-p 21 0.0001658670 7.898...e-06 org-super-agenda--get-marker 54 0.0001629399 3.017...e-06 delete-dups 111 0.0001343219 1.210...e-06 org-at-heading-p 9 0.000128871 1.4319e-05 string-equal 361 0.0001168969 3.238...e-07 org-super-agenda--make-agenda-header 13 0.0001151760 8.859...e-06 org-add-prop-inherited 142 0.0001142290 8.044...e-07 org-check-agenda-file 8 0.000110448 1.3806e-05 org-agenda-get-day-face 7 0.000110437 1.577...e-05 org-get-agenda-file-buffer 8 0.000107414 1.342675e-05 org-agenda-files 3 0.000102652 3.421...e-05 org-downcase-keep-props 90 0.0001008230 1.120...e-06 org-super-agenda--get-priority-cookie 15 9.460...e-05 6.307e-06 org-get-todo-face 31 9.423...e-05 3.039...e-06 org-find-base-buffer-visiting 8 8.7492e-05 1.09365e-05 org-habit-get-faces 140 8.080...e-05 5.771...e-07 org-remove-uninherited-tags 176 7.580...e-05 4.307...e-07 goto-char 31 7.1014e-05 2.290...e-06 org-super-agenda--transform-group-order 7 6.5543e-05 9.363...e-06 org-agenda-add-time-grid-maybe 7 5.703...e-05 8.147...e-06 org-compile-prefix-format 1 5.2164e-05 5.2164e-05 org-link-unescape 4 4.4823e-05 1.120575e-05 listp 133 3.960...e-05 2.977...e-07 org-habit-duration-to-days 5 3.9446e-05 7.8892e-06 org-make-options-regexp 1 3.6054e-05 3.6054e-05 org-agenda-mark-header-line 1 3.3066e-05 3.3066e-05 org-element--collect-affiliated-keywords 4 3.2661e-05 8.16525e-06 org-habit-get-priority 5 2.9857e-05 5.9714e-06 org-agenda-get-category-icon 31 2.673...e-05 8.624...e-07 org-face-from-face-or-color 61 2.1803e-05 3.574...e-07 org-reduced-level 40 2.1692e-05 5.423e-07 org-link-get-parameter 26 2.062...e-05 7.930...e-07 buffer-name 57 1.9875e-05 3.486...e-07 org-defkey 5 1.9835e-05 3.966...e-06 string-to-char 56 1.929...e-05 3.446...e-07 org-string-nw-p 5 1.7894e-05 3.578...e-06 s-wrap 7 1.6955e-05 2.422...e-06 org-link-expand-abbrev 2 1.5491e-05 7.7455e-06 map-keymap 2 1.4987e-05 7.4935e-06 search-forward 7 1.403...e-05 2.004...e-06 string-lessp 41 1.394...e-05 3.400...e-07 org-agenda-reset-markers 1 1.3927e-05 1.3927e-05 org-agenda-deadline-face 7 1.2592e-05 1.798...e-06 search-backward 5 1.1253e-05 2.2506e-06 org-agenda-span-name 5 9.758e-06 1.9516e-06 outline-previous-heading 1 8.678e-06 8.678e-06 buffer-modified-p 10 4.618e-06 4.617...e-07 buffer-file-name 13 4.608...e-06 3.544...e-07 buffer-live-p 11 4.473e-06 4.066...e-07 buffer-local-value 10 4.278e-06 4.278...e-07 buffer-base-buffer 10 3.858e-06 3.858...e-07 org-agenda-set-mode-name 1 3.118e-06 3.118e-06 org-remove-flyspell-overlays-in 4 2.926...e-06 7.315...e-07 buffer-size 8 2.791...e-06 3.489...e-07 org-key 5 2.707e-06 5.414e-07 org-unbracket-string 1 2.692e-06 2.692e-06 save-place-to-alist 1 2.276e-06 2.276e-06 org-element--cache-put 4 1.907e-06 4.7675e-07 org-property-inherit-p 2 1.414e-06 7.07e-07 org-agenda-ndays-to-span 2 1.293...e-06 6.465...e-07 maphash 1 1.171e-06 1.171e-06 org-set-sorting-strategy 1 1.165e-06 1.165e-06 org-file-menu-entry 1 1.12e-06 1.12e-06 string-width 2 1.093e-06 5.465e-07 org-time-stamp-format 1 1.054e-06 1.054e-06 org-agenda-fit-window-to-buffer 1 9.31e-07 9.31e-07 org-font-lock-add-tag-faces 1 9.13e-07 9.13e-07 org-agenda-span-to-ndays 1 8.3e-07 8.3e-07 org-element-property 2 8.12e-07 4.06e-07 org-agenda-mark-clocking-task 1 8.03e-07 8.03e-07 org-tag-alist-to-groups 1 7.55e-07 7.55e-07 org-element-type 1 6.14e-07 6.14e-07 org-agenda-use-sticky-p 1 4.73e-07 4.73e-07 mapatoms 1 0 0.0 #+end_example *** Profile org-element-map #+BEGIN_SRC elisp (elp-profile 1 (with-current-buffer (find-buffer-visiting "~/org/main.org") (org-element-parse-buffer 'headline))) #+END_SRC #+RESULTS: #+begin_example org-element--parse-elements 1002 4.1612395469 0.0041529336 org-element-parse-buffer 1 0.859981956 0.859981956 org-element--current-element 1225 0.8236391779 0.0006723585 org-element-headline-parser 1225 0.7952382879 0.0006491741 org-end-of-subtree 1225 0.5557043549 0.0004536362 line-end-position 1995 0.0751104350 3.764...e-05 re-search-forward 3743 0.0516547359 1.380...e-05 org-outline-level 1225 0.0477962079 3.901...e-05 org-back-to-heading 1225 0.0469223529 3.830...e-05 outline-back-to-heading 1225 0.0450936199 3.681...e-05 org-element--get-node-properties 1225 0.0434517140 3.547...e-05 line-beginning-position 2003 0.0334129610 1.668...e-05 org-element--get-time-properties 1225 0.0306394040 2.501...e-05 org-get-limited-outline-regexp 2072 0.0140703559 6.790...e-06 org-element-timestamp-parser 249 0.0124751890 5.010...e-05 outline-next-heading 847 0.0115396399 1.362...e-05 org-at-heading-p 2227 0.0109473260 4.915...e-06 outline-on-heading-p 3452 0.0101107139 2.928...e-06 string-match 4594 0.0064121759 1.395...e-06 mapcar 30 0.0041008740 0.0001366958 #+end_example ** with/without ts.el [2019-08-11 Sun 15:39] These results seem to show a minor performance improvement by using ~ts~, and the code is simpler. #+BEGIN_SRC elisp ;; (require 'ts) (org-ql--defpred ts-ts (&key from to _on) ;; The underscore before `on' prevents "unused lexical variable" warnings, because we ;; pre-process that argument in a macro before this function is called. "Return non-nil if current entry has a timestamp in given period. If no arguments are specified, return non-nil if entry has any timestamp. If FROM, return non-nil if entry has a timestamp on or after FROM. If TO, return non-nil if entry has a timestamp on or before TO. If ON, return non-nil if entry has a timestamp on date ON. FROM, TO, and ON should be strings parseable by `parse-time-string' but may omit the time value." ;; TODO: DRY this with the clocked predicate. ;; NOTE: FROM and TO are actually expected to be Unix timestamps. The docstring is written ;; for end users, for which the arguments are pre-processed by `org-ql-select'. ;; FIXME: This assumes every "clocked" entry is a range. Unclosed clock entries are not handled. (cl-macrolet ((next-timestamp () `(when (re-search-forward org-element--timestamp-regexp end-pos t) (ts-parse-org (match-string 0)))) (test-timestamps (pred-form) `(cl-loop for next-ts = (next-timestamp) while next-ts thereis ,pred-form))) (save-excursion (let ((end-pos (org-entry-end-position))) (cond ((not (or from to)) (re-search-forward org-element--timestamp-regexp end-pos t)) ((and from to) (test-timestamps (and (ts<= from next-ts) (ts<= next-ts to)))) (from (test-timestamps (ts<= from next-ts))) (to (test-timestamps (ts<= next-ts to)))))))) #+END_SRC *** :from #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("old ts" (org-ql "~/org/inbox.org" (ts :from "2017-01-01"))) ("ts.el ts" (org-ql "~/org/inbox.org" (ts-ts :from "2017-01-01"))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | ts.el ts | 1.32 | 1.299966 | 0 | 0 | | old ts | slowest | 1.713027 | 0 | 0 | *** :on #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("old ts" (org-ql "~/org/inbox.org" (ts :on "2019-05-14"))) ("ts.el ts" (org-ql "~/org/inbox.org" (ts-ts :on "2019-05-14"))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | ts.el ts | 1.17 | 0.557281 | 0 | 0 | | old ts | slowest | 0.652149 | 0 | 0 | *** :to #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("old ts" (org-ql "~/org/inbox.org" (ts :to "2019-01-01"))) ("ts.el ts" (org-ql "~/org/inbox.org" (ts-ts :to "2019-01-01"))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | ts.el ts | 1.01 | 1.300084 | 0 | 0 | | old ts | slowest | 1.312208 | 0 | 0 | *** Without timestamp argument #+BEGIN_SRC elisp (bench-multi-lexical :times 1 :ensure-equal t :forms (("old ts" (org-ql "~/org/inbox.org" (ts))) ("ts.el ts" (org-ql "~/org/inbox.org" (ts-ts))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | ts.el ts | 1.14 | 2.251801 | 0 | 0 | | old ts | slowest | 2.560280 | 0 | 0 | #+BEGIN_SRC elisp (bench-multi-lexical :times 20 :ensure-equal t :forms (("old ts" (org-ql "~/src/emacs/org-ql/tests/data.org" (ts))) ("ts.el ts" (org-ql "~/src/emacs/org-ql/tests/data.org" (ts-ts))))) #+END_SRC #+RESULTS: | Form | x faster than next | Total runtime | # of GCs | Total GC runtime | |----------+--------------------+---------------+----------+------------------| | ts.el ts | 1.05 | 0.103714 | 0 | 0 | | old ts | slowest | 0.108663 | 0 | 0 | * References :PROPERTIES: :TOC: :include descendants :depth 1 :END: :CONTENTS: - [[#github---ndwarshuisorg-sql-sql-backend-for-emacs-org-mode][GitHub - ndwarshuis/org-sql: SQL backend for Emacs Org-Mode]] - [[#john-kitchin-on-rewriting-the-org-agenda-code][John Kitchin on rewriting the Org agenda code]] - [[#nicolas-goazious-org-element-cache-implementation][Nicolas Goaziou's org-element cache implementation]] - [[#uniform-structured-syntax-metaprogramming-and-run-time-compilation][Uniform Structured Syntax, Metaprogramming and Run-time Compilation]] :END: ** [[https://github.com/ndwarshuis/org-sql][GitHub - ndwarshuis/org-sql: SQL backend for Emacs Org-Mode]] [2020-01-04 Sat 09:03] ** [[gnus:gmane.emacs.orgmode#CAJ51EToLCm5zDLKu8XeuqEWrLhHZF+OoNkviPSivZbFttzF8=A@mail.gmail.com][John Kitchin on rewriting the Org agenda code]] :Emacs:Org: :PROPERTIES: :archive.is: http://archive.is/33R9M :END: [2019-10-28 Mon 08:28] Originally from [[id:90a68535-2403-4ba9-a117-60be1862628f][this entry in my notes]]. #+BEGIN_QUOTE From: John Kitchin Subject: Re: [Orgmode] Slow speed of week and month views Newsgroups: gmane.emacs.orgmode To: Karl Voit Cc: "emacs-orgmode@gnu.org" Date: Sat, 5 Aug 2017 18:17:09 -0400 (4 hours, 11 minutes, 38 seconds ago) I can think of two possibilities for a future approach (besides a deep dive on profiling the current elisp to improve the speed there). They both involve some substantial coding though, and would probably add dependencies. I am curious what anyone things about these, or if there are other ideas. One is to use the new dynamic module capability to write an org parser in C, or a dedicated agenda function, which would presumably be faster than in elisp. This seems hard, and for me would certainly be a multiyear project I am sure! The downside of this is the need to compile the module. I don't know how easy it would be to make this work across platforms with the relatively easy install org-mode currently has. This could have a side benefit though of a c-lib that could be used by others to expand where org-mode is used. The other way that might work is to rely more heavily on a cached version of the files, perhaps in a different format than elisp, that is faster to work with. The approach I have explored in this is to index org files into a sqlite database. The idea then would be to generate the agenda from a sql query. I use something like this already to "find stuff in orgmode anywhere". One of the reasons I wrote this is the org-agenda list of files isn't practical for me because my files are so scattered on my file system. I had a need to be able to find TODOs in research projects in a pretty wide range of locations. The code I use is at https://github.com/jkitchin/scimax/blob/master/org-db.el, and from one database I can find headlines, contacts, locations, TODO headlines across my file system, all the files that contain a particular link, and my own recent org files. This approach relies on emacsql, and a set of hook functions to update the database whenever a file is changed. It is not robust, e.g. the file could be out of sync with the db if it is modified outside emacs, but this works well enough for me so far. Updated files get reindexed whenever emacs is idle. It was a compromise on walking the file system all the time or daily, or trying to use inotify and you can always run a command to prune/sync all the files any time you want. sqlite is ok, but with emacsql you cannot put strings in it directly (at least when I wrote the org-db code), which has limited it for full-text search so far. Also with text, the db got up to about 0.5 GB in size, and started slowing down. So it doesn't have text in it for now. It has all the other limitations of sqlite too, limited support for locking, single process.... I am moderately motivated to switch from sqlite to MongoDB, but the support for Mongo in emacs is pretty crummy (I tried writing a few traditional interfaces, but the performance was not that good, and limited since Mongo uses bson, and it is just not the same as json!). Why Mongo? Mostly because the Mongo query language is basically json and easy to generate in Emacs, unlike sql. Also, it is flexible and easy to adapt to new things, e.g. indexing src-blocks or tables or whatever org-element you want. (And I want to use Mongo for something else too ;). Obviously these all add dependencies, and might not be suitable for the core org-mode distribution. But I do think it is important to think about ways to scale org-mode while maintaining compatibility with the core. The main point of the database was to get a query language, persistence and good performance. I have also used caches to speed up using bibtex files, and my org-contacts with reasonable performance. These have been all elisp, with no additional dependencies. Maybe one could do something similar to keep an agenda cache that is persistent and updated via hook functions. Thoughts? John #+END_QUOTE + [[https://github.com/m00natic/cl-fdbq][GitHub - m00natic/cl-fdbq: SQL-like operations over fixed field DBs]] ** Nicolas Goaziou's =org-element= cache implementation [2020-01-04 Sat 08:31] https://code.orgmode.org/bzg/org-mode/src/master/lisp/org-element.el#L4817 I guess I haven't been keeping up, because I'm not sure when exactly he made or committed this, or what his plans for it are. It's currently disabled by default. ** [[https://m00natic.github.io/lisp/manual-jit.html][Uniform Structured Syntax, Metaprogramming and Run-time Compilation]] * Testing [2020-01-08 Wed 07:15] Subtree moved from =tests/data.org=. #+BEGIN_SRC elisp (cl-defun ap/org-tweak-timestamps (&key (offset 0) epoch-ts) "Advance all timestamps in the current buffer as if the earliest one was on today. OFFSET changes which timestamp (in chronological order) is set to today. Or, if EPOCH-TS is non-nil, use it as the new zero-point for today." (let* ((tss (->> (org-with-wide-buffer (goto-char (point-min)) (cl-loop while (re-search-forward org-ts-regexp-both nil t) collect (ts-parse-org (match-string 0)))) (-sort #'ts<))) (epoch-ts (if epoch-ts (ts-parse-org epoch-ts) (nth offset tss))) (difference-secs (ts-diff (ts-now) epoch-ts)) (days (floor (/ difference-secs 86400)))) (org-with-wide-buffer (goto-char (point-min)) (while (re-search-forward org-ts-regexp-both nil t) (let* ((ts (ts-parse-org (match-string 0))) (timed-p (string-match-p (rx (repeat 2 digit) ":" (repeat 2 digit) (or ">" "]")) (match-string 0))) (type (cond ((string-prefix-p "<" (match-string 0)) 'active) ((string-prefix-p "[" (match-string 0)) 'inactive) (t (error "Unknown ts type")))) (brackets (cl-ecase type ('active (cons "<" ">")) ('inactive (cons "[" "]")))) (format-string (concat (car brackets) (if timed-p "%Y-%m-%d %a %H:%M" "%Y-%m-%d %a") (cdr brackets))) (new-ts (ts-adjust 'day days ts)) (new-ts-string (ts-format format-string new-ts))) (replace-match new-ts-string t t nil 0)))) (message "Tweaked from epoch: %s" (ts-format epoch-ts)))) #+END_SRC #+BEGIN_SRC elisp (org-time-string-to-absolute (org-entry-get (point) "SCHEDULED")) #+END_SRC #+BEGIN_SRC elisp :results none ;; Setup code (require 'org-super-agenda) (org-super-agenda-mode 1) (require 'org-habit) (setq org-todo-keywords '((sequence "TODO(t!)" "TODAY(a!)" "NEXT(n!)" "STARTED(s!)" "IN-PROGRESS(p!)" "UNDERWAY(u!)" "WAITING(w@)" "SOMEDAY(o!)" "MAYBE(m!)" "|" "DONE(d@)" "CANCELED(c@)") (sequence "CHECK(k!)" "|" "DONE(d@)") (sequence "TO-READ(r!)" "READING(R!)" "|" "HAVE-READ(d@)") (sequence "TO-WATCH(!)" "WATCHING(!)" "SEEN(!)"))) (with-current-buffer "test.org" (revert-buffer)) (defmacro with-org-today-date (date &rest body) "Run BODY with the `org-today' function set to return simply DATE. DATE should be a date-time string (both date and time must be included)." (declare (indent defun)) `(let ((day (date-to-day ,date)) (orig (symbol-function 'org-today))) (unwind-protect (progn (fset 'org-today (lambda () day)) ,@body) (fset 'org-today orig)))) #+END_SRC #+BEGIN_SRC elisp :results none (defun diary-sunrise () (let ((dss (diary-sunrise-sunset))) (with-temp-buffer (insert dss) (goto-char (point-min)) (search-forward ",") (buffer-substring (point-min) (match-beginning 0))))) (defun diary-sunset () (let ((dss (diary-sunrise-sunset)) start end) (with-temp-buffer (insert dss) (goto-char (point-min)) (search-forward ", ") (setq start (match-end 0)) (search-forward " at") (setq end (match-beginning 0)) (goto-char start) (capitalize-word 1) (buffer-substring start end)))) #+END_SRC *Note:* Removing tests from here as they're added to =test.el=. #+BEGIN_SRC elisp (org-super-agenda--test-with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/emacs/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:name "Time grid items in all-uppercase with RosyBrown1 foreground" :time-grid t :transformer (--> it (upcase it) (propertize it 'face '(:foreground "RosyBrown1")))) (:name "Priority >= C items underlined, on black background" :face (:background "black" :underline t) :not (:priority>= "C") :order 100)))) (org-agenda nil "a"))) (org-super-agenda--test-with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/emacs/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:name none :time-grid t) (:name "Should be all-uppercase RosyBrown1 on black" :face (:background "black" :foreground "RosyBrown1") :transformer #'upcase :not (:priority>= "C") :order 100)))) (org-agenda nil "a"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:name none :time-grid t) (:name none :not (:priority>= "C") :order 100)))) (org-agenda nil "a"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:name "Items with child TODOs" :children todo)))) (org-agenda nil "a"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:name "Items with child TODOs" :children "CHECK")))) (org-agenda nil "a"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-agenda-custom-commands '(("u" "Super view" ((agenda "" ((org-super-agenda-groups '((:name "Today" :time-grid t :scheduled today :deadline today))))) (todo "" ((org-super-agenda-groups '((:name "Projects" :children t) (:discard (:anything t))))))))))) (org-agenda nil "u"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-agenda-custom-commands '(("u" "Super view" ((agenda "" ((org-super-agenda-groups '((:name "Today" :time-grid today))))) (todo "" ((org-super-agenda-groups '((:name "Projects" :children t) (:discard (:anything t))))))))))) (org-agenda nil "u"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-agenda-custom-commands '(("u" "Super view" ((agenda "" ((org-super-agenda-groups '((:name "Schedule" :time-grid t :date today) (:name "Due today" :deadline today) (:name "Due soon" :deadline t))))) (todo "" ((org-agenda-overriding-header "") (org-super-agenda-groups '((:name "Projects" :children t) (:discard (:anything t))))))))))) (org-agenda nil "u"))) (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) (org-super-agenda-groups '((:scheduled (before "2017-07-06"))))) (org-agenda nil "a"))) #+END_SRC #+BEGIN_SRC elisp (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:todo "WAITING"))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) (org-todo-list))) (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:todo "SOMEDAY"))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) (org-tags-view nil "Emacs"))) (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:todo "CHECK"))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) ;; org-search-view doesn't seem to set the todo-state property, so the matcher doesn't work (org-search-view nil "Emacs"))) (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:regexp ("moon" "mars")))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) (org-search-view nil "space"))) (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:todo "SOMEDAY"))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) (org-agenda-list nil nil 'day))) #+END_SRC ** Agenda censoring For sharing screenshots of the agenda without revealing private data. #+BEGIN_SRC elisp (defun org-agenda-sharpie () "Censor the text of items in the agenda." (interactive) (let (regexp old-heading new-heading properties) ;; Save face properties of line in agenda to reapply to changed text (setq properties (text-properties-at (point))) ;; Go to source buffer (org-with-point-at (org-find-text-property-in-string 'org-marker (buffer-substring (line-beginning-position) (line-end-position))) ;; Save old heading text and ask for new text (line-beginning-position) (unless (org-at-heading-p) ;; Not sure if necessary (org-back-to-heading)) (setq old-heading (when (looking-at org-complex-heading-regexp) (match-string 4)))) (setq new-heading (read-from-minibuffer "Overwrite visible heading with: ")) (add-text-properties 0 (length new-heading) properties new-heading) ;; Back to agenda buffer (save-excursion (when (and old-heading new-heading) ;; Replace agenda text (let ((inhibit-read-only t)) (goto-char (line-beginning-position)) (when (search-forward old-heading (line-end-position)) (replace-match new-heading 'fixedcase 'literal))))))) #+END_SRC ** Agenda examining This helps a lot. #+BEGIN_SRC elisp (defun data-debug-show-string-with-properties (s) (with-current-buffer (get-buffer-create "argh") (erase-buffer) (print s (current-buffer)) ;; Convert string reader representations to plain lists that can be set (cl-loop for (match replace) in '(("#(" "'(") ("#<" "'(") (">" ")")) do (progn (goto-char (point-min)) (while (search-forward match nil 'noerror) (replace-match replace 'fixedcase 'literal)))) ;; Surround content in a list which `argh' is set to, then eval ;; the buffer to do it (goto-char (point-min)) (insert "(setq argh (list '") (delete-forward-char 2) (goto-char (point-max)) (insert "))") ;; Okay, sure, eval'ing the buffer is dangerous and bad and wrong. ;; But this is the only way I can find to make this work. (Maybe ;; `text-properties-at' could be used to get actual lists...) (eval-buffer) (data-debug-show-stuff argh "argh") ;; (switch-to-buffer (current-buffer)) )) (defun data-debug-show-current-line-with-properties () (interactive) (data-debug-show-string-with-properties (buffer-substring (line-beginning-position) (line-end-position)))) (with-current-buffer "*Org Agenda*" (data-debug-show-string-with-properties (seq-subseq (split-string (buffer-string) "\n") 0 5))) #+END_SRC ** Auto categories #+BEGIN_SRC elisp (let ((org-super-agenda-groups '((:auto-category t)))) (org-agenda-list nil nil 'day)) #+END_SRC ** Auto grouping #+BEGIN_SRC elisp (with-org-today-date "2017-07-05 00:00" (let ((org-super-agenda-groups '((:auto-group t))) (org-agenda-files (list "~/src/org-super-agenda/test/test.org"))) (org-agenda-list nil nil 'day))) #+END_SRC ** Date #+BEGIN_SRC elisp (with-org-today-date "2017-07-05 00:00" (-let* ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) ((sec minute hour day month year dow dst utcoff) (list 0 0 0 5 7 2017 3 t nil)) (last-day-of-month ;; A hack that seems to work fine (1+ (calendar-last-day-of-month month year))) (target-date (format "%d-%02d-%02d" year month last-day-of-month)) (org-super-agenda-groups `((:deadline (before ,target-date)) (:discard (:anything t))))) (org-todo-list))) (with-org-today-date "2017-07-05 00:00" (-let* ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-agenda-span 'day) ((sec minute hour day month year dow dst utcoff) (list 0 0 0 5 7 2017 3 t nil)) (last-day-of-month (calendar-last-day-of-month month year)) (target-date (format "%d-%02d-%02d" year month last-day-of-month)) (org-super-agenda-groups `((:deadline (before ,target-date)) (:discard (:anything t))))) (org-todo-list))) #+END_SRC ** Effort #+BEGIN_SRC elisp (with-org-today-date "2017-07-05 00:00" (let ((org-agenda-files (list "~/src/org-super-agenda/test/test.org")) (org-super-agenda-groups '((:effort< "0:06")))) (org-agenda-list nil nil 'day))) #+END_SRC ** Misc *** let-plist I don't need this right now, but it might come in handy here or elsewhere. #+BEGIN_SRC elisp (defmacro osa/let-plist (keys plist &rest body) "`cl-destructuring-bind' without the boilerplate for plists." ;; See https://emacs.stackexchange.com/q/22542/3871 ;; I really don't understand why Emacs doesn't have this already. ;; So many things come close to it: pcase, pcase-let, map-let, ;; cl-destructuring-bind, -let...but none of them let you simply ;; bind all the values of a plist to variables with the same name as ;; their keys. You always have to type the name of the key twice. ;; For example, compare these two forms: ;; (-let (((&keys :from from :to to :date date :subject subject) email)) ;; (list from to date subject)) ;; (osa/let-plist (:from :to :date :subject) email ;; (list from to date subject)) ;; Now, sure, sometimes you need to bind values to differently named ;; variables. But when you don't, I know which one I prefer. (declare (indent defun)) (setq keys (cl-loop for key in keys collect (intern (replace-regexp-in-string (rx bol ":") "" (symbol-name key))))) `(cl-destructuring-bind (&key ,@keys &allow-other-keys) ,plist ,@body)) #+END_SRC ** Profiling #+BEGIN_SRC elisp :results none (defmacro profile-it (times &rest body) `(let (output) (dolist (p '("org-super-agenda-" "map" "org-" "string-" "s-" "buffer-" "append" "delq" "map" "list" "car" "save-" "outline-" "delete-dups" "sort" "line-" "nth" "concat" "char-to-string" "rx-" "goto-" "when" "search-" "re-")) (elp-instrument-package p)) (dotimes (x ,times) ,@body) (elp-results) (elp-restore-all) (point-min) (forward-line 20) (delete-region (point) (point-max)) (setq output (buffer-substring-no-properties (point-min) (point-max))) (kill-buffer) (delete-window) output)) #+END_SRC * [#C] COMMENT Config :noexport: :LOGBOOK: CLOCK: [2021-06-18 Fri 07:38]--[2021-06-18 Fri 21:51] => 14:13 :END: ** Org settings #+PROPERTY: LOGGING nil #+TODO: TODO MAYBE NEXT PROJECT UNDERWAY WAITING | DONE(d) CANCELED ** File-local variables # before-save-hook: ((lambda () (when (fboundp 'unpackaged/org-fix-blank-lines) (unpackaged/org-fix-blank-lines t))) (lambda () (when (fboundp 'ap/org-sort-entries-recursive-multi) (let ((narrowed-p (buffer-narrowed-p)) (olp (org-get-outline-path 'with-self)) (relative-pos (- (point) (save-excursion (org-back-to-heading) (point))))) (widen) (goto-char (point-min)) (ap/org-sort-entries-recursive-multi '(?a ?p ?o)) (goto-char (org-find-olp olp 'this-buffer)) (when narrowed-p (org-narrow-to-subtree)) (forward-char relative-pos)))) org-update-all-dblocks org-make-toc) # Local Variables: # before-save-hook: ((lambda () (when (fboundp 'unpackaged/org-fix-blank-lines) (unpackaged/org-fix-blank-lines t))) ap/org-sort-before-save-hook org-update-all-dblocks org-make-toc) # org-ql-ask-unsafe-queries: nil # End: