Wild brush strokes across canvas

Hands invisible

A cool Starry Night

Why IDEs?

Programs—even small ones—can be incredibly dense and complex. Beyond a point, they cannot be understood as mere textual entities. Nor can the human mind keep track of a self-updating correct emulation of their live meaning.

This is why people have built, and continue to build, a vast array of tools to help us understand and manipulate meanings of programming languages, their surroundings, and their runtimes.

Integrated Development Environments (IDEs) provide bundles of these tools and services to assist the programmer. Tools and services that:

• are aware of code syntax and semantics, project layouts and structures
• are aware of error checks and safeties the language provides (or need bolted-on)
• can build, publish, use, run, observe, analyze software artifacts
• help experiment with code, e.g. change code as we step through a debugger
• these days, even unburden us from pinching StackOverflow answers ².

What does "Integrated" mean?

A big task calls for a little philosophical indulgence. I think there are at least two lenses we can use to explore this question.

One lens: Tools integrating with us as humans.

Emacs, our building material, lets us mould it in our image, all the way from the guts of the lisp machine, to packages we use, to how various modes play off each other, to interoperability with the OS and third party software, to our muscle memory and our brains ³.

The other lens: Integrating with how other things are integrated.

For example, a programming language along with its ecosystem. A plaintext notebook integrating TODOs, checklists, calendar, live code evaluation, export to other formats. A deep integration into code review and CI/CD ⁴.

Perhaps a third lens: is us integrating back with the development environment…

…by learning the tools, using them better, switching them out when we meet their limits. For example, Kotlin is IntelliJ-only. So I just port my daily driver Emacs shortcuts to IntelliJ, run it in "Zen" mode. I know it works because A) colleagues don't believe that it is, in fact, IntelliJ until I restore all the menus, file tree, and sundry visual cues, and because B) my muscle memory is oblivious to the differences. It simply continues taking care of me. ⁵

John Carmack opines

We can configure our Emacs for this kind of deep integration, with the help of specialist tools designed for each language + ecosystem + runtime.

I will make an appeal to authority to set the touchstone for IDE requirements.

Ok, motivation! Let's make working config now!

Integration Level One: Language as Plain Text

Programs, for better or worse, continue to be written in plain text. Our Emacs already is a pretty good general-purpose text editor that also "just works" for any text based programming language. Neat!

• projectile for project-aware directory navigation
• magit for version control
• avy + key-chord to fluidly navigate / select text units
• flyspell for spellchecks
• yasnippet for boilerplate templates
• expand-region for incremental selection of units of text
• multiple-cursors to edit structured text
• imenu to display top-level names (vars, methods)
• wgrep for grep-powered search/replace across multiple files
• eldoc to surface function doc-strings and argument lists
• paren + smartparens for structural editing support, which works uniformly across Lispy languages, as well as for data representations in most other languages (e.g. quoted strings, JSON data, python tuples and dicts etc.)

However we can and should do better. Much better! ⁶

Integration Level Two: Language as Structured Material (LSP FTW!)

We write and modify code not as plain text, but as structural elements of the programming language at hand; viz. its syntax, semantics, idioms, patterns, conventions, method or function call graphs, object hierarchies, compiler feedback etc.

Historically, language-aware editors and IDEs have been purpose-built for individual languages. Emacs has historically been extended to new languages, because we can ⁷. In all cases, everyone has had to implement the same set of features from scratch, such as, auto-complete, documentation on hover, go to definition, code browsing, project browsing etc.

Recently, the Language Server Protocol (LSP) Project changed the game, and quickly became foundational infrastructure for code editors ⁸. So much so that Emacs 29 baked in the eglot language server client. We also have the Emacs LSP project that gives us lsp-mode, and other packages with which to design a general-purpose IDE experience.

I have been itching to design my baseline programming workflow around LSP. Familiarity with Emacs lsp-mode led me to choose it. Though I will probably switch to eglot whenever I upgrade my Emacs. I am using:

• lsp-mode: Emacs client/library for the Language Server Protocol
• lsp-ui: inline UI display of flycheck diagnostics, LSP code actions, code lenses, documentation etc.
• lsp-ivy: interactive ivy interface to lsp-mode's workspace symbol functionality

Optionally (not sure about these two yet):

• lsp-treemacs: treemacs-driven views of project files, symbols, errors, call graphs etc.
• dap-mode: "Debug Adapter Protocol", optionally, to integrate language specific debuggers

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Programming languages
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(use-package lsp-mode
  ;; ref: https://emacs-lsp.github.io/lsp-mode/page/installation/#use-package
  :ensure t
  :init
  (setq lsp-keymap-prefix "C-c C-l")
  :hook ((clojure-mode clojurescript-mode clojurec-mode) . lsp-deferred)
  :hook (lsp-mode . lsp-enable-which-key-integration)
  :custom
  ;; LSP "workspace" dirs:
  ;; nb. "workspace" seems to be a confusing concept. It is a VSCode concept
  ;; that lsp-mode superseded as "session", because lsp-mode was already using
  ;; the word "workspace" in some other context. See: `lsp-describe-session'.
  ;; https://github.com/emacs-lsp/lsp-mode/discussions/3095
  ;; "workspace" directories still seem to server some purpose (no idea what),
  ;; and seem to be language specific.
  (lsp-clojure-workspace-dir
   (file-name-as-directory (expand-file-name "workspace"
                                             adi/dotemacs-dir)))
  :config
  (setq lsp-server-install-dir (file-name-as-directory ; install local to dotemacs
                                (expand-file-name "lsp" adi/dotemacs-cache-dir))
        ;; Perf. tweaks. Ref: https://emacs-lsp.github.io/lsp-mode/page/performance/
        lsp-idle-delay 0.500 ; bump higher if lsp-mode gets sluggish
        lsp-log-io nil
        ; lsp-enable-indentation nil ; set 'nil' to use cider indentation instead of lsp
        ; lsp-enable-completion-at-point nil; set 'nil' to use cider completion instead of lsp

        ;; No semgrep. https://emacs-lsp.github.io/lsp-mode/page/lsp-semgrep/
        ;; IDK why semgrep is on by default, docs are thin on configuring it
        ;; I don't want the error 'Command "semgrep lsp" is not present on the path.'
        ;; because I don't want to "pip install semgrep --user".
        lsp-semgrep-server-command nil)
  ;; LANGUAGE SPECIFIC SETTINGS
  ;; clojure-lsp: cf. https://clojure-lsp.io/clients/#emacs
  (add-to-list 'lsp-language-id-configuration
                `(clojurex-mode . "clojure"))
  :commands (lsp lsp-deferred))

(use-package lsp-ui
  :ensure t
  :after lsp-mode
  :commands lsp-ui-mode
  :bind (:map lsp-ui-mode-map ; h/t github.com/bbatsov/prelude
              ([remap xref-find-definitions] . #'lsp-ui-peek-find-definitions)
              ([remap xref-find-references] . #'lsp-ui-peek-find-references)
              ("C-c C-l ." . 'lsp-ui-peek-find-definitions)
              ("C-c C-l ?" . 'lsp-ui-peek-find-references)
              ("C-c C-l r" . 'lsp-rename)
              ("C-c C-l x" . 'lsp-workspace-restart)
              ("C-c C-l w" . 'lsp-ui-peek-find-workspace-symbol)
              ("C-c C-l i" . 'lsp-ui-peek-find-implementation)
              ("C-c C-l d" . 'lsp-describe-thing-at-point)
              ("C-c C-l e" . 'lsp-execute-code-action))
  :config
  ; h/t github.com/bbatsov/prelude
  (setq lsp-ui-sideline-enable t
        lsp-ui-doc-enable t
        lsp-ui-peek-enable t
        lsp-ui-peek-always-show t))

(use-package lsp-ivy
  :ensure t
  :after lsp-mode
  :commands lsp-ivy-workspace-symbol)

;; treemacs is cool, but I'm not sure I want it yet.
;; cf: https://github.com/emacs-lsp/lsp-treemacs
;; and https://github.com/Alexander-Miller/treemacs
;; (use-package lsp-treemacs
;;   :ensure t
;;   :after lsp-mode
;;   :commands lsp-treemacs-errors-list
;;   :config
;;   (setq treemacs-space-between-root-nodes nil))

;; dap-mode, optionally to use LANGUAGE-specific debuggers
;; cf. https://emacs-lsp.github.io/lsp-mode/page/installation/#use-package
;; (use-package dap-mode)
;; (use-package dap-LANGUAGE :ensue t :after dap-mode) ; load dap adapter for LANGUAGE

Integration Level Three: A language and its ecosystem: Clojure(Script) FTW!

LSP can solve a good chunk of the overall IDE design problem, but plenty is left for more specialised tools. Often, a language will have properties, runtimes, ecosystems which is out of scope for the language server project. For those needs, language-specific Emacs packages will likely be available.

Using Clojure as a practical example, here are some features we expect.

• Editing and linting support for all Clojure dialects, viz. Clojure, ClojureScript, Babashka, Cljc files, and EDN.
• Tight integration with REPLs, test tools, debugger, profiler.
• And hopefully some automagical support for Clojure-like languages such as janet-lang, ferret-lang.

Here are two demos of rock-solid professional Clojure IDE experiences.

We are well-served by the clojure-emacs and clojure-lsp projects. These, with allied packages enhance our Emacs's baseline programming experience, with Clojure focused features and capabilities.

clojure-mode: foundational Clojure programming support

This major mode provides syntax highlighting, indentation, navigation, and refactoring support for Clojure, ClojureScript, and mixed cljc code.

It also provides language-specific hooks that other modes can use to add more behaviours and features. For example, I want to defer the start of language servers until after a major mode is activated, so I set the hook in the lsp-mode config (see above).

(use-package clojure-mode
  ;; Brings in clojure-mode for Clj, clojurescript-mode for Cljs,
  ;; and clojurec-mode for Cljc
  :ensure t
  ;; Hook into subword-mode to work with CamelCase tokens like Java classes
  ;; h/t suvratapte/dot-emacs-dot-d
  :hook ((clojure-mode . subword-mode)
         (clojure-mode . yas-minor-mode))

  :config
  (setq clojure-indent-style 'align-arguments)
  :blackout "Clj")

clojure-lsp: for a static-analysis style workflow, no REPL needed

clojure-lsp is a language server, and Emacs lsp-mode already knows about it (see configuration above).

clojure-lsp helps us "navigate, identify and fix errors, perform refactors and much more!", with features like:

• Autocomplete
• Jump to definition/implementation
• Find references
• Renaming
• Code actions
• Errors
• Automatic ns cleaning
• Lots of Refactorings
• Code lens
• Semantic tokens (syntax highlighting)
• Linting (via clj-kondo)
• Call hierarchy
• Java interop

All of this is available without even firing up a Clojure REPL, or adding any other package. For example, if we used only clojure-mode, we would have had to add linting support with flycheck-clj-kondo for Clojure and flycheck-joker for ClojureScript. And if we used only CIDER, then for better auto-complete using CIDER, we would use ac-cider.

However static analysis is no substitute for programming Clojure interactively, against a live REPL. The excellent CIDER package enhances that for us.

CIDER: Putting the "Interactive" in the IDE

In the nearly half century old tradition of Lisps, Smalltalks, and APLs, Clojure programming is a highly interactive exercise. We converse with the live runtime. Supporting this requires its own kind of tooling.

CIDER: is "the Clojure(Script) Interactive Development Environment that Rocks!". Wherever CIDER and clojure-lsp offer similar features, I pick clojure-lsp, only adding what is unique to CIDER, viz.:

• Enhanced REPL experience and REPL session manager
• Value inspector
• Interactive Debugger
• Profiler (CIDER profiler)
• Session tracking (cider-spy + cider-spy-nrepl)
• Test runner integration
• Minibuffer code evaluation
• Macro expansion
• Smart namespace reloading
• Refactor intelligently with clj-refactor

Optionally, I will bring in clj-refactor to patch up corner cases where cider, and lsp-mode may both fall short.

(use-package clojure-mode
  ;; Brings in clojure-mode for Clj, clojurescript-mode for Cljs,
  ;; and clojurec-mode for Cljc
  :ensure t
  ;; Hook into subword-mode to work with CamelCase tokens like Java classes
  ;; h/t suvratapte/dot-emacs-dot-d
  :hook ((clojure-mode . subword-mode)
         (clojure-mode . yas-minor-mode))

  :config
  (setq clojure-indent-style 'align-arguments)
  :blackout "Clj")

(use-package cider
  ;; Note: Ensure CIDER and lsp-mode play well together, as we use both.
  ;; - LSP for more static-analysis-y services (completions, lookups, errors etc.),
  ;; - CIDER for "live" runtime services (enhanced REPL, interactive debugger etc.).
  :ensure t
  :after clojure-mode
  :init
  ;; Use clojure-lsp for eldoc and completions
  ;; h/t cider docs and ericdallo/dotfiles/.config/doom/config.el
  (remove-hook 'eldoc-documentation-functions #'cider-eldoc)
  (remove-hook 'completion-at-point-functions #'cider-complete-at-point)
  :custom
  (cider-preferred-build-tool 'clj)
  :bind
  (:map cider-mode-map
        ("C-c C-l" . nil))
  :config
  ;; settings h/t suvratapte/dot-emacs-dot-d
  (setq cider-repl-pop-to-buffer-on-connect nil
        cider-show-error-buffer t
        cider-auto-select-error-buffer t
        cider-repl-history-file (expand-file-name "cider-history"
                                                  adi/dotemacs-savefile-dir)
        cider-repl-wrap-history t
        cider-prompt-for-symbol nil
        cider-repl-use-pretty-printing t
        nrepl-log-messages nil
        ;; play nice with lsp-mode
        ;; h/t ericdallo/dotfiles/.config/doom/config.el
        cider-font-lock-dynamically nil ; use lsp semantic tokens
        cider-eldoc-display-for-symbol-at-point nil ; use lsp
        cider-prompt-for-symbol nil ; use lsp
        cider-use-xref nil ; use lsp
        ;; Maybe customize variables for cider-jack-in
        ;; https://docs.cider.mx/cider/basics/up_and_running.html
        )
  :blackout)

;; clj-refactor can go where clojure-lsp refactor can't go
(use-package clj-refactor
  ;; config h/t ericdallo/dotfiles doom emacs config
  :after clojure-mode
  :config
  (setq cljr-warn-on-eval nil
        cljr-eagerly-build-asts-on-startup nil
        cljr-add-ns-to-blank-clj-files nil ; use lsp
        cljr-magic-require-namespaces
        '(("s"   . "schema.core")
          ("pp" . "clojure.pprint"))))

Clojure with org-mode for live demos and more

As it happens, I do all my conference talks as live demos (What can I say, I like to live dangerously and embrace the demofails :)). The upshot of using org-mode is that I can publish my talks as plaintext org files that others can read or use, as well as static PDF or html files, optionally with in-line "results capture".

Here is one such talk I gave last year, and its associated blog post.

(use-package org
  :ensure nil
  :config
  (setq org-export-coding-system 'utf-8
        org-babel-clojure-backend 'cider)
  (org-babel-do-load-languages
   'org-babel-load-languages
   '((shell . t)
     (clojure . t)
     (clojurescript . t)
     (sql .t )
     (sqlite . t)
     (plantuml . t))))

(use-package ob-clojurescript
  :blackout)

(use-package org-tree-slide
  ;; Simple org outline based presentation mode
  ;; ref: https://github.com/takaxp/org-tree-slide
  :ensure t
  :bind (("<f8>" . 'org-tree-slide-mode)
         ("S-<f8>" . 'org-tree-slide-skip-done-toggle)
         :map org-tree-slide-mode-map
         ("<f9>" . 'org-tree-slide-move-previous-tree)
         ("<f10>" . 'org-tree-slide-move-next-tree)
         ("<f11>" . 'org-tree-slide-content))
  :config
  (setq org-tree-slide-skip-outline-level 4))

Assist Emacs with Graphical Interactive Development

Clojure programs model the world in terms of composite data structures; hash-maps, vectors, sequences, streams and so forth. And we also access host platform objects, classes, metadata. And we also like to traverse / inspect / visualise any of those entities from different angles, to further our understanding of what's actually going on in the live runtime.

Emacs is great for text UIs (e.g. magit), but not for rich graphical UIs. CIDER affords navigable views into much of this stuff right inside Emacs. And graphical tools like vlaaad's Reveal, Cognitect REBL by Datomic Team, or Chris Badahdah's browser-based portal data navigator level up our runtime visualisations and interactions to a whole other level.

Wire everything up using Clojure Deps and CLI tools

And finally, we have to integrate back with our programming ecosystem's build tooling. I have chosen to switch away from Leiningen, to the new-ish CLI tools by the Clojure team, that come bundled with Clojure these days. From the official Deps and CLI Guide:

Clojure provides command line tools for:

• Running an interactive REPL (Read-Eval-Print Loop)
• Running Clojure programs
• Evaluating Clojure expressions

My CLI tools invocation looks like this:

adi@tardis:~/src/github/adityaathalye/planet_coloniser
\_$ clj -M:dev/base:repl/clj:repl/reveal
nREPL server started on port 34139 on host localhost - nrepl://localhost:34139
nREPL 1.0.0
Clojure 1.10.3
OpenJDK 64-Bit Server VM 18.0.2-ea+9-Ubuntu-222.04
Interrupt: Control+C
Exit:      Control+D or (exit) or (quit)
user=>

And below is my global deps.edn configuration that declare a few standard clj aliases (e.g. :dev/base, :repl/clj, :repl/reveal) that I want across all Clojure(Script) projects. These configurations let our Emacs play well with CIDER middleware, clj-refactor, Reveal etc.

{;; Ref. deps configs by Sean Corfield and practicalli
 ;; Lots of stuff taken from there.
 ;; https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn
 ;; https://github.com/practicalli/clojure-cli-config/blob/main/deps.edn

 ;; Enumerate aliases as:
 ;; clj -T:deps aliases

 ;; Start a typical session as:
 ;; clj -M:dev/base:repl/clj:repl/reveal

 :paths ["src"]

 :mvn/repos
 {"central" {:url "https://repo1.maven.org/maven2/"}
  "clojars" {:url "https://repo.clojars.org/"}}

 :aliases
 {:build {:deps {io.github.clojure/tools.build {:mvn/version "0.9.5"}}
          :ns-default build}

  ;; LOCAL DEVELOPMENT
  :dev/base
  {:extra-paths ["dev"]}

  :dev/test
  {:extra-paths ["test"]
   :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                io.github.cognitect-labs/test-runner {:git/tag "v0.5.1" :git/sha "dfb30dd"}}}

  :dev/tools
  {:extra-deps {criterium/criterium {:mvn/version "RELEASE"}}}

  ;; CIDER AND REPL THINGS
  ;; https://docs.cider.mx/cider/basics/middleware_setup.html
  ;; nb. we should get nrepl and refactor-nrepl bundled with cider-nrepl
  ;; - for clojure-only: clj -M:repl/clj
  ;; - for clj and cljs: clj -M:repl/clj:repl/cljs
  ;; - for reveal GUI:   clj -M:repl/clj:repl/cljs:repl/reveal
  :repl/clj
  {:extra-deps {cider/cider-nrepl {:mvn/version "0.37.0"}
                org.clojure/tools.deps {:mvn/version "0.18.1354"}
                org.clojure/tools.namespace {:mvn/version "RELEASE"}}
   :main-opts ["-m" "nrepl.cmdline"
               "--middleware" "[cider.nrepl/cider-middleware]"
               "--interactive"]}

  :repl/cljs
  {:extra-deps {org.clojure/clojurescript {:mvn/version "1.10.339"}
                cider/piggieback {:mvn/version "0.5.3"}}
   :main-opts ["-m" "nrepl.cmdline"
               "--middleware" "[cider.piggieback/wrap-cljs-repl]"
               "--interactive"]}

  ;; :repl/refactor
  ;; {:extra-deps {refactor-nrepl/refactor-nrepl {:mvn/version "3.9.0"}}
  ;;  :main-opts ["--middleware" "[refactor-nrepl.middleware/wrap-refactor]"]}

  :repl/reveal ;; https://vlaaad.github.io/reveal/setup#clj--nrepl-middleware
  {:extra-deps {vlaaad/reveal {:mvn/version "1.3.280"}}
   ;; optional: preferences
   ;; :jvm-opts ["-Dvlaaad.reveal.prefs={:theme,:light}"]
   :main-opts ["-m" "nrepl.cmdline"
               "--middleware" "[vlaaad.reveal.nrepl/middleware]"
               "--interactive"]}}

Bonus demos and references

Clojure tooling evolution

The evolution of the Emacs tooling for Clojure (2014), Bozhidar Batsov.

A session dedicated to the evolution of CIDER (the Clojure dev environment for Emacs) and all the new features that have been added since I took over the project exactly one year ago.

CIDER was already cool circa 2014, and has come a long way since then!

The Future of Clojure Tooling (2018), Bozhidar Batsov is a nice follow-up to his 2014 talk. By 2018, LSP had arrived and debates about "REPL Powered" versus "Static Analysis Powered" tooling were in the air. We now have the best of both worlds, because even though those world overlap, they are not, in fact, in conflict!

Clojurists like Static Analysis and REPL Powered Tools

As you have seen so far, both LSP and CIDER enhance our Clojure programming life in Emacs. The fantastic Cursive IDE for IntelliJ is the original model of a seamless static+REPL powered Clojure IDE. If I weren't already invested in Emacs, I would have picked Cursive.

Cursive: A different type of IDE (2014) , Colin Fleming.

In contrast to the majority of Clojure development environments, Cursive uses static analysis of the source code to work its magic rather than taking advantage of the REPL.

Debugging Clojure Code With Cursive, Colin Fleming

A very good showcase of a rock-solid debugger and debugging experience. Cursive provides a complete JVM debugger based on the one provided in IntelliJ, including breakpoints, stepping and expression evaluation.

Emacs is a first-rate C++ IDE, John

I feel compelled to include this section because I set up this post by quoting famous C++ developer and a lover of great IDEs, John Carmack.

Even without LSP, Emacs was configurable as a first-rate IDE for large scale codebases: CppCon 2015: Emacs as a C++ IDE - Atila Neves.

Now that we have LSP, the C++ IDE story is even better!

Our plan is complete!

We have come a long way!

All the big pieces are in place, and the general design looks good to me. I will add enhancements, and fix bad ideas and bugs as I go along.

• [✓] Set the very preliminaries.
• [✓] Set up package management. I'll probably stick with the old familiars; elpa and melpa. I'm not sure about straight.el at this time.
• [✓] Choose use-package to get and configure each package. I like how neat configs are, when defined with use-package.
• [✓] Unexpectedly refactor the whole thing.
• [✓] Make completions and "getting about" work (the right mix of ivy, consul, swiper, company, helm, imenu). Someone mentioned newer alternatives to helm. Have a look at that.
• [✓] Fix general text editing stuff (keybindings, multiple cursors, snippets etc.)
• [✓] Add support for favourite programming languages.
  • [✓] Emacs Lisp (built-in + smartparens, eldoc etc.)
  • [✓] Clojure (clojure-mode, clojure-lsp, CIDER, and lispy editing packages)
  • [✓] Bash (lsp-bash)
  • Others will be configured on an as-needed basis.
• [✓] org-mode specifics that suit how I use org
  • [✓] org-babel (currently for Clojure, elisp, shell, SQL and some other langs)
  • [✓] tree-slide for presentations
• then let's see…

This was educational, fun, satisfying, and useful.

I published it, and call it done (for now)!

# https://github.com/adityaathalye/dotemacs
ln -s ~/src/github/adityaathalye/dotemacs ~/.emacs.d

Footnotes