Julia Tools and Editors

Florian Oswald

This is lecture is a slightly modified version of https://lectures.quantecon.org/jl/tools_editors.html Thank you to the amazing Quantecon.org team!

Co-authored with Arnav Sood.

While Jupyter notebooks are a great way to get started with the language, eventually you will want to use more powerful tools

We’ll discuss a few of them here, such as

  • Text editors like Atom, which come with rich Julia support for debugging, documentation, git integration, plotting and inspecting data, and code execution
  • The Julia REPL, which has specialized modes for package management, shell commands, and help
  • The Docker setup we mentioned in getting started, which provides a painless pre-configured environment on your machine

Note that we assume you’ve already completed the getting started lecture

Preliminary Setup

Creating a Startup File (Recommended)

Whenever the Julia compiler or REPL starts, it will look for a file called startup.jl (see Julia Manual)

We provide a file here which does two things

  • Makes the REPL shell mode “sticky,” so you don’t need to keep running ; for new commands
  • Loads the Revise.jl package on startup, which lets you see changes you make to a package in real-time (i.e., no need to quit the REPL, open again, and load again)

The location for the file is relative to your default Julia environment (e.g. ~/.julia/config/startup.jl or C:\Users\USERNAME\.julia\config\startup.jl on Windows)

Recall that you can find the location of the ~/.julia directory by running

In [1]:
DEPOT_PATH[1]
Out[1]:
"/Users/florian.oswald/.julia"

Note: On Mac, this won’t be visible in the Finder unless you specifically enable that option, but you can get to it by running cd .julia; open . from a new terminal. Alternatively, with your Finder window open at HOME, just press Shift + CMD + . to see all hidden files.

To add Revise.jl:

  • In the julia terminal, type the following
] add Revise REPL; precompile
  • Create the ~/.julia/config/ directory if necessary in the terminal or file explorer
  • Download the file startup.jl into that directory
  • For convenience, you may find it useful on your operating system to change the directory where the REPL starts

On Windows, if you have a shortcut on your desktop or on the taskbar, you could: (1) right-click on the icon; (2) right click on the “julia” text; (3) choose “Properties”, and (4) change the “Start In” to be something such as C:\Users\YOURUSERNAME\Documents

The REPL

Previously, we discussed basic use of the Julia REPL (“Read-Evaluate-Print Loop”)

Here, we’ll consider some more advanced features

Shell Mode

  • Hitting ; brings you into shell mode, which lets you run bash commands (PowerShell on Windows)
  • Notice that by default the REPL goes back to standard mode after your shell command exists - not so with our modification from above in startup.jl!
  • To go back to standard mode, type CTRL + c
In [2]:
; pwd
/Users/florian.oswald/Dropbox/teaching/ScPo/ScPo-CompEcon/CoursePack/Notebooks

You can also use Julia variables from shell mode: the shell can "read" the julia variable x!

In [3]:
x = 2
Out[3]:
2
In [4]:
; echo $x
2

Package Mode

Hitting ] brings you into package mode

  • ] add Expectations will add a package (here, Expectations.jl)
  • Likewise, ] rm Expectations will remove that package
  • ] st will show you a snapshot of what you have installed
  • ] up will (intelligently) upgrade versions of your packages
  • ] precompile will precompile everytihng possible

You can get a full list of package mode commands by running

In [5]:
] ?
  Welcome to the Pkg REPL-mode. To return to the julia> prompt, either press
  backspace when the input line is empty or press Ctrl+C.

  Synopsis

  pkg> cmd [opts] [args]

  Multiple commands can be given on the same line by interleaving a ; between
  the commands.

  Commands

  activate: set the primary environment the package manager manipulates

  add: add packages to project

  build: run the build script for packages

  develop: clone the full package repo locally for development

  free: undoes a pin, develop, or stops tracking a repo

  gc: garbage collect packages not used for a significant time

  generate: generate files for a new project

  help: show this message

  instantiate: downloads all the dependencies for the project

  pin: pins the version of packages

  precompile: precompile all the project dependencies

  preview: previews a subsequent command without affecting the current state

  registry-add: add package registries

  registry-remove: remove package registries

  registry-status: information about installed registries

  registry-update: update package registries

  remove: remove packages from project or manifest

  resolve: resolves to update the manifest from changes in dependencies of
  developed packages

  status: summarize contents of and changes to environment

  test: run tests for packages

  update: update packages in manifest

On some operating systems (such as OSX) REPL pasting may not work for package mode, and you will need to access it in the standard way (i.e., hit ] first and then run your commands)

Help Mode

Hitting ? will bring you into help mode

The key use case is to find docstrings for functions and macros, e.g.

? print

Note that objects must be loaded for Julia to return their documentation, e.g.

? @test

will fail, but

using Test
? @test

will succeed.

Atom

  • As discussed previously, eventually you will want to use a fully fledged text editor.
  • The most feature-rich one for Julia development is Atom, with the Juno package

There are several reasons to use a text editor like Atom, including

  • Git integration (more on this in the next lecture)
  • Painless inspection of variables and data
  • Easily run code blocks, and drop in custom snippets of code
  • Integration with Julia documentation and plots

  • sublime is another good alternative (much fewer features than Juno, but very stable. Just send lines of code into the console with sendText)

Installation and Configuration

  1. Download and Install Atom from the Atom website
  2. (Optional, but recommended): Change default Atom settings

    • Use Ctrl-, (CMD , on Mac) to get the Settings pane
    • Choose the Packages tab
    • Type line-ending-selector into the Filter and then click “Settings” for that package

      • Change the default line ending to LF (only necessary on Windows)
    • Choose the Editor tab

      • Turn on Soft Wrap
      • Set the Tab Length default to 4

Installing Juno

  1. Use Ctrl-, to get the Settings pane
  2. Go to the Install tab
  3. Type uber-juno into the search box and then click Install on the package that appears
  4. Wait while Juno installs dependencies
  5. When it asks you whether or not to use the standard layout, click yes

At that point, you should see a built-in REPL at the bottom of the screen and be able to start using Julia and Atom

Troubleshooting

Sometimes, Juno will fail to find the Julia executable (say, if it’s installed somewhere nonstandard, or you have multiple)

To do this

  1. Ctrl-, to get Settings pane, and select the Packages tab
  2. Type in julia-client and choose Settings
  3. Find the Julia Path, and fill it in with the location of the Julia binary
  • To find the binary, you could run Sys.BINDIR in the REPL, then add in an additional /julia to the end of the screen output
  • e.g. C:\Users\YOURUSERNAME\AppData\Local\Julia-1.0.1\bin\julia.exe on Windows as /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia on OSX

See the setup instructions for Juno if you have further issues

Standard Layout

If you follow the instructions, you should see something like this when you open a new file

If you don’t, simply go to the command palette and type “Julia standard layout”

The bottom pane is a standard REPL, which supports the different modes above

The “workspace” pane is a snapshot of currently-defined objects

For example, if we define an object in the REPL

In [6]:
x = 2
Out[6]:
2

Our workspace should read

The ans variable simply captures the result of the last computation

The Documentation pane simply lets us query Julia documentation

The Plots pane captures Julia plots output (the code is as follows)

using Plots
gr(fmt = :png);
data = rand(10, 10)
h = heatmap(data)

Note: The plots feature is not perfectly reliable across all plotting backends, see the Basic Usage page

Other Features

  • Shift + Enter will evaluate a highlighted selection or line (as above)
  • The run symbol in the left sidebar (or Ctrl+Shift+Enter) will run the whole file

See basic usage for an exploration of features, and the FAQ for more advanced steps (e.g. using with Docker)

Package Environments

  • Julia’s package manager lets you set up Python-style “virtualenvs,” or subsets of packages that draw from an underlying pool of assets on the machine.
  • This way, you can work with (and specify) the dependencies (i.e., required packages) for one project without worrying about impacts on other projects.
    • Your project A may depend on package xzy
    • Your project B, however, may depend on an old version of package xzy!
  • This is fundamental to ensure reproducibility of results!

  • An environment is a set of packages specified by a Project.toml (and optionally, a Manifest.toml)

  • A registry is a git repository corresponding to a list of (typically) registered packages, from which Julia can pull (for more on git repositories, see version control)
  • A depot is a directory, like ~/.julia, which contains assets (compile caches, registries, package source directories, etc.)

Essentially, an environment is a dependency tree for a project, or a “frame of mind” for Julia’s package manager

  • We can see the default (v1.0) environment as such:
In [7]:
] st
    Status `~/.julia/environments/v1.1/Project.toml`
  [aae01518] BandedMatrices v0.8.1
  [6e4b80f9] BenchmarkTools v0.4.2
  [a134a8b2] BlackBoxOptim v0.4.0
  [324d7699] CategoricalArrays v0.5.2
  [3da002f7] ColorTypes v0.7.5
  [34da2185] Compat v1.4.0
  [a93c6f00] DataFrames v0.17.0
  [1313f7d8] DataFramesMeta v0.4.0
  [5721bf48] DataVoyager v0.3.1
  [01453d9d] DiffEqDiffTools v0.7.1
  [31c24e10] Distributions v0.16.4
  [2fe49d83] Expectations v1.0.2
  [442a2c76] FastGaussQuadrature v0.3.2
  [1a297f60] FillArrays v0.4.0
  [9d5cd8c9] FixedEffectModels v0.7.1
  [587475ba] Flux v0.7.1
  [f6369f11] ForwardDiff v0.10.2
  [38e38edf] GLM v1.0.2
  [28b8d3ca] GR v0.37.0
  [7073ff75] IJulia v1.15.2
  [43edad99] InstantiateFromURL v0.2.1
  [a98d9a8b] Interpolations v0.11.1
  [b6b21f68] Ipopt v0.5.1
  [4076af6c] JuMP v0.18.5
  [5ab0869b] KernelDensity v0.5.1
  [b964fa9f] LaTeXStrings v1.0.3
  [23fbe1c1] Latexify v0.6.0
  [5078a376] LazyArrays v0.5.1
  [0fc2ff8b] LeastSquaresOptim v0.7.1
  [093fc24a] LightGraphs v1.2.0
  [76087f3c] NLopt v0.5.1
  [2774e3e8] NLsolve v3.0.1
  [429524aa] Optim v0.17.2
  [d96e819e] Parameters v0.10.3
  [14b8a8f1] PkgTemplates v0.4.1
  [91a5bcdd] Plots v0.22.5
  [f27b6e38] Polynomials v0.5.2
  [92933f4c] ProgressMeter v0.9.0
  [1fd47b50] QuadGK v2.0.3
  [fcd29c91] QuantEcon v0.15.0
  [1a8c2f83] Query v0.10.1
  [612083be] Queryverse v0.2.0
  [ce6b1742] RDatasets v0.6.1
  [d519eb52] RegressionTables v0.2.0
  [295af30f] Revise v1.0.2
  [f2b01f46] Roots v0.7.4
  [60ddc479] StatPlots v0.9.1
  [90137ffa] StaticArrays v0.10.2
  [2913bbd2] StatsBase v0.27.0
  [3eaba693] StatsModels v0.4.0
  [37b6cedf] Traceur v0.2.1
  [0ae4a718] VegaDatasets v0.5.0
  [112f6efa] VegaLite v0.5.2
  [37e2e46d] LinearAlgebra 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [8dfed614] Test 
  • We can also create and activate a new environment
In [8]:
] generate ExampleEnvironment
Generating project ExampleEnvironment:
    ExampleEnvironment/Project.toml
    ExampleEnvironment/src/ExampleEnvironment.jl
  • And go to it
In [9]:
; cd ExampleEnvironment
/Users/florian.oswald/Dropbox/teaching/ScPo/ScPo-CompEcon/CoursePack/Notebooks/ExampleEnvironment
  • To activate the directory, simply
In [10]:
] activate .

where “.” stands in for the “present working directory”

  • Let’s make some changes to this Environment now
  • after activate-ing and environment, all package manager actions are applied to that environment:
In [11]:
] add Expectations Parameters
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[1mFetching: [========================================>]  100.0 %.0 % Resolving package versions...
  Updating `~/Dropbox/teaching/ScPo/ScPo-CompEcon/CoursePack/Notebooks/ExampleEnvironment/Project.toml`
  [2fe49d83] + Expectations v1.0.2
  [d96e819e] + Parameters v0.10.3
  Updating `~/Dropbox/teaching/ScPo/ScPo-CompEcon/CoursePack/Notebooks/ExampleEnvironment/Manifest.toml`
  [7d9fca2a] + Arpack v0.3.0
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.3
  [34da2185] + Compat v1.4.0
  [864edb3b] + DataStructures v0.15.0
  [31c24e10] + Distributions v0.16.4
  [2fe49d83] + Expectations v1.0.2
  [442a2c76] + FastGaussQuadrature v0.3.2
  [e1d29d7a] + Missings v0.4.0
  [bac558e1] + OrderedCollections v1.0.2
  [90014a1f] + PDMats v0.9.6
  [d96e819e] + Parameters v0.10.3
  [1fd47b50] + QuadGK v2.0.3
  [79098fc4] + Rmath v0.5.0
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v0.7.2
  [2913bbd2] + StatsBase v0.27.0
  [4c63d2b9] + StatsFuns v0.7.0
  [30578b45] + URIParser v0.4.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [4607b0f0] + SuiteSparse 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

Note the lack of commas!

  • To see the changes, simply open the ExampleEnvironment directory in an editor like Atom

The Project TOML should look something like this

name = "ExampleEnvironment"
uuid = "f92c96ca-1a46-11e9-1b82-d538e194f99a"
authors = ["Florian Oswald <florian.oswald@gmail.com>"]
version = "0.1.0"

[deps]
Expectations = "2fe49d83-0758-5602-8f54-1f90ad0d522b"
Parameters = "d96e819e-fc66-5662-9728-84c9c7592b0a"

We can also

In [12]:
] precompile
Precompiling project...
Precompiling Parameters
┌ Info: Recompiling stale cache file /Users/florian.oswald/.julia/compiled/v1.1/Parameters/ycYNs.ji for Parameters [d96e819e-fc66-5662-9728-84c9c7592b0a]
└ @ Base loading.jl:1184
Precompiling ExampleEnvironment
┌ Info: Precompiling ExampleEnvironment [5f706bf6-2171-11e9-03f1-797435e3197f]
└ @ Base loading.jl:1186

Note The TOML files are independent of the actual assets (which live in ~/.julia/packages, ~/.julia/dev, and ~/.julia/compiled)

You can think of the TOML as specifying demands for resources, which are supplied by the ~/.julia user depot

  • To return to the default Julia environment, simply
    ] activate
    
    without any arguments!
  • Finally, to clean up:
    ; cd ..
      ; rm -rf ExampleEnvironment
    

InstantiateFromURL

With this knowledge, we can explain the operation of the setup block

using InstantiateFromURL

# activate the QuantEcon environment
activate_github("QuantEcon/QuantEconLecturePackages", tag = "v0.9.5");

What this activate_github function does is

  1. Download the TOML from that repo to a directory called .projects
  2. ] activate that environment, and
  3. ] instantiate and ] precompile, if necessary