Packages, Testing, and Continuous Integration

Florian Oswald

This is lecture is a slightly modified version of https://lectures.quantecon.org/jl/testing.html Thank you to the amazing Quantecon.org team!

This lecture discusses structuring a project as a Julia module, and testing it with tools from GitHub.

Benefits include:

  1. Specifying dependencies (and their versions) so that your project works across Julia setups and over time
  2. Being able to load your project’s functions from outside without copy/pasting
  3. Writing tests that run locally, and automatically on the GitHub server
  4. Having GitHub test your project across operating systems, Julia versions, etc.

Setting Up your Github Accounts

Travis CI

  • Travis is a service that automatically tests your projects.
  • First, we need to make sure that your GitHub account is set up with Travis CI and Codecov
  • As a reminder, make sure you signed up for the GitHub Student Developer Pack or Academic Plan if eligible
  • Navigate to the travis-ci.com website and click “sign up with GitHub” – supply your credentials
  • If you get stuck, see the Travis tutorial

Codecov

Codecov is a service that tells you how comprehensive your tests are (i.e., how much of your code is actually tested)

To sign up, visit the Codecov website, and click “sign up”

Next, click “add a repository” and enable private scope (this allows Codecov to service your private projects)

The result should be

This is all we need for now

Julia Setup

In [1]:
using InstantiateFromURL

# activate the QuantEcon environment
activate_github("QuantEcon/QuantEconLectureAllPackages", tag = "v0.9.7");

Note: Before these steps, make sure that you’ve either completed the version control lecture or run

Note: Throughout this lecture, important points and sequential workflow steps are listed as bullets

git config --global user.name "Your Name"

To set up a project on Julia:

In [2]:
using PkgTemplates
  • Create a template for your project

This specifies metadata like the license we’ll be using (MIT by default), the location (~/.julia/dev by default), etc.

In [4]:
ourTemplate = Template(;user="ScPo-CompEcon", plugins = [TravisCI(), Codecov()])
Out[4]:
Template:
  → User: ScPo-CompEcon
  → Host: github.com
  → License: MIT (florian oswald 2019)
  → Package directory: ~/.julia/dev
  → Minimum Julia version: v1.1
  → SSH remote: No
  → Commit Manifest.toml: No
  → Plugins:
    • Codecov:
      → Config file: None
      → 3 gitignore entries: "*.jl.cov", "*.jl.*.cov", "*.jl.mem"
    • TravisCI:
      → Config file: Default
      → 0 gitignore entries

Note: Make sure you replace the ScPo-CompEcon with your GitHub ID

  • Create a specific project based off this template
In [6]:
generate("ExamplePackage.jl", ourTemplate)
Generating project ExamplePackage:
    /Users/florian.oswald/.julia/dev/ExamplePackage/Project.toml
    /Users/florian.oswald/.julia/dev/ExamplePackage/src/ExamplePackage.jl
┌ Info: Initialized Git repo at /Users/florian.oswald/.julia/dev/ExamplePackage
└ @ PkgTemplates /Users/florian.oswald/.julia/packages/PkgTemplates/YbOlC/src/generate.jl:29
┌ Info: Set remote origin to https://github.com/ScPo-CompEcon/ExamplePackage.jl
└ @ PkgTemplates /Users/florian.oswald/.julia/packages/PkgTemplates/YbOlC/src/generate.jl:48
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[1mFetching: [========================================>]  100.0 %.0 % Resolving package versions...
  Updating `~/.julia/dev/ExamplePackage/Project.toml`
  [8dfed614] + Test 
  Updating `~/.julia/dev/ExamplePackage/Manifest.toml`
  [2a0f44e3] + Base64 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/dev/ExamplePackage/Project.toml`
 [no changes]
  Updating `~/.julia/dev/ExamplePackage/Manifest.toml`
  [2a0f44e3] - Base64 
  [8ba89e20] - Distributed 
  [b77e0a4c] - InteractiveUtils 
  [56ddb016] - Logging 
  [d6f4376e] - Markdown 
  [9a3f8284] - Random 
  [9e88b42a] - Serialization 
  [6462fe0b] - Sockets 
  [8dfed614] - Test 
 Resolving package versions...
┌ Info: Committed 8 files/directories: src/, Project.toml, test/, REQUIRE, README.md, LICENSE, .travis.yml, .gitignore
└ @ PkgTemplates /Users/florian.oswald/.julia/packages/PkgTemplates/YbOlC/src/generate.jl:73
  Updating `~/.julia/environments/v1.1/Project.toml`
  [63b64609] + ExamplePackage v0.1.0 [`~/.julia/dev/ExamplePackage`]
  Updating `~/.julia/environments/v1.1/Manifest.toml`
  [63b64609] + ExamplePackage v0.1.0 [`~/.julia/dev/ExamplePackage`]
┌ Info: New package is at /Users/florian.oswald/.julia/dev/ExamplePackage
└ @ PkgTemplates /Users/florian.oswald/.julia/packages/PkgTemplates/YbOlC/src/generate.jl:84

If we navigate to the package directory (shown in the output), we should see something like the following

Note: On Mac, this may be hidden; you can either start a terminal, cd ~ and then cd .julia, or make hidden files visible in the Finder

Adding a Project to Github

The next step is to creat a repository online where to store our code.

  • go to github.com, login, and click on new repository

We’ll want the following settings

In particular

  • The repo you create should have the same name as the project we added
  • We should leave the boxes unchecked for the README.md, LICENSE, and .gitignore, since these are handled by PkgTemplates

Then, if you use Github Desktop

  • Drag and drop your folder from your ~/.julia/dev directory to GitHub Desktop
  • Click the “publish branch” button to upload your files to GitHub

Else, if you use the commandline, do

cd ~/.julia/dev/ExamplePackage
git add .
git commit -m 'initial commit'
git push origin master

If you navigate to your git repo (ours is here), you should see something like

Note: Be sure that you don’t separately clone the repo you just added to another location (i.e., to your desktop)

A key note is that you have some set of files on your local machine (here in ~/.julia/dev/ExamplePackage.jl) and git is plugged into those files

For convenience, you might want to create a shortcut to that location somewhere accessible

Adding a Project to the Julia Package Manager

  • We also want Julia’s package manager to be aware of the project
  • In order to make it visible to the package manager, we need to execute the Pkg-mode command dev

  • Open a REPL in the newly created project directory, either by noting the path printed above, or by running the following in a REPL

In [7]:
# first we go to where our package is on our computer
cd(joinpath(DEPOT_PATH[1], "dev", "ExamplePackage"))   # Note the lack of `.jl`!
# ] goes into pkg mode!
In [13]:
] activate
In [14]:
] dev .
 Resolving package versions...
  Updating `~/.julia/environments/v1.1/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.1/Manifest.toml`
 [no changes]
In [15]:
] st
    Status `~/.julia/environments/v1.1/Project.toml`
  [c52e3926] Atom v0.7.14
  [aae01518] BandedMatrices v0.8.1
  [6e4b80f9] BenchmarkTools v0.4.2
  [a134a8b2] BlackBoxOptim v0.4.0
  [ad839575] Blink v0.10.1
  [324d7699] CategoricalArrays v0.5.2
  [3da002f7] ColorTypes v0.7.5
  [34da2185] Compat v1.4.0
  [8f4d0f93] Conda v1.2.0
  [2023c74b] CourseMatch v0.1.0 [`../../dev/CourseMatch`]
  [a93c6f00] DataFrames v0.17.0
  [1313f7d8] DataFramesMeta v0.4.0
  [864edb3b] DataStructures v0.15.0
  [5721bf48] DataVoyager v0.3.1
  [01453d9d] DiffEqDiffTools v0.7.1
  [0c46a032] DifferentialEquations v6.3.0
  [31c24e10] Distributions v0.16.4
  [63b64609] ExamplePackage v0.1.0 [`../../dev/ExamplePackage`]
  [442a2c76] FastGaussQuadrature v0.3.2
  [1a297f60] FillArrays v0.4.0
  [9d5cd8c9] FixedEffectModels v0.7.1
  [587475ba] Flux v0.7.1
  [f6369f11] ForwardDiff v0.10.2
  [38e38edf] GLM v1.0.2
  [28b8d3ca] GR v0.37.0
  [7073ff75] IJulia v1.15.2
  [43edad99] InstantiateFromURL v0.2.1
  [a98d9a8b] Interpolations v0.11.1
  [b6b21f68] Ipopt v0.5.1
  [4076af6c] JuMP v0.18.5
  [e5e0dc1b] Juno v0.5.4
  [5ab0869b] KernelDensity v0.5.1
  [b964fa9f] LaTeXStrings v1.0.3
  [23fbe1c1] Latexify v0.6.0
  [5078a376] LazyArrays v0.5.1
  [0fc2ff8b] LeastSquaresOptim v0.7.1
  [093fc24a] LightGraphs v1.2.0
  [739be429] MbedTLS v0.6.8
  [76087f3c] NLopt v0.5.1
  [2774e3e8] NLsolve v3.0.1
  [47be7bcc] ORCA v0.2.1
  [429524aa] Optim v0.17.2
  [3b7a836e] PGFPlots v3.0.3
  [d96e819e] Parameters v0.10.3
  [14b8a8f1] PkgTemplates v0.4.1
  [f0f68f2c] PlotlyJS v0.12.3
  [91a5bcdd] Plots v0.22.5
  [f27b6e38] Polynomials v0.5.2
  [92933f4c] ProgressMeter v0.9.0
  [438e738f] PyCall v1.18.5
  [d330b81b] PyPlot v2.7.0
  [1fd47b50] QuadGK v2.0.3
  [fcd29c91] QuantEcon v0.15.0
  [1a8c2f83] Query v0.10.1
  [612083be] Queryverse v0.2.0
  [6f49c342] RCall v0.13.1
  [ce6b1742] RDatasets v0.6.1
  [d519eb52] RegressionTables v0.2.0
  [295af30f] Revise v1.0.2
  [f2b01f46] Roots v0.7.4
  [ed01d8cd] Sobol v1.1.0
  [60ddc479] StatPlots v0.9.1
  [90137ffa] StaticArrays v0.10.2
  [2913bbd2] StatsBase v0.27.0
  [3eaba693] StatsModels v0.4.0
  [37b6cedf] Traceur v0.2.1
  [0ae4a718] VegaDatasets v0.5.0
  [112f6efa] VegaLite v0.5.2
  [44d3d7a6] Weave v0.7.1
  [37e2e46d] LinearAlgebra 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [8dfed614] Test 

Using the Package Manager

Now, from any Julia terminal in the future, we can run

using ExamplePackage

To use its exported functions

We can also get the path to this by running

using ExamplePackage
pathof(ExamplePackage) # returns path to src/ExamplePackage.jl

Project Structure

Let’s unpack the structure of the generated project

  • The first directory, .git, holds the version control information
  • The src directory contains the project’s source code – it should contain only one file (ExamplePackage.jl), which reads
module ExamplePackage

greet() = print("Hello World!")

end # module
  • Likewise, the test directory should have only one file (runtests.jl), which reads
using ExamplePackage
using Test

@testset "ExamplePackage.jl" begin
    # Write your own tests here.
end

In particular, the workflow is to export objects we want to test (using ExamplePackage), and test them using Julia’s Test module

The other important text files for now are

  • Project.toml and Manifest.toml, which contain dependency information

In particular, the Project.toml contains a list of dependencies, and the Manifest.toml specifies their exact versions and sub-dependencies

  • The .gitignore file (which may display as an untitled file), which contains files and paths for git to ignore

Project Workflow and Dependencies

Environments

As before, the .toml files define an environment for our project, or a set of files which represent the dependency information

The actual files are written in the TOML language, which is a lightweight format to specify configuration options

This information is the name of every package we depend on, along with the exact versions of those packages

This information (in practice, the result of package operations we execute) will be reflected in our ExamplePackage.jl directory’s TOML, once that environment is activated (selected)

This allows us to share the project with others, who can exactly reproduce the state used to build and test it

See the Pkg3 docs for more information

Pkg Operations

For now, let’s just try adding a dependency

  • Activate the package environment (to be run from the base, v1.0 environment)
In [16]:
] activate ExamplePackage
This tells Julia to write the results of package operations to `ExampleProject`’s TOML,
and use the versions of packages specified there

Note that the base environment isn’t special, except that it’s what’s loaded by a freshly-started REPL or Jupyter notebook

  • Add a package
In [17]:
] add Expectations
 Resolving package versions...
  Updating `~/.julia/dev/ExamplePackage/Project.toml`
  [2fe49d83] + Expectations v1.0.2
  Updating `~/.julia/dev/ExamplePackage/Manifest.toml`
  [7d9fca2a] + Arpack v0.3.0
  [9e28174c] + BinDeps v0.8.10
  [b99e7846] + BinaryProvider v0.5.3
  [34da2185] + Compat v2.0.0
  [864edb3b] + DataStructures v0.15.0
  [31c24e10] + Distributions v0.16.4
  [2fe49d83] + Expectations v1.0.2
  [442a2c76] + FastGaussQuadrature v0.3.2
  [e1d29d7a] + Missings v0.4.0
  [bac558e1] + OrderedCollections v1.0.2
  [90014a1f] + PDMats v0.9.6
  [1fd47b50] + QuadGK v2.0.3
  [79098fc4] + Rmath v0.5.0
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v0.7.2
  [2913bbd2] + StatsBase v0.29.0
  [4c63d2b9] + StatsFuns v0.8.0
  [30578b45] + URIParser v0.4.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [4607b0f0] + SuiteSparse 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

We can track changes in the TOML, as before

Here’s the Manifest.toml

We can also run other operations, like ] up, ] precompile, etc.

Package operations are listed in detail in the tools and editors lecture

Recall that, to quit the active environment and return to the base (v1.0), simply run

In [18]:
] activate

Writing Code

The basic idea is to work in tests/runtests.jl, while reproducible functions should go in the src/ExamplePackage.jl

For example, let’s say we add Distributions.jl

In [19]:
] activate ExamplePackage
In [20]:
] add Distributions
 Resolving package versions...
  Updating `~/.julia/dev/ExamplePackage/Project.toml`
  [31c24e10] + Distributions v0.16.4
  Updating `~/.julia/dev/ExamplePackage/Manifest.toml`
 [no changes]

and edit the source (paste this into the file itself ) to read as follows

module ExamplePackage

greet() = print("Hello World!")

using Expectations, Distributions

function foo(μ = 1., σ = 2.)
    d = Normal(μ, σ)
    E = expectation(d)
    return E(x -> sin(x))
end

export foo

end # module
In [21]:
] activate
In [22]:
using ExamplePackage
ExamplePackage.greet()
┌ Info: Recompiling stale cache file /Users/florian.oswald/.julia/compiled/v1.1/ExamplePackage/bt9ZU.ji for ExamplePackage [63b64609-ed96-5e0f-b5ca-5e4fd3ab9311]
└ @ Base loading.jl:1184
Hello World!
foo() # exported, so don't need to qualify the namespace

Note: If you didn’t follow the instructions to add a startup file, you may need to quit your REPL and load the package again

Collaborative Work

For someone else to get the package, they simply need to

  • Run the following command
] dev https://github.com/ScPo-CompEcon/ExamplePackage.jl.git   
# using your github ID!

This will place the repository inside their ~/.julia/dev folder

  • Drag-and-drop the folder to GitHub desktop in the usual way

Recall that the path to your ~/.julia folder is

In [15]:
DEPOT_PATH[1]
Out[15]:
"/Users/florian.oswald/.julia"

They can then collaborate as they would on other git repositories

In particular, they can run

] activate ExamplePackage  # to activate
] instantiate  # to get all required packages onto their machine
] precompile   # precompile all packages
] test         # run the packages unit tests

Unit Testing

It’s important to make sure that your code is well-tested

There are a few different kinds of test, each with different purposes

  • Unit testing makes sure that individual pieces of a project work as expected
  • Integration testing makes sure that they fit together as expected
  • Regression testing makes sure that behavior is unchanged over time

In this lecture, we’ll focus on unit testing

In general, well-written unit tests (which also guard against regression, for example by comparing function output to hardcoded values) are sufficient for most small projects

The Test Module

Julia provides testing features through a built-in package called Test, which we get by using Test

The basic object is the macro @test

In [16]:
using Test
@test 1 == 1
@test 1  1
Out[16]:
Test Passed

Tests will pass if the condition is true, or fail otherwise

If a test is failing, we should flag it with @test_broken as below

In [17]:
@test_broken 1 == 2
Out[17]:
Test Broken
  Expression: 1 == 2

This way, we still have access to information about the test, instead of just deleting it or commenting it out

There are other test macros, that check for things like error handling and type-stability

Advanced users can check the Julia docs

Example

Let’s add some unit tests for the foo() function we defined earlier

Our tests/runtests.jl file should look like this

As before, this should be pasted into the file directly

using ExamplePackage
using Test

@test foo() == 0.11388071406436832
@test foo(1, 1.5) == 0.2731856314283442
@test_broken foo(1, 0) # tells us this is broken

And run it by typing ] test into an activated REPL (i.e., a REPL where you’ve run ] activate ExamplePackage)

Test Sets

By default, the runtests.jl folder starts off with a @testset

This is useful for organizing different batches of tests, but for now we can simply ignore it

To learn more about test sets, see the docs

Running Tests

There are a few different ways to run the tests for your package

  • Run the actual runtests.jl, say by hitting shift-enter on it in Atom
  • From a fresh (v1.0) REPL, run ] test ExamplePackage
  • From an activated (ExamplePackage) REPL, simply run ] test (recall that you can activate with ] activate ExamplePackage)

Continuous Integration with Travis

Setup

By default, Travis should have access to all your repositories and deploy automatically

This includes private repos if you’re on a student developer pack or an academic plan (Travis detects this automatically)

To change this, go to “settings” under your GitHub profile

Click “Applications,” then “Travis CI,” then “Configure,” and choose the repos you want to be tracked

Build Options

By default, Travis will compile and test your project (i.e., “build” it) for new commits and PRs for every tracked repo with a .travis.yml file

We can see ours by opening it in Atom

# Documentation: http://docs.travis-ci.com/user/languages/julia/
language: julia
os:
- linux
- osx
julia:
- 1.0
- nightly
matrix:
allow_failures:
    - julia: nightly
fast_finish: true
notifications:
email: false
after_success:
- julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'

This is telling Travis to build the project in Julia, on OSX and Linux, using Julia v1.0 and the latest (“nightly”)

It also says that if the nightly version doesn’t work, that shouldn’t register as a failure

Note You won’t need OSX unless you’re building something Mac-specific, like iOS or Swift

You can delete those lines to speed up the build, likewise for the nightly Julia version

Working with Builds

As above, builds are triggered whenever we push changes or open a pull request

For example, if we push our changes to the server and then click the Travis badge (the one which says “build”) on the README, we should see something like

Note that you may need to wait a bit and/or refresh your browser

This gives us an overview of all the builds running for that commit

To inspect a build more closely (say, if it fails), we can click on it and expand the log options

Note that the build times here aren’t informative, because we can’t generally control the hardware to which our job is allocated

We can also cancel specific jobs, either from their specific pages or by clicking the grey “x” button on the dashboard

Lastly, we can trigger builds manually (without a new commit or PR) from the Travis overview

To commit without triggering a build, simply add “[ci skip]” somewhere inside the commit message

Travis and Pull Requests

One key feature of Travis is the ability to see at-a-glance whether PRs pass tests before merging them

This happens automatically when Travis is enabled on a repository

For an example of this feature, see this PR in the Games.jl repository

Code Coverage

Beyond the success or failure of our test suite, we also want to know how much of our code the tests cover

The tool we use to do this is called Codecov

Setup

You’ll find that Codecov is automatically enabled for public repos with Travis

For private ones, you’ll need to first get an access token

Add private scope in the Codecov website, just like we did for Travis

Navigate to the repo settings page (i.e., https://codecov.io/gh/quanteconuser/ExamplePackage.jl/settings for our repo) and copy the token

Next, go to your Travis settings and add an environment variable as below

Interpreting Results

Click the Codecov badge to see the build page for your project

This shows us that our tests cover 50% of our functions in src//

Note: To get a more detailed view, we can click the src// and the resultant filename

Note: Codecov may take a few minutes to run for the first time

This shows us precisely which methods (and parts of methods) are untested

Additional Notes

  • The JuliaCI organization provides more Julia utilities for continuous integration and testing

Review

To review the workflow for creating, versioning, and testing a new project end-to-end

  1. Create the local package directory using the PkgTemplates.jl
  2. Add that package to the Julia package manager, by opening a Julia REPL in the ~/.julia/dev/ExamplePackage.jl, making sure the active environment is the default one (v1.0), and hitting ] dev .
  3. Drag-and-drop that folder to GitHub Desktop
  4. Create an empty repository with the same name on the GitHub server
  5. Push from GitHub Desktop to the server
  6. Open the original project folder (e.g., ~/.julia/dev/ExamplePackage.jl) in Atom / your editor
  7. Make changes, test, iterate on it, etc. As a rule, functions like should live in the src/ directory once they’re stable, and you should export them from that file with export func1, func2. This will export all methods of func1, func2, etc.
  8. Commit them in GitHub Desktop as you go (i.e., you can and should use version control to track intermediate states)
  9. Push to the server, and see the Travis and Codecov results (note that these may take a few minutes the first time)

Exercises

Exercise 1

Following the instructions for a new project, create a new package on your github account called NewtonsMethod.jl

In this package, you should create a simple package to do Newton’s Method using the code you did in the Newton’s method exercise in Introductory Examples

In particular, within your package you should have two functions

  • newtonroot(f, f′; x₀, tol = 1E-7, maxiter = 1000)
  • newtonroot(f; x₀, tol = 1E-7, maxiter = 1000)

Where the second function uses Automatic Differentiation to call the first.

The package should include

  • implementations of those functions in the /src directory
  • comprehensive set of tests
  • project and manifest files to replicate your development environment
  • automated running of the tests with Travis CI in GitHub

For the tests, you should have at the very minimum

  • a way to handle non-convergence (e.g. return back nothing as discussed in error handling
  • several @test for the root of a known function, given the f and analytical f' derivatives
  • tests of those roots using the automatic differentiation version of the function
  • test of finding those roots with a BigFloat and not just a Float64
  • test of non-convergence for a function without a root (e.g. $ f(x) = 2 + x^2 $ )
  • test to ensure that the maxiter is working (e.g. what happens if you call maxiter = 5
  • test to ensure that tol is working

And anything else you can think of. You should be able to run ] test for the project to check that the test-suite is running, and then ensure that it is running automatically on Travis CI

Push a commit to the repository which breaks one of the tests and see what the Travis CI reports after running the build