Packages, Testing, and Continuous Integration¶
Florian Oswald
This is lecture is a slightly modified version of https://lectures.quantecon.org/jl/testing.html Thank you to the amazing Quantecon.org team!
This lecture discusses structuring a project as a Julia module, and testing it with tools from GitHub.
Benefits include:
- Specifying dependencies (and their versions) so that your project works across Julia setups and over time
- Being able to load your project’s functions from outside without copy/pasting
- Writing tests that run locally, and automatically on the GitHub server
- Having GitHub test your project across operating systems, Julia versions, etc.
Setting Up your Github Accounts¶
Travis CI¶
- Travis is a service that automatically tests your projects.
- First, we need to make sure that your GitHub account is set up with Travis CI and Codecov
- As a reminder, make sure you signed up for the GitHub Student Developer Pack or Academic Plan if eligible
- Navigate to the travis-ci.com website and click “sign up with GitHub” – supply your credentials
- If you get stuck, see the Travis tutorial
Codecov¶
Codecov is a service that tells you how comprehensive your tests are (i.e., how much of your code is actually tested)
To sign up, visit the Codecov website, and click “sign up”
Next, click “add a repository” and enable private scope (this allows Codecov to service your private projects)
The result should be
This is all we need for now
Julia Setup¶
using InstantiateFromURL
# activate the QuantEcon environment
activate_github("QuantEcon/QuantEconLectureAllPackages", tag = "v0.9.7");
Note: Before these steps, make sure that you’ve either completed the version control lecture or run
Note: Throughout this lecture, important points and sequential workflow steps are listed as bullets
git config --global user.name "Your Name"
To set up a project on Julia:
- Load the PkgTemplates package
using PkgTemplates
- Create a template for your project
This specifies metadata like the license we’ll be using (MIT by default), the location (~/.julia/dev by default), etc.
ourTemplate = Template(;user="ScPo-CompEcon", plugins = [TravisCI(), Codecov()])
Note: Make sure you replace the ScPo-CompEcon with your GitHub ID
- Create a specific project based off this template
generate("ExamplePackage.jl", ourTemplate)
If we navigate to the package directory (shown in the output), we should see something like the following
Note: On Mac, this may be hidden; you can either start a terminal, cd ~ and then cd .julia, or make hidden files visible in the Finder
Adding a Project to Github¶
The next step is to creat a repository online where to store our code.
- go to github.com, login, and click on new repository
We’ll want the following settings
In particular
- The repo you create should have the same name as the project we added
- We should leave the boxes unchecked for the
README.md,LICENSE, and.gitignore, since these are handled byPkgTemplates
Then, if you use Github Desktop
- Drag and drop your folder from your
~/.julia/devdirectory to GitHub Desktop - Click the “publish branch” button to upload your files to GitHub
Else, if you use the commandline, do
cd ~/.julia/dev/ExamplePackage
git add .
git commit -m 'initial commit'
git push origin master
If you navigate to your git repo (ours is here), you should see something like
Note: Be sure that you don’t separately clone the repo you just added to another location (i.e., to your desktop)
A key note is that you have some set of files on your local machine (here in ~/.julia/dev/ExamplePackage.jl) and git is plugged into those files
For convenience, you might want to create a shortcut to that location somewhere accessible
Adding a Project to the Julia Package Manager¶
- We also want Julia’s package manager to be aware of the project
In order to make it visible to the package manager, we need to execute the Pkg-mode command
devOpen a REPL in the newly created project directory, either by noting the path printed above, or by running the following in a REPL
# first we go to where our package is on our computer
cd(joinpath(DEPOT_PATH[1], "dev", "ExamplePackage")) # Note the lack of `.jl`!
# ] goes into pkg mode!
] activate
] dev .
] st
Using the Package Manager¶
Now, from any Julia terminal in the future, we can run
using ExamplePackage
To use its exported functions
We can also get the path to this by running
using ExamplePackage
pathof(ExamplePackage) # returns path to src/ExamplePackage.jl
Project Structure¶
Let’s unpack the structure of the generated project
- The first directory,
.git, holds the version control information - The
srcdirectory contains the project’s source code – it should contain only one file (ExamplePackage.jl), which reads
module ExamplePackage
greet() = print("Hello World!")
end # module
- Likewise, the
testdirectory should have only one file (runtests.jl), which reads
using ExamplePackage
using Test
@testset "ExamplePackage.jl" begin
# Write your own tests here.
end
In particular, the workflow is to export objects we want to test (using ExamplePackage), and test them using Julia’s Test module
The other important text files for now are
Project.tomlandManifest.toml, which contain dependency information
In particular, the Project.toml contains a list of dependencies, and the Manifest.toml specifies their exact versions and sub-dependencies
- The
.gitignorefile (which may display as an untitled file), which contains files and paths forgitto ignore
Project Workflow and Dependencies¶
Environments¶
As before, the .toml files define an environment for our project, or a set of files which represent the dependency information
The actual files are written in the TOML language, which is a lightweight format to specify configuration options
This information is the name of every package we depend on, along with the exact versions of those packages
This information (in practice, the result of package operations we execute) will
be reflected in our ExamplePackage.jl directory’s TOML, once that environment is activated (selected)
This allows us to share the project with others, who can exactly reproduce the state used to build and test it
See the Pkg3 docs for more information
Pkg Operations¶
For now, let’s just try adding a dependency
- Activate the package environment (to be run from the base,
v1.0environment)
] activate ExamplePackage
- This tells Julia to write the results of package operations to `ExampleProject`’s TOML,
- and use the versions of packages specified there
Note that the base environment isn’t special, except that it’s what’s loaded by a freshly-started REPL or Jupyter notebook
- Add a package
] add Expectations
We can track changes in the TOML, as before
Here’s the Manifest.toml
We can also run other operations, like ] up, ] precompile, etc.
Package operations are listed in detail in the tools and editors lecture
Recall that, to quit the active environment and return to the base (v1.0), simply run
] activate
Writing Code¶
The basic idea is to work in tests/runtests.jl, while reproducible functions should go in the src/ExamplePackage.jl
For example, let’s say we add Distributions.jl
] activate ExamplePackage
] add Distributions
and edit the source (paste this into the file itself ) to read as follows
module ExamplePackage
greet() = print("Hello World!")
using Expectations, Distributions
function foo(μ = 1., σ = 2.)
d = Normal(μ, σ)
E = expectation(d)
return E(x -> sin(x))
end
export foo
end # module
] activate
using ExamplePackage
ExamplePackage.greet()
foo() # exported, so don't need to qualify the namespace
Note: If you didn’t follow the instructions to add a startup file, you may need to quit your REPL and load the package again
Collaborative Work¶
For someone else to get the package, they simply need to
- Run the following command
] dev https://github.com/ScPo-CompEcon/ExamplePackage.jl.git
# using your github ID!
This will place the repository inside their ~/.julia/dev folder
- Drag-and-drop the folder to GitHub desktop in the usual way
Recall that the path to your ~/.julia folder is
DEPOT_PATH[1]
They can then collaborate as they would on other git repositories
In particular, they can run
] activate ExamplePackage # to activate
] instantiate # to get all required packages onto their machine
] precompile # precompile all packages
] test # run the packages unit tests
Unit Testing¶
It’s important to make sure that your code is well-tested
There are a few different kinds of test, each with different purposes
- Unit testing makes sure that individual pieces of a project work as expected
- Integration testing makes sure that they fit together as expected
- Regression testing makes sure that behavior is unchanged over time
In this lecture, we’ll focus on unit testing
In general, well-written unit tests (which also guard against regression, for example by comparing function output to hardcoded values) are sufficient for most small projects
The Test Module¶
Julia provides testing features through a built-in package called Test, which we get by using Test
The basic object is the macro @test
using Test
@test 1 == 1
@test 1 ≈ 1
Tests will pass if the condition is true, or fail otherwise
If a test is failing, we should flag it with @test_broken as below
@test_broken 1 == 2
This way, we still have access to information about the test, instead of just deleting it or commenting it out
There are other test macros, that check for things like error handling and type-stability
Advanced users can check the Julia docs
Example¶
Let’s add some unit tests for the foo() function we defined earlier
Our tests/runtests.jl file should look like this
As before, this should be pasted into the file directly
using ExamplePackage
using Test
@test foo() == 0.11388071406436832
@test foo(1, 1.5) == 0.2731856314283442
@test_broken foo(1, 0) # tells us this is broken
And run it by typing ] test into an activated REPL (i.e., a REPL where you’ve run ] activate ExamplePackage)
Running Tests¶
There are a few different ways to run the tests for your package
- Run the actual
runtests.jl, say by hittingshift-enteron it in Atom - From a fresh (
v1.0) REPL, run] test ExamplePackage - From an activated (
ExamplePackage) REPL, simply run] test(recall that you can activate with] activate ExamplePackage)
Continuous Integration with Travis¶
Setup¶
By default, Travis should have access to all your repositories and deploy automatically
This includes private repos if you’re on a student developer pack or an academic plan (Travis detects this automatically)
To change this, go to “settings” under your GitHub profile
Click “Applications,” then “Travis CI,” then “Configure,” and choose the repos you want to be tracked
Build Options¶
By default, Travis will compile and test your project (i.e., “build” it) for new commits and PRs for every tracked repo with a .travis.yml file
We can see ours by opening it in Atom
# Documentation: http://docs.travis-ci.com/user/languages/julia/
language: julia
os:
- linux
- osx
julia:
- 1.0
- nightly
matrix:
allow_failures:
- julia: nightly
fast_finish: true
notifications:
email: false
after_success:
- julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'
This is telling Travis to build the project in Julia, on OSX and Linux, using Julia v1.0 and the latest (“nightly”)
It also says that if the nightly version doesn’t work, that shouldn’t register as a failure
Note You won’t need OSX unless you’re building something Mac-specific, like iOS or Swift
You can delete those lines to speed up the build, likewise for the nightly Julia version
Working with Builds¶
As above, builds are triggered whenever we push changes or open a pull request
For example, if we push our changes to the server and then click the Travis badge (the one which says “build”) on the README, we should see something like
Note that you may need to wait a bit and/or refresh your browser
This gives us an overview of all the builds running for that commit
To inspect a build more closely (say, if it fails), we can click on it and expand the log options
Note that the build times here aren’t informative, because we can’t generally control the hardware to which our job is allocated
We can also cancel specific jobs, either from their specific pages or by clicking the grey “x” button on the dashboard
Lastly, we can trigger builds manually (without a new commit or PR) from the Travis overview
To commit without triggering a build, simply add “[ci skip]” somewhere inside the commit message
Setup¶
You’ll find that Codecov is automatically enabled for public repos with Travis
For private ones, you’ll need to first get an access token
Add private scope in the Codecov website, just like we did for Travis
Navigate to the repo settings page (i.e., https://codecov.io/gh/quanteconuser/ExamplePackage.jl/settings for our repo) and copy the token
Next, go to your Travis settings and add an environment variable as below
Interpreting Results¶
Click the Codecov badge to see the build page for your project
This shows us that our tests cover 50% of our functions in src//
Note: To get a more detailed view, we can click the src// and the resultant filename
Note: Codecov may take a few minutes to run for the first time
This shows us precisely which methods (and parts of methods) are untested
Review¶
To review the workflow for creating, versioning, and testing a new project end-to-end
- Create the local package directory using the
PkgTemplates.jl - Add that package to the Julia package manager, by opening a Julia REPL in the
~/.julia/dev/ExamplePackage.jl, making sure the active environment is the default one(v1.0), and hitting] dev . - Drag-and-drop that folder to GitHub Desktop
- Create an empty repository with the same name on the GitHub server
- Push from GitHub Desktop to the server
- Open the original project folder (e.g.,
~/.julia/dev/ExamplePackage.jl) in Atom / your editor - Make changes, test, iterate on it, etc. As a rule, functions like should live in the
src/directory once they’re stable, and you should export them from that file withexport func1, func2. This will export all methods offunc1,func2, etc. - Commit them in GitHub Desktop as you go (i.e., you can and should use version control to track intermediate states)
- Push to the server, and see the Travis and Codecov results (note that these may take a few minutes the first time)
Exercises¶
Exercise 1¶
Following the instructions for a new project, create a new package on your github account called NewtonsMethod.jl
In this package, you should create a simple package to do Newton’s Method using the code you did in the Newton’s method exercise in Introductory Examples
In particular, within your package you should have two functions
newtonroot(f, f′; x₀, tol = 1E-7, maxiter = 1000)newtonroot(f; x₀, tol = 1E-7, maxiter = 1000)
Where the second function uses Automatic Differentiation to call the first.
The package should include
- implementations of those functions in the
/srcdirectory - comprehensive set of tests
- project and manifest files to replicate your development environment
- automated running of the tests with Travis CI in GitHub
For the tests, you should have at the very minimum
- a way to handle non-convergence (e.g. return back
nothingas discussed in error handling - several
@testfor the root of a known function, given thefand analyticalf'derivatives - tests of those roots using the automatic differentiation version of the function
- test of finding those roots with a
BigFloatand not just aFloat64 - test of non-convergence for a function without a root (e.g. $ f(x) = 2 + x^2 $ )
- test to ensure that the
maxiteris working (e.g. what happens if you callmaxiter = 5 - test to ensure that
tolis working
And anything else you can think of. You should be able to run ] test for the project to check that the test-suite is running, and then ensure that it is running automatically on Travis CI
Push a commit to the repository which breaks one of the tests and see what the Travis CI reports after running the build