This is a guide for enclone developers, both inside and outside 10x Genomics.
There is no reason to read this, unless you want to experimentally modify enclone, or are curious. We distribute pre-built binaries for enclone, and that's all you need if you just want to run the existing code.
enclone has the following software and data components:
~/enclone
), unless you
choose the small
option, in which case you get only a subset of it. The reason
for having a separate repo for data is to keep the enclone repo itself small.
large
option; the colossus
option results in downloading of additional GEX data.
enclone is written in Rust.
We used Rust because the language makes it extremely easy to use Rust code written by other people, because Rust has its own build system, and because it is fast and safe.
Technical details. The repo is itself a Rust workspace, with a several
subdirectories that comprise Rust crates. Each of these crates has its own
Cargo.toml
file. Modification of dependency
versions in these files files is done by modifying a file master.toml
at the
top level,
and then typing sync_to_master
, which invokes a program (part of the repo) to
sync each Cargo.toml
to master.toml
.
enclone can be built and run on an x86-64 Linux server or a Mac. It can also be built under Windows, but we do not provide instructions for this.
The following software are needed to build and test enclone:
Rust
. Detailed instructions on how to install Rust
can be found here. You can
confirm that you have successfully installed the Rust compiler by running
rustc --version
.cargo-license
. Probably this is needed only for internal use. You can install
it using
cargo install cargo-license
, assuming that you have already installed Rust.cargo-sweep
. Install via cargo install cargo-sweep
.git
, curl
and wget
samtools
.xcode-select --install
, however this can be very slow. An alternative
method is to get the package from the Apple developer site at
https://developer.apple.com/download/all/?q=xcode. Downloading and installing from there still takes some time, but at a minimum,
it is more transparent in what it's doing.homebrew
, you can
install these as follows:
brew install graphviz
brew install samtools
enclone
repository using
git clone git@github.com:10XGenomics/enclone.git
If you have a very slow internet connection, you might wish to download only the current version
(and not the history) using
git clone --depth=1 git@github.com:10XGenomics/enclone.git
cd enclone
and then ./build
. Depending on your computer, this
will take roughly 5-10 minutes. Subsequent invocations (after code changes) will be much
faster. You can also run cargo b
, which is faster, but omits some steps.
Please let us know if you encounter difficulties with compilation.enclone/target/debug
to your PATH
, and
make sure this comes before ~/bin
, which was where the precompiled
version of enclone was downloaded by the install command. You may need to close and reopen
your terminal window to get the change to PATH
to take effect.enclone has an extensive test suite, with three components as described here:
# | command | content | minutes: large Linux server | minutes: Mac | CI coverage | minutes: CI |
---|---|---|---|---|---|---|
1 | ./test |
hundreds of small tests | 2 | 7 | partial, but runs on both Linux and Mac (running all tests on Mac is overkill) | 60 |
2 | enclone.test |
one big test | 2.5 | N/A | full | |
3 | test_vis |
many tests of enclone visual using screenshots | N/A | 1 | none | N/A |
To varying degrees, the tests can either be run from the command line or through a CI which
is triggered by pull requests. The CI runs via GitHub Actions and Jenkins. Because it is so slow
we tend not to use it. The GitHub Actions part of the CI test is given by
.github/workflows/test.yaml
. We strongly encourage that when tests are added to the
CI, they are also added to test 1, as otherwise we force people to use the CI to test code
changes.
Notes on test 1.
cargo t test_enclone_d
../test linkless
.enclone_core/src/testlist.rs
enclone_main/tests/enclone_test*.rs
test
.cargo/config
.Notes on test 2. See the file enclone/enclone.test
.
Notes on test 3.
touch enclone/enclone_core/defs.rs
on the Linux server and then cargo b
.
Otherwise youβll get an error message.enclone_visual/src/bin/test_vis.rs
.There are miscellaneous gnarly issues that we do not have automated testing for:
NOPAGER
accidentally broke at one point and we
don't want that to recur. Some test cases that could be verified:
enclone BCR=...
enclone BCR=... NOPAGER
enclone help all
enclone help all NOPAGER
enclone help faq
enclone help faq NOPAGER
.enclone_visual/src/testsuite.rs
.
There are some other major things that are not tested:
|
---|
enclone provides a few built-in mechanisms for profiling:
COMP
to your command. This yields a summary of wall-clock
time by code section, and also tracks peak memory usage. There is also a version
COMP2
that prints a bit more. Run this with NOPAGER
if you want
to see the lines printed as they happen.PROFILE
to your command. This yields a list of tracebacks
based on random interrupts. These tracebacks are manicured and each is shown with its
multiplicity. When this option is used, the internal paging mechanism in enclone is turned
off, and you may wish to pipe the output to less -r
.CTRLC
to your command. Then upon CTRL-C, execution will
be stopped and a traceback printed. In general this is only useful if your code seems to
be extremely slow or perhaps is stuck in an infinite loop.It is usually convenient to add the argument NOPRINT
when using these profiling
options, so as to suppress other output.
If the code crashes, you should get a clean traceback. If this happens on the distributed version of enclone or the master version, that's a bug that should be reported.
If you run out of disk space while building, you may get weird error messages from the linker,
in which case you should fix your disk space problem. You might also need to
cargo clean
.
Somewhat rarely, you'll get weird error messages from the linker, with no apparent reason.
If this happens, cargo clean
should fix the problem.
This is done by editing the files in the subdirectory pages
, and never by
editing .html
files directly. All the .html
files are
auto-generated.
There is a script ./local_view
that copies the site to
your public_html
directory so that it can be previewed.
enclone binaries for OSX and Linux can be released by βpushing a buttonβ. This only works at 10x Genomics.
First, if the release warrants announcement, edit pages/history.html.src
to note
the relevant changes. Running build
will then update the actual file that goes on the
site.
Run the enclone test suite.
From master on enclone, from a Linux server, type
start_release
This takes about five minutes. It causes GitHub Actions to initiate creation of release binaries.
That takes about thirty-five minutes. When start_release
finishes, it launches
release_nanny
in the background, which at the appropriate time runs some final steps.
In principle the following should work:
but when we tested this, we got errors. This should be resolvable but we're not sure how. Also note that this will clobber the results of an ordinaryrustup target add x86_64-pc-windows-gnu
cargo b --target=x86_64-pc-windows-gnu
cargo b
. In principle
this can be avoided by specifying the target for it too.