Despite its relative size, this is very significant change
to Ceres.
Why
===
Up till now, when the user chose SPARSE_NORMAL_CHOLESKY,
the Jacobian was evaluated in a CompressedRowSparseMatrix,
which was then use to compute the normal equations which were
passed to a sparse linear algebra library for factorization.
The reason to do this was because in the case of SuiteSparse,
we were able to pass the Jacobian matrix directly without
computing the normal equations and SuiteSparse/CHOLMOD did the
normal equation computation.
This turned out to be slow, so Cheng Wang implemented a high
performance version of the matrix-matrix multiply to compute
the normal equations, and all the sparse linear algebra libraries
now are passed the normal equations.
So that raises the question, as to what the best representation
of the Jacobian which is suitable for the normal equation computation.
Turns out BlockSparseMatrix is ideal. It brings two advantages.
1. Jacobian evaluation into a BlockSparseMatrix is considerably
faster when using a BlockSparseMatrix than
CompressedRowSparseMatrix. This is because we save on a bunch
of memory copies.
2. To make the matrix multiplication fast and use the block structure
Cheng Wang had to essentially make the CompressedRowSparseMatrix
carry a bunch of sidecar information about the block sparsity,
essentially making it behave like a BlockSparseMatrix. The resulting
code had fairly complicated indexing and complicated the semantics
of CompressedRowSparseMatrix. The new InnerProductComputer class
does away with all that and once this CL goes in, I will be able to
remove all that code and simplify the semantics of
CompressedRowSparseMatrix.
Changes
=======
1. Use InnerProductComputer in SparseNormalCholeskySolver.
2. Change the evaluator instantiated for SPARSE_NORMAL_CHOLESKY with
static sparsity inside evaluator.cc
3. The former change necessitates that we change ProblemImpl::Evaluate
to create the evaluate it needs on its own, because it was
depending on passing "SPARSE_NORMAL_CHOLESKY" as linear solver type
to the evaluator factor to get an Evaluator which can use
CompressedRowSparseMatrix objects for storing the Jacobian.
4. Update the tests for SparseNormalCholeskySolver.
5. Separate out the tests for DynamicSparseNormalCholeskySolver into its
own file.
Change-Id: I2ef7ef8fbfbb4967d0c1ec2068c1c778248fdf5b
SparseCholesky is an interface to sparse cholesky factorization
routines across sparse linear algebra libraries. Each sparse
linear algebra library is responsible for implementing its own
instance of this interface.
As a result the various places - SparseNormalCholeskySolver,
SparseSchurComplementSolver and VisibilityBasedPreconditioner
are significantly simplified.
Change-Id: I8b465705eae83bba9e1adfffcc741a05c70faf2e
Now that there is a single piece of code doing the outer product
computation for all three sparse linear algebra backends, move
this code one level up the call stack and there by make the actual
per-library solver code shorter and simpler.
Also fix a minor omission in the outer product computation code
where row/column blocks were not being copied over to the
outer product matrix.
Change-Id: I22a7967bdc659385b741901afefa7af312e676e5
By adding an enum to CompressedRowSparseMatrix, which indicates
whether the matrix is unsymmetric, upper or lower triangular
we are able to improve the readability and fix some minor
bugs in the way some matrix manipulation code was being
called.
Thank to William Rucklidge for this suggestion.
Change-Id: I355c90d11cd5d31f5a25741b0bda4fc4583e9095
1. Add stype to the outerproduct computation to control the output
matrix in upper or lower triangular matrix. For SuiteSparse,
upper triangular matrix is generated. SuiteSparse can directly use
this matrix format for cholesky without matrix transpose overhead.
2. Change the outerproduct computation to block multiplication. This
reduces the computation complexity for the sort in preprocessing, also
allows formulation of the block outerproduct computation as dense Eigen
block matrix multiplication.
3. Solve 32 Tango problems on Qualcomm MSM8994 Cortex-A53 (1.55GHz)
before change: 140 seconds
after change: 131 seconds
Change-Id: I8054114cef911de6a303310a448821ca296e4744
Print information about the fill-in when using Eigen to perform
Simplicial sparse Cholesky factorization.
Change-Id: I09013b7f4ed2f6c55ae8ce8922447e23d63580a8
The outer product computation logic in SparseNormalCholeskySolver
does not work well with dynamic sparsity. The overhead of computing
the sparsity pattern of the normal equations is only amortized if
the sparsity is constant. If the sparsity can change from call to call
SparseNormalCholeskySolver will actually be more expensive.
For Eigen and for CXSparse we now explicitly compute the normal
equations using their respective matrix-matrix product routines and solve.
Change-Id: Ifbd8ed78987cdf71640e66ed69500442526a23d4
The previous implementation incorrectly cached the outer product matrix
pattern even when `dynamic_sparsity = true`.
Change-Id: I1e58315a9b44f2f457d07c56b203ab2668bfb8a2
Since Ceres is moving to using GitHub for issues, and the Google
Code URL in the current copyright header will soon become invalid,
update all the headers.
Change-Id: I1fce70375d1bcf098591f07b4d8f01a5c1e0789c
Simplifying some of the template handling, and remove the use
of SelfAdjointView as it is not needed. The solver itself takes
an argument for where the data is actually stored.
The performance of SparseNormalCholesky with EIGEN_SPARSE
seems to be on par with CX_SPARSE.
Change-Id: I69e22a144b447c052b6cbe59ef1aa33eae2dd9e3
SPARSE_NORMAL_CHOLESKY and SPARSE_SCHUR can now be used
with EIGEN_SPARSE as the backend.
The performance is not as good as CXSparse. This needs to be
investigated. Is it because the quality of AMD ordering that
we are computing is not as good as the one for CXSparse? This
could be because we are working with the scalar matrix instead
of the block matrix.
Also, the upper/lower triangular story is not completely clear.
Both of these issues will be benchmarked and tackled in the
near future.
Also included in this change is a bunch of cleanup to the
SparseNormalCholeskySolver and SparseSchurComplementSolver
classes around the use of the of defines used to conditionally
compile out parts of the code.
The system_test has been updated to test EIGEN_SPARSE also.
Change-Id: I46a57e9c4c97782696879e0b15cfc7a93fe5496a
- Previously we passed all compile options to Ceres via add_definitions
in CMake. This was fine for private definitions (used only by Ceres)
but required additional work for public definitions to ensure they
were correctly propagated to clients via CMake using
target_compile_definitions() (>= 2.8.11) or add_definitions().
- A drawback to these approaches is that they did not work for chained
dependencies on Ceres, as in if in the users project B <- A <- Ceres,
then although the required Ceres public compile definitions would
be used when compiling A, they would not be propagated to B.
- This patch replaces the addition of compile definitions via
add_definitions() with an autogenerated config.h header which
is installed with Ceres and defines all of the enabled Ceres compile
options.
- This removes the need for the user to propagate any compile
definitions in their projects, and additionally allows post-install
inspect of the options with which Ceres was compiled.
Change-Id: Idbdb6abdad0eb31e7540370e301afe87a07f2260
The standard sparse normal Cholesky solver assumes a fixed
sparsity pattern which is useful for a large number of problems
presented to Ceres. However, some problems are symbolically dense
but numerically sparse i.e. each residual is a function of a
large number of parameters but at any given state the residual
only depends on a sparse subset of them. For these class of
problems it is faster to re-analyse the sparsity pattern of the
jacobian at each iteration of the non-linear optimisation instead
of including all of the zero entries in the step computation.
The proposed solution adds the dynamic_sparsity option which can
be used with SPARSE_NORMAL_CHOLESKY. A
DynamicCompressedRowSparseMatrix type (which extends
CompressedRowSparseMatrix) has been introduced which allows
dynamic addition and removal of elements. A Finalize method is
provided which then consolidates the matrix so that it can be
used in place of a regular CompressedRowSparseMatrix. An
associated jacobian writer has also been provided.
Changes that were required to make this extension were adding the
SetMaxNumNonZeros method to CompressedRowSparseMatrix and adding
a JacobianFinalizer template parameter to the ProgramEvaluator.
Change-Id: Ia5a8a9523fdae8d5b027bc35e70b4611ec2a8d01
When using sparse cholesky factorization to solve the linear
least squares problem:
Ax = b
There are two sources of computational complexity.
1. Computing H = A'A
2. Computing the sparse Cholesky factorization of H.
Doing 1. using CX_SPARSE is particularly expensive, as it uses
a generic cs_multiply function which computes the structure of
the matrix H everytime, reallocates memory and does not take
advantage of the fact that the matrix being computed is a symmetric
outer product.
This change adds a custom symmetric outer product algorithm for
CompressedRowSparseMatrix.
It has a symbolic phase, where it computes the sparsity structure
of the output matrix and a "program" which allows the actual
multiplication routine to determine exactly which entry in the
values array each term in the product contributes to.
With these two bits of information, the outer product H = A'A
can be computed extremely fast without any reasoning about
the structure of H.
Further gains in efficiency are made by exploiting the block
structure of A.
With this change, SPARSE_NORMAL_CHOLESKY with CX_SPARSE as the
backend results in > 300% speedup for some problems.
The symbolic analysis phase of the solver is a bit more expensive
now but the increased cost is made up in 3-4 iterations.
Change-Id: I5e4a72b4d03ba41b378a2634330bc22b299c0f12
CompressedRowSparseMatrix can store the row and column block structure
but the AppendRows and DeleteRows methods did not pay attention to them.
This meant that it was possible to get to a CompressedRowSparseMatrix
whose block structure did not match the contents of the matrix.
This change fixes this problem.
Change-Id: I1b3c807fc03d8c049ee20511e2bc62806d211b81
A lot of error checking cruft has accumulated over the years
in the various linear solvers. This change makes the error reporting
more robust and consistent across the various solvers.
Preconditioners are not covered by this change and will be the
subject of a future change.
Change-Id: Ibeb2572a1e67758953dde8d12e3abc6d1df9052d
1. Move LinearSolverTerminationType to ceres::internal.
2. Add FATAL_ERROR as a new enum to LinearSolverTerminationType.
3. Pipe SuiteSparse errors via a LinearSolverTerminationType so
to distinguish between fatal and non-fatal errors.
4. Update levenberg marquardt and dogleg strategies to deal
with FATAL_ERROR.
5. Update trust_region_minimizer to terminate when FATAL_ERROR
is encountered.
6. Remove SuiteSparse::SolveCholesky as it screws up the error
handling.
7. Fix all clients calling SuiteSparse to handle the result of
SuiteSparse::Cholesky correctly.
8. Remove fatal failures in SuiteSparse when symbolic factorization
fails.
9. Fix all clients of SuiteSparse to deal with null symbolic factors.
This is a temporary fix to deal with some production problems. A more
extensive cleanup and testing regime will be put in place in a
subsequent CL.
Change-Id: I1f60d539799dd95db7ecc340911e261fa4824f92
Main purpose of this is to make implementation files free from
endless ifdef blocks every time this libraries are needed to be
included. This would hopefully prevent compilation errors in
the future caused by missing ifdef around header include.
This also includes some stubs added to suitesparse/cxsparse
headers to make code even more free from ifdefs.
Change-Id: Ic8554e7df31d8c4751583fe004b99e71b3c9087b
1. When a LAPACK implementation is present, then
DENSE_QR, DENSE_NORMAL_CHOLESKY and DENSE_SCHUR
can use it for doing dense linear algebra operations.
2. The user can switch dense linear algebra libraries
by setting Solver::Options::dense_linear_algebra_library_type.
3. Solver::Options::sparse_linear_algebra_library is now
Solver::Options::sparse_linear_algebra_library_type to be consistent
with all the other enums in Solver::Options.
4. Updated documentation as well as Solver::Summary::FullReport
to reflect these changes.
Change-Id: I5ab930bc15e90906b648bc399b551e6bd5d6498f
1. CX_SPARSE supports pre-ordering of the jacobian.
2. Add support for constrained approximate minimum degree ordering
for SuiteSparse versions >= 4.2.0
3. Using 2, support for pre-ordering for SPARSE_SCHUR when used
with SUITE_SPARSE.
4. Using 2, support for user orderings in SPARSE_NORMAL_CHOLESKY.
5. Minor cleanups in documentation and code all around.
6. Test update and refactoring.
Change-Id: Ibfe3ac95d59d54ab14d1d60a07f767688070f29f
Sparse Cholesky factorization algorithms use a fill-reducing
ordering to permute the columns of the Jacobian matrix. There
are two ways of doing this.
1. Compute the Jacobian matrix in some order and then have the
factorization algorithm permute the columns of the Jacobian.
2. Compute the Jacobian with its columns already permuted.
The first option incurs a significant memory penalty. The
factorization algorithm has to make a copy of the permuted
Jacobian matrix.
Starting with this change Ceres pre-permutes the columns of the
Jacobian matrix and generally speaking, there is no performance
penalty for doing so.
In some rare cases, it is worth using a more complicated
reordering algorithm which has slightly better runtime
performance at the expense of an extra copy of the Jacobian
matrix. Setting Solver::Options::use_postordering to true
enables this tradeoff.
This change also removes Solver::Options::use_block_amd
as an option. All matrices are ordered using their block
structure. The ability to order them by their scalar
sparsity structure has been removed.
Here is what performance on looks like on some BAL problems.
Memory
======
HEAD pre-ordering
16-22106 137957376.0 113516544.0
49-7776 56688640.0 46628864.0
245-198739 1718005760.0 1383550976.0
257-65132 387715072.0 319512576.0
356-226730 2014826496.0 1626087424.0
744-543562 4903358464.0 3957878784.0
1024-110968 968626176.0 822071296.0
Time
====
HEAD pre-ordering
16-22106 3.8 3.7
49-7776 1.9 1.8
245-198739 82.6 81.9
257-65132 14.0 13.4
356-226730 98.8 95.8
744-543562 325.2 301.6
1024-110968 42.1 37.1
Change-Id: I6b2e25f3fed7310f88905386a7898ac94d37467e
Break up CreateReducedProgram into smaller functions in
preparation for more sophisticated ordering strategies.
Change-Id: Ic3897522574fde770646d747fe383f5dbd7a6619
1. CreateSparseMatrixTransposeView now returns a struct instead
of a pointer.
2. Add AnalyzeCholeskyWithNaturalOrdering.
Change-Id: If27a5502949c3994edd95be0d25ec7a0d1fa1ae1
1. Silence CHOLMOD's indefiniteness warnings.
2. Add a comment about how the error handling in suitesparse.cc
needs to be improved.
3. Move the analysis logging into suitesparse.cc and out of the
three callsites.
Change-Id: Idd396b8ea4bf59fc1ffc7f9fcbbc7b38ed71643c
1. Add -DLINE_SEARCH_MINIMIZER to CMake to make the line search
minimizer optional.
2. Better handling of -DSUITESPARSE/-DCXSPARSE in top level cmake
file.
3. Disable code which will never be used if SuiteSparse and/or
CXSparse is not available.
4. Update build docs.
5. Update jni/Android.mk
6. Minor lint cleanup from William Rucklidge.
Change-Id: If60460a858000df82faed7a6bb056dd2bfdde562
1. Add an ExecutionSummary object to record execution
information about Ceres objects.
2. Add an EventLogger object to log events in a function call.
3. Add a ScopedExecutionTimer object to log times in ExecutionSummary.
4. Instrument ProgramEvaluator and all the linear solvers
to report their timing statistics.
5. Connect the timing statistics to Summary::FullReport.
6. Add high precision timer on unix systems using
gettimeofday() call.
7. Various minor clean ups all around.
Change-Id: I5e09804b730b09535484124be7dbc1c58eccd1d4
Average factorization times for bundle adjustment test problem:
SuiteSparse: 0.2794 s.
CXSparse: 0.4039 s.
CXSparse cached: 0.2399 s.
CXSparse will still be slower, though, because it has to compute
the transpose and J^T * J.
Change-Id: If9cdaa3dd520bee84b56e5fd4953b56a93db6bde
This is a preliminary, but full, port of Ceres to Windows.
Currently all tests compile and run, with only system_test
failing to work correctly due to a path issue.
Change-Id: I4152c1588bf51ffd7f4d9401ef9759f5d28c299c
Fix broken build and verbosity issues.
1. While cleaning up the last CL, I broke a macro.
2. cholmod_common_print was being called too often.
Change-Id: Ia76d8863c72f31b0c02977094b22035ceef835cf
By virtue of the modeling layer in Ceres being block oriented,
all the matrices used by Ceres are also block oriented.
When doing sparse direct factorization of these matrices, the
fill-reducing ordering algorithms can either be run on the
block or the scalar form of these matrices. Running it on the
block form exposes more of the super-nodal structure of the
matrix to the Cholesky factorization routines. This leads to
substantial gains in factorization performance.
This changelist adds support for approximate minimium degree
orderings to be computed on the block structure of the
Schur complement matrix. This affects, SchurComplementSolver
and VisibilityBasedPreconditioner and SparseNormalCholesky
when using SuiteSparse.
A bool, use_block_amd has been added to Solver::Options and
bundle_adjuster.cc has been updated to allow testing with it.
When combined with a multithreaded Schur elimination, speed ups
can be seen quite uniformly across the board. For some problems
this can be dramatic, reducing the factorization time from 70
seconds down to 17 seconds.
Change-Id: I15ebb0afcbc85ada032ec8d179ee3a2f7c8d3e46
1. Added support for CXSparse - SparseNormalCholesky and
SchurComplementSolver support SuiteSparse and CXSparse now.
I am not sure I will add suport for visibility based
preconditioning using CXSparse. Its not a high priority.
2. New enum SparseLinearAlgebraLibraryType which allows the user
to indicate which sparse linear algebra library should be used.
3. Updated tests for SolverImpl and system_test.
4. Build system changes to automatically detect CXSparse and
link to it by default -- just like SuiteSparse.
5. Minor bug fixes dealing in the cmake files and VBP.
6. Changed the order of the system test.
7. Deduped the unsymmetric linear solver test.
Change-Id: I33252a103c87b722ecb7ed7b5f0ae7fd91249244
unnecessarily complexity in the structure of linear solvers and preconditioners.
This is the first step towards cleaning up the Preconditioner interface.
2. Minor tweaks and cleanups to the various linear solvers.
Keir Mierle