Implemented templated invocation routines for ParallelFor backends
in order to improve loop body inlining.
Several modifications of ParallelFor implementation using CXX threads:
- Index order changed from interleaved to sequential
- Static task scheduling replaced with dynamic (controlled by
kWorkBlocksPerThread)
- Changed index retrieval to atomic
Modifications of OpenMP backend:
- Changed loop scheduling to guided
Changing index order from interleaved to sequential in parallel seem
to significantly improve run-times of parallel loops, for example in
evaluation of jacobian and residuals.
Other modifications provide minor improvements for unbalanced
sub-problem lengths and parallel for loops with small number of
computation per operation.
Single-threaded performance was improved by avoiding costs of
wrapping parallel loop bodies in std::function.
On BAL dataset the following improvements in time consumed for
evaluation of residuals or jacobian and residuals were observed:
OLD NEW OLD/NEW
dataset threads r J r J r J
problem-257-65132-pre.txt 1 0.025 0.079 0.025 0.074 1.016 1.056
problem-257-65132-pre.txt 2 0.030 0.062 0.022 0.050 1.333 1.246
problem-257-65132-pre.txt 4 0.023 0.052 0.014 0.034 1.592 1.515
problem-257-65132-pre.txt 8 0.015 0.035 0.010 0.025 1.477 1.401
problem-257-65132-pre.txt 16 0.011 0.027 0.008 0.019 1.365 1.377
problem-356-226730-pre.txt 1 0.150 0.442 0.147 0.412 1.017 1.070
problem-356-226730-pre.txt 2 0.155 0.322 0.100 0.281 1.542 1.145
problem-356-226730-pre.txt 4 0.129 0.291 0.089 0.196 1.439 1.485
problem-356-226730-pre.txt 8 0.091 0.184 0.066 0.139 1.381 1.319
problem-356-226730-pre.txt 16 0.070 0.148 0.055 0.110 1.272 1.340
problem-1723-156502-pre.txt 1 0.084 0.243 0.082 0.229 1.023 1.063
problem-1723-156502-pre.txt 2 0.088 0.188 0.055 0.154 1.589 1.222
problem-1723-156502-pre.txt 4 0.072 0.159 0.049 0.108 1.475 1.475
problem-1723-156502-pre.txt 8 0.050 0.105 0.037 0.077 1.348 1.368
problem-1723-156502-pre.txt 16 0.038 0.083 0.030 0.062 1.269 1.344
problem-1778-993923-pre.txt 1 0.621 1.777 0.609 1.667 1.018 1.065
problem-1778-993923-pre.txt 2 0.621 1.273 0.415 1.199 1.494 1.061
problem-1778-993923-pre.txt 4 0.514 1.140 0.361 0.786 1.421 1.449
problem-1778-993923-pre.txt 8 0.365 0.808 0.277 0.559 1.319 1.443
problem-1778-993923-pre.txt 16 0.279 0.608 0.223 0.441 1.252 1.379
problem-13682-4456117-pre.txt 1 3.877 10.726 3.738 10.082 1.037 1.063
problem-13682-4456117-pre.txt 2 3.310 7.170 2.423 6.448 1.366 1.111
problem-13682-4456117-pre.txt 4 3.070 6.344 2.064 4.474 1.486 1.417
problem-13682-4456117-pre.txt 8 2.051 4.612 1.527 3.133 1.343 1.472
problem-13682-4456117-pre.txt 16 1.549 3.453 1.218 2.488 1.271 1.387
Run time in seconds for a single evaluation, using evaluation_benchmark
numactl -N 0 -m 0 ./bin/evaluation_benchmark --bal_root ${path_to_BAL}
Evaluation was performed on 28-core CPU.
Note: performance when running across numa-nodes degrades in both old
and proposed implementations, thus the test was executed limiting memory
and compute resources allocation to a single numa-node.
Change-Id: Ia195580bdab9d05c95ac983bfe37b045eecfaf49
* All Cuda* objects now take in a ContextImpl* during
construction, and save the context instead of individual
handles.
* Since we no longer use the legacy default stream, we need to
explicitly synchronize the stream before performing GPU->CPU
transfers, and CudaBuffer is responsible for such synchronization
when asked to perform GPU to CPU transfers.
* Remove all manual syncs and relegate syncing to CudaBuffer
before performing GPU to CPU transfers.
Change-Id: Ic73cb24174a1e09842827323280e90241716cc20
The value of x_norm_ was computed and then incorrectly set to -1.
This meant that ParameterToleranceReached was using the incorrect
value till such time as the minimizer made its first successful
step.
This change removes the member variable and just computes
the norm of x inside ParameterToleranceReached.
Fixes https://github.com/ceres-solver/ceres-solver/issues/885
Thanks to Anton Adanasyev for reporting this.
Change-Id: Ib4d52a45c2d925557ce2c5b57de8a9fa37da6c70
Previously some matrices used Block to keep track of
row/column block sizes and some would just use ints, and
then compute the position of each row and column from it.
By uniformly using Block everywhere, we reduce duplicate
computation and data copies.
I also cleaned up a bunch of c++17 related stuff as I edited
these files.
Change-Id: I4c86b1593fd4c91f9057fbb38314f62f303e0477
* The algorithm enum value for SpMV is now selected based on the
version of the CUDA runtime that Ceres is compiled against.
Change-Id: I2e0e39f1cbdb8ac26d2a9d45f4ebfc09b96d872b
Instead of iterating row-wise, use the row block structure
to update the preconditioner one row block at a time.
While benchmarking, I also found an opportunity to speed up
BlockSparseJacobiPreconditiner.
Before
-----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------
BM_BlockSparseJacobiPreconditioner 132122392 ns 132103000 ns 5
BM_BlockCRSJacobiPreconditionerBA 73365217 ns 73335800 ns 10
BM_BlockSparseJacobiPreconditionerUnstructured 92407762 ns 92407714 ns 7
BM_BlockCRSJacobiPreconditionerUnstructured 72256367 ns 72256400 ns 10
After
-----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------
BM_BlockSparseJacobiPreconditionerBA 39723456 ns 39722222 ns 18
BM_BlockCRSJacobiPreconditionerBA 46561625 ns 46561667 ns 15
BM_BlockSparseJacobiPreconditionerUnstructured 52676208 ns 52676167 ns 12
BM_BlockCRSJacobiPreconditionerUnstructured 54430564 ns 54421462 ns 13
Change-Id: I6da8e044b26f0c24481deee5722322a606e9e5ca
1. Add CreateFakeBundleAdjustmentJacobian to create
bundle adjustment structured jacobians.
2. sparse_linear_operator_benchmark -> spmv_benchmark
3. Refactor spmv_benchmark to use CreateFakeBundleAdjustmentJacobian.
4. Add BA and unstructured Jacobian variants of spmv benchmarks
5. Add BA and unstructured Jacobian benchmarks for the block Jacobi
preconditioner.
6. Fix BlockSparseMatrix::ToCompressedRowSparseMatrix to add the
row and column block structure to the output.
Change-Id: I737a3d7d82dad665b1f2e7886c2c3aedd702ffcd
Use the more modern form of accessing the data array of a vector
rather than grabbing the pointer to the first element. The latter
can lead to errors if the vector is of zero length.
Change-Id: Ifc8fc969b06b3ba1a9385e8a3a8d5c50b25db5a8
* Added GPU device and CUDA compute capability identification.
* Added GpuMemoryAvailable() to aid downstream optimizations based
on GPU memory availability.
Change-Id: I326dc1e4b7a6a7f5571b7e5479eb9aa300ad1075
1. Use hardware_concurrency to configure number of threads.
2. Use user ordering instead of automatic ordering.
Fixes https://github.com/ceres-solver/ceres-solver/issues/874
Change-Id: I4d7d69612e1ee40358943019a0c3a909e419c4e4
With -flto=auto, GCC emits multiple warnings in release builds such as
In function ‘make_unique’,
inlined from ‘Create’ at ceres-solver/internal/ceres/scratch_evaluate_preparer.cc:43:75,
inlined from ‘CreateEvaluatePreparers’ at ceres-solver/internal/ceres/compressed_row_jacobian_writer.h:95:66,
inlined from ‘__ct ’ at ceres-solver/internal/ceres/program_evaluator.h:120:9,
inlined from ‘Evaluate.constprop.isra’ at ceres-solver/internal/ceres/problem_impl.cc:695:65:
/usr/include/c++/12.1.1/bits/unique_ptr.h:1080:30: warning: argument 1 value ‘18446744073709551615’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
1080 | { return unique_ptr<_Tp>(new remove_extent_t<_Tp>[__num]()); }
| ^
because a signed integer is used to specify the size of allocated arrays
instead of the expected unsigned (specifically, std::size_t) without
checking for negative values at the call site.
Change-Id: I923b1d074241535426bfea041568ef1dc7f3ec86
1. Rename BlockJacobiPreconditionet to BlockSparseJacobiPreconditioner.
2. Add CompressedRowSparseJacobiPreconditioner which is a
block Jacobi preconditioner for CompressedRowSparseMatrix objects.
3. Re-write the tests to be more comprehensive.
Change-Id: Icbc91f9ad2cefaad593c11397f8cdcf805d7e118
* Added CudaCgnrSolver, a new CUDA-accelerated CGNR.
* To use CudaCgnrSolver, the user must select CGNR as the linear_solver
and CUDA_SPARSE as the sparse_linear_algebra_library.
* Updated ConjugateGradientSolver to work with an array of pointers to
scratch to support CudaVectors as scratch.
* Moved CUDA initialization to run in Solver::Solve as needed.
Some performance comparisons on an Ubuntu 20.04 desktop with an
Intel i9-9940X CPU @ 3.30GHz, and an nVidia Quadro RTX 6000,
all configurations run with 24 threads, and 10 iterations.
=================================================
CGNR + CUDA_SPARSE + IDENTITY Preconditioner
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.724755e+06
Change 2.546725e+08
Minimizer iterations 11
Successful steps 7
Unsuccessful steps 4
Time (in seconds):
Preprocessor 4.020158
Residual only evaluation 1.567092 (10)
Jacobian & residual evaluation 7.847130 (7)
Linear solver 31.688898 (10)
Minimizer 46.834987
Postprocessor 0.353974
Total 51.209120
=================================================
SPARSE_SCHUR (CPU) + SUITE_SPARSE + AMD
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.651617e+06
Change 2.547457e+08
Minimizer iterations 11
Successful steps 11
Unsuccessful steps 0
Time (in seconds):
Preprocessor 35.812003
Residual only evaluation 1.658980 (10)
Jacobian & residual evaluation 12.218799 (11)
Linear solver 76.409992 (10)
Minimizer 98.809773
Postprocessor 0.372712
Total 134.994489
=================================================
ITERATIVE_SCHUR (CPU) + JACOBI Preconditioner
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.684447e+06
Change 2.547128e+08
Minimizer iterations 11
Successful steps 8
Unsuccessful steps 3
Time (in seconds):
Preprocessor 15.331614
Residual only evaluation 1.606114 (10)
Jacobian & residual evaluation 8.502166 (8)
Linear solver 351.910080 (10)
Minimizer 368.797327
Postprocessor 0.363536
Total 384.492478
=================================================
CGNR + CUDA_SPARSE + IDENTITY Preconditioner
problem-13682-4456117-pre.txt
=================================================
Cost:
Initial 1.126372e+09
Final 2.269329e+07
Change 1.103678e+09
Minimizer iterations 11
Successful steps 7
Unsuccessful steps 4
Time (in seconds):
Preprocessor 19.140087
Residual only evaluation 8.721920 (10)
Jacobian & residual evaluation 41.955923 (7)
Linear solver 214.121861 (10)
Minimizer 296.636890
Postprocessor 1.971827
Total 317.748804
Change-Id: I3a09f31aa6903f661e91f595afd39d427583e856
* Added CudaSparseMatrix to manage and operate on sparse matrices with
cuSparse.
* Added tests for CudaSparseMatrix.
* Added a new sparse linear operator benchmark.
Change-Id: Id09df46de3b40be1f14441528088b54dab5844af
CompressedRowSparseMatrix.
Since the conversion from BlockSparseMatrix to CompressedRowSparseMatrix
is not used in any performance-critical context, this CL simplifies it
by re-using existing conversions.
Change-Id: I51263bc95cc056efb31961ccda548cd7be35b2a4
Clang emits the following warning:
internal/ceres/schur_complement_solver.cc:71:51: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
virtual ~BlockRandomAccessSparseMatrixAdapter() final {}
Change-Id: I9da96711fd1e924ce8504988ff171cb476b6222f
Add an option to use schur power series expansion for initialization
of pcg solution in ITERATIVE_SCHUR linear solver.
Change-Id: Ifb8bce02bc5f5ceebc74f961eefd3f6dd2ffab4a
Use same instance of a PRNG throughout by passing it to methods and
functions as an argument to generate random numbers without breaking the
sequence.
Change-Id: Ib024bbc1ea2d14e4b9afb71857856a5fb77b1667
The code that verifies that the linear solver is configuration
specified by the user has grown into a rat's nest. This CL
attempts to bring some order to this madness.
Fixes https://github.com/ceres-solver/ceres-solver/issues/852
Change-Id: I3f34c0e27da13a6412117dee43ef2d9ec3835b64
These changes came about from testing the power bundle adjustment
integration CL.
1. Allow Solver::Options::max_linear_solver_iterations == 0.
2. Simplify the logic for when inverse(F'F) is computed.
3. norm_b -> norm_rhs in ConjugateGradientsSolver.
Change-Id: I50c19e1f24a4cc08ed60e3a3032b96b37bcada9f
Implementation of "Power Bundle Adjustment for Large-Scale 3D
Reconstruction" by Weber et. al. added in the form of preconditioner.
Change-Id: Ie85526a5fc46f74256f6dfe9173c3571f7160f3a
Sameer Agarwal