Parallel implementations for right-multiply by dense vector for:
- Partitioned matrix view
- Block-sparse matrix
- CRS matrix (non-symmetric only)
When coupled with non-interleaving indexes in parallel for, this
simple aproach provides a reasonable speedup.
For example, in CRS case difference with GPGPU approach reduces
closer to memory throughput ratio for high enough core count.
./bin/spmv_benchmark
-------------------------------------------------------------------
Benchmark Time
-------------------------------------------------------------------
BM_BlockSparseRightMultiplyAndAccumulateBA/1 28.5 ms
BM_BlockSparseRightMultiplyAndAccumulateBA/2 15.7 ms
BM_BlockSparseRightMultiplyAndAccumulateBA/4 9.01 ms
BM_BlockSparseRightMultiplyAndAccumulateBA/8 5.60 ms
BM_BlockSparseRightMultiplyAndAccumulateBA/16 3.86 ms
BM_BlockSparseRightMultiplyAndAccumulateBA/28 3.84 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/1 23.8 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/2 15.0 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/4 8.01 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/8 4.02 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/16 2.39 ms
BM_BlockSparseRightMultiplyAndAccumulateUnstructured/28 1.68 ms
BM_BlockSparseLeftMultiplyAndAccumulateBA 30.7 ms
BM_BlockSparseLeftMultiplyAndAccumulateUnstructured 41.5 ms
BM_CRSRightMultiplyAndAccumulateBA/1 24.1 ms
BM_CRSRightMultiplyAndAccumulateBA/2 13.6 ms
BM_CRSRightMultiplyAndAccumulateBA/4 8.70 ms
BM_CRSRightMultiplyAndAccumulateBA/8 5.34 ms
BM_CRSRightMultiplyAndAccumulateBA/16 3.99 ms
BM_CRSRightMultiplyAndAccumulateBA/28 4.00 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/1 21.1 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/2 10.83 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/4 5.88 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/8 3.68 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/16 2.21 ms
BM_CRSRightMultiplyAndAccumulateUnstructured/28 1.71 ms
BM_CRSLeftMultiplyAndAccumulateBA 23.6 ms
BM_CRSLeftMultiplyAndAccumulateUnstructured 22.5 ms
BM_CudaRightMultiplyAndAccumulateBA 0.679 ms
BM_CudaRightMultiplyAndAccumulateUnstructured 0.480 ms
BM_CudaLeftMultiplyAndAccumulateBA 0.774 ms
BM_CudaLeftMultiplyAndAccumulateUnstructured 0.361 ms
./bin/partitioned_matrix_view_benchmark
-----------------------------------------------------------------
Benchmark Time
-----------------------------------------------------------------
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/1 18.5 ms
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/2 10.7 ms
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/4 6.34 ms
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/8 4.26 ms
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/16 3.86 ms
BM_PatitionedViewRightMultiplyAndAccumulateE_Static/28 3.75 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/1 18.8 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/2 11.9 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/4 6.94 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/8 4.41 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/16 3.63 ms
BM_PatitionedViewRightMultiplyAndAccumulateF_Static/28 3.86 ms
Timings correspond to intel 8176 cpu and 2080ti nvidia gpu,
with OpenMP threading backend.
Change-Id: Idc07d0563103d057ca3c8412de81a7823fe232af
A previous change to fix a bug related to how x_norm was being
initialized changed the semantics of the TrustRegionMinimizer
loop. With the bug, the parameter_tolerance was being ignored
till the first successful step was encountered.
parameter_tolerance based convergence is a hack anyways, so
restoring the previous semantics is reasonable and will preserve
existing code/tests that depend on this behaviour.
Change-Id: Ia00ca6c47f77e74dee64ad746f4299a9eab0eb7e
* Use the context stream, not the default stream, since all
CUDA operations now operate on the context stream.
Change-Id: I24c872c467f13ee4d276fc04d6822b79a18070fc
Benchmark for evaluation of residuals and evaluation of both residuals
and jacobian.
For each input file (in BAL format) specified on the command line two
sets of benchmarks are created for different number of threads.
BAL data is preloaded before starting benchmark.
Usage:
./bin/evaluation_benchmark [flags] input_1.txt ... input_N.txt
Change-Id: I543df65f483e3b186f52785b68bd5e2b3e3552a3
Conversions function include Euler Angles to / from Rotation Matrices
and Quaternions. They are generalized for any Euler convention that can
be specified in the arguments. Algorithm is from "Euler angle
conversion", Ken Shoemake, Graphics Gems IV
Change-Id: I7f9ddc0b8d686efca16299d2ba374295744376ce
Implemented templated invocation routines for ParallelFor backends
in order to improve loop body inlining.
Several modifications of ParallelFor implementation using CXX threads:
- Index order changed from interleaved to sequential
- Static task scheduling replaced with dynamic (controlled by
kWorkBlocksPerThread)
- Changed index retrieval to atomic
Modifications of OpenMP backend:
- Changed loop scheduling to guided
Changing index order from interleaved to sequential in parallel seem
to significantly improve run-times of parallel loops, for example in
evaluation of jacobian and residuals.
Other modifications provide minor improvements for unbalanced
sub-problem lengths and parallel for loops with small number of
computation per operation.
Single-threaded performance was improved by avoiding costs of
wrapping parallel loop bodies in std::function.
On BAL dataset the following improvements in time consumed for
evaluation of residuals or jacobian and residuals were observed:
OLD NEW OLD/NEW
dataset threads r J r J r J
problem-257-65132-pre.txt 1 0.025 0.079 0.025 0.074 1.016 1.056
problem-257-65132-pre.txt 2 0.030 0.062 0.022 0.050 1.333 1.246
problem-257-65132-pre.txt 4 0.023 0.052 0.014 0.034 1.592 1.515
problem-257-65132-pre.txt 8 0.015 0.035 0.010 0.025 1.477 1.401
problem-257-65132-pre.txt 16 0.011 0.027 0.008 0.019 1.365 1.377
problem-356-226730-pre.txt 1 0.150 0.442 0.147 0.412 1.017 1.070
problem-356-226730-pre.txt 2 0.155 0.322 0.100 0.281 1.542 1.145
problem-356-226730-pre.txt 4 0.129 0.291 0.089 0.196 1.439 1.485
problem-356-226730-pre.txt 8 0.091 0.184 0.066 0.139 1.381 1.319
problem-356-226730-pre.txt 16 0.070 0.148 0.055 0.110 1.272 1.340
problem-1723-156502-pre.txt 1 0.084 0.243 0.082 0.229 1.023 1.063
problem-1723-156502-pre.txt 2 0.088 0.188 0.055 0.154 1.589 1.222
problem-1723-156502-pre.txt 4 0.072 0.159 0.049 0.108 1.475 1.475
problem-1723-156502-pre.txt 8 0.050 0.105 0.037 0.077 1.348 1.368
problem-1723-156502-pre.txt 16 0.038 0.083 0.030 0.062 1.269 1.344
problem-1778-993923-pre.txt 1 0.621 1.777 0.609 1.667 1.018 1.065
problem-1778-993923-pre.txt 2 0.621 1.273 0.415 1.199 1.494 1.061
problem-1778-993923-pre.txt 4 0.514 1.140 0.361 0.786 1.421 1.449
problem-1778-993923-pre.txt 8 0.365 0.808 0.277 0.559 1.319 1.443
problem-1778-993923-pre.txt 16 0.279 0.608 0.223 0.441 1.252 1.379
problem-13682-4456117-pre.txt 1 3.877 10.726 3.738 10.082 1.037 1.063
problem-13682-4456117-pre.txt 2 3.310 7.170 2.423 6.448 1.366 1.111
problem-13682-4456117-pre.txt 4 3.070 6.344 2.064 4.474 1.486 1.417
problem-13682-4456117-pre.txt 8 2.051 4.612 1.527 3.133 1.343 1.472
problem-13682-4456117-pre.txt 16 1.549 3.453 1.218 2.488 1.271 1.387
Run time in seconds for a single evaluation, using evaluation_benchmark
numactl -N 0 -m 0 ./bin/evaluation_benchmark --bal_root ${path_to_BAL}
Evaluation was performed on 28-core CPU.
Note: performance when running across numa-nodes degrades in both old
and proposed implementations, thus the test was executed limiting memory
and compute resources allocation to a single numa-node.
Change-Id: Ia195580bdab9d05c95ac983bfe37b045eecfaf49
* All Cuda* objects now take in a ContextImpl* during
construction, and save the context instead of individual
handles.
* Since we no longer use the legacy default stream, we need to
explicitly synchronize the stream before performing GPU->CPU
transfers, and CudaBuffer is responsible for such synchronization
when asked to perform GPU to CPU transfers.
* Remove all manual syncs and relegate syncing to CudaBuffer
before performing GPU to CPU transfers.
Change-Id: Ic73cb24174a1e09842827323280e90241716cc20
The value of x_norm_ was computed and then incorrectly set to -1.
This meant that ParameterToleranceReached was using the incorrect
value till such time as the minimizer made its first successful
step.
This change removes the member variable and just computes
the norm of x inside ParameterToleranceReached.
Fixes https://github.com/ceres-solver/ceres-solver/issues/885
Thanks to Anton Adanasyev for reporting this.
Change-Id: Ib4d52a45c2d925557ce2c5b57de8a9fa37da6c70
Previously some matrices used Block to keep track of
row/column block sizes and some would just use ints, and
then compute the position of each row and column from it.
By uniformly using Block everywhere, we reduce duplicate
computation and data copies.
I also cleaned up a bunch of c++17 related stuff as I edited
these files.
Change-Id: I4c86b1593fd4c91f9057fbb38314f62f303e0477
* The algorithm enum value for SpMV is now selected based on the
version of the CUDA runtime that Ceres is compiled against.
Change-Id: I2e0e39f1cbdb8ac26d2a9d45f4ebfc09b96d872b
Instead of iterating row-wise, use the row block structure
to update the preconditioner one row block at a time.
While benchmarking, I also found an opportunity to speed up
BlockSparseJacobiPreconditiner.
Before
-----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------
BM_BlockSparseJacobiPreconditioner 132122392 ns 132103000 ns 5
BM_BlockCRSJacobiPreconditionerBA 73365217 ns 73335800 ns 10
BM_BlockSparseJacobiPreconditionerUnstructured 92407762 ns 92407714 ns 7
BM_BlockCRSJacobiPreconditionerUnstructured 72256367 ns 72256400 ns 10
After
-----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------------------------
BM_BlockSparseJacobiPreconditionerBA 39723456 ns 39722222 ns 18
BM_BlockCRSJacobiPreconditionerBA 46561625 ns 46561667 ns 15
BM_BlockSparseJacobiPreconditionerUnstructured 52676208 ns 52676167 ns 12
BM_BlockCRSJacobiPreconditionerUnstructured 54430564 ns 54421462 ns 13
Change-Id: I6da8e044b26f0c24481deee5722322a606e9e5ca
1. Add CreateFakeBundleAdjustmentJacobian to create
bundle adjustment structured jacobians.
2. sparse_linear_operator_benchmark -> spmv_benchmark
3. Refactor spmv_benchmark to use CreateFakeBundleAdjustmentJacobian.
4. Add BA and unstructured Jacobian variants of spmv benchmarks
5. Add BA and unstructured Jacobian benchmarks for the block Jacobi
preconditioner.
6. Fix BlockSparseMatrix::ToCompressedRowSparseMatrix to add the
row and column block structure to the output.
Change-Id: I737a3d7d82dad665b1f2e7886c2c3aedd702ffcd
Use the more modern form of accessing the data array of a vector
rather than grabbing the pointer to the first element. The latter
can lead to errors if the vector is of zero length.
Change-Id: Ifc8fc969b06b3ba1a9385e8a3a8d5c50b25db5a8
* Added GPU device and CUDA compute capability identification.
* Added GpuMemoryAvailable() to aid downstream optimizations based
on GPU memory availability.
Change-Id: I326dc1e4b7a6a7f5571b7e5479eb9aa300ad1075
1. Use hardware_concurrency to configure number of threads.
2. Use user ordering instead of automatic ordering.
Fixes https://github.com/ceres-solver/ceres-solver/issues/874
Change-Id: I4d7d69612e1ee40358943019a0c3a909e419c4e4
With -flto=auto, GCC emits multiple warnings in release builds such as
In function ‘make_unique’,
inlined from ‘Create’ at ceres-solver/internal/ceres/scratch_evaluate_preparer.cc:43:75,
inlined from ‘CreateEvaluatePreparers’ at ceres-solver/internal/ceres/compressed_row_jacobian_writer.h:95:66,
inlined from ‘__ct ’ at ceres-solver/internal/ceres/program_evaluator.h:120:9,
inlined from ‘Evaluate.constprop.isra’ at ceres-solver/internal/ceres/problem_impl.cc:695:65:
/usr/include/c++/12.1.1/bits/unique_ptr.h:1080:30: warning: argument 1 value ‘18446744073709551615’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
1080 | { return unique_ptr<_Tp>(new remove_extent_t<_Tp>[__num]()); }
| ^
because a signed integer is used to specify the size of allocated arrays
instead of the expected unsigned (specifically, std::size_t) without
checking for negative values at the call site.
Change-Id: I923b1d074241535426bfea041568ef1dc7f3ec86
1. Rename BlockJacobiPreconditionet to BlockSparseJacobiPreconditioner.
2. Add CompressedRowSparseJacobiPreconditioner which is a
block Jacobi preconditioner for CompressedRowSparseMatrix objects.
3. Re-write the tests to be more comprehensive.
Change-Id: Icbc91f9ad2cefaad593c11397f8cdcf805d7e118
* Added CudaCgnrSolver, a new CUDA-accelerated CGNR.
* To use CudaCgnrSolver, the user must select CGNR as the linear_solver
and CUDA_SPARSE as the sparse_linear_algebra_library.
* Updated ConjugateGradientSolver to work with an array of pointers to
scratch to support CudaVectors as scratch.
* Moved CUDA initialization to run in Solver::Solve as needed.
Some performance comparisons on an Ubuntu 20.04 desktop with an
Intel i9-9940X CPU @ 3.30GHz, and an nVidia Quadro RTX 6000,
all configurations run with 24 threads, and 10 iterations.
=================================================
CGNR + CUDA_SPARSE + IDENTITY Preconditioner
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.724755e+06
Change 2.546725e+08
Minimizer iterations 11
Successful steps 7
Unsuccessful steps 4
Time (in seconds):
Preprocessor 4.020158
Residual only evaluation 1.567092 (10)
Jacobian & residual evaluation 7.847130 (7)
Linear solver 31.688898 (10)
Minimizer 46.834987
Postprocessor 0.353974
Total 51.209120
=================================================
SPARSE_SCHUR (CPU) + SUITE_SPARSE + AMD
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.651617e+06
Change 2.547457e+08
Minimizer iterations 11
Successful steps 11
Unsuccessful steps 0
Time (in seconds):
Preprocessor 35.812003
Residual only evaluation 1.658980 (10)
Jacobian & residual evaluation 12.218799 (11)
Linear solver 76.409992 (10)
Minimizer 98.809773
Postprocessor 0.372712
Total 134.994489
=================================================
ITERATIVE_SCHUR (CPU) + JACOBI Preconditioner
problem-1778-993923-pre.txt
=================================================
Cost:
Initial 2.563973e+08
Final 1.684447e+06
Change 2.547128e+08
Minimizer iterations 11
Successful steps 8
Unsuccessful steps 3
Time (in seconds):
Preprocessor 15.331614
Residual only evaluation 1.606114 (10)
Jacobian & residual evaluation 8.502166 (8)
Linear solver 351.910080 (10)
Minimizer 368.797327
Postprocessor 0.363536
Total 384.492478
=================================================
CGNR + CUDA_SPARSE + IDENTITY Preconditioner
problem-13682-4456117-pre.txt
=================================================
Cost:
Initial 1.126372e+09
Final 2.269329e+07
Change 1.103678e+09
Minimizer iterations 11
Successful steps 7
Unsuccessful steps 4
Time (in seconds):
Preprocessor 19.140087
Residual only evaluation 8.721920 (10)
Jacobian & residual evaluation 41.955923 (7)
Linear solver 214.121861 (10)
Minimizer 296.636890
Postprocessor 1.971827
Total 317.748804
Change-Id: I3a09f31aa6903f661e91f595afd39d427583e856
* Added CudaSparseMatrix to manage and operate on sparse matrices with
cuSparse.
* Added tests for CudaSparseMatrix.
* Added a new sparse linear operator benchmark.
Change-Id: Id09df46de3b40be1f14441528088b54dab5844af
Mike Vitus