Post by Viral ShahIt could be because of memory usage. I have 1 TB RAM on the machine I was
doing. If you were running into swap, it would certainly take much longer.
I will try the other version as soon as the machine is available for me to
use (some admin issues), and also look into speeding things up if possible.
If the generation of the data is in your control, you can just generate it
pre-sorted or in CSC format. I just need to check if we can shortcut
pre-sorted data and generate the sparse matrix quickly.
-viral
Post by Ryan GardnerHmmm. That is much better than I was getting. Thanks Viral.
Was it much faster for you to create the column-index, row-index, and
value arrays? I would still expect them to be roughly on par in terms of
speed.
Post by Ryan GardnerI ran the sprand example, and it took 290 seconds on a machine with
enough RAM. Given that it is creating a matrix with half a billion
nonzeros, this doesnât sound too bad.
Post by Ryan Gardner-viral
Post by Ryan GardnerI've got 16GB of RAM on this machine. Largely, my question, with
admittedly little knowledge of the internal structure of the sparse arrays,
is why generating the actual SparseMatrixCSC is so much slower than
generating what is essentially another sparse matrix representation
consisting of the indices and values. (I realize that once we start
swapping, which will happen in my example, things slow down a ton, but even
the sprand I mention was slow.) Do you observe the same results? Is the
reason for the difference clear to someone else?
Post by Ryan GardnerPost by Ryan GardnerThanks for all the comments. These are helpful. It had not crossed
my mind that I could control the data type of the indices.
Post by Ryan GardnerPost by Ryan GardnerUsing the SparseMatrixCSC constructor directly would probably be very
helpful. Do you learn about that constructor from looking at source code
or do you see it somewhere else?
Post by Ryan GardnerPost by Ryan GardnerIf you're assembling the matrix in row-sorted column-major order and
there's no duplication, then you can also skip the conversion work by using
the SparseMatrixCSC constructor directly.
Post by Ryan GardnerPost by Ryan GardnerCould you post your code? Will avoid me writing the same. :-)
Was building the vectors taking all the time, or was it in building
the sparse matrix from the triples? Triples to CSC conversion is an
expensive operation, and we have spent a fair amount of time making it
fast. Of course, there could be more opportunities at speeding the code.
Post by Ryan GardnerPost by Ryan Gardner-viral
Post by Dominique OrbanDowngrading the 700,000 to 70,000 for the sake of not waiting all
night, the original implementation takes about 4.3 seconds on my laptop.
sparse matrix takes about 3.8 seconds. This may be due to conversion from
triple to csc format?!
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique Orbanps: using the original size of 700,000, Julia reports a memory usage
of 11.8GB.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique OrbanI believe the memory requirement should be 700000*700*16 (64-bit
nonzeros and row indices) + 700001*8 (64-bit column pointers) = 7.8 GB.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique OrbanThis can be brought down a bit by using 32-bit index values and
64-bit floats, but then you need 5.8 GB. Finally, if you use 32-bit index
values with 32-bit floats, you can come down to 4GB. The Julia sparse
matrix implementation is quite flexible and allows you to easily do such
things.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique Orbanjulia> s = sparse(int32(1:10), int32(1:10), 1.0);
julia> typeof(s)
SparseMatrixCSC{Float64,Int32} (constructor with 1 method)
julia> s = sparse(int32(1:10), int32(1:10), float32(1.0));
julia> typeof(s)
SparseMatrixCSC{Float32,Int32} (constructor with 1 method)
-viral
Sorry for pointing out a probably obvious problem, but as there are
others that might try debug this issue on their laptop, I ask how much
memory do you have? 700000*700 floats + indexes, will spend a minimum of 11
GB (if my math is correct) and possibly more if the asymptotic storage
requirement is more than 2 Int64 + 1 Float64 per stored value.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique OrbanIvar
kl. 01:46:22 UTC+2 onsdag 30. april 2014 skrev Ryan Gardner
Creating sparse arrays seems exceptionally slow.
I can set up the non-zero data of the array relatively quickly. For
example, the following code takes about 80 seconds on one machine.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique Orbanvec_len = 700000
row_ind = Uint64[]
col_ind = Uint64[]
value = Float64[]
for j = 1:700000
for k = 1:700
ind = k*50
push!(row_ind, ind)
push!(col_ind, j)
push!(value, 5.0)
end
end
but then
a = sparse(row_ind, col_ind, value, 700000, 700000)
takes more than at least about 30 minutes. (I never let it finish.)
It doesn't seem like the numbers I'm using should be that far off
the scale. Is there a more efficient way I should be doing what I'm doing?
Am I missing something and asking for something that really is
impractical?
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique OrbanIf not, I may be able to look into the sparse matrix code a little
this weekend.
Post by Ryan GardnerPost by Ryan GardnerPost by Dominique OrbanThe never-finishing result is the same if I try
sprand(700000, 700000, .001)
or if I try to set 700000*700 values in a sparse matrix of zeros
directly. Thanks.