Matrix Completion via Successive Low-rank Matrix Approximation

In this paper, a successive low-rank matrix approximation algorithm is presented for the matrix completion (MC) based on hard thresholding method, which approximate the optimal low-rank matrix from rank-one matrix step by step. The algorithm enables the distance between the matrix with the observed elements and the projection on low-rank manifold to be minimum. The optimal low-rank matrix with observed elements is obtained when the distance is zero. In theory, convergence and convergent error of the new algorithm are analyzed in detail. Furthermore, some numerical experiments show that the algorithm is more e ff ective in CPU time and precision than the orthogonal rank-one matrix pursuit(OR1MP) algorithm and the augmented Lagrange multiplier (ALM) method when the sampling rate is low.


Introduction
The study of recovering an unknown low-rank or approximately low-rank matrix from incomplete samples of its entries has aroused great interest in machine learning( [1], [2], [3]), imaging inpainting( [4], [5], [6]), computer vision( [7]), control( [8], [9]), and so on.This problem is well known as the MC problem, mathematically, min rank(X), where rank(X) denotes the rank of matrix X.where X, D ∈ R m×n , P Ω (D) is the observed matrix, Ω is the set of all index pairs (i, j) of observed entries, P Ω is the orthogonal projector onto Ω.
For the problem (1), wang presented the orthogonal rank-one matrix pursuit algorithm, which only the top singular vector pair is calculated in each iteration and can obtain an ϵ-feasible solution in only O(log(1/ϵ)) iterations with less computation (see [20]).But the algorithm is not satisfied in its precision.This motivates that the new algorithm is designed to improve the precision.In this paper, we present a simple and efficient algorithms to solve the lowrank matrix completion problem (1).Repeat iteration to decrease the error between the matrix with the known elements and its corresponding elements of the projection matrix on {r}-dimensional manifold generated by hard thresholding method.If the error is not reduced and not less than the given precision, more better approximation on {r+1}-dimensional manifold is completed.Thus, repeat the operation till convergence.
The rest of the paper is organized as follows.The successive low-rank matrix approximation algorithms is presented in Section 2. Convergence and convergent error are discussed in Section 3. Section 4 shows the performance of the algorithm compared to the OR1MP and ALM algorithms and a conclusion remark.
Notation.Let X = (x 1 , ..., x n ) ∈ R m×n be an m × n real matrix, where x i ∈ R m×1 .X ij denotes the (i, j)th entry of X.The nuclear norm of a matrix X is denoted by ∥X∥ * , the Frobenius norm by ∥X∥ F .r(X) denotes the rank of a matrix X. Ω ⊂ {1, ..., m} × {1, ..., n} is the indices of the observed entries of a matrix X, Ω is the complementary set of Ω. P Ω is the projection operator onto Ω. [U, Σ, V] = lansvd(Y, r, L) denotes computing the top-r singular pairs of the matrix Y by using the Lanczos method.

Successive Low-rank Matrix Approximation Algorithm
In first, the distance is introduced between the matrix with the observed elements and its projection onto the r-dimensional manifold as following, Obviously, if d(Y * , r) = 0, Y * is the optimal solution of (1).
Y * is researched by repeated iterations in the matrix set with the observed elements until d(Y, r) to be minimum.Thus, d(Y * , r) = 0 can be obtained as rank r raises.The key steps of the algorithm are given in Algorithm 1 in the following.

Convergence Analysis
In this section, the iteration error and convergence of the new algorithm are discussed. where obviously, According to KKT condition, if Y * is the optimal solution, ∂d(Y * ,r) ∂Y = P Ω (U * ), where U * is the Lagrange multiplier.This implies The lemma has been proved.Theorem 1 Let {Y k } be the matrix sequence generated by Algorithm 1. Then Proof.Suppose that Y k is generated by Algorithm ??, Theorem 3 Let Y k be the matrix sequence generated by Algorithm 1 and 0 < c = max k c k < 1.Then the following inequality holds On the other hand, from Lemma 3 it has Thus, (8) holds true for k and r.
Remark: The proof of theorem 3 is divided to two parts, one is the distance shrinks (c 2 ) m i by repeatedly iteration on the {i − 1}-dimensional manifold, another is the distance which shrinks (1 − 1 n−i+1 ) on the {i}dimensional manifold.
In the experiments, "iteration" denotes the number of iterations, "rank" denotes the rank of the recovered solution, sr denotes the sampling ratio and sr = p/n 2 , where p is the number of observed entries.Let sr = 0.1, 0.3, 0.5, respectively.The relative error is rel.err= . For each (n, r), repeat the following procedure 5 times, and the reported results are for a mean.The rank-r matrix is generated by D = LR T , where L ∈ R n×r , R ∈ R n×r are independent matrices with independent identically distributed (i.i.d.) Gaussian entries.For Algorithms 1, the empirical parameter set c = 0.95.For Table 1, the algorithms stops once the termination criterion ϵ = ∥P Ω (X)−P Ω (D)∥ F ∥P Ω (D)∥ F is less than 10 −4 .Since the OR1MP algorithm does not perform well, the maximum number of iterations of OR1MP algorithm is set to 100 (this is because the rank of the matrix generated by OR1MP method is up to 100).
A brief comparison of the algorithms is presented in Table 1.We can see that the SLRMA algorithm achieve better than the OR1MP in the accuracy.The CPU time of the SLRMA algorithm is less than other algorithms.Although ALM method has the least iterations, it is not dominant in the CPU time.As the sampling rate declines, the SLRMA algorithm appears to converge well especially for large-scale matrix completion with less CPU time.Furthermore, except for the OR1MP algorithm, all the other algorithms can recover the rank exactly.
For further comparison, two experiments on a 1000 × 1000 matrix are tested the efficiency of the SLRMA algorithm (see Figure 1).It is shown from the figures that when the rank is fixed, the smaller the sampling ratio is, the less effectiveness the SLRMA algorithm runs; and when the sampling ratio is fixed, the higher the rank is, the less effectiveness the SLRMA algorithm runs.
For the paining image, the SLRMA algorithm has the same effect as the ALM algorithm (see Figure 2), but the SLRMA algorithm costs less time than the ALM algorithm.

Conclusion
In this paper, we first propose an low-rank matrix approximation algorithm.In the algorithm, by introducing the distance between the feasible matrix and its projection onto the r-dimensional manifold (where r is less than or equal to the rank of the completion matrix), the optimal low-rank matrix can be found by multi-step iterations until the minimum distance equal to zero.Meanwhile, Numerical experiments show that the new algorithms achieve much better performance in precision than the OR1MP algorithm and faster than the ALM algorithm, especially for large-scale matrix completion problem.As the observation ratio decreasing, the SLRMA algorithm consumes the least CPU time than the other algorithms.Besides, the SLRMA algorithm can recover the rank of the matrix being recovered exactly.

Figure 1 .
Figure 1.convergence behavior of the relative error in SLRMA runs.

Table 1 .
Numerical results on random matrix completion problem