The stored deformation energy in the dislocation structures in a polycrystalline metal can provide a sufficient driving force to move grain boundaries during annealing. In this paper, a thermodynamically-consistent three-dimensional, finite-strain and dislocation density-based crystal viscoplasticity constitutive theory has been developed to describe the distribution of stored energy and dislocation density in a polycrystalline metal. The developed constitutive equations have been numerically implemented into the Abaqus finite element package via writing a user material subroutine. The simulations have been performed using both the simple Taylor model and the full micromechanical finite element model. The theory and its numerical implementation are then verified using the available data in literature regarding the physical experiments of the single crystal aluminum. As an application of the developed constitutive model, the relationship between the stored energy and the strain induced grain boundary migration in aluminum polycrystals has been investigated by the Taylor model and also, the full finite element model. The obtained numerical results indicated that the Taylor model could not precisely simulate the distribution of the stored deformation energy within the polycrystalline microstructure; consequently, the strain induced grain boundary migration. This is due to the fact that the strain induced grain boundary migration in a plastically deformed polycrystalline microstructure is principally dependent on the spatial distribution of the stored deformation energy rather than the overall stored energy value.