Holding onto the pointer is not a good idea, and it is read-only as direct manipulation of the world matrix is not supported. However, this is useful for passing the matrix to the GPU.
And start working on scene management.