Java SDK usage for usecases involving inactive users

When you have  a BoxUser or BoxFolder Object created using the BOXAPIConnection Object of  an inactive user  the objects are unusable.

Method calls on the BoxUser/BoxFolder object throw 403/Unauthorized error.

If I want to for e.g iterate the list of folders an inactive user has, what  is the way out?