Unable to implement Oauth + specific folder permission + server fetch of all files

We have to built a feature in our product where we will ask our users to grant access of one of their folders using OAuth flow.

Once we have their access_token with that folder view & download permission, we use this access_token in our backend APIs (server call) and fetch list of files under this folder. Then these files are visible in our portal (thumbnail view/preview) and users can click on them to view the bigger version of file or can download them.

Solutions we tried:
1- We created a client app with OAuth flow (with just read permission) but there wan't any way to grant access of any specific folder (we don't want them to grant access of their whole box account). Also there was one more issue in this flow, we were not able to get list of all files/folders present on root level and without write permission we were not able to fetch download link.

2- Then we tried to create Server app with client credentials (thought of using Users and Group section for permission). Here, the Oauth flow will not work (because it's a server app and we were okay with that unless we have specifc folder access). We created service account (followed all 3 prerequisite) and with 5 keys mentioned in box documentation. Here, we were 400 bad request in response.

I tried all possible solutions and already wasted weeks in it, my question is how can I achieve Oauth with specific folder permission (without write permission if possible) where am able to fetch files on server and also able to download them.

Any help will be appreciated.