Use access token from Azure AD app to use with sharepoint REST API

I have an Azure AD app registered and my web app requires using both graph-api and REST API to fetch sharepoint account information.

For eg:

https://domain.sharepoint.com/_api/Web/RoleAssignments

Problem is the access token i fetch for graph isnt valid for REST. I basically want to avoid having to register my app on the:

https://domain-admin.sharepoint.com/_layouts/15/appinv.aspx

Which asks for an xml block to be added to allow Full Access.

Desired: Single AD app to be used for both graph-api and rest api implementation. Separate tokens are fine.