I'm managing a server running Windows Server 2008 R2 for a client and after investigating some high memory usage, I discovered that the Windows Internal Database SQL instance is using over half of the server's RAM.

Capping the instance's memory usage is easy, but unfortunately my user account (which is a member of domain admins as well as the WSUS administrators group, and WSUS is the reason for this instance existing) does not have permission to make any changes to the database. The built in "sa" account is disabled, with an unknown password (as windows authentication is the default anyway), so I'm at a loss as to how I can give myself enough permissions to manage this instance.

Are there any other Windows administrators here who have run into that same issue, or are familiar with altering permissions on these types of databases?

sql database is used by what program? you can search the web on how to reset or enable "sa" account.

Some sql database programs, does use a lot of memory in the system. You might consider adding some RAM in your system.

have you checked for any malware or virus in your system?

but "sa" account maybe disabled for some reason, if you reset the password database functionality might be affected. Be careful though!