ScTagQuery: Mapping Service Hosting Threads With Their Owner Service

Today I want to introduce another utility for Vista and Windows Server 2008 called ScTagQuery (short for Service Controller Tag Query), a tool which will allow you identify to which running service a certain thread inside a service hosting process (e.g Svchost.exe) belongs to, in order to help with identifying which services may be using up your CPU, or to better understand the stack trace of a service thread.

If you’ve ever had to deal with a service process on your system taking up too much CPU usage, memory, or other kinds of resources, you’ll know that Task Manager isn’t particularly helpful at finding the offending service because it only lists which services are running inside which service hosting process, but not which ones are consuming CPU time (in fact, some Svchosts host more than a dozen services!)

Process Explorer can also display this information, as well as the names of the DLLs containing the services themselves. Combined with the ability to display threads inside processes, including their starting address (which for some service threads, identifies the service they are associated with by corresponding to their service DLL) and call stack (which helps identify additional threads that started in a generic function but entered a service-specific DLL), Process Explorer makes it much easier to map each thread to its respective service name. With the cycles delta column, the actual thread causing high CPU usage can reliably be mapped to the broken or busy service (in the case of other high resource usage or memory leaks, more work with WinDBG and other tools may be required in order to look at thread stacks and contexts).

However, with each newer Windows release, the usage of worker pool threads, worker COM or RPC communication threads, and other generic worker threads has steadily increased (this is a good thing – it makes services more scalable and increases performance in multiprocessor environments), making this technique a lot less useful. These threads belong to their parent wrapper routines (such as those in ole32.dll or rpcrt4.dll) and cannot be mapped to a service by looking at the start address. The two screenshots below represent a service hosting process on my system which contains 6 services — yet only one of the DLLs identified by Process Explorer is visible in the list of threads inside the process (netprofm.dll).

Again, there’s nothing wrong with using worker threads as part of your service, but it simply makes identifying the owner of the actual code doing the work a lot harder. To solve this need, Windows Vista adds a new piece of information called the Service Tag. This tag is contained in the TEB of every thread (internally called the Sub-Process Tag), and is currently used in threads owned by service processes as a way to link them with their owning service name.

When each service is registered on a machine running Windows Vista or later, the Service Control Manager (SCM) assigns a unique numeric tag to the service (in ascending order). Then, at service creation time, the tag is assigned to the TEB of the main service thread. This tag will then be propagated to every thread created by the main service thread. For example, if the Foo service thread creates an RPC worker thread (note: RPC worker threads don’t use the thread pool mechanism – more on that later), that thread will have the Service Tag of the Foo service.

ScTagQuery uses this Sub-Process Tag as well as the SCM’s tag database and undocumented Query Mapping API (I_ScQueryTagInformation) to display useful information in tracking down rogue services, or simply to glean a better understanding of the system’s behavior. Let’s look at the option it supports and some scenarios where they would be useful.

During a live session, the -p, -s and -d options are most useful. The former will display a list of all service threads which have a service tag inside the given process, then map that tag to the service name. It can also function without a process ID, causing it to display system-wide data (enumerating each active process). The -s option, on the other hand, dumps which service tags are active for the process, but not the actual threads linked to them — it then links these tags to their service name. Finally, the -d option takes a DLL name and PID and displays which services are referencing it. This is useful when a thread running code inside a DLL doesn’t have an associated service tag, but the SCM does know which service is using it.

The -a and -n options are particularly useful when you’ve obtained a service tag from looking at a crash dump yourself, and run the tool on the same system after a reboot. The -t option will let you map the service name if you know the PID of the service. If the PID changed or is otherwise unrecoverable, the -a option will dump the entire SCM tag database, which includes services which are stopped at the moment. Because these mappings are persistent across reboots, you’ll be able to map the thread with the service this way.

On the other hand, if all you’re dealing with is one thread, and you want to associate it to its service, the -t option lets you do just that.

Back to that same svchost.exe we were looking at with Process Explorer, here are some screenshots of ScTagQuery identifying the service running code inside rpcrt4.dll, as well as the service referencing the fundisc.dll module.

Note that Thread Pool worker threads do not have a Service Tag associated to them, so the current version of ScTagQuery on Vista cannot yet identify a service running code inside a worker pool thread inside ntdll.dll.

Finally, I should mention that ScTagQuery isn’t the only tool which uses Service Tags to help with troubleshooting and system monitoring: Netstat, the tool which displays the state of TCP/IP connections on a local machine, has also been improved to use servicre tags to better identify who owns an open port (Netstat uses new information returned by various Iphlpapi.dll APIs which now store the service tag of every new connection). With the -b option, Netstat is now able to display the actual service name which owns an active connection, and not just the process name (which would likely just be svchost.exe).

The tool page for ScTagQuery is located here. You can download ScTagQuery in both 32-bit and 64-bit versions from this link. Windows Vista or higher is required.