Do you really need to hook it?

An evergreen question 'How should I hook...? I want to monitor/ block...' How many of us really look up before asking this question? Hooking has been a classic old way of doing things, but time changes and so do the techniques. Hooking is powerful, but it is bad. Use hooking when you have no other alternative. I have seen many people use hooking just out of ignorance; they don't know that a documented way exists to the same thing.

Remember the classic old tool Regmon? The initial versions of regmon used system call hooking. But then as newer techniques with better features supported got introduced, Regmon started using them. Regmon uses system call hooking till Windows XP and uses registry callback mechanism Windows 2003 onwards.

 

I like to give the example of registry filtering for making people understand when they should use hooking. Let's take 3 tasks:

1. Monitoring registry calls.
   
2. Blocking registry operations.
   
3. Modifying data for a registry operation.
   

 

Let's first take the example of a person who has just started Windows programming and has become aware of the concept of hooking. The first thing that might come in his mind is that all three tasks are do-able by system call hooking of the registry operations. But is it really so? He needs to ask himself two things: Which versions of OS (NT/ 2K/ XP/ 2K3, Vista/ 2K8) do I need to support along with the platforms (x86/ x64/ IA64)? And if that answer includes x64 or IA64, his idea of system call hooking has blown apart by now. Because system call hooking is not supported on 64 bit Windows. A special component of the OS called the Patch Guard prevents system call hooking on 64 bit Windows. But be very sure that on NT and 2K you have no other mechanism other than hooking for doing any of the three tasks.

Before we go any further, let me just tell you that there is registry callback mechanism in Windows starting from Windows XP. But how useful it actually is depends on the functionality that we want to achieve. Anyways, let's take each task one by one.

1. Monitoring registry calls: Just by the look of it, we feel that if registry callback mechanism is present the bare minimum support that it should provide is that of giving plain notification like callbacks with both the pre-operation callback and a post-operation callback. But life is not so good! The registry callback mechanism of Windows XP is like an engineering project that a student tries to get over with without thinking much about the required functionality. The point is that the registry callback mechanism in Windows XP does not even provide post callbacks for all operations. So what now? Does the developer have an option? Yes, of course. He should use system call hooking for Windows XP. But what about Windows XP x64? I have seen many people ask this question. But the fact is that Windows XP x64 is actually not Windows XP from the inside; it's just Windows XP from the outside. Windows XP x64 is build from the same source code as Windows 2003, and hence is different from Windows XP x86. So, all the functionality that Windows 2003 x64 has, Windows XP x64 also has. Now, what about Windows 2003, Vista and 2008? The registry callback mechanism in 2003 is better than XP but not as good as in Vista. But this task of monitoring most of the registry calls can be achieved in both OS by the callback mechanism.
   
2. Blocking registry operations: As such the blocking of registry calls can be achieved by callback mechanism in XP, 2003 and Vista/ 2008. However, since the monitoring functionality itself cannot be achieved properly in XP, hooking still remains the option for XP. For 2003 and Vista/ 2008, the callback mechanism does a good job.
   
3. Modifying data for a registry operation: This is one task that gets a little tricky and questionable. For XP, hooking still remains the technique for achieving this. For Vista/ 2008, the callback mechanism supports it properly. However, the problem comes for Windows 2003. You get stuck in both the techniques: The callback mechanism does not support modification of parameters (though the documentation said that the parameters could be modified. But that is wrong!) And if you choose hooking you won't be able to support Windows 2003 64 bit. Here comes a need for business decision! What does a customer require and what is do-able?
   

Above mentioned example is one of the simplest ones. However, in reality I have seen that some security products really need to hook. But again, they fail to provide that functionality in their 64 bit versions because the hooking mechanism fails there. Again, I have seen some people that do not want to do things in a documented fashion; they WANT to hook! This approach is not good. There can be an endless discussion on hooking vs. documented way, but I believe that the decision should be taken based on the functionality required and the built-in features available from the OS to do it in a documented fashion. Though interesting, but don't always put a hook because that may make you a crook!