By ARC Labs contributor, Dylan Michalak, Threat Research Intern

Sleep obfuscation is a term encompassing malware that waits for some time period to avoid detection. It could include extended sleeping, where malware will wait an extended time (10+ minutes) to start executing to evade shorter sandbox analysis. It could also encompass logic bombs, malicious code that is set to execute on a specific trigger or at a scheduled time, which could be days, weeks, or months into the future. Recently, however, it most refers to a category of in-memory obfuscation techniques that encrypts memory-resident malicious code and data, suspends execution for a pre-determined amount of time, decrypts the memory, executes the malicious code, then encrypts itself again start the cycle over. All in-memory sleep obfuscation techniques function similarly at a more abstract level, but differences in their implementations mean that they produce different telemetry. A method to detect one may not work for the others, and it’s important to be aware those differences when investigating them as a potential threat.

The focus of this research will be on malware that uses regular sleep and encryption cycles to hide its behavior. Initial research involves looking at the source code and analyzing the behavior of three popular open-source sleep-obfuscation implementations:

Ekko

Ekko works by using Windows Timer Queues to queue encryption, sleep and decryption. Timers are queued to call NtContinue, which is passed thread contexts set to execute functions that handle changing memory permissions, encryption, and decryption during the sleep cycle.

One potential detection that didn’t work out was based on how a CreateRemoteThread function was called when executing the POC. Below is a screenshot of Microsoft Sentinel logs showing CreateRemoteThreadApiCall events from Ekko. It would be possible to differentiate this behavior from applications that use Timer Queues and CreateRemoteThread legitimately.

Unfortunately, these events are only logged when executing the Ekko proof of concept. When running a payload that uses Ekko, generated by the Havoc Command and Control client, these security events are not traced.

After tracing the function calls of the exposed kernel32 function “SetTimerQueueTimer” used by Ekko, there is an NtTraceEvent call, a function used by windows to trace and log security events. This instance of NtTraceEvent may be used to record SetTimerQueue function calls, which are critical to Ekko’s functionality. This will be addressed further in the Debugging/Decompiling section.

Cronos

Possible IOCs:

Cronos works by setting WaitableTimer objects and using a ROP chain to keep waiting for APCs made by each WaitableTimer, to hide its function calls used for sleep obfuscation.

Specifically, WaitableTimers, a type of timer object that waits for a set time and then calls a completion routine (an Asynchronous Procedure Call), are set with completion routines that contain functions core to the sleep obfuscation cycle. The rop.asm file in the source-code contains the critical “QuadSleep” function which, four times, allocates memory to the thread’s stack, pushes the addresses of ROP gadgets onto the stack that will supply parameter values necessary for calling SleepEx, and pushes the address to call SleepEx. When called, SleepEx is called four times, each time it waits for the next WaitableTimer to finish, and then executes the completion routine that it contains.

One potential detection is based around the QueueUserApcRemoteApiCall DeviceEvent logs that it consistently generates while it runs in Sentinel. This will be covered more in-depth later in the “Detections” section.

FOLIAGE

This is the oldest of the three techniques looked at, but it still has widespread popularity. FOLIAGE directly queues a series of user mode Asynchronous Procedure Calls that sequentially pass thread contexts (these are used to alter what the thread is currently executing) that are used to call core sleep obfuscation functions that handle decryption, changing memory permissions, and sleeping.

Due to the fact that FOLIAGE uses many library functions that can directly create threads and set thread context (NtQueueApcThread, NtCreateThread), it regularly generates SetThreadContextRemoteApiCall event logs that can be used to detect its behavior. Once again, the usage of these logs will be covered more in-depth in “Detections”.

Debugging/Decompiling

When looking at the libraries used by the POCs themselves, Binary Defense primarily focused on Ekko and Cronos. The goal was to find NtTraceEvent or related Windows event tracing function calls that could potentially be leveraged to detect these sleep obfuscation methods natively. All analysis was done on Windows 11 Enterprise Edition.

Launching the Ekko executable, attaching WinDbg, and setting breakpoints on ntdll!NtTraceEvent and other ETW functions didn’t show any sort of event tracing during the sleep process. Starting the executable from WinDbg, on the other hand, did break several times on NtTraceEvent calls as seen in the below screenshot during debugging:

These calls, however, only come from up the callstack from RtlUserThreadStart calls at the start of program execution, indicating that the event tracing was only related the start of the new process/thread and not the use of TimerQueues, encryption libraries, or anything else directly indicative of sleep-based obfuscation.

The TimerQueue functions used by Ekko are exposed in the windows kernel32 library. Generally, these functions wrap functions of the same name in kernelbase, which houses a less abstract implementation. These kernelbase functions are implemented with ntdll library functions, which are even lower level and eventually lead to system calls. After looking through decompiled windows libraries containing TimerQueue functions (ntdll, kernelbase, kernel32), one potential security event tracing call was found:

CreateTimerQueueTimer (kernel32) -> CreateTimerQueueTimer (Kernelbase) -> RtlCreateTimer (ntdll) -> TpSetTimerEx (ntdll) -> TppSetTimer (ntdll) -> TppETWTimerSet:

This event trace seems like it could potentially be leveraged to build a detection around the use of Timer Queues. However, it never gets reached in the execution of the program and doesn’t correlate with any actions recorded in Microsoft Sentinel. Given the increased popularity of incorporating sleep-obfuscation techniques into payloads, with some C2s directly incorporating techniques like Ekko or Foliage, adding more native event tracing capabilities that could be leveraged to detect patterns of encryption, waiting, and decryption seems like it would be valuable.

When looking at it with WinDbg, the Cronos POC executable also does not appear to result in any ETW telemetry during the actual runtime of the sleep obfuscation cycle.

Detections

For investigating detections for sleep obfuscation, payloads generated by the Havoc C2 were used. Havoc is an open-source command and control server and client made by C5pider, who also happens to be the creator of Ekko. Most, if not all default payloads have already been signatured, so for the purpose of this testing Defender alerts about executables matching known malicious signatures will be ignored.

Memory Scanners

Although resource-intensive and not widely applicable for businesses, memory scanners are a consistent tool to identify potential sleep obfuscation. Although the purpose of sleep obfuscation is to hide from in-memory detection, the nature of encrypting entire PEs (Portable Executables) in memory and modifying the permissions of that memory results in some strong indicators of compromise (IoC).

Moneta is a popular open-source memory scanner, which looks at the PE structures of committed memory regions, image files on disk, and several other data sources to identify anomalous memory and IoCs. Using it to look at active Havoc executable payloads utilizing Ekko and Foliage generated the following output:

Moneta lists several IOCs, which are indicators of malicious activity, for the Ekko malware sample in-memory. “Modified PE header” is due to the memory of the portable executable being encrypted during the scan. Additionally, “Inconsistent +x between disk and memory” (+x meaning “executable”, specifying a region of memory executes code rather than just containing data) is likely due to Ekko changing the memory permissions to only read/write while it is encrypted, which is inconsistent with the memory being allocated for a Portable Executable (.exe).

In the case of a process injection as opposed to a standalone executable, a memory scan may be less straightforward, as the whole process in memory would no longer have fully incorrect permissions and an only partially encrypted process is significantly less likely to be malicious.

PE-Sieve is another memory scanner that was used to analyze the sleep obfuscated payloads. It is primarily useful for detecting injected PEs, inline hooks, and hollowed processes.

Its output is less clear, but the “scans” section, shown in Figure 7, does show that the PE header in-memory is malformed, potentially indicating encryption.

POC Detections

CFG-Find-Hidden-Shellcode is an open-source proof of concept tool made to utilize an unintentional consequence of the CFG bitmap, a relatively new Windows security feature. The bitmap only updates when a region is marked as executable, not when it is marked as non-executable, which inadvertently keeps a record of all private memory regions that were previously executable.

Theoretically, this should be able to detect Ekko, Cronos, and FOLIAGE as they all alter memory permissions during the sleep cycle. However, during testing it never identified the Havoc Ekko or FOLIAGE payloads, and it wasn’t able to identify any of the POCs either.

Hunt-Sleeping-Beacons is an open-source proof of concept tool that looks for suspicious APCs and suspicious timers among other things to attempt to identify malware utilizing sleep obfuscation. During testing it somewhat consistently was able to detect the running POCs for FOLIAGE, Ekko, and Cronos, but never identified any of the Havoc payloads, demonstrating that the C2 demons have applied bypasses.

SIEM Detections

The most efficient detection would be a query that uses logs already collected by existing security monitoring tools, such as Microsoft defender or System Monitor. Luckily, the Cronos POC as well as the FOLIAGE Havoc payload both consistently generate event logs in Microsoft Defender and Sentinel during their runtime:

As shown in Figure 8 and 9, both Cronos and FOLIAGE regularly generate some API call action logs. Cronos’ use of WaitableTimers and NtContinue generates QueueUserApcRemoteApiCall logs and SetThreadContextRemoteApiCall logs. The FOLIAGE payload consistently generates only SetThreadContextRemoteApiCall logs during its runtime. Havoc does not have the capability to generate a payload that uses Cronos sleep obfuscation, so this detection is only tested on the POC, for Cronos.

A query that looks for repetition of these actions by a single process, as well as one that correlates these actions with remote server connection attempts could be a way to detect payloads that use Cronos or FOLIAGE in Microsoft Defender:

As seen above in Figure 10, the query is able to successfully identify the Havoc FOLIAGE payload connected to the C2. Running a query with no network correlation is also able to detect the Cronos POC, which has no related network connection.

Unfortunately, the implementation of Ekko doesn’t result in repeated action logs being generated in a similar way. The only DeviceEvent logs generated are just the ones related to Microsoft Defender identifying the Havoc payload’s static signature, which isn’t useful for detecting process behavior.

Conclusion

In-memory sleep obfuscation and related detections is a research topic that can take many different directions. Although the current existing telemetry logged by Microsoft Defender and Sysmon is limited and the final detections can only target two of the three investigated techniques, they hopefully provide a good basis for building future detections around the behavior of sleep cycles. No matter what techniques or APIs are used to perform the encrypt, sleep, decrypt cycle, there should theoretically always be the potential to identify and detect that behavior through existing or future telemetry collected in those APIs.

While memory scanners and other POC tools can be useful to gain additional insight into the likelihood of an identified piece of software to be sleep-obfuscated malware, they aren’t practical as an automated, regular detection. It will be important for businesses to enable logging for sleep and encryption APIs, whether that’s through Defender, System Monitor, or some other endpoint agent, in order to detect malware hidden with a sleep mask

Interestingly, one shared potential point of detection that was not thoroughly investigated is the NtContinue function that all the investigated techniques use to execute thread contexts with a single-parameter function. If there is a consistent way to differentiate this malicious usage of NtContinue from normal usage, that could lead to a very comprehensive detection that covers many existing sleep obfuscation techniques.