Słomkowski's technical musings

Playing with software, hardware and touching the sky with a paraglider.

ABBYY FineReader 6 no-hardware-dongle hack. Investigating Rainbow/SafeNet iKey 1000 software protection dongle

Originally published on , 5 min read, ~1050 words.

Several years ago, I had to adapt some ancient in-house software to work in a modern environment. This piece of software had a dependency on ABBYY FineReader 6 (released in 2002) as an OCR engine. This specific version of FineReader was secured against piracy by a software protection dongle: Rainbow iKey 1000.

The software I mentioned needed to be used through a Windows Remote Desktop connection. However, the dongles apparently leverage Windows’ smart card subsystem, which makes them cumbersome to use over remote desktop—the dongle had to be plugged into the client, not the server! Additionally, due to their age, some of the dongles we owned began failing.

To ensure compatibility, I needed to find a way to bypass the dongles—namely, write a crack. This proved to be quite easy.

Rainbow/SafeNet iKey 1000

SafeNet, or formerly Rainbow, iKey 1000 is a Windows 9x-era hardware security token (dongle). It plugs into the USB port and provides authentication through a challenge-response mechanism. One may also store a small amount of user data on it.

Older dongles, branded Rainbow iKey 1000.
Newer dongle: SafeNet iKey 1000.
Backside of SafeNet iKey 1000.

The following resources are required to work with the dongles:

This technology is quite ancient and niche, so the information gradually disappears from the web. I have included below some additional materials I could find:

Examining FineReader 6 libraries

The FineReader 6 toolkit consists of several DLLs. According to the iKey 1000 Series Developer’s Guide, all API function names start with the prefix ikey_. I looked for them in all the FineReader DLLs:

strings --print-file-name *.dll | grep ikey_

The output was:

FREngine.dll: ikey_CreateContext
FREngine.dll: ikey_OpenDevice
FREngine.dll: ikey_CloseDevice
FREngine.dll: ikey_DeleteContext
FREngine.dll: ikey_OpenFile
FREngine.dll: ikey_MD5_CHAP
FREngine.dll: ikey_Read

These functions are dynamically loaded from the iKey API library. On 32-bit systems, iKeyAPI.dll is located in C:\Windows\System32. On 64-bit systems, iKeyAPI64.dll is located in C:\Windows\System32, and iKeyAPI.dll is located in C:\Windows\SysWOW64. Since FineReader 6 is a 32-bit application, I worked with the 32-bit library.

Wrapping the library iKeyAPI.dll

To understand how FineReader authenticates with the USB dongle, I wrapped the original library with my own custom DLL. My library exports functions with names identical to the ikey_ functions used by FineReader. These functions subsequently call the original iKey functions and print debug messages.

First, I found iKeyAPI.dll in the Windows system and renamed it to iKeyAPI-orig.dll. The original library was later loaded into the process by a LoadLibrary() call. My own wrapper library was named iKeyAPI.dll, so FineReader could find it. I put it in the same directory as the other FineReader DLLs. Windows looks for libraries in accordance with the DLL search order.

I leveraged the Mingw-w64 compiler under Linux and created a CMake library project in the same way as shown in my tutorial. From the iKey 1000 SDK, I took the iKeyAPI.h header file and put it in the project directory.

The prototypes and pointers to the original ikey_ functions and the function initOrigFunctions() to initialize them:

#include <windows.h>
#include "iKeyAPI.h"

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_CreateContext)(
    IKEY_HANDLE *PtrContextHandle,
    unsigned long Flags,
    unsigned long ApiVersion);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_OpenDevice)(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    void *pAppId);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_CloseDevice)(
    IKEY_HANDLE ContextHandle);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_DeleteContext)(
    IKEY_HANDLE ContextHandle);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_OpenFile)(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    unsigned long FileId,
    PIKEY_FILEINFO FileInfo,
    unsigned long SizeOfFileInfo);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_MD5_CHAP)(
    IKEY_HANDLE ContextHandle,
    unsigned long FileId,
    unsigned char Identifier,
    unsigned char *Text,
    unsigned long TextSize,
    unsigned char *Digest);

extern "C" typedef IKEY_STATUS IKEY_API (*proto_ikey_Read)(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    unsigned long Offset,
    unsigned char *Buffer,
    unsigned long BytesToRead,
    unsigned long *BytesRead);

static proto_ikey_CreateContext orig_CreateContext = nullptr;
static proto_ikey_OpenDevice orig_OpenDevice = nullptr;
static proto_ikey_CloseDevice orig_CloseDevice = nullptr;
static proto_ikey_DeleteContext orig_DeleteContext = nullptr;
static proto_ikey_OpenFile orig_OpenFile = nullptr;
static proto_ikey_MD5_CHAP orig_MD5_CHAP = nullptr;
static proto_ikey_Read orig_Read = nullptr;

static void initOrigFunctions() {
    const HMODULE hLib = LoadLibrary("iKeyAPI-orig.dll");
    if (hLib == nullptr) {
        MessageBox(nullptr, "Error: cannot find iKeyAPI-orig.dll", "Error", MB_OK | MB_ICONERROR);
        exit(1);
    }

    auto GetProcAddressAndCheck = [&](const LPCSTR procName) {
        return GetProcAddress(hLib, procName);
    };

    orig_CreateContext = reinterpret_cast<proto_ikey_CreateContext>(GetProcAddressAndCheck("ikey_CreateContext"));
    orig_OpenDevice = reinterpret_cast<proto_ikey_OpenDevice>(GetProcAddressAndCheck("ikey_OpenDevice"));
    orig_CloseDevice = reinterpret_cast<proto_ikey_CloseDevice>(GetProcAddressAndCheck("ikey_CloseDevice"));
    orig_DeleteContext = reinterpret_cast<proto_ikey_DeleteContext>(GetProcAddressAndCheck("ikey_DeleteContext"));
    orig_OpenFile = reinterpret_cast<proto_ikey_OpenFile>(GetProcAddressAndCheck("ikey_OpenFile"));
    orig_MD5_CHAP = reinterpret_cast<proto_ikey_MD5_CHAP>(GetProcAddressAndCheck("ikey_MD5_CHAP"));
    orig_Read = reinterpret_cast<proto_ikey_Read>(GetProcAddressAndCheck("ikey_Read"));
}

Next, I defined the wrapper functions themselves. I decided to call initOrigFunctions() from within the ikey_CreateContext() function. Note that one cannot initialize the orig_ pointers from DllMain(), because calling LoadLibrary() is not allowed there.

Each wrapper function calls the original function and prints a message containing the arguments it receives:

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_CreateContext(
    IKEY_HANDLE *PtrContextHandle,
    unsigned long Flags,
    unsigned long ApiVersion) {
    if (orig_CreateContext == nullptr) {
        initOrigFunctions();
    }

    LogMessage("Calling ikey_CreateContext(Flags=" + std::to_string(Flags) + ", ApiVersion=" + std::to_string(ApiVersion) + ")");         
    return orig_CreateContext(PtrContextHandle, Flags, ApiVersion);
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_OpenDevice(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    void *pAppId) {
    LogMessage("Calling ikey_OpenDevice(Flags=" + std::to_string(Flags) + ")");
    return orig_OpenDevice(ContextHandle, Flags, pAppId);
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_CloseDevice(
    IKEY_HANDLE ContextHandle) {
    LogMessage("Calling ikey_CloseDevice()");
    return orig_CloseDevice(ContextHandle);
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_DeleteContext(
    IKEY_HANDLE ContextHandle) {
    LogMessage("Calling ikey_DeleteContext()");
    return orig_DeleteContext(ContextHandle);
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_OpenFile(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    unsigned long FileId,
    PIKEY_FILEINFO FileInfo,
    unsigned long SizeOfFileInfo) {
    LogMessage("Calling ikey_OpenFile(Flags=" + std::to_string(Flags) +
               ", FileId=" + std::to_string(FileId) +
               ", SizeOfFileInfo=" + std::to_string(SizeOfFileInfo) + ")");
    return orig_OpenFile(ContextHandle, Flags, FileId, FileInfo, SizeOfFileInfo);
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_MD5_CHAP(
    IKEY_HANDLE ContextHandle,
    unsigned long FileId,
    unsigned char Identifier,
    unsigned char *Text,
    unsigned long TextSize,
    unsigned char *Digest) {
    const auto status = orig_MD5_CHAP(ContextHandle, FileId, Identifier, Text, TextSize, Digest);

    std::ostringstream textOss;
    for (std::size_t i = 0; i < TextSize; ++i) {
        textOss << std::hex
                << std::setw(2)
                << std::setfill('0')
                << static_cast<int>(Text[i]);
    }

    std::ostringstream digestOss;
    for (std::size_t i = 0; i < 16; ++i) {
        digestOss << std::hex
                << std::setw(2)
                << std::setfill('0')
                << static_cast<int>(Digest[i]);
    }

    LogMessage("Calling ikey_MD5_CHAP(FileId=" + std::to_string(FileId) +
               ", Identifier=" + std::to_string(Identifier) +
               ", TextSize=" + std::to_string(TextSize) +
               ", Text=" + textOss.str() +
               ", Digest=" + digestOss.str() + ")");
    return status;
}

extern "C" __declspec(dllexport) IKEY_STATUS IKEY_API ikey_Read(
    IKEY_HANDLE ContextHandle,
    unsigned long Flags,
    unsigned long Offset,
    unsigned char *Buffer,
    unsigned long BytesToRead,
    unsigned long *BytesRead) {
    const auto status = orig_Read(ContextHandle, Flags, Offset, Buffer, BytesToRead, BytesRead);

    std::ostringstream oss;
    for (std::size_t i = 0; i < *BytesRead; ++i) {
        oss << std::hex
                << std::setw(2)
                << std::setfill('0')
                << static_cast<int>(Buffer[i]);
    }
    LogMessage("Calling ikey_Read(Flags=" + std::to_string(Flags) +
               ", Buffer=" + oss.str() +
               ", Offset=" + std::to_string(Offset) +
               ", BytesToRead=" + std::to_string(BytesToRead) + ")");
    return status;
}

I compiled the DLL and ran the main application. This produced the following diagnostic output in the log file:

Calling ikey_CreateContext(Flags=0, ApiVersion=512)
Calling ikey_OpenDevice(Flags=1)
Calling ikey_OpenFile(Flags=16, FileId=1, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=41424259592046696e6552656164657220456e67696e6500, Offset=0, BytesToRead=24)
Calling ikey_OpenFile(Flags=16, FileId=2, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=414242595900, Offset=0, BytesToRead=6)
Calling ikey_OpenFile(Flags=16, FileId=3, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=465245352d303430303132372d35323839380000, Offset=0, BytesToRead=20)
Calling ikey_OpenFile(Flags=16, FileId=4, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=465245352d333230313230372d35303232390000, Offset=0, BytesToRead=20)
Calling ikey_OpenFile(Flags=0, FileId=5, SizeOfFileInfo=44)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=37473b44344b3d3e4e3f424c44534155, Digest=eb2dc3f6204098d0fee80f4404c47424)
Calling ikey_OpenFile(Flags=16, FileId=4, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=465245352d333230313230372d35303232390000, Offset=0, BytesToRead=20)
Calling ikey_CloseDevice()
Calling ikey_DeleteContext()
Calling ikey_CreateContext(Flags=0, ApiVersion=512)
Calling ikey_OpenDevice(Flags=1)
Calling ikey_OpenFile(Flags=16, FileId=1, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=41424259592046696e6552656164657220456e67696e6500, Offset=0, BytesToRead=24)
Calling ikey_OpenFile(Flags=16, FileId=2, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=414242595900, Offset=0, BytesToRead=6)
Calling ikey_OpenFile(Flags=16, FileId=3, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=465245352d303430303132372d35323839380000, Offset=0, BytesToRead=20)
Calling ikey_OpenFile(Flags=16, FileId=4, SizeOfFileInfo=44)
Calling ikey_Read(Flags=0, Buffer=465245352d333230313230372d35303232390000, Offset=0, BytesToRead=20)
Calling ikey_OpenFile(Flags=0, FileId=5, SizeOfFileInfo=44)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=3331343c3b3a3a48414a4f3e44405455, Digest=358aa13d45e5ffc624ad2335bef40eda)
Calling ikey_CloseDevice()
Calling ikey_DeleteContext()

I ran the application several more times and compared the log files using the diff command. I noticed that all ikey_ functions are called with the same arguments every time, except for ikey_MD5_CHAP():

Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=36433248374b3f484d3f4f4345414643, Digest=eeaf71fc92694abfb454a4f49e38517f)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=364545473848494b4a4d3f4c3e525250, Digest=c4f6e48af361ffa3db8b3895ccad5b5b)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=3732473b494a3a403f3f3d4f423e5147, Digest=289fec4962501c9605f28b6f2f41b7fd)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=37473b44344b3d3e4e3f424c44534155, Digest=eb2dc3f6204098d0fee80f4404c47424)

Citing the docs:

ikey_MD5_CHAP: This function performs the MD5-CHAP operation, a variation of MD5 hashing

This is understandable. FineReader reads some files from the dongle and doesn’t modify them, so the result for read functions is always the same. However, the CHAP protocol requires that the challenge be different for each call, otherwise the authentication would be vulnerable to a replay attack. Ideally, the challenge should be a unique, cryptographically secure random number.

But what would a lazy programmer do to get a random number? That’s right, in FineReader, they simply used rand(). And how to make the result of rand() deterministic? One has to call srand() with a known seed before the first call to rand(). I tried putting it in ikey_CreateContext():

srand(0); // call to srand() from GNU libc, wrong
return orig_CreateContext(PtrContextHandle, Flags, ApiVersion);

But that didn’t work, because I just called a different srand() than the one FineReader uses. Remember that my project uses Mingw-w64, which imports GNU libc. However, FineReader uses the Microsoft Visual C++ library. Unless I compile with the Microsoft compiler, I need to import srand() from the MSVC library:

extern "C" typedef void (*proto_srand)(unsigned int seed);

HMODULE hLib = LoadLibrary("MSVCRT.DLL");
if (hLib == nullptr) {
    MessageBox(nullptr, "Error: cannot find MSVCRT.DLL", "Error", MB_OK | MB_ICONERROR);
    exit(1);
}

proto_srand orig_srand = (proto_srand) GetProcAddress(hLib, "srand");
if (orig_srand == nullptr) {
    MessageBox(nullptr, "Error: cannot find procedure srand()", "Error", MB_OK | MB_ICONERROR);
    exit(1);
}

Finally, I can call the correct srand() just before the original ikey_CreateContext():

orig_srand(0); // call to srand() from MSVCRT.DLL
return orig_CreateContext(PtrContextHandle, Flags, ApiVersion);

This made the challenge bytes always identical. Success!

Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=35453737343c3e4b4e3a4c434f4e4048, Digest=6d627836ed78a0abea5c2a24b13ba658)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=35453737343c3e4b4e3a4c434f4e4048, Digest=6d627836ed78a0abea5c2a24b13ba658)
Calling ikey_MD5_CHAP(FileId=5, Identifier=87, TextSize=16, Text=35453737343c3e4b4e3a4c434f4e4048, Digest=6d627836ed78a0abea5c2a24b13ba658)

I demonstrated a way to force FineReader to always call the ikey_ functions with the same challenge, so I can subsequently remove the dependency on the dongle completely and emulate the functions with a simple lookup table.

Writing the crack

The code is published on GitHub. What is going on here is basically a replay attack. I copied the file contents from the dongle and the response bytes into C arrays. The fake ikey_ functions simply retrieve the contents from them and call srand(0) at the proper place. That is trivial.

Keep in mind that the challenge array contents (Text) depend on the exact implementation of the rand() function. The stored Digest array will need to be updated if the implementation of rand() changes. I tested it under Windows 10 Home, Windows 10 Pro and Windows 7 Ultimate 32-bit and it worked, so it is probably safe to assume it won’t change. For good measure, I include the msvcrt.dll in the repository.

Summing up

I’ve written a simple no-dongle crack which didn’t even require delving into assembly language or function hooking or investigating the library under IDA Pro or Ghidra. If the FineReader developers had abstained from using standard rand() and srand(), the cracking would be much harder. It is trivial to inject an additional srand() call to make the random numbers generated by rand() deterministic.