llir 0.1.0

Low level mouse and keyboard input recorder and player.

LLIR

Low level input recorder or LLIR is a Python package for recording low level mouse and keyboard events and replaying them back.

Why? Have you ever wondered why packages like pyautogui or pynput or even programs like AutoHotKey don't work with games? They either can't send input to games altogether or the mouse input "works" but it's completely broken.

Well, I created this package to address these issues. Using low level (hence the name) methods and APIs to record and playback the input, it works with a lot of games and also in normal application.

Table of Content

• LLIR
  • Table of Content
  • Installation
  • Usage
    • Start recording
    • Stop recording
    • Start replaying
      • To start replaying a macro that was recorded during this run
      • To start replaying a macro that was saved to a file
    • Stop replaying
    • Stop listening
    • Changing the default shortcuts
    • Keeping old macros
  • Notes
  • Limitations
  • How it works
  • What this package isn't
  • License

Installation

This package supports Python >= 3.10, and it has been tested on Windows 10 and 11, though it should work on Windows 7+.

To install simply run:

pip install llir

If you can't use Python 3.10+, you can modify the syntax for the type hints or use from future import __anotations__ and change any type imports that don't exist in the typing package to typing_extensions.

Usage

The main entry point to the package is the LowLevelInputRecorder class. import as the following:

from llir import LowLevelInputRecorder

There are two ways to use it, programmatically or using shortcuts. in either case, you must first call start_listening:

recorder = LowLevelInputRecorder()
recorder.start_listening()

Now the class is ready to accept actions.
If wish to control the actions using shortcuts, these are all the lines of code you need to run.

Start recording

To start recording mouse and keyboard input so you can replay them back later, call start_recording or press the shortcut for starting recording as specified in start_recording_shortcut which can be defined at class creation. The default shortcut is left ctrl + left arrow.

Note: You can only start recording if start_listening has been called and the class isn't replaying a macro.

Stop recording

Call stop_recording or the press the start_recording_shortcut (default is left ctrl + right arrow). This method can be called at any time, it will call the correct thing only the class is currently recording.

Start replaying

To start replaying a macro that was recorded during this run

Call start_replaying or press the start_replaying_shortcut (default is left ctrl + down arrow).

Note: You can only start replaying a macro if the class isn't currently recording input and start_listening has been called.

To start replaying a macro that was saved to a file

If you haven't recorded any input yet, you can replay an already recorded macro that was saved to a file by specifying the file path where the macro was saved to at class initialization, for example:

recorder = LowLevelInputRecorder(save_file_path="path/to/file")
recorder.start_listening()
recorder.start_replaying()

Stop replaying

Call stop_replaying or press the stop_replaying_shortcut (default is left ctrl + up arrow).

Stop listening

To stop listening for input and effectively shutdown the class, call stop_listening or press the stop_shortcut (default is left ctrl + left shift + x).

Note: You can't perform any other actions after this method is called. To start the class again so you can record/replay input, start_listening must be called again.

Changing the default shortcuts

To change the default shortcuts, pass the ScanCodes for the keys you want to be pressed to trigger as specific action to the class initializer seperated by a plus sign.

For example, to set the shortcut for starting recording to left shift and the letter x:

recorder = LowLevelInputRecorder(start_recording_shortcut="0x002A+0x002D")

Where "0x002A" is the ScanCode for the left shift key and 0x002D is the ScanCode for the letter "X".

A list of scan codes can be found at the official Microsoft documentation.

Keeping old macros

The default behaviour is discarding any previously recorded input when start_recording is called.
If you wish to keep any old macros that were recorded, set append_new_input to True when initializing the class.

Notes

1. To record keyboard and mouse events and be able to replay them on a different window (application), the python process that ran the script must be on the same level of privilege as that application, with some notable exceptions. So for example, if you run a game as an administrator, you must also run Python (through your IDE or CMD) as an administrator to be able to record keyboard and mouse inputs and replay them in that window. A notable exception to this is Task Manager, even if you launch it normally, you need admin privileges to "send input" to it.
2. The only blocking function in this class is start_replaying for replaying back the input, the other functions return immediately, so you will need to place time.sleep() calls for example after starting to record.
3. You can't use the class after you called stop_listening unless you call start_listening() again.
4. Some games block the SendInput API which this packages uses under the hood to replay back the recorded input. These games are usually online games that do this to protect against hacking, usage of this package with such games will not work and might get you banned.

Limitations

1. Touchpads:
   1. Windows treats touchpads differently than mouses, so touchpad specific functionality aside than basic functionality isn't supported, like gestures (3 finger touch for example).
   2. Moving, left and right clicks always work.
   3. However, scrolling only works correctly if it's recorded, but sometimes, depending on the application, it's not recorded, as if the application had "swallowed" it. It works correctly in Pycharm and Minecraft but not in Firefox for example.
2. Windows UI shortcuts: Shortcuts like alt-tapping to switch windows or Windows + D to show the desktop don't work.

How it works

There are different ways to listen for keyboard, mouse and other hardware devices input in Windows, such as LowLevelMouseProc callback.
The problem with most "user-mode" methods is that they either can't record input happening in other applications or they don't record input at a low enough level.
Not recording at a "low enough level" has some caveats, one of which is that the mouse input will be altered by the operating system, like applying acceleration and custom scroll distance and threshold.

The other possible solution might then be to create a custom driver or an interceptor, but that's too much work and Python isn't really the language for doing that.\

Well, there's one last method we haven't discussed yet, the Raw Input API.
The Raw Input API can be used to get low level device input (like a mouse, touchpad or keyboard) before they are processed by the operating system.
Also, a lot of games, especially newer ones use this API to get device input.

Why care about getting "low level" mouse and keyboard input?
Most games work correctly regardless of keyboard layout/language (a modification done by the Operating System). Additionally most games lock the cursor to the middle of the screen, rendering methods such as recording and setting the cursor position useless.
Recording low level input events allows us to obtain scan codes and mouse movement deltas, which then can be sent to games in the same format they understand.

This packages creates a window that's invisible and registers raw mouse and keyboard input devices to it as input sinks, meaning that we get input events even if other programs are in the foreground (focused)¹.
Then the recorded raw input is played back using the SendInput API.

If you still want to know more about the inner workings of this, check the function documentation and comments in raw_input.py and send_input.py, they contain some useful information and gotchas when working with these APIs.

One last thing that this package does is special handling when replaying back the input, it tries to the best it can to replay the recorded events in exactly the same time they were recorded in.
In other words, if the time difference between the first event and second event is 500 ms, it will attempt to execute the second event at exactly 500 ms after the first event.
Why is this important? Well for general automation and normal applications it's not an issue for there to be extra delay between the recorded events times and the execution time.
However, in games this is most likely an issue. Hopefully an example would make this more clear: Suppose the recorded macro was moving the mouse to the right by 5 pixels (remember? mouse deltas) every 100 ms for 5 seconds, given the same starting position, on the desktop it doesn't matter if replaying back the macro took 10 seconds and the delay between each input was 200 ms, the cursor will end up at the same location as the one when the macro was recorded.
But in a game, the player's camera will be pointing at a different location than the one were it ended pointing at when recording the macro due to game mechanics, how it handles input and acceleration and movement delta threshold settings.
The variable execution "lag" effect will be more troublesome the longer the macro is, playing back a 1-second macro might take 1.05 seconds which might not be a problem, but playing back a 100-second macro might take 120 seconds, which will most likely be a problem.
To mitigate this, a hybrid approach of short sleeps and busy-waiting to achieve timing accuracy used. This is more heavy on the CPU than simply using time.sleep(time_difference_between_this_event_and_previous_event), but it's not that much (2% CPU usage on an old 6 core Ryzen 2600 processor) but is much more accurate.
A better algorithm for achieving this "real-time" constraint probably exists, nonetheless I couldn't find any useful information on this topic.

^{¹ This is subject to program privilege, check the notes for more information.}

What this package isn't

This package is not intended as a replacement for other Python packages such as pynput, pyautogui, keyboard or dedicated automation programs like AutoHotKey.
These solutions offer the functionality to programmatically define mouse and keyboard (and much more) actions and then execute them using a user-friendly interface.
This package is only intended for recording and replaying raw mouse and keyboard inputs, even though you can use it as the packages mentioned above, it's not as easy or convenient to do so.

License

This project is licensed under the MIT license.