This blog discuss about my research work and projects related to artificial intelligence, Internet of things, Embedded systems,Data science, Machine learning and deep learning
The concept of a virtual mouse using hand gestures is to control the movement of the mouse cursor on a computer screen without using a physical mouse. Instead, the user can use hand gestures captured by a camera or sensor to move the cursor and perform actions like clicking and scrolling.
The basic idea is to detect the position of the hand in the camera frame and track the movement of the hand to move the mouse cursor accordingly. Hand gestures can be recognized by analyzing the movement and position of the fingers and the palm.
Various computer vision techniques and machine learning algorithms can be used to implement this concept. OpenCV and Mediapipe are two popular libraries that provide hand tracking and gesture recognition functionality. These libraries use deep learning models to detect and track the position of the hand and fingers in real-time video frames.
Once the position of the hand is tracked, the movement of the hand can be mapped to the movement of the mouse cursor on the computer screen. The position of the fingers can be used to simulate mouse clicks, right-clicks, and scrolling actions.
The virtual mouse using hand gestures is a convenient and intuitive way to control the computer without the need for a physical mouse. It can be particularly useful for people with disabilities or those who find it difficult to use a traditional mouse. Additionally, it can be used in situations where a physical mouse is not available, such as in a presentation or a remote desktop environment.
Moving cursor : Middle and index are opened. Remaining finger closed.
Double click : Joining middle and index finger.
Right click: folding index finger alone and middle finger unfold.
Left click: folding middle finger and index finger unfold.
Dragging folder : closing all the fingers
To create a virtual mouse using hand gesture recognition with OpenCV and
Mediapipe, you can follow these steps:
Install OpenCV and Mediapipe libraries.
Source Code:
# Imports
import cv2
import mediapipe as mp
import pyautogui
import math
from enum import IntEnum
from ctypes import cast, POINTER
from comtypes import CLSCTX_ALL
from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume
from google.protobuf.json_format import MessageToDict
import screen_brightness_control as sbcontrol
pyautogui.FAILSAFE = False
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
# Gesture Encodings
class Gest(IntEnum):
# Binary Encoded
FIST = 0
PINKY = 1
RING = 2
MID = 4
LAST3 = 7
INDEX = 8
FIRST2 = 12
LAST4 = 15
THUMB = 16
PALM = 31
# Extra Mappings
V_GEST = 33
TWO_FINGER_CLOSED = 34
PINCH_MAJOR = 35
PINCH_MINOR = 36
# Multi-handedness Labels
class HLabel(IntEnum):
MINOR = 0
MAJOR = 1
# Convert Mediapipe Landmarks to recognizable Gestures
class HandRecog:
def __init__(self, hand_label):
self.finger = 0
self.ori_gesture = Gest.PALM
self.prev_gesture = Gest.PALM
self.frame_count = 0
self.hand_result = None
self.hand_label = hand_label
def update_hand_result(self, hand_result):
self.hand_result = hand_result
def get_signed_dist(self, point):
sign = -1
if self.hand_result.landmark[point[0]].y < self.hand_result.landmark[point[1]].y:
A virtual keyboard using OpenCV is a graphical interface that allows users to type without the need for a physical keyboard. It provides a visual representation of a keyboard on a screen, and users can select keys using a pointer or by using their fingers in front of a camera.
The concept of a virtual keyboard using OpenCV involves using a computer's webcam to capture live video of the user's hand gestures. The captured video is then processed using the OpenCV library to detect the user's hand landmarks and track their movement.
Once the hand landmarks are detected, the position of the user's fingers can be used to identify the key on the virtual keyboard that the user wants to select. The selected key can then be sent to the computer's operating system as if it was typed on a physical keyboard.
To create a virtual keyboard using OpenCV, the following steps are typically involved:
Capture video from the computer's webcam
Preprocess the video to remove noise and improve image quality
Detect hand landmarks using OpenCV and MediaPipe library
Track the movement of the user's fingers
Map the position of the fingers to the keys on the virtual keyboard
Send the selected key to the operating system as keyboard input
Overall, a virtual keyboard using OpenCV can provide a convenient and accessible way for users to type without the need for a physical keyboard. It can be useful in situations where a physical keyboard is not available or when users have difficulty using a physical keyboard due to disability or injury.
To create a virtual keyboard using OpenCV, Python, and MediaPipe, you can follow these general steps:
Tesseract is a cross-OS optical character recognition (OCR) engine developed by HP in the 1980s, and since 2006, maintained by Google as an open-source project with high marks for accuracy in reading raw image data into digital characters. The project has been continuously developed and now offers OCR supported by LSTM neural networks for highly improved results. In this session, we’ll use the Python wrapper for Tesseract to first test drive OCR on images through code before connecting our solution to a live IP video feed from your smartphone processed through OpenCV, and then translating the resultant text stream into audible form with gTTS (Google Text-To-Speech), enabling our mashup program to automatically read out loud from any script it ‘sees’.
Prerequisites: —Python IDE such as PyCharm / VScode
