Iphone Mirroring实现AI打DNF手游的技术路径与训练模型

Iphone Mirroring圆了我AI打DNF手游的梦

契机

⚙ 最近痴迷于打地下城手游，太浪费时间了，本着编程解放生产力的思想，开始想着能不能用程序来打地下城，就是开外挂！当然任何形式的开外挂都是不倡导的，这里只是探寻技术方案可行性，输出的成品并不能直接使用！为啥要等Iphone Mirroring出来完成这件事呢，因为我的设备是Iphone+Mac，虽然Mac可以通过PlayCover安装破解IPA，但是不支持微信登陆，现在有了Iphone Mirroring一切问题迎刃而解。

iphone mirroring

• iphone设备打开Beta更新，且更新到ios18.0最新版本
• mac更新到版本15.0 Beta版(24A5279h)
• 保持ihone和mac在同一局域网下

技术路径

• 基于python实现屏幕点击，屏幕录制
• 通过yolo模型识别怪物，人物等
• 通过计算人物到怪物的xy距离，移动到怪物身边平A

准备训练模型数据

录制打怪视频

使用OBS软件选择窗口录制，程序采集

录制完成后需要更改下编码格式为mp4

#把mkv转为mp4
ffmpeg -i xxxx.mkv -codec copy xxxx.mp4

将打怪视频转为图片

mkdir -p ./images
#从视频中每隔2秒提取一帧并保存为图像文件
ffmpeg -i xxxx.mp4 -vf fps=1/2 ./images/%d.jpg

开始打标签

#安装label-studio
docker run -dit -p 18080:8080 heartexlabs/label-studio:latest
#进入localhost:18080

创建项目

选择自己需要打的标签如下 ！这里最好使用英文标签

import本地文件开始打标，挨个打上就行

最后保存打标结果，导出yolo格式的训练数据

得到一个zip包这就是我们需要训练的数据

开始训练模型

#安装python环境
pip install ultralytics

#找一个猫咪的照片比如cat.jpg到当前目录

#直接命令行运行
yolo predict model=yolov8n.pt source='cat.jpg'

#输出以下
#会直接去github下载yolov8n.pt模型文件
Downloading https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n.pt to 'yolov8n.pt'...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.25M/6.25M [00:01<00:00, 5.85MB/s]
Ultralytics YOLOv8.2.28 🚀 Python-3.9.6 torch-2.3.0 CPU (Apple M1 Pro)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs
image 1/1 /Users/y/PycharmProjects/pythonProject1/cat.jpg: 384x640 1 cat, 76.0ms
Speed: 4.7ms preprocess, 76.0ms inference, 6.3ms postprocess per image at shape (1, 3, 384, 640)
Results saved to runs/detect/predict

#结果为
./runs/detect/predict/cat.jpg

#解压刚才label-studio输出的zip

#当前目录写一个game.yml如下

path: /Users/y/Downloads/project-1-at-2024-06-04-14-44-0205e0ec  
train: images 
val: images  

names:
  0: me
  1: enemy

#开始训练，batch: 批大小 epoches: 训练轮次
yolo detect train data=game.yml model=yolov8n.yaml batch=10 epochs=100

#出现以下字样代表正在训练
Image sizes 640 train, 640 val
Using 0 dataloader workers
Logging results to runs/detect/train4
Starting training for 100 epochs...
Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
0%|          | 0/1 [00:02<?, ?it/s]

#训练完成后会在./runs/detect/train中出现训练结果
#在weights目录中有两个pt模型文件。last代表最新的，best代表最好的

#再随便找个游戏图片验证下
yolo predict model=best.pt  source='game.jpg'

集成代码

以下代码表示实时读取窗口画面分析，并没有移动打怪逻辑

import pytesseract
import pyautogui
from PIL import Image, ImageOps, ImageDraw, ImageGrab
import Quartz
import os
from datetime import datetime
import cv2
from ultralytics import YOLO
import numpy as np
from PIL import ImageGrab

model = YOLO('https://b2.7b2.com/best.pt')
# 类别对应的名字，及rgb颜色
clazz_dict = {
    0: ('close-door', (255, 0, 0)),
    1: ('enemy', (0, 255, 0)),
    2: ('me', (0, 0, 255)),
    3: ('open-door', (255, 255, 0)),
    4: ('sth', (13, 27, 211)),
}

def ai_boxes(image):
    result = model(image)[0]
    boxes = result.boxes.data.cpu().numpy().tolist()
    for x1, y1, x2, y2, conf, cls in boxes:
        x1, y1, x2, y2, cls = int(x1), int(y1), int(x2), int(y2), int(cls)
        cls_name, cls_rgb = clazz_dict[cls]
        # 增加框
        cv2.rectangle(image, (x1, y1), (x2, y2), cls_rgb, 2)
        # 增加文字
        cv2.putText(image, cls_name, (x1, y1), cv2.FONT_HERSHEY_COMPLEX, 1, cls_rgb, 2)
    return image

def get_window_id_by_name(window_name):
    window_list = Quartz.CGWindowListCopyWindowInfo(Quartz.kCGWindowListOptionOnScreenOnly, Quartz.kCGNullWindowID)
    for window in window_list:
        if window_name in window.get('kCGWindowName', ''):
            return window['kCGWindowNumber']
    return None

def cgimage_to_pil_image(cgimage):
    width = Quartz.CGImageGetWidth(cgimage)
    height = Quartz.CGImageGetHeight(cgimage)
    bpr = Quartz.CGImageGetBytesPerRow(cgimage)
    bpp = Quartz.CGImageGetBitsPerPixel(cgimage)
    bpc = Quartz.CGImageGetBitsPerComponent(cgimage)
    pixeldata = Quartz.CGDataProviderCopyData(Quartz.CGImageGetDataProvider(cgimage))

    # Convert to PIL.Image
    pil_image = Image.frombuffer("RGBA", (width, height), pixeldata, "raw", "RGBA", bpr, 1)
    return pil_image

def capture_window_screenshot(window_id):
    screenshot = Quartz.CGWindowListCreateImage(
        Quartz.CGRectNull, 
        Quartz.kCGWindowListOptionIncludingWindow,
        window_id,
        Quartz.kCGWindowImageDefault
    )
    image = cgimage_to_pil_image(screenshot)
    return image

def main():
    window_name = ""  # 替换为目标窗口的名称
    window_id = get_window_id_by_name(window_name)

    if window_id:
        print(f"Found window '{window_name}' with ID {window_id}")
        while True:
            pil_image = capture_window_screenshot(window_id)
            image = np.array(pil_image)  # Convert PIL image to NumPy array
            image = cv2.cvtColor(image, cv2.COLOR_RGBA2RGB)  # Convert RGBA to RGB
            image = ai_boxes(image)
            cv2.imshow("video", image)
            # 按"Esc"键退出
            if cv2.waitKey(1) & 0xFF == 27:
                break
    else:
        print(f"No window found with name '{window_name}'")

if __name__ == "__main__":
    main()

总结

• yolo模型现在训练十分简单
• label-studio完美的打标软件
• 要是能ai打标就好了

写到最后