Lesson Applications of Robotics II - Artificial Intelligence - ثالث ثانوي

Lesson 3 Applications of Robotics II Link to digital lesson www.ien.edu.sa Robotics, Computer Vision and Al Computer vision and robotics are two cutting-edge fields of technology that together are rapidly changing the way people live and work. When combined, they open up a vast array of possibilities for automation, manufacturing, and developing other applications. Al is a key component of both computer vision and robotics, enabling machines to learn and adapt to their environment over time. By using Al algorithms, robots can analyze and interpret vast amounts of visual data, allowing them to make decisions and take actions in real time. Al also enables robots to improve their performance and accuracy over time, as they learn from their experiences and adjust their behavior accordingly. This means that robots with computer vision and Al capabilities can perform increasingly complex tasks with greater efficiency and accuracy. In this lesson, you will upgrade the initial drone project from the previous lesson to use computer vision in order to detect human figures near the house. These figures can be perceived as hostile in a real life scenario and the drone, using its camera, acts as surveillance system. This example can easily be applied and implemented to various other buildings, infrastructure, private and company properties, such as factories and energy plants. To detect the human figures, you will be using the OpenCV library for Python. OpenCV (Open Source Computer Vision Library) is an open-source computer vision library that provides a range of computer vision and image processing algorithms as well as a set of programming tools for developing applications in these areas. & OpenCV OpenCV can be used in robotics for tasks such as object detection and tracking, 3D reconstruction, and navigation. Its features also include object detection and recognition, face detection and recognition, image and video processing, camera calibration, machine learning, and more. OpenCV is widely used in research and development projects in fields such as robotics, automation, surveillance, and medical imaging. It is also used in commercial applications such as face recognition, video surveillance, and augmented reality. Adult Make Adult Ministry of Eduition 328 2024-1446 Adult Male Adult Male Figure 6.16: Real-time Humans detection Adult Male Adult Male Adult Male Adult Male Adult Mal Adult

Let's take a look at the changes you will be making to add computer vision functionalities to your drone. Adding a Timer Capturing, processing and saving an image can be computationally expensive if calculated for every frame of the simulation. This is why you will be adding a timer that will be used so these actions are only performed every 5 seconds. # time intervals used for adjustments in order to reach the target altitude t1 = self.getTime() # time intervals between each detection for human figures t2 = self.getTime() Creating a Folder The captured images in which human figures are detected will be saved in a folder. This is done as part of the security surveillance archive, so the images can be examined in the future. First you must retrieve, with the getcwd() function, the path of the controller's current working directory (the folder the controller is in) in order for the program to know where to place the new folder. The new folder is named "detected" and the path's name is concatenated with the folder's name string with the path.join() function. The last step is to check whether the folder already exists and if not, the folder is created. # gets the current working directory cwd = os.getcwd() # sets the name of the folder where the images # with detected humans will be stored folder_name "detected" #joins the current working directory and the new folder name folder_path = os.path.join(cwd, folder_name) if not os.path.exists(folder_path): # creates the folder if it doesn't exist already os.makedirs(folder_path) else: print (f" Folder \"detected\" created!") print (f" Folder \"detected\" already exists!") Image Processing Now it is time to retrieve (read) the image from the device so as to process it before attempting detection. Notice that everything related to the image processing and up to its saving happens only every 5 seconds as it is inside the "self.getTime() - t2 > 5.0" condition. # initiates the image processing and detection routine every 5 seconds t2 > 5.0: if self.getTime() - # retrieves image array from camera cameraImg = self.camera.getImageArray() وزارة التعليم Ministry of Education 2024-1446 329

height -240- وزارة التعليم Ministry of Education 330 2024-1446 After the image is checked that it was retrieved successfully, the algorithm proceeds to modify some properties of the image. The image is 3-dimensional; it has dimensions of height, width and the color channels. The drone's camera captures images of 240 pixels in height and 400 pixels in width. It also uses 3 color channels to save the image information: red, green and blue. In order to be used for detection, the image has to be manipulated first. For the functions to be applied properly later, it has to fit a particular structure. In this case, the sequence of the dimensions has to change from (height, width, color channels) to (color channels, height, width) by using the transpose() function. This function is given, as arguments, the camera, image cameralmg, and the new sequence (2, 0, 1), assuming the original order was (0, 1, 2). The dimension sizes have to be adjusted too after the change in sequence. The reshape() function is used in the same manner, but with the respective dimension sizes (3, 240, 400) as the second argument. # reshapes image array to (channels, height, width) format cameraImg = np.transpose(cameraImg, (2, 0, 1)) cameraImg = np.reshape(cameraImg, (3, 240, 400)) 0 1 2 height 240 width 400 color channels 3 color channels 3 height 240 width 400 2 0 1 Original color image 400 width Figure 6.17: The dimensions' sequence change 240- -400- RGB matrix color channels Figure 6.18: The dimensions of the image

ة التعليم Next, the image has to be changed to grayscale, as needed by the detection algorithm, but before that it must be stored in an Image object and have its 3 color channels combined. Here the color channels have to be merged and stored with the merge() function in reverse sequence, meaning in BGR (Blue, Green, Red) instead of RGB (Red, Green, Blue), (2, 1, 0) instead of (0, 1, 2) respectively. # creates RGB image from merged channels img Image.new('RGB', (400, 240)) img = = cv2.merge((cameraImg[2], cameraImg[1], cameraImg[0])) Finally, the image is converted to grayscale with the cvtColor() function using the COLOR_BGR2GRAY argument, to change from BGR to grayscale. # converts image to grayscale gray cv2.cvtColor(np. uint8(img), cv2.COLOR_BGR2GRAY) = Human Silhouette Detection For the detection, you will use the Haar Cascade classifier. The Haar Cascade classifier is a machine learning-based object detection algorithm used to identify objects in images or videos. To use it, you need to train a machine learning model with a set of images that have the object you want to find, and others that do not. The algorithm looks for certain patterns in the pictures to determine where the object is. This algorithm is often used to find things like faces or people walking in a video. However, it might not work well in some situations where the object is partially/fully occluded or exposed to low illumination. The classifier in your project is particularly trained for human detection. The haarcascade_fullbody. xml file provided to you is the pre-trained machine learning model you will use and part of the OpenCV library. It is given as argument to the Cascade Classifier() object, and the function detectMultiScale() is called after to perform the detection. #loads and applies the Haar cascade classifier to detect humans in image human_cascade = cv2.Cascade Classifier('haarcascade_fullbody.xml') humans human_cascade.detectMultiScale(gray) = original image Ministry of Education 2024-1446 extracted silhouette Figure 6.19: An example of human silhouette detection 331

Drone Report and Saving of the Detected Images The final addition to your controller is a simple report system given by the drone in the form of printing a message on the console when a human form is detected and saving the image to the folder you created before. The variable humans holds the rectangles (bounding boxes) inside which humans are detected, if they are found. The rectangles are defined by 4 variables: the pair of x and y, the two coordinates in the picture of the top left corner of the rectangle, and the pair of w and h, the width and height of the rectangle. For all detections found in the image, the function rectangle() marks the humans with a blue rectangle. The function takes as parameters the image, the top left corner (x, y) and bottom right corner (x+w, y+h) of the rectangle, and the rectangle's color and its width. Here, the rectangle is blue (B-255, G=0, R=0) and its width is 2. The report system will retrieve the current date and time by using the datetime.now() function and print it on the console, along with the drone's coordinates at the time of the report. (x, y) (x+w, y+h) W Figure 6.20: The variables of the rectangle The date and time format is slightly modified by inserting dashes (-) and undersores ( _ ) to be used as part of saved file's name and then saved in the folder with the function imwrite(). When everything is completed, the getTime() function resets the timer. #loop, through detected human images, annotates them with a bounding box # and prints a timestamp and an info message on the console for (x, y, w, h) in humans: # the image, the top left corner, the bottom right corner, color and width of the rectangle cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2) current_time = datetime.now() print(current_time) print("Found a person in coordinates [{:.2f}, {:.2f}]" .format(x_pos, y_pos)). # saves annotated image to file with timestamp current_time = current_time.strftime("%Y-%m-%d %H-%M-%S") filename f"detected/IMAGE_{current_time}.png" = cv2.imwrite(filename, img) t2 = self.getTime() وزارة التعليم Ministry of Education 332 2024-1446 In a string, the notation {:.2f} is used as a placeholder for a floating-point number with two decimal places. Here, two placeholders are used for the two variables, x_pos and y_pos.

After adding all these functionalities, the run() function of your controller should look like this: def run(self): # time intervals used for adjustments in order to reach the target altitude t1 = self.getTime() # time intervals between each detection for human figures t2 = self.getTime() roll_disturbance = 0 pitch disturbance = 0 yaw_disturbance = 0 # specifies the patrol coordinates waypoints = [[-30, 20], [-60, 30], [-75, 0], [-40, -10]] # target altitude of the drone in meters self.target_altitude = 8 #gets the current working directory cwd = os.getcwd() # sets the name of the folder where the images # with detected humans will be stored folder_name = "detected" #joins the current working directory and the new folder name folder_path = os.path.join(cwd, folder_name) if not os.path.exists(folder_path): # creates the folder if it doesn't exist already else: os.makedirs(folder_path) print(f" Folder \"detected\" created!") print(f"Folder \"detected\" already exists!") while self.step(self.time_step) != -1: وزارة التعليم #reads sensors roll, pitch, yaw = self.imu.getRoll PitchYaw() x_pos, y_pos, altitude = self.gps.getValues(). roll_acceleration, pitch_acceleration, _ = self.gyro.getValues() self.current_pose = [x_pos, y_pos, altitude, roll, pitch, yaw] if altitude > self.target_altitude - 1: # as soon as it reaches the target altitude, # computes the disturbances to go to the given waypoints if self.getTime() - t1 > 0.1: yaw_disturbance, pitch_disturbance = self.move_to_target( waypoints) t1 = self.getTime() # initiates the image processing and detection routine every 5 seconds if self.getTime() - t2 > 5.0: # retrieves image array from camera cameraImg = self.camera.getImageArray() # checks if image is successfully retrieved if cameraImg: Ministry of Education 2024-1446 333

# reshapes image array to (channels, height, width) format cameraImg = np. transpose(cameraImg, (2, 0, 1)) cameraImg = np.reshape(cameraImg, (3, 240, 400)) # creates RGB image from merged channels img Image.new('RGB', (400, 240)) = img = cv2.merge((camera Img[2], cameraImg[1], camera Img[0])) # converts image to grayscale gray = cv2.cvtColor(np. uint8(img), cv2.COLOR_BGR2GRAY) # loads and applies the Haar cascade classifier to detect humans in image human_cascade = cv2.Cascade Classifier('haarcascade_fullbody.xml') humans = human_cascade.detect MultiScale(gray) #loop, through detected human images, annotates them with a bounding box # and prints a timestamp and an info message on the console for (x, y, w, h) in humans: cv2. rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2) current_time = datetime.now() print(current_time) print("Found a person in coordinates [{: .2f}, {:.2f}]" .format(x_pos, y_pos)) # saves annotated image to file with timestamp current_time = current_time.strftime("%Y-%m-%d %H-%M-%S") filename f" detected/IMAGE_{current_time}.png" = cv2.imwrite(filename, img) t2 = self.getTime() # calculates the desired input values for roll, pitch, yaw, # and altitude using various constants and disturbance values roll input = self.K_ROLL_P* clamp(roll, -1, 1) pitch_input = = + roll_acceleration + roll_disturbance self.K_PITCH_P * clamp(pitch, -1, 1) yaw_input yaw_disturbance + pitch_acceleration + pitch_disturbance clamped_difference_altitude = clamp(self.target_altitude vertical_input = altitude self.K_VERTICAL_OFFSET, -1, 1) self.K_VERTICAL_P* pow(clamped_difference_altitude, 3.0) # calculates the motors' input values based on the desired roll, pitch, yaw, and altitude values front_left_motor_input = self.K_VERTICAL_THRUST vertical_input - yaw_input + pitch_input front_right_motor_input = self.K_VERTICAL_THRUST + - roll input + vertical_input + yaw_input + pitch_input + roll_input rear_left_motor_input = self.K_VERTICAL_THRUST + vertical_input + yaw_input - pitch_input - roll input rear_right_motor_input = self.K_VERTICAL_THRUST + vertical_input - yaw_input - pitch_input + roll_input # sets the velocity of each motor based on the motors' input values calculated above self.front_left_motor.setVelocity (front_left_motor_input) self.front_right_motor.setVelocity(-front_right_motor_input) self.rear_left_motor.setVelocity(-rear_left_motor_input) self. rear_right_motor.setVelocity (rear_right_motor_input) وزارة التعليم Ministry of Education 334 2024-1446

Now, run the simulation to see your drone taking off and patroling around the house. Notice the new console outputs and the images created in the folder. File Edit View Simulation Build Overlays Tools Help DB Mavic 2 PRO demo во 0.00.16768 0.00 IMPORTABLE EXTERNPROTO Floor "floor Pedestrian "pedestrian(2) Pedestrian "pedestrian(1) Road "read" Pedestrian "pedestrian Windmill "windmill Windmill "windmill Windmill "windmill(2) SmallManor "small manor SquareManhole "manhole CardboardBox "cardboard box" TeslaModel3Simple "vehicle" Forest Mavic Pro "Mavic 2 PRO translation-1.13 0.327 8.27 rotation-0.00798 0.00138 12.96 name "Mavic 2 PRO controller "drone controller controllerArgs window generics customData" supervisor FALSE synchronization TRUE ® llers\drone controller\drone controller-py DEBBO Qa drone, controller.py 20 Copyright 2995-2023 Cyberbottes Ltd. 3 Licensed under the Apache License, V 4 you may not use this file except in c 5 You may obtain a copy of the License 7# https://ow.apache.org/licenses/ Unless required by applicable Law.or distributed under the License is dist WITHOUT WARRANTIES OR CONDITIONS OF A See the License for the specific lang 15 Limitations under the License. 10 11 12 14 15 16 from controller inpert Robot 17 import numpy as np Used for authen 18 import os Used for folder creation 19 import cv2 Used for teage manipulat 20 from PIL import Inage Used for ina 21 from datetime import datetime Used 22 23 Auxlltory function used for calculati 24 def clamp(value, value min, value_max): return sin(max(value, value_min). 25 26 battery Console-All INFO: drone controller: Starting controlleri python.exe -u drone controller.py Folder "detected" created! 2023-04-20 10:50:17.093902 Found a person in coordinates [-0.20, 0.13] 2023-04-20 10:50:22.546951 Found a person in coordinates [-0.01, 0.25] Console - All INFO: drone_controller: Starting controller: python.exe -u drone_controller.py Folder "detected" created! 2023-04-20 10:50:17.093902 Found a person in coordinates [-0.20, 0.13] 2023-04-20 10:50:22.546951 Figure 6.21: Console outputs Found a person in coordinates [-0.81, 0.25] وزارة التعليم Ministry of Education 2024-1446 Open File... X This PC > Documents A12 U3_Drone controllers drone controller Search drone controller Organize New folder * Quick access PC Creative Cloud Files This PC Network The folder has been created. drone controller. haarcascade_full py body.xml detected File name: drone controller.py All Files (") Open Cancel IMAGE 2023-04-20 10-50-17.png IMAGE 2023-04-20 10-50-22.png Figure 6.22: Folder creation and images saved containing detections 60 335

Exercises 1 Modify your controller to not check if the folder already exists in the path. Does it create any complications in the execution of the simulation? 2 Modify your controller to perform a detection every 10 seconds. Do you notice any difference in the frequency of the console prints and the images saved? وزارة التعليم Ministry of Education 336 2024-1446

3 What would happen to the image output if you merged the color dimensions in the normal sequence instead of the reversed one? Write down your observations below. 4 Experiment with the fourth and fifth arguments of the rectangle() function. Write down your observations below. 5 Modify your controller to also print the drone's roll, pitch and yaw values when detecting a person. وزارة التعليم Ministry of Education 2024-1446 337