Lesson Building a Smart Waste Management Solution - Internet of Things - ثاني ثانوي

292 Lesson 3 Building a Smart Waste Management Solution Linx to digital lesson www.jen.es sa Smart Waste Management and Data Analysis In the previous lesson, you created a smart waste management prototype for a garbage can with an Arduino microcontroller that monitors its environment, generates data from sensors, and publishes this data as messages to an MQTT topic. This data needs to be collected and processed to generate insights from it. وزارة التعليم 3073-1445 MQTT Broker Collect Write 200 Read (json) matt_receiver.py data_analysis.ipynb Figure 7 21 Smart Waste project data analysis solution In this lesson, you will create a Python script that will subscribe to the same topic of the MQTT broker to which the messages were published. These messages will be collected, and each time the garbage can is filled, a report will be generated and stored in another file. This file will only contain the data from the generated reports. After that, you will create a Jupyter notebook that will open the file, analyze the data and generate insights and data visualizations. These two files will be named mqtt_receiver.py and data_analysis.ipynb, respectively. The first script will write the recorded data into a JSON file, and the second script will read from the JSON file and perform the data analysis.

Creating the JSON Data File We are going to create a JSON data file with an empty array. The mqtt_receiver.py script will append each generated report as a JSON object to the array. The data analysis.py will open this JSON file, read the JSON array contents and perform the data analysis operations. Open PyCharm and in your working directory create a new file named data.json. Inside that file create an empty array object as shown in the box below. The mqtt_receiver.py will append JSON objects of the generated reports to the array presented below. Save and close the data.json file. [] Create a new Python file called mqtt_receiver.py and at the beginning of your code import the following packages. ■ datetime: Create timestamps for the messages that we send. ⚫json: Work with JSON objects. • paho.mqtt.client: Create clients that communicate with MQTT brokers. ⚫os: Work with files on your computer from datetime import datetime import json import paho.mqtt.client as mq from os import path Create the following variables data_file and data_file_objects which will be used to interact with the JSON data file. data file "your_file_path" # Absolute path to the JSON data file data_file_objects = [] # This contains the objects from the JSON data file وزارة التعليم 023-1445 293

294 Create the following variables which will be used for the MQTT client that we will create. CLIENT_ID is the name we will give to our client, MQTT BROKER is the address of the public broker provided by EMQX that we will connect to. TOPIC is the name of the topic that the client will subscribe to. PORT is the default server port to connect to the MQTT broker. FLAG_CONNECTED is a flag variable we will use on an event handler function later. CLIENT_ID = "RECEIVER_01" MQTT BROKER # Variables to setup MQTT client # ID of the client "broker.emqx.io" # Address of the broker TOPIC "waste/drops" # Topic to subscribe to PORT = 1883 # Default server port FLAG CONNECTED = False # Connection flag Create the following variables messages_stack and reports which will be used to store information from the published messages. messages stack = [] # The array with the messages per can filling reports = [] # The array with all the generated report objects Create the following event handler function that prints a confirmation message to the terminal about whether or not the connection to the client was successful. The function's arguments are default arguments that must be used to bind that function to the appropriate event handler provided by the paho.mqtt.client library. def on_connect(client, userdata, flags, rc): global FLAG CONNECTED # Access the FLAG CONNECTED variable وزارة التعليم - if rc == 0: # If rc is the client connected successfully FLAG CONNECTED = True ort("Connected to MQTT Broker!") print("Failed to connect to MQTT Broker!")

Create the following event handler function that triggers each time a message is published to the subscribed topic. The function's arguments are default arguments that must be used to bind that function to the appropriate event handler provided by the paho.mqtt.client library. def on message(client, userdata, msg): global messages_stack # Access the messages_stack variable #Decode the message payload payload = str(msg.payload.decode()) # Convert the payload to a JSON object and append it # to the messages stack payload object = json. Loads (payload) messages stack.append(payload_object) # When you receive a message, print it to the terminal print("MESSAGE RECEIVED ----||\n") print("Payload: " str(payload_object)) # If the payload object has the can filled flag set to True # generate a report for the filled can if payload object["can_filled"] == True: generate report() This Python script that serves the role of an MQTT receiver can collect mesages from multiple Arduino publishers at the same time. This solution can be further expanded to process JSON data with even more helds containing information about the publishers as well. وزارة التعليم 2173-1466 Figure 722 Python script surws the car of in MOTTEL 295

296 Create the following generate_report function which will create a report JSON object and append it to the data file every time that a message is received that indicates that the garbage can is full. def generate report(): global data file objects # Access data_file_objects variable جميلية التعليم 173-1445 global messages_slack # Access messages stack variable global reports #Access reports variable # Getting the first and last objects from the messages stack first_msg =messages_stack[0] last_msg = messages_stack[len(messages_stack) - 1] # Converting the string attributes of the messages objects to datetime time_filled timestamp = last_msg["timestamp"] first timestamp = datetime. strptime(first_msg["timestamp"],"%H:%M:%S") last timestamp = datetime.strptime(last msg "timestamp"], "%H:%M:%S") garbage drops last_msgi "garbage_drops"] # Calculating the time to fill by comparing the timestamps. # of the first and last fillings time_delta = Last timestamp time to fill = first_timestamp time delta.total_seconds() / 60 report id = len(reports) # This will be used for object indexing # The JSON object that will be appended to the JSON data file report = "id": report id, "timestamp": Lime filled timestamp, "garbage drops": garbage_drops, "time to fill" time_to_fill

# Append the new report to the objects of the data file # and write the data_file_objects array to the data file data file objects.append(report) with open(data_file, 'w') as file: json.dump(data_file_objects, file, indent-4, separators-(,,: 19) # Append the report object to the reports array and to the JSON data file # and clear the messages stack reports.append(report) messages stack = [0] In the main part of the program, we will check if the data exists, we will open it and then we will initialize the MQTT client, bind the on_connect and on_message event handlers to the above functions, connect to the specified MQTT broker, subscribe to the specified topic and listen for incoming messages. # Check if the data file exists if path.isfile(data_file) is False: raise Exception("Data file not found") # Read the contents of the JSON data file with open(data file) as fp: data file objects = json.load(fp) client mq.client (CLIENT_ID) client on connect on connect client.on message on message client.connect(MQTT BROKER, PORT) client.subscribe(TOPIC, 0) liegt op forever() # Initialize an MQTT client #Bind the on connect event handler #Bind the on message event handler # Connect on the specified MQTT broker # Subscribe to the specified topic # Listen continuously for messages وزارة التعليم 211773-1445 297

298 Complete Code from datetime import datetime import json import paho.mqtt.client as ma from os import path data_file = "your_file_path" # Absolute path to the JSON data file data_file_objects. = [] # This contains the objects from the JSON data file # Variables to setup MQTT client CLIENT ID="RECEIVER_01" #ID of the client MQTT BROKER = "broker.emqx.io" # Address of the broker TOPIC "waste/drops" # Topic to subscribe to # Default server port FLAG CONNECTED = False: # Connection flag PORT = 1883 messages stack = [] # The array with the messages per can filling reports = [] #The array with all the generated report objects def on connect(client, userdata, flags, rc): global FLAG CONNECTED # Access the FLAG CONNECTED variable if rc == 0: else: # If re is the client connected successfully FLAG CONNECTED = True print("Connected to MQTT Broker!") print("Failed to connect to MQTT Broker!") def on message(client, userdata, msg): buat messages stack # Access the messages_stack variable حرارة النص اليد 23-1445

#Decode the message payload payload str(msg.payload.decode()) # Convert the payload to a JSON object and append it # to the messages stack payload_object = json.Loads (payload) messages stack.append(payload_object) #When you receive a message, print it to the terminal print("MESSAGE RECEIVED ----||\n") print("Payload: " str(payload_object)) # If the payload object has the can filled flag set to True # generate a report for the filled can if payload object["can_filled"] == True: generate report() def generate report(): global data file objects global messages_stack global reports # Access data_file_objects variable # Access messages_stack variable # Access reports variable # Getting the first and last objects from the messages stack first_msg = messages stack[0] last_msg = messages_stack len(messages_stack) - 1] # Converting the string datetimes to datetime objects first timestamp = datetime.strptime(first_msgf "timestamp"], "%H:%M:%S"). last timestamp = datetime.strptime(last_msg["timestamp"], "%H:%M:%S") garbage drops = last_msg["garbage_drops"] calculing the time to fill by comparing the timestamps # of the first and last fillings phdelta last timestamp first timestamp 15 - בקוע 299

300 time to fill = time delta.total_seconds() / 60 report_id = len(reports) # This will be used for object indexing # The JSON object that will be appended to the JSON data file report ( } "id": report id, "garbage drops": garbage drops, "time_to_fill" time_to_fill # Append the new report to the objects of the data file # and write the data_file_objects array to the data file data file objects.append(report) with open(data_file, 'w') as file: json.dump(data_file_objects, file, indent-4, separators-('','')) # Append the report object to the reports array and to the JSON data file # and clear the messages stack reports.append(report) messages stack = [] # Check if the data file exists if path.isfile(data_file) is False: raise Exception("Data file not found") # Read the contents of the JSON data file with open(data_file) as fp: data_file_objects = json.load(fp) client = mq.Client(CLIENT_ID) client.on_connect = on_connect client.on message on message # Initialize an MQTT client # Bind the on_connect event handler # Bind the on message event handler client someNOST BROKER, PORT) # Connect on the specified MQTT broker client subscribe(TOPIC, 6) pop_forever() # Subscribe to the specified topic #Listen continuously for messages דאו - כקו

Data Analysis in Jupyter Notebook Now we will use a Jupyter Notebook to perform data analysis operations on the JSON data file. Because it would take too long to collect the data needed to perform data analysis, there is a prefilled JSON dataset that we will use. This dataset simulates leaving. the Arduino prototype running for a long period of time. The JSON file is available for download here: http://binary-academy.com/dnid/KSA/IOTZ/U3_L3_DATA.json In the beginning, we will import the required libraries and read the JSON data from the file. import os import pandas as pd import matplotlib.pyplot as plt # Library used for data manipulation #library used for plotting data # The data that will be used, extracted from the JSON dataset data = pd.read_json('U3_13_DATA. json', 'records', convert_dates=['timestamp']) We will then describe the dataset to extract statistical properties. data.describe().round(0) This is the downloaded data file that must be placed in the same project folder. id garbage drops time_to_fill وزارة التعليم J173-1445 count 50.0 50.0 50.0 mean 24.0 54.0 1520 std 15.0 30.0 100.0 min 0.0 20 5.0 25% 12.0 30.0 60.0 50% 24.0 550 147.0 75% 37.0 78.0 235.0 max 49.0 100.0 376.0 Figure 7.23 Data descrip 301

We will create two histograms grouped by the garbage drops and the time_to_fill properties. # Create histograms for the data using 8 groupings hist hist data.hist(['garbage drops'], figsize=(10,5),bins=8) data.hist(['time_to_fill'], figsize-(10,6),bins-8) Figure 1.24 Histoga We will then create two stem plots to display the garbage_drops and time_to_fill values at each time interval.. # Create stem plots for the data with diamond-shaped ('D') markers plt.stem(data['timestamp'], data['time_to_fill'], marker fmt='D'); plt.stem(data['timestamp'], data['garbage drops' ], marker fmt='D'); وابة التعليم 302 2123-1865 Figure 7.25: Stem prots

garbage_drops 20 Finally, we will create two plots that group the number of garbage_drops for each hour of the day and for each hour of the week. # Create 2 plots, one that groups mean garbage amount by hour and one by day. fig, (ax1, ax2) plt.subplots(2, figsize=(12, 8)) fig, supylabel('garbage_drops') data.groupby(data["timestamp" ).dt.hour)["garbage drops"].mean().plot(kind='bar', rot-0, xlabel= 'Hour of the day', ax=ax1); #Monday 0, Sunday = 6 data.groupby(data["timestamp" ).dt.day_of_week)["garbage drops"].mean(). plot(kinds 'bar', rot-8, ax-ax2); 100 80 III .. 2 3 5 7 6 Hour of the day 14 16 17 19 وزارة التعليم 2103-1445 . 1mestamp Figure 7 26 Group by plots ا 20 21 22 23 303

304 Exercises 1 Create a diagram that illustrates the relationship between the two Python scripts and the JSON file that holds the data. 2 Create a Python script that connects to three topics and write an on_connect event handler that prints the configuration info and the topics to which the client is subscribed to the terminal. حرارة التعليم 021-1445

3 Update the on_message object to print info to the terminal about the client that published the data and the topic that the data was received from Create a new JSON file that will hold all the values from the messages stack and use the generate_report() function to append the values of the messages_stack to the new JSON file. 5 In the Jupyter notebook, create a new scatter plot for the same data that you processed in the lesson. 6 Add another Python script that will receive the messages that you published from the script in exercise 6 of lesson 2. When you receive a message print the info about the publisher, the receiver, and the subscribed topic to the terminal. حرارة التعليم 00-1445 305