Skip to main content

exercice mapreduce avec python (moyenne d'une variable)


Écrire un programme python FriendByAge.py pour trouver le nombre moyen d’amis dans un réseau social selon l’âge.

on a les variables       userID name age nbFriend
           
Données : socialfriends.csv dans le dossier 
https://drive.google.com/open?id=1fSt2aFZryUiKBxI3aJXQnpezJ3xntESP


le code python se fait comme suit

 

from mrjob.job import MRJob

import numpy


class MRFriendByAge(MRJob):

    

    def mapper(self,key,line):

        (userID,name,age,number) = line.split(',')

        k = int(number)

        yield age,k


    def reducer(self,age,number1):

            sum = 0 

            count = 0

            for n in number1:

                sum = sum + n

                count = count + 1


            average1  = sum / count  


          

         

            yield age,   average1 

        



if __name__ == '__main__':

    MRFriendByAge.run()




comme vous le voyer pour gérer notre programme nous avons du créer une méthode mapper et une auttre reducer.  la méthode mapper nus a permis de splitter les donner et de les distribues  donc dans cette méthode on rentre les données ligne par ligne et il sort chaque variable age avec un tableau représentant les valeurs nbfriend associées et ce sont ces éléments que nous allons passer en paramètre a reducer qui va nous sortir le résultat 



!python FriendByAge.py  /home/cloudera/Downloads/socialfriends.csv

No configs found; falling back on auto-configuration
Creating temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460
Running step 1 of 1...
Streaming final output from /tmp/FriendByAge.cloudera.20180213.155551.376460/output...
"18"    343
"19"    213
"20"    165
"21"    350
"22"    206
"23"    246
"24"    233
"25"    197
"26"    242
"27"    228
"28"    209
"29"    215
"30"    235
"31"    267
"32"    207
"33"    325
"34"    245
"35"    211
"36"    246
"37"    249
"38"    193
"39"    169
"40"    250
"41"    268
"42"    303
"43"    230
"44"    282
"45"    309
"46"    223
"47"    233
"48"    281
"49"    184
"50"    254
"51"    302
"52"    340
"53"    222
"54"    278
"55"    295
"56"    306
"57"    258
"58"    116
"59"    220
"60"    202
"61"    256
"62"    220
"63"    384
"64"    281
"65"    298
"66"    276
"67"    214
"68"    269
"69"    235
Removing temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460...


Comments

Post a Comment

Popular posts from this blog

Observability with grafana and prometheus (SSO configutation with active directory)

How to Set Up Grafana Single Sign-On (SSO) with Active Directory (AD) Grafana is a powerful tool for monitoring and visualizing data. Integrating it with Active Directory (AD) for Single Sign-On (SSO) can streamline access and enhance security. This tutorial will guide you through the process of configuring Grafana with AD for SSO. Prerequisites Active Directory Domain : Ensure you have an AD domain set up. Domain: bazboutey.local AD Server IP: 192.168.170.212 Users: grafana (for binding AD) user1 (to demonstrate SSO) we will end up with a pattern like this below Grafana Installed : Install Grafana on your server. Grafana Server IP: 192.168.179.185 Administrator Privileges : Access to modify AD settings and Grafana configurations. Step 1: Configure AD for LDAP Integration Create a Service Account in AD: Open Active Directory Users and Computers. Create a user (e.g., grafana ). Assign this user a strong password (e.g., Grafana 123$ ) and ensure it doesn’t expire. Gather Required AD D...
Post a Comment
Read more

Deploying a Scalable Monitoring Stack Lab on AWS using Terraform and Ansible

Deploying a Scalable Monitoring Stack Lab on AWS using Terraform and Ansible Introduction Effective monitoring is a cornerstone of cloud infrastructure management, ensuring high availability and performance. This guide provides a professional walkthrough on deploying Prometheus , Grafana , and Node Exporter on AWS using Terraform for infrastructure provisioning and Ansible for configuration management. This lab will create a prometheus server and a grafana server, It will install node exporter on both server. You should be able to see the metrics in grafana, we already install a node exporter dashboard for the user. The diagram below will give you an idea of what the architecture will look like If you want to replicate this lab, you can find the complete code repository here: GitHub - MireCloud Terraform Infra .  Infrastructure Setup with Terraform 1. Creating a Dedicated VPC To ensure isolation, we define a VPC named Monitoring with a CIDR block of 10.0.0.0/16 . reso...
Post a Comment
Read more

Building a Static Website on AWS with Terraform

The Journey to a Fully Automated Website Deployment A few weeks ago, I found myself needing to deploy a simple static website . Manually setting up an S3 bucket, configuring permissions, and linking it to a CloudFront distribution seemed like a tedious process. As someone who loves automation, I decided to leverage Terraform to simplify the entire process. Why Terraform? Infrastructure as Code (IaC) is a game-changer. With Terraform, I could:  Avoid manual setup errors  Easily reproduce and  Automate security best practices Instead of clicking through AWS settings, I wrote a few Terraform scripts and deployed everything in minutes. Let me walk you through how I did it!  Architecture Overview The architecture consists of three main components: User:  The end user accesses the website via a CloudFront URL.  CloudFront Distribution:  Acts as a content delivery network (CDN) to distribute content efficiently, reduce latency, and enhance security. It ...
Post a Comment
Read more