Skip to main content

exercice mapreduce avec python (moyenne d'une variable)


Écrire un programme python FriendByAge.py pour trouver le nombre moyen d’amis dans un réseau social selon l’âge.

on a les variables       userID name age nbFriend
           
Données : socialfriends.csv dans le dossier 
https://drive.google.com/open?id=1fSt2aFZryUiKBxI3aJXQnpezJ3xntESP


le code python se fait comme suit

 

from mrjob.job import MRJob

import numpy


class MRFriendByAge(MRJob):

    

    def mapper(self,key,line):

        (userID,name,age,number) = line.split(',')

        k = int(number)

        yield age,k


    def reducer(self,age,number1):

            sum = 0 

            count = 0

            for n in number1:

                sum = sum + n

                count = count + 1


            average1  = sum / count  


          

         

            yield age,   average1 

        



if __name__ == '__main__':

    MRFriendByAge.run()




comme vous le voyer pour gérer notre programme nous avons du créer une méthode mapper et une auttre reducer.  la méthode mapper nus a permis de splitter les donner et de les distribues  donc dans cette méthode on rentre les données ligne par ligne et il sort chaque variable age avec un tableau représentant les valeurs nbfriend associées et ce sont ces éléments que nous allons passer en paramètre a reducer qui va nous sortir le résultat 



!python FriendByAge.py  /home/cloudera/Downloads/socialfriends.csv

No configs found; falling back on auto-configuration
Creating temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460
Running step 1 of 1...
Streaming final output from /tmp/FriendByAge.cloudera.20180213.155551.376460/output...
"18"    343
"19"    213
"20"    165
"21"    350
"22"    206
"23"    246
"24"    233
"25"    197
"26"    242
"27"    228
"28"    209
"29"    215
"30"    235
"31"    267
"32"    207
"33"    325
"34"    245
"35"    211
"36"    246
"37"    249
"38"    193
"39"    169
"40"    250
"41"    268
"42"    303
"43"    230
"44"    282
"45"    309
"46"    223
"47"    233
"48"    281
"49"    184
"50"    254
"51"    302
"52"    340
"53"    222
"54"    278
"55"    295
"56"    306
"57"    258
"58"    116
"59"    220
"60"    202
"61"    256
"62"    220
"63"    384
"64"    281
"65"    298
"66"    276
"67"    214
"68"    269
"69"    235
Removing temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460...


Comments

Post a Comment

Popular posts from this blog

FastAPI Instrumentalisation with prometheus and grafana Part1 [Counter]

welcome to this hands-on lab on API instrumentation using Prometheus and FastAPI! In the world of modern software development, real-time API monitoring is essential for understanding usage patterns, debugging issues, and ensuring optimal performance. In this lab, we’ll demonstrate how to enhance a FastAPI-based application with Prometheus metrics to monitor its behavior effectively. We’ve already set up the lab environment for you, complete with Grafana, Prometheus, and a PostgreSQL database. While FastAPI’s integration with databases is outside the scope of this lab, our focus will be entirely on instrumentation and monitoring. For those interested in exploring the database integration or testing , you can review the code in our repository: FastAPI Monitoring Repository . What You’ll Learn In this lab, we’ll walk you through: Setting up Prometheus metrics in a FastAPI application. Instrumenting API endpoints to track: Number of requests HTTP methods Request paths Using Grafana to vi...
Post a Comment
Read more

Join Ubuntu 20.04 to Active Directory with SSSD and SSH Access

Join Ubuntu 20.04 to Active Directory with SSSD and SSH Access  Overview This guide walks you through joining an Ubuntu 20.04 machine to an Active Directory domain using SSSD, configuring PAM for AD user logins over SSH, and enabling automatic creation of home directories upon first login. We’ll also cover troubleshooting steps and verification commands. Environment Used Component Value Ubuntu Client       ubuntu-client.bazboutey.local Active Directory FQDN   bazboutey.local Realm (Kerberos)   BAZBOUTEY.LOCAL AD Admin Account   Administrator Step 1: Prerequisites and Package Installation 1.1 Update system and install required packages bash sudo apt update sudo apt install realmd sssd libnss-sss libpam-sss adcli \ samba-common-bin oddjob oddjob-mkhomedir packagekit \ libpam-modules openssh-server Step 2: Test DNS and Kerberos Configuration Ensure that the client can resolve the AD domain and discover services. 2.1 Test domain name resol...
Post a Comment
Read more

Observability with grafana and prometheus (SSO configutation with active directory)

How to Set Up Grafana Single Sign-On (SSO) with Active Directory (AD) Grafana is a powerful tool for monitoring and visualizing data. Integrating it with Active Directory (AD) for Single Sign-On (SSO) can streamline access and enhance security. This tutorial will guide you through the process of configuring Grafana with AD for SSO. Prerequisites Active Directory Domain : Ensure you have an AD domain set up. Domain: bazboutey.local AD Server IP: 192.168.170.212 Users: grafana (for binding AD) user1 (to demonstrate SSO) we will end up with a pattern like this below Grafana Installed : Install Grafana on your server. Grafana Server IP: 192.168.179.185 Administrator Privileges : Access to modify AD settings and Grafana configurations. Step 1: Configure AD for LDAP Integration Create a Service Account in AD: Open Active Directory Users and Computers. Create a user (e.g., grafana ). Assign this user a strong password (e.g., Grafana 123$ ) and ensure it doesn’t expire. Gather Required AD D...
Post a Comment
Read more