Skip to main content

exercice mapreduce avec python (moyenne d'une variable)


Écrire un programme python FriendByAge.py pour trouver le nombre moyen d’amis dans un réseau social selon l’âge.

on a les variables       userID name age nbFriend
           
Données : socialfriends.csv dans le dossier 
https://drive.google.com/open?id=1fSt2aFZryUiKBxI3aJXQnpezJ3xntESP


le code python se fait comme suit

 

from mrjob.job import MRJob

import numpy


class MRFriendByAge(MRJob):

    

    def mapper(self,key,line):

        (userID,name,age,number) = line.split(',')

        k = int(number)

        yield age,k


    def reducer(self,age,number1):

            sum = 0 

            count = 0

            for n in number1:

                sum = sum + n

                count = count + 1


            average1  = sum / count  


          

         

            yield age,   average1 

        



if __name__ == '__main__':

    MRFriendByAge.run()




comme vous le voyer pour gérer notre programme nous avons du créer une méthode mapper et une auttre reducer.  la méthode mapper nus a permis de splitter les donner et de les distribues  donc dans cette méthode on rentre les données ligne par ligne et il sort chaque variable age avec un tableau représentant les valeurs nbfriend associées et ce sont ces éléments que nous allons passer en paramètre a reducer qui va nous sortir le résultat 



!python FriendByAge.py  /home/cloudera/Downloads/socialfriends.csv

No configs found; falling back on auto-configuration
Creating temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460
Running step 1 of 1...
Streaming final output from /tmp/FriendByAge.cloudera.20180213.155551.376460/output...
"18"    343
"19"    213
"20"    165
"21"    350
"22"    206
"23"    246
"24"    233
"25"    197
"26"    242
"27"    228
"28"    209
"29"    215
"30"    235
"31"    267
"32"    207
"33"    325
"34"    245
"35"    211
"36"    246
"37"    249
"38"    193
"39"    169
"40"    250
"41"    268
"42"    303
"43"    230
"44"    282
"45"    309
"46"    223
"47"    233
"48"    281
"49"    184
"50"    254
"51"    302
"52"    340
"53"    222
"54"    278
"55"    295
"56"    306
"57"    258
"58"    116
"59"    220
"60"    202
"61"    256
"62"    220
"63"    384
"64"    281
"65"    298
"66"    276
"67"    214
"68"    269
"69"    235
Removing temp directory /tmp/FriendByAge.cloudera.20180213.155551.376460...


Comments

Post a Comment

Popular posts from this blog

ExternalDNS on Kubernetes - mirecloud homelab part 4

MireCloud Home Lab · DevOps ExternalDNS on Kubernetes Automatic Sync with BIND via RFC2136 How to fully automate DNS management in a bare-metal Kubernetes homelab using Cilium, BIND, and HashiCorp Vault. 📅 February 22, 2026 ⏱ ~10 min read 🔧 ExternalDNS v0.20.0 ☸ Kubernetes v1.34 Kubernetes v1.34 ExternalDNS v0.20.0 Cilium Gateway API BIND (RFC2136) TSIG / HMAC-SHA256 HashiCorp Vault External Secrets Operator ArgoCD (GitOps) cert-manager When you run a Kubernetes homelab with multiple exposed services — Grafana, Keycloak, ArgoCD, PgAdmin — you quickly find yourself maintaining DNS entries in BIND manually . It's repetitive, prone to errors, and breaks the GitOps flow. The solution is ExternalDNS . This controller monitors your Services, Ingresses, and HTTPRoutes in real-time, automatically pushing DNS updates to BIND as soon as a route is created. No mo...
Post a Comment
Read more

FastAPI Instrumentalisation with prometheus and grafana Part1 [Counter]

welcome to this hands-on lab on API instrumentation using Prometheus and FastAPI! In the world of modern software development, real-time API monitoring is essential for understanding usage patterns, debugging issues, and ensuring optimal performance. In this lab, we’ll demonstrate how to enhance a FastAPI-based application with Prometheus metrics to monitor its behavior effectively. We’ve already set up the lab environment for you, complete with Grafana, Prometheus, and a PostgreSQL database. While FastAPI’s integration with databases is outside the scope of this lab, our focus will be entirely on instrumentation and monitoring. For those interested in exploring the database integration or testing , you can review the code in our repository: FastAPI Monitoring Repository . What You’ll Learn In this lab, we’ll walk you through: Setting up Prometheus metrics in a FastAPI application. Instrumenting API endpoints to track: Number of requests HTTP methods Request paths Using Grafana to vi...
Post a Comment
Read more

Kubernetes Secrets: Risks, Reality, and Secure Design - MireCloud Homelab Part 7

MireCloud Series · Part 7 · DevSecOps Your Kubernetes Secrets Are Not Safe. Here's the Forensic Proof — and the Fix. Why your Vault + ESO setup leaks plaintext credentials into etcd, and how to close the gap with Encryption at Rest and the Vault CSI Driver. Zero Trust ⏱ ~15 min read ☸ Kubernetes v1.34 Reproducible lab Kubernetes v1.34 HashiCorp Vault External Secrets Operator Secrets Store CSI Driver AES-CBC / AES-GCM / KMS v2 etcdctl ArgoCD Cilium Gateway API ▸ TL;DR If you run HashiCorp Vault with External Secrets Operator on Kubernetes, your application secrets are stored as plaintext bytes in etcd . Any operator with kubectl get secrets , any backup of your control plane, or any disk that ever held an etcd snapshot can recover them in seconds. This post proves it with a hexdump from a live cluster, then closes the gap with two layered controls: Encryption at Rest to prot...
Post a Comment
Read more