DATA SCIENCE TECHNOLOGY STACK

ASSESS SUPERSTEP

6.0 OBJECTIVES

This chapter will make you understand the practical concepts of:

• Assess superstep

• Python NetworkX Library used to draw network routing graphs

• Python Schedule library to schedule various jobs

 

6.1 ENGINEERING A PRACTICAL ASSESS SUPERSTEP

Let us first consider an example of Network routing:

Python uses a library called NetworkX for network routing.

 

To use NetworkX library, first install the library on your machine by using following command on your command prompt:

pip install Network X

 

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

Network X provides:

• tools for the study of the structure and dynamics of social, biological, and infrastructure networks;

• a standard programming interface and graph implementation that is suitable for many applications;

• a rapid development environment for collaborative, multidisciplinary projects;

• an interface to existing numerical algorithms and code written in C, C++, and FORTRAN; and the ability to painlessly work with large nonstandard data sets.

With NetworkX you can load and store networks in standard and nonstandard data formats, generate many types of random and classic 63 networks, analyze network structure, build network models, design new network algorithms, draw networks, and much more.

 

Graph Theory:

In the Graph Theory, a graph has a finite set of vertices (V) connected to two-elements (E).

Each vertex (v) connecting two destinations, or nodes, is called a link or an edge. Consider the Graph of bike paths below: sets {K,L}, {F,G}, {J,H}, {H,L}, {A,B}, and {C,E} are examples of edges.

 

The total number of edges for each node is the degree of that node. In the Graph above, M has a degree of 2 ({M,H} and {M,L}) while B has a degree of 1 ({B,A}). Degree is described formally as:

 Connections through use of multiple edges are called paths. {F, H, M, L, H, J, G, I} is an example of a path. A simple path is when a path does not repeat a node — formally known as Eulerian path. {I, G, J, H, F} is an example of a simple path. The shortest simple path is called Geodesic. Geodesic between I and J is {I, G, J} or {I, K, J}. Finally, a cycle is when a path’s start and end points are the same (ex. {H,M,L,H}). In some notebooks, a cycle is formally referred to as Eulerian cycle.

Not all networks in a Graph system are interconnected. This disconnection is when components are formed. As shown in the graph below, a component is formed only when every node has a path to other nodes.

Neo4J’s book on graph algorithms provides a clear summary

For example:

# ### Creating a graph

# Create an empty graph with no nodes and no edges.

 

import networkx as nx

G = nx.Graph()

 

# By definition, a `Graph` is a collection of nodes (vertices) along with identified pairs of

# nodes (called # edges, links, etc). In NetworkX, nodes can be any [hashable] object e.g., a # text string, an image, an

# XML object, another Graph, a customized node object, etc.

# # Nodes

# The graph `G` can be grown in several ways. NetworkX includes many graph generator

# functions # and facilities to read and write graphs in many formats.

# To get started # though we’ll look at simple manipulations. You can add one node at a

# time,

G.add_node(1)

# or add nodes from any [iterable] container, such as a list

G.add_nodes_from([2, 3])

# Nodes from one graph can be incorporated into another:

H = nx.path_graph(10)

G.add_nodes_from(H)

# `G` now contains the nodes of `H` as nodes of `G`.

# In contrast, you could use the graph `H` as a node in `G`

G.add_node(H)

# The graph `G` now contains `H` as a node. This flexibility is very powerful as it allows

# graphs of graphs, graphs of files, graphs of functions and much more. It is worth thinking

# about how to structure # your application so that the nodes are useful entities. Of course

# you can always use a unique identifier

 # in `G` and have a separate dictionary keyed by

# identifier to the node information if you prefer.

 

# # Edges

# `G` can also be grown by adding one edge at a time,

 

G.add_edge(1, 2)

e = (2, 3)

G.add_edge(*e) # unpack edge tuple*

 

# by adding a list of edges,

G.add_edges_from([(1, 2), (1, 3)])

 

# or by adding any ebunch of edges. An *ebunch* is any iterable container of edge-tuples.

# An edge-tuple can be a 2-tuple of nodes or a 3-tuple with 2 nodes followed by an edge

# attribute dictionary, e.g.,

# `(2, 3, {'weight': 3.1415})`. Edge attributes are discussed further below. G.add_edges_from(H.edges)

# There are no complaints when adding existing nodes or edges. 66

# For example, after removing all # nodes and edges, G.clear()

# we add new nodes/edges and NetworkX quietly ignores any that are already present.

 

G.add_edges_from([(1, 2), (1, 3)])

G.add_node(1)

G.add_edge(1, 2)

G.add_node("spam") # adds node "spam"

G.add_nodes_from("spam") # adds 4 nodes: 's', 'p', 'a', 'm'

G.add_edge(3, 'm')

 

# At this stage the graph `G` consists of 8 nodes and 3 edges, as can be seen by:

 

G.number_of_nodes()

G.number_of_edges()

 

# # Examining elements of a graph

# We can examine the nodes and edges. Four basic graph properties facilitate reporting:

#`G.nodes`,

# `G.edges`, `G.adj` and `G.degree`. These are set-like views of the nodes, edges, neighbors

# (adjacencies), and degrees of nodes in a graph. They offer a continually updated read-only

#view into the graph structure. They are also dict-like in that you can look up node and edge

#data attributes via the views and iterate with data attributes using methods `.items()`,

#`.data('span')`.

# If you want a specific container type instead of a view, you can specify one.

# Here we use lists, though sets, dicts, tuples and other containers may be better in other

#contexts.

 

list(G.nodes)

list(G.edges)

list(G.adj[1]) # or list(G.neighbors(1))

G.degree[1] # the number of edges incident to 1

 

# One can specify to report the edges and degree from a subset of all nodes using an

#nbunch.

# An *nbunch* is any of: `None` (meaning all nodes), a node, or an iterable container of nodes that is # not itself a node in the graph.

 

G.edges([2, 'm'])

G.degree([2, 3])

 

# # Removing elements from a graph

# One can remove nodes and edges from the graph in a similar fashion to adding.

# Use methods `Graph.remove_node()`, `Graph.remove_nodes_from()`,

#`Graph.remove_edge()`

# and `Graph.remove_edges_from()`, e.g.

 

G.remove_node(2)

G.remove_nodes_from("spam")

list(G.nodes)

G.remove_edge(1, 3)

 

# # Using the graph constructors

# Graph objects do not have to be built up incrementally - data specifying

# graph structure can be passed directly to the constructors of the various graph classes.

# When creating a graph structure by instantiating one of the graph

# classes you can specify data in several formats.

 

G.add_edge(1, 2)

H = nx.DiGraph(G) # create a DiGraph using the connections from G

list(H.edges())

edgelist = [(0, 1), (1, 2), (2, 3)]

H = nx.Graph(edgelist)

 

# # What to use as nodes and edges

# You might notice that nodes and edges are not specified as NetworkX

# objects. This leaves you free to use meaningful items as nodes and

# edges. The most common choices are numbers or strings, but a node can

# be any hashable object (except `None`), and an edge can be associated

# with any object `x` using `G.add_edge(n1, n2, object=x)`.

 

# As an example, `n1` and `n2` could be protein objects from the RCSB Protein Data Bank,

#and `x` # could refer to an XML record of publications detailing experimental observations

#of their interaction.

# We have found this power quite useful, but its abuse can lead to surprising behavior

#unless one is # familiar with Python.

# If in doubt, consider using `convert_node_labels_to_integers()` to obtain a more

traditional graph with # integer labels. Accessing edges and neighbors

# In addition to the views `Graph.edges`, and `Graph.adj`, access to edges and neighbors is

#possible using subscript notation.

 

G = nx.Graph([(1, 2, {"color": "yellow"})])

G[1] # same as G.adj[1]

G[1][2]

G.edges[1, 2]

 

# You can get/set the attributes of an edge using subscript notation

# if the edge already exists

 

G.add_edge(1, 3)

G[1][3]['color'] = "blue"

G.edges[1, 2]['color'] = "red"

G.edges[1, 2]

 

# Fast examination of all (node, adjacency) pairs is achieved using

# `G.adjacency()`, or `G.adj.items()`.

# Note that for undirected graphs, adjacency iteration sees each edge twice.

 

FG = nx.Graph()

FG.add_weighted_edges_from([(1, 2, 0.125), (1, 3, 0.75), (2, 4, 1.2), (3, 4, 0.375)])

for n, nbrs in FG.adj.items():

     for nbr, eattr in nbrs.items():

      wt = eattr['weight']

       if wt < 0.5: print(f"({n}, {nbr}, {wt:.3})")

# Convenient access to all edges is achieved with the edges property

 

for (u, v, wt) in FG.edges.data('weight'):

       if wt < 0.5:

         print(f"({u}, {v}, {wt:.3})")

 

# # Adding attributes to graphs, nodes, and edges

#

# Attributes such as weights, labels, colors, or whatever Python object you like,

# can be attached to graphs, nodes, or edges.

#

# Each graph, node, and edge can hold key/value attribute pairs in an associated

# attribute dictionary (the keys must be hashable). By default these are empty,

# but attributes can be added or changed using `add_edge`, `add_node` or direct

# manipulation of the attribute dictionaries named `G.graph`, `G.nodes`, and

# `G.edges` for a graph `G`.

# ## Graph attributes

# Assign graph attributes when creating a new graph

 

G = nx.Graph(day="Friday")

G.graph

 

# Or you can modify attributes later

 

G.graph['day'] = "Monday"

G.graph

 

# # Node attributes

# Add node attributes using `add_node()`, `add_nodes_from()`, or `G.nodes`

 

G.add_node(1, time='5pm')

G.add_nodes_from([3], time='2pm')

G.nodes[1]

G.nodes[1]['room'] = 714

G.nodes.data()

 

# Note that adding a node to `G.nodes` does not add it to the graph, use

# `G.add_node()` to add new nodes. Similarly for edges.

# # Edge Attributes

# Add/change edge attributes using `add_edge()`, `add_edges_from()`,

# or subscript notation.

 

G.add_edge(1, 2, weight=4.7 )

G.add_edges_from([(3, 4), (4, 5)], color='red')

G.add_edges_from([(1, 2, {'color': 'blue'}), (2, 3, {'weight': 8})])

G[1][2]['weight'] = 4.7

G.edges[3, 4]['weight'] = 4.2

 

# The special attribute `weight` should be numeric as it is used by

# algorithms requiring weighted edges.

# Directed graphs

# The `DiGraph` class provides additional methods and properties specific

# to directed edges, e.g.,

# `DiGraph.out_edges`, `DiGraph.in_degree`,

# `DiGraph.predecessors()`, `DiGraph.successors()` etc.

# To allow algorithms to work with both classes easily, the directed versions of

# `neighbors()` is equivalent to `successors()` while `degree` reports

# the sum of `in_degree` and `out_degree` even though that may feel

# inconsistent at times.

 

DG = nx.DiGraph()

DG.add_weighted_edges_from([(1, 2, 0.5), (3, 1, 0.75)])

DG.out_degree(1, weight='weight')

DG.degree(1, weight='weight')

list(DG.successors(1))

list(DG.neighbors(1))

 

# Some algorithms work only for directed graphs and others are not well

# defined for directed graphs. Indeed the tendency to lump directed

# and undirected graphs together is dangerous. If you want to treat

# a directed graph as undirected for some measurement you should probably

# convert it using `Graph.to_undirected()` or with

 

H = nx.Graph(G) # create an undirected graph H from a directed graph G

 

# # Multigraphs

# NetworkX provides classes for graphs which allow multiple edges

# between any pair of nodes. The `MultiGraph` and

# `MultiDiGraph`

# classes allow you to add the same edge twice, possibly with different

# edge data. This can be powerful for some applications, but many

# algorithms are not well defined on such graphs.

# Where results are well defined,

# e.g., `MultiGraph.degree()` we provide the function. Otherwise you

# should convert to a standard graph in a way that makes the measurement well defined

 

MG = nx.MultiGraph()

MG.add_weighted_edges_from([(1, 2, 0.5), (1, 2, 0.75), (2, 3, 0.5)])

dict(MG.degree(weight='weight'))

GG = nx.Graph()

for n, nbrs in MG.adjacency():

      for nbr, edict in nbrs.items():

         minvalue = min([d['weight'] for d in edict.values()])

          GG.add_edge(n, nbr, weight = minvalue)

 

nx.shortest_path(GG, 1, 3)

 

# # Graph generators and graph operations

# In addition to constructing graphs node-by-node or edge-by-edge, they

# can also be generated by

# 1. Applying classic graph operations, such as:

# 1. Using a call to one of the classic small graphs, e.g.,

# 1. Using a (constructive) generator for a classic graph, e.g.,

# like so:

 

K_5 = nx.complete_graph(5)

K_3_5 = nx.complete_bipartite_graph(3, 5)

barbell = nx.barbell_graph(10, 10)

lollipop = nx.lollipop_graph(10, 20)

 

# 1. Using a stochastic graph generator, e.g, like so:

 

er = nx.erdos_renyi_graph(100, 0.15)

ws = nx.watts_strogatz_graph(30, 3, 0.1)

ba = nx.barabasi_albert_graph(100, 5)

red = nx.random_lobster(100, 0.9, 0.9)

 

# 1. Reading a graph stored in a file using common graph formats,

# such as edge lists, adjacency lists, GML, GraphML, pickle, LEDA and others.

 

nx.write_gml(red, "path.to.file")

mygraph = nx.read_gml("path.to.file")

 

# For details on graph formats see Reading and writing graphs

# and for graph generator functions see Graph generators

# # Analyzing graphs

# The structure of `G` can be analyzed using various graph-theoretic functions such as:

 

G = nx.Graph()

G.add_edges_from([(1, 2), (1, 3)])

G.add_node("spam") # adds node "spam"

list(nx.connected_components(G))

sorted(d for n, d in G.degree())

nx.clustering(G)

 

# Some functions with large output iterate over (node, value) 2-tuples.

# These are easily stored in a [dict](https://docs.python.org/3/library/stdtypes.html#dict) # structure if you desire.

 

sp = dict(nx.all_pairs_shortest_path(G))

sp[3]

 

# See Algorithms for details on graph algorithms supported.

# # Drawing graphs

# NetworkX is not primarily a graph drawing package but basic drawing with

# Matplotlib as well as an interface to use the open source Graphviz software

# package are included. These are part of the `networkx.drawing` module and will

# be imported if possible.

# First import Matplotlib’s plot interface (pylab works too)

 

import matplotlib.pyplot as plt

# To test if the import of `networkx.drawing` was successful draw `G` using one of

 

G = nx.petersen_graph()

plt.subplot(121)

nx.draw(G, with_labels=True, font_weight='bold')

plt.subplot(122)

nx.draw_shell(G, nlist=[range(5, 10), range(5)], with_labels=True,

font_weight='bold')

 

# when drawing to an interactive display. Note that you may need to issue a Matplotlib

 

plt.show()

 

options = {

      'node_color': 'black',

      'node_size': 100,

       'width': 3,

}

plt.subplot(221)

nx.draw_random(G, **options

plt.subplot(222)

nx.draw_circular(G, **options)

plt.subplot(223)

nx.draw_spectral(G, **options)

plt.subplot(224)

nx.draw_shell(G, nlist=[range(5,10), range(5)], **options)

 

# You can find additional options via `draw_networkx()` and

# layouts via `layout`.

# You can use multiple shells with `draw_shell()`.

 

G = nx.dodecahedral_graph()

shells = [[2, 3, 4, 5, 6], [8, 1, 0, 19, 18, 17, 16, 15, 14, 7], [9, 10, 11, 12, 13]]

nx.draw_shell(G, nlist=shells, **options)

 

# To save drawings to a file, use, for example

nx.draw(G)

plt.savefig("path.png")

# writes to the file `path.png` in the local directory.

Output:

G = nx.petersen_graph()

plt.subplot(121)

nx.draw(G, with_labels=True, font_weight='bold')

plt.subplot(122)

nx.draw_shell(G, nlist=[range(5, 10), range(5)], with_labels=True,

font_weight='bold')

 

plt.show()

      options = {

          'node_color': 'black',

          'node_size': 100,

 

'width': 3,

}

plt.subplot(221)

nx.draw_random(G, **options)

plt.subplot(222)

nx.draw_circular(G, **options)

plt.subplot(223)

nx.draw_spectral(G, **options)

plt.subplot(224)

nx.draw_shell(G, nlist=[range(5,10), range(5)], **options

 G = nx.dodecahedral_graph()

shells = [[2, 3, 4, 5, 6], [8, 1, 0, 19, 18, 17, 16, 15, 14, 7], [9, 10, 11, 12, 13]]

nx.draw_shell(G, nlist=shells, **options)

nx.draw(

nx.draw(G)

plt.savefig("path.png")

Python:

Python Schedule Library:

Schedule is in-process scheduler for periodic jobs that use the builder pattern for configuration. Schedule lets you run Python functions (or any other callable) periodically at pre-determined intervals using a simple, human-friendly syntax.

Schedule Library is used to schedule a task at a particular time every day or a particular day of a week. We can also set time in 24 hours format that when a task should run. Basically, Schedule Library matches your systems time to that of scheduled time set by you. Once the scheduled time and system time matches the job function (command function that is scheduled ) is called.

 

Installation:

$ pip install schedule

schedule.Scheduler class:

• schedule.every(interval=1) : Calls every on the default scheduler instance. Schedule a new periodic job.

• schedule.run_pending() : Calls run pending on the default scheduler instance. Run all jobs that are scheduled to run.

• schedule.run_all(delay_seconds=0) : Calls run_all on the default scheduler instance. Run all jobs regardless if they are scheduled to run or not. schedule.idle_seconds() : Calls idle_seconds on the default scheduler instance.

• schedule.next_run() : Calls next_run on the default scheduler instance. Datetime when the next job should run.

• schedule.cancel_job(job) : Calls cancel_job on the default scheduler instance. Delete a scheduled job.

• schedule.Job(interval, scheduler=None) class A periodic job as used by Scheduler.

 

Parameters:

• interval: A quantity of a certain time unit

• scheduler: The Scheduler instance that this job will register itself with once it has been fully configured in Job.do().

 

Basic methods for Schedule.job:

• at(time_str) : Schedule the job every day at a specific time. Calling this is only valid for jobs scheduled to run every N day(s). Parameters: time_str – A string in XX:YY format. Returns: The invoked job instance

• do(job_func, *args, **kwargs) : Specifies the job_func that should be called every time the job runs. Any additional arguments are passed on to job_func when the job runs. Parameters: job_func – The function to be scheduled. Returns: The invoked job instance

• run() : Run the job and immediately reschedule it. Returns: The return value returned by the job_func

• to(latest) : Schedule the job to run at an irregular (randomized) interval. For example, every(A).to(B).seconds executes the job function every N seconds such that A <= N <= B.

 

For example

# Schedule Library imported

import schedule

import time

 

# Functions setup

def placement():

print("Get ready for Placement at various companies")

 

def good_luck():

      print("Good Luck for Test")

def work():

       print("Study and work hard")

def bedtime():

      print("It is bed time go rest")

 

def datascience():

print("Data science with python is fun")

 

# Task scheduling

# After every 10mins datascience() is called.

schedule.every(10).minutes.do(datascience

 

# After every hour datascience() is called.

schedule.every().hour.do(datascience)

 

# Every day at 12am or 00:00 time bedtime() is called.

schedule.every().day.at("00:00").do(bedtime)

 

# After every 5 to 10 mins in between run work()

schedule.every(5).to(10).minutes.do(work)

 

# Every monday good_luck() is called

schedule.every().monday.do(good_luck)

 

# Every tuesday at 18:00 placement() is called

schedule.every().tuesday.at("18:00").do(placement

 

# Loop so that the scheduling task

# keeps on running all time.

while True:

 

# Checks whether a scheduled task

# is pending to run or not

schedule.run_pending()

time.sleep(1)

 

6.2 REFERENCES

• Python for Data Science For Dummies, by Luca Massaron John Paul Mueller (Author),

• ISBN-13 : 978-8126524938, Wiley

• Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, 2nd Edition

• by William McKinney (Author), ISBN-13 : 978-9352136414 , Shroff/O'Reilly

• Data Science From Scratch: First Principles with Python, Second Edition by Joel Grus,

• ISBN-13 : 978-9352138326, Shroff/O'Reilly 77

• Data Science from Scratch by Joel Grus, ISBN-13 : 978-1491901427 ,O′Reilly

• Data Science Strategy For Dummies by Ulrika Jagare, ISBN-13 : 978-8126533367 , Wiley

• Pandas for Everyone: Python Data Analysis, by Daniel Y. Chen, ISBN-13 : 978-

• 9352869169, Pearson Education

• Practical Data Science with R (MANNING) by Nina Zumel, John Mount, ISBN-13 : 978-

• 9351194378, Dreamtech Press

 

6.3 UNIT END QUESTION

1. Write Python program to create the network routing diagram from the given data.

2. Write a Python program to build directed acyclic graph.

3. Write a Python program to pick the content for Bill Boards from the given data.

4. Write a Python program to generate visitors data from the given csv file.