Search

With this chapter, you will begin to take the first steps in the world of data analysis, seeing in detail all the concepts and processes that make up this discipline. The concepts discussed in this chapter will be helpful background for the following chapters, where these concepts and procedures will be applied in the form of Python code, through the use of several libraries that will be discussed in just as many chapters.

In the previous chapter, you got familiar with the pandas library and with all the basic functionalities that it provides for the data analysis. You have seen that DataFrame and Series are the heart of this library. These are the material on which to perform all manipulations of data, calculations, and analysis.

2017 was a special year for deep learning. In addition to the great experimental results obtained thanks to the algorithms developed, deep learning has seen its glory in the release of many frameworks with which to develop numerous projects. Some of you will certainly already know this branch of machine learning, others you have certainly heard someone mention it. Given the great importance that deep learning is taking in data processing and analysis techniques, I found it important to add this new chapter in the second edition of this book.

So far you have seen how to apply the techniques of data analysis to Pandas dataframes containing numbers and strings. Indeed, data analysis is not limited to numbers and strings, because images and sounds can also be analyzed and classified.

In the previous chapters, the analysis of data was centered entirely on numerical and tabulated data, while in the previous one we saw how to process and analyze data in textual form. This book rightfully closes by introducing the last aspect of data analysis: image analysis.

With this chapter, you can finally get into the heart of this book: the pandas library. This fantastic Python library is a perfect tool for anyone who wants to practice data analysis using Python as a programming language.

In this book, you have seen various analysis techniques and numerous examples that worked on data in numerical or tabular form, which is easily processed through mathematical expressions and statistical techniques. But most of the data is composed of text, which responds to grammatical rules (or sometimes not even that :)) that differ from language to language. In text, the words and the meanings attributable to the words (as well as the emotions they transmit) can be a very useful source of information.

NumPy is a basic package for scientific computing with Python and especially for data analysis. In fact, this library is the basis of a large amount of mathematical and scientific Python packages, and among them, as you will see later in the book, the pandas library. This library, totally specialized for data analysis, is fully developed using the concepts introduced by NumPy. In fact, the built-in tools provided by the standard Python library could be too simple or inadequate for most of the calculations in the data analysis.

LaTeX is extensively used in Python. In this appendix there are many examples that can be useful to represent LaTeX expressions inside Python implementations. This same information can be found at the link http://matplotlib.org/users/mathtext.html .

One type of data that’s easier to find on the net is meteorological data. Many sites provide historical data on many meteorological parameters such as pressure, temperature, humidity, rain, etc. You only need to specify the location and the date to get a file with datasets of measurements collected by weather stations. These data are a source of a wide range of information. As you read in the first chapter of this book, the purpose of data analysis is to transform the raw data into information and then convert it into knowledge.

In this chapter you will see how to extend the capabilities of the graphical representation including the JavaScript D3 library within your IPython Notebooks. This library has enormous potential graphics and allows you to build graphical representations that even the matplotlib library cannot represent.

In the chain of processes that make up the data analysis, the construction phase of predictive models and their validation are done by a powerful library called scikit-learn. In this chapter you will see some examples that will illustrate the basic construction of predictive models with some different methods.

In the previous chapter you have seen how to acquire data from data sources such as databases or files. Once you have the data in DataFrame format, they are ready to be manipulated. The manipulation of the data has the purpose of preparing the data so that they can be more easily subjected to analysis. In fact, their manipulation will depend a lot on purposes of those who must carry out the analysis, and it will be performed for making more explicit the information you are looking for. Especially in preparation for the next phase, the data must be ready to the data visualization that will follow in the next chapter.

In this short but nonetheless important chapter, you will look at candlestick charts. This type of chart is based on a particular data format (OHLC, or open-high-low-close). Thus, you will need to implement a parser to read OHLC data from an external file. Moreover, another nontrivial aspect that you need to solve is how to deal with date and time data.

This appendix provides guidelines on how to use XAMPP and Aptana Studios together to create a development environment on your PC that will allow you to develop, run, and fix the examples given in the book.

This chapter begins the third part of the book, concerning the D3 library. This library has a separate section of the book dedicated to it because it differs in many aspects from the jqPlot and Highcharts libraries. In the various sections of this chapter, and as you delve deeper into the aspects of the library in the next chapters, you’ll be able to appreciate that D3 has a unique and innovative structure. First of all, it does not use jQuery, but it reproduces all the features necessary for data visualization. Whereas in the jqPlot and Highcharts libraries, chart components are already created, requiring the users only to adjust their properties via the options object, D3 has virtually the opposite approach.

Funnel charts are used to show the progressive reduction of data as they go down one level to the next. The chart consists of an inverted pyramid, or funnel, divided into different levels. Each level has its own area, which is proportional to a given percentage value. A funnel chart is similar to a pie chart in that both express a whole divided into its constituent parts. But, the funnel chart specifies levels, which succeed one another in a very precise sequence. This sequence may express a hierarchical order, the steps of a process, and so on. A pie chart cannot do this.

In the previous chapter, you have just seen how bar charts represent a certain category of data. You have also seen that starting from the same data structure, depending on your intentions you could choose one type of chart rather than another in order to accentuate particular aspects of the data. For instance, in choosing a normalized stacked bar chart, you wanted to focus on the percentage of income that each sector produces in its country.

When we need to represent data or qualitative structures graphically in order to show a relationship—to make a comparison or highlight a trend—we make use of charts. A chart is a graphic structure consisting of symbols, such as lines, in a line chart; bars, in a bar chart; or slices, in a pie chart. Charts serve as valid tools that can help us discern and understand the relationships underlying large quantities of data. It is easier for humans to read graphic representations, such as a chart, than raw numeric data. Nowadays, use of charts has become common practice in a wide variety of professional fields as well as in many other aspects of daily life. For this reason, charts have come to take on many forms, depending on the stucture of the data and the phenomenon that is being highlighted. For example, if you have data separated into different groups and want to represent the percentage of each, with respect to the total, you usually display these groups of data in a pie chart or a bar chart. In contrast, if you want to show the trend of a variable over time, a line chart is typically the best choice.

Pie charts and donut charts are an excellent way to show the breakdown of data into their constituent parts. A pie chart is a circular chart divided into sectors, or “slices,” and its main purpose is to illustrate their relative proportions: the arc length of each slice is proportional to the quantity it represents. A donut chart is very similar to a pie chart but has a hole in the center and supports the comparison of multiple series. In this chapter, you will look at both kinds of charts. The chapter concludes with a discussion of multidimentionsional pie charts.

In this chapter, you will learn about scatterplot charts. Whenever you have a set of data pairs [x, y] and you want to analyze their distribution in the xy plane, you will refer to this type of chart. Thus, you will see first how to make this type of chart using the D3 library. In the first example, you will begin reading a TSV (tab-separated values) file containing more than one series of data, and through them, you will see how to achieve a scatterplot.

This appendix shows the complete list of available plug-ins in the jqPlot distibution (see Table B-1). Not all these plug-ins have been treated in this book; for more information, please visit the jqPlot web site ( www.jqplot.com ).

Candlestick charts are widely used in the analysis of a currency over time or of price movements. This chart consists of a series of vertical bars, called candlesticks. They show the opening, closing, lowest, and highest price in a given time period (see Figure 12-1). For this reason, this kind of chart is often called an OHLC chart (when it reports open-high-low-close values) or an HLC chart (when it reports onlyhigh-low-close values).

In this chapter, you’ll exploit the capability to represent the charts within some specific containers, often referred to as widgets, such as tabbed panels and accordions, provided by some libraries, including jQuery UI. This enables you to exploit the great potential of the jQuery UI widgets to further improve the way in which your charts are represented.

You have seen how to handle real-time charts with jqPlot, and in this chapter, you will implement the same example, using the D3 library. Indeed, you will create a line chart that displays the real-time values generated from a function that simulates an external source of data. The data will be generated continuously, and therefore the line chart will vary accordingly, always showing the latest situation.

This chapter covers a type of chart that you have not yet read about: the radar chart. First you will get to know what it is, including its basic features, and how to create one using the SVG elements provided by the D3 library.

In this chapter, you will see how, using the D3 library, you can build the most commonly used type of chart: the bar chart. As a first example, you will start from a simple bar chart to practice the implementation of all the components using scalar vector graphic (SVG) elements.

Once you have dealt with all the graphical aspects of a chart, it is time to analyze input data in more detail. In the previous chapters, you started assigning the values of input data to arrays. Then you made use of tabulated data contained within CSV or TSV files. This step has allowed you to be able to handle most of the data for the visualization, but in reality there are more complex data structures, in a hierarchical form, such as trees.

Following in the footsteps of the jqPlot framework, a new JavaScript library is catching up, called Highcharts. This is a commercial product and was completed in late 2009 by the Norwegian company Highsoft Solutions AS. As this book is being written, this new library is at version 3.0.1 and is increasingly being offered in the market as a new solution for the professional representation of charts.

In the previous chapter, you observed the most basic use of jqPlot, in which a series of data serves to plot a line, with no need for any additional options. You saw that in order to create the most basic type of chart, a line chart, , you do not need to include plug-ins.