Search

Identifying frequencies and patterns of missing values and preliminary cleaning of missing data are common and fundamental parts of statistical data management. I offer a new command, missings, as a replacement for, and extension of, my previous commands nmissing (Cox 1999, Stata Technical Bulletin 49: 7–8; 2001a, Stata Technical Bulletin 60: 2–3; 2003, Stata Journal 3: 449; 2005, Stata Journal 5: 607) and dropmiss (Cox 2001b, Stata Technical Bulletin 60: 7–8; 2008, Stata Journal 8: 594). I also make comparisons with similar commands.

The problem of handling Roman numerals in Stata is used to illustrate issues arising in the handling of classification codes in character string form and their numeric equivalents. The solutions include Stata programs and Mata functions for conversion from numeric to string and from string to numeric. Defining acceptable input and trapping and flagging incorrect or unmanageable inputs are key concerns in good practice. Regular expressions are especially valuable for this problem.

any problems in data management center on relating values to values in other observations, either within a dataset as a whole or within groups such as panels. This column reviews some basic Stata techniques helpful for such tasks, including the use of subscripts, summarize, by:, sum(), cond(), and egen. Several techniques exploit the fact that logical expressions yield 1 when true and 0 when false. Dividing by zero to yield missings is revealed as a surprisingly valuable device.

A standard task in data management is producing a list of variable names showing which variables have specific properties, such as being of string type, or having value labels attached, or having a date format. A first-principles strategy is to loop over variables, checking one by one which have the property or properties concerned. I discuss this strategy in detail with a variety of examples. A canned alternative is offered by the official command ds or by a new command, findname, published formally with this column.

Distinct observations are those different with respect to one or more variables, considered either individually or jointly. Distinctness is thus a key aspect of the similarity or difference of observations. It is sometimes confounded with uniqueness. Counting the number of distinct observations may be required at any point from initial data cleaning or checking to subsequent statistical analysis. We review how far existing commands in official Stata offer solutions to this issue, and we show how to answer questions about distinct observations from first principles by using the by prefix and the egen command. The new distinct command is offered as a convenience tool.

Counting panels, and more generally groups, is sometimes possible in Stata through a reduction command (e.g., collapse, contract, statsby) that produces a smaller dataset or through a tabulation command. Yet there are also many problems, especially with irregular sets of observations for varying times, that do not yield easily to this approach. This column focuses on techniques for answering such questions while maintaining the same data structure. Especially useful are the Stata commands by: and egen and indicator variables constructed for the purpose. With by: we often exploit the fact that subscripts are defined within group, not within dataset. egen functions are often used to produce group-level statistics. Tagging each group just once ensures that summaries, including counts, are of groups, not individual observations.

Spells in time series (and more generally in any kind of one-dimensional series) may be defined as sequences of observations that are homogeneous in some sense. For example, a categorical variable may remain in the same state, or values of a measured variable may satisfy the same true–false condition. Devices for working with spells in Stata include marking the start of each spell with indicator variables and tagging spells with integer codes. Panel data are easy to handle with the by: prefix. Some kinds of spell identification require two passes through the data, as when only spells of some minimum length are of interest or short gaps are tolerable within spells. Many questions concerning spells are easy to answer given careful use of by: and appropriate sort order, selection of just 1 observation from each panel or spell, and appreciation of the many functions written for egen. Gaps before, between, and after spells can also be important, and I suggest a convention for handling them.

The count command has one simple role, to count observations in general or that satisfy some condition(s). This task can be useful when some larger problem pivots on counting, especially if count is used with a loop over observations or variables. I use various problems, mostly of data management, as examples. I also make comparisons with the use of _N, summarize, and egen for the same or similar problems.