Search

Thesis (Ph. D.)--University of Rochester. Department of Brain & Cognitive Sciences and Computer Science, 2020., Perception is often characterized as an inference process in which the brain unconsciously reasons about causes in the world based on their sensory effects. According to this “Bayesian Brain” hypothesis, perception is an expression of beliefs about latent variables in a statistical model of the world, and sensory neurons implement some form of approximate inference algorithm. The initial chapters of this dissertation summarize and link existing neural models of inference and suggest new ways to categorize and interpret them. Subsequent chapters derive predictions for both behavioral and neural consequences of approximate inference in the special case of visual categorization tasks. In particular, when approximate prior expectations impinge on sensory representations (a feature of optimal inference), a behavioral consequence is the emergence of temporal biases, and a neural consequence is task-dependent noise covariability. The predictions for behavioral biases are verified in human subjects using a novel pair of visual psychophysics tasks. The predictions for neurons are explored using multi-electrode recordings from primary visual cortex of macaque monkeys, but with inconclusive results. We conclude with discussion of the nature of inference in the brain more broadly and sketch alternative perspectives on the results presented here.

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2013., Image segmentation is a fundamental problem in computer vision. It can be defined as the process of labeling each pixel in the image such that pixels with the same label share some common pre-defined attribute. Thus, the image is broken into groups of pixels that are easier to analyze for the task at hand. In this thesis, we focus on the problem of object segmentation where the pre-defined attributes are objects. Markov random fields (MRF) are undirected graphical models with a probability distribution attached to it. The nodes in the graph are random variables and the interaction between these random variables follow the Markov property. MRFs have generative properties. The discriminative version of MRFs called conditional random fields (CRF) have a conditional distribution attached to the graphs. Both MRFs and CRFs are useful tools for modeling image segmentation problems. In this thesis, we explore the use of conditional random fields and hybrid graphical models consisting of generative and discriminative parts for image segmentation. In the first part of the thesis, we consider the problem of multi-class segmentation of the abdominal organs in medical images. The graphical models are 3D graphs and we use different features to improve segmentation on these medical images. In the second part of the thesis, we segment people from videos. The method makes no assumption about camera motion or location, scale or motion patterns of the person. In the third part of the thesis, we focus on the problem of inference in graphs. Inference is the process of prediction of labels of the graph and is also needed to learn the parameters of the model. We discuss a new method to do inference in graphs that improves the existing dual decomposition method.

Thesis (Ph. D.)--University of Rochester. Department of Computer Science, 2020., Building systems for understanding creative language is a notoriously difficult problem in AI. The task not only demands machine learning algorithms that incorporate world knowledge, reasoning, and perception but also requires substantial effort in annotating training data. Furthermore, it is difficult to compare models for creative language generation due to a lack of evaluation metrics and unified, shared tasks. In this thesis, we explore the generation and detection of creative language through two areas where creativity is involved: (i) general and topical (news based) humor generation; and (ii) creative slang in politically-oriented online speech. In the first area, we develop datasets and algorithms for humor generation (i) in Mad Libs®, where the goal is to fill in the blanks in a short story to make it funny, and (ii) in news headlines, where our objective is to achieve humor by applying small edits to original headlines. Next, we significantly reduce the cost of humor data collection by designing a competitive game that incentivizes players to generate high-quality humor. Then, we host a shared task on humorous headline detection, attracting over 50 participating research teams, thereby getting the research community more involved in pursuing computational humor. In the second area, we investigate the generation and usage of creative slang in online political discourse among reader communities within and across major news sources. This domain provides an interesting intersection of humor and hate speech because many slang terms are both hateful and intended to be humorous, as we shall see in “political portmanteau”. Finally, as a next step for generating long-form creative text, we develop a deep-learning based text generation framework that (i) augments the source text using various retrieved information, (ii) generates several candidate outputs, and (iii) ranks these candidates to select the best output. Such a framework can be useful for high precision c

Thesis (Ph. D.)--University of Rochester. Department of Computer Science, 2021., Mental health can be conceptualized as a complex, emergent phenomenon resulting from the interaction between biological, psychological, social, environmental, and other contextual factors. Traditionally, care providers assess one’s mental health when an individual consults a medical or behavioral health professional. Different events in our quotidian life trigger various emotional and cognitive responses. Since much of our lives is now lived online, activities on digital platforms may provide valuable insights and understandings of an individual’s mental states. Recently, social computer scientists have attempted to model mental health phenomena based on data derived from social media activities. We propose a framework to incorporate online activities from an alternate source, and along with longitudinal ground truth data about an individual for detecting various mental health problems. We hypothesize that both the routine experiences of daily living, as well as specific, significant life events, invoke particular patterns that can be detected in online activities. A close monitoring of these experiences and events, along with search activity and other online behavior, can inform systems to provide early diagnosis and detection of related mental health conditions. In this dissertation, we study daily online behaviors from Google search and YouTube platforms to build predictive models for different mental health conditions. First, we build a lightweight cloud-based framework capable of combining groundtruth mental health phenomena and daily online activities to enable researchers build models for different mental health issues. Using this platform, ran a study for two months where we collected Google search engine engagement logs from a college population and showed that search logs contained strong signals that could identify individuals with current low self-esteem. Next, we use Google search history to demonstrate that search engine logs contain significant signal

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2016, Until recently, the areas of perception, language and knowledge reasoning have been studied primarily in isolation. Activity recognition has been no exception; over the past years we have seen a variety of different models with the goal of accurately representing and inferring activities from sensor data. However, recent advances in sensor technology, computational capabilities and data storage have facilitated the collection of large-scale, multimodal datasets. We look at some of our efforts in collecting multimodal datasets in heavily instrumented spaces, which we call “smart task spaces.” Modes such as language and commonsense knowledge also have great potential for augmenting recognition in situations where sensor data is lacking. In particular, language can be used as a form of indirect labeling for activities when no labeled data exists. We present a system where new actions can be learned from text descriptions and multiple instances of video without explicit labeled data. We also present a general framework for complex event recognition that is well-suited for integrating information that varies in detail and granularity. We introduce the current state of the art in fine-grained activity recognition, and introduce a framework that integrates multiple modes of input into a single system. Our system uses Markov logic which encourages the combined use of probabilistic perceptual features with commonsense knowledge, represented using logic statements. Through our framework, we show that are able to effectively integrate natural language and commonsense knowledge in our activity models.

Thesis (Ph. D.)--University of Rochester. Department of Computer Science, 2016., Collaboration is crucial to everything from product development in the workplace to research design in academia, yet there is no consensus on best practice when it comes to teaching collaborative skills. We explore one promising option: collaborative scaffolding in educational video games. Through this methodology, we can impart collaborative skills alongside topics like math and science in a medium already proven to be beneficial. Before approaching our study design, we had to reconcile the large and often contradictory volume of literature on collaboration’s definition and elements. From our review of over 25 studies and surveys, we developed a multilevel taxonomy with four main components: group quality, coordination, communication, and support. Researchers can utilize the taxonomy to isolate elements of collaboration relevant to their interests, and educators can use it to help identify collaborative skills that students lack. It can even act as a design guide for educational game developers. For our own work, we chose to focus on scaffolding two subcomponents of support: direct and indirect helping. We ran mixed-factorial trials with over 70 local middle school students to observe behaviors and attitudes before, during, and after collaborative video game play. To gather data, facilitate play, and present scaffolding to students, we made several modifications to an existing educational physics game, including the creation of a scaffolding architecture and the implementation of a data collection infrastructure. Researchers and game developers can build on our designs and learn from the challenges we encountered to help develop and evaluate future scaffolding systems. During trials, we recorded audio, made direct observations, and logged in-game events. From our data, we found that collaborative scaffolding had no impact on helping behaviors, but, more interestingly, had a negative impact on motivation and attitudes toward collaborati

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2014., In this dissertation we propose extensions to Markov logic networks that can improve inference and learning by exploiting deterministic constraints, expert knowledge or se- quential/temporal structure. The dissertation consists of the description of four different approaches. First, we discuss how we can speed up inference and hence learning in Markov logic networks when deterministic constraints are present, using constraint programming techniques. Second, we propose a new probabilistic model (slice normal- ized dynamic Markov logic networks) for reasoning in sequential (temporal) domains, which is more suitable for particle filtering than dynamic conditional random fields. In the third part of the dissertation, we propose a framework for parameter learning in Markov logic networks that can combine the possibly inconsistent knowledge of a domain expert with the information obtained from training data according to Bayesian statistics. Finally, we propose two possible ways of extending propositional Markov logic with modal operators to get a framework that allows reasoning about beliefs and knowledge of agents. Most of our results are general enough to hold also for random fields, and exponential families of probability distributions, due to their close relationship to Markov logic networks.

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2012., Until recently, complex phenomena---such as human behavior and disease epidemics---have been modeled primarily at an aggregate level. Detailed studies have been limited to small domains encompassing only a few subjects, as scaling the methods involved poses considerable challenges in terms of cost, human effort required, computational bottlenecks, and data sources available. With the surge of online social media and sensor networks, the abundance of interesting and publicly accessible data is beginning to increase. However, we also need the ability to reason about it efficiently. The underlying theme of this thesis is the unification and data mining of diverse, noisy, and incomplete sensory data over large numbers of individuals. We show that the mined patterns can be leveraged in predictive models of human behavior and other phenomena at a large scale. We find that raw sensory data linked with the content of users' online communication, the explicit as well as the implicit online social interactions, and interpersonal relationships are rich information sources upon which strong machine learning models can be built. Example domains where such models apply include understanding human activities, predicting people's location and social ties from their online behavior, and predicting the emergence of global epidemics from day-to-day interpersonal interactions.

Thesis (Ph. D.)--University of Rochester. Department of Computer Science, 2017., Relational theories are a powerful representational tool for expressing problems in a vast array of domains from planning and circuit verification to link prediction and social networks. Many systems, such as Markov Logic Networks, theorem provers, model counters, verifiers, and more, use relational theories to express their problem domains. Early algorithms for inference over relational theories relied on viewing the problem at the level of propositional atoms. However, a great deal of recent research has been on lifting these algorithms to exploit the structure that exists in these problems at higher levels. In this work, I give techniques for exploiting the symmetrical structure that is common in problems expressed as relational theories. I give formal definitions of classes of symmetries, algorithms for discovering and exploiting them, and empirical results that speak to the efficacy of these methods.

Thesis (Ph. D.)--University of Rochester. Dept. of Computer Science, 2014., The considerable impact that intelligent prompting systems could have on the lives of people with cognitive disabilities is gaining wide interest. Such intelligent systems would be able to support a variety of cognitive tasks by quickly recognizing different situations, performing autonomous decision making, and taking appropriate actions. While an intelligent prompting system offers significant benefits, it also proves to be an extremely challenging design paradigm. A successful prompting system that meets the needs of practical use requires the ability to trade off conflicting objectives, make decisions in the face of all kinds of uncertainties, and support a wide range of prompting behaviors that can adapt to diverse user needs and preferences. To handle all these complexities in the prompting domain, researchers have been studying the use of POMDPs (partially observable Markov decision processes) to formulate the problem in a well-formed theoretic framework. However, solving a POMDP problem in a large domain is quite challenging and even intractable. In this dissertation, we are addressing this challenge by proposing a hierarchical POMDP model that runs on a set of individual option-based MDPs (Markov decision processes). Options are temporal extended actions that can facilitate learning by exploiting the problem structure and supporting the specification of a variety of complex prompting routines. This approach is then combined with a dual control heuristic to convert a solution of the underlying completely observable control algorithm to the POMDP’s. To further enhance the system’s performance, we have developed several novel techniques that work in ways complementary to each other. These include (i) adaptive options that optimize the timing of the prompt and quickly adapt to different user behaviors, the performance of which is shown based on both simulation and human subject experiments; (ii) a selective-inquiry based dual control algorith