Search

In 2019, COVID-19 quickly spread across the world, infecting billions of people and disrupting the normal lives of citizens in every country. Governments, organizations, and research institutions all over the world are dedicating vast resources to research effective strategies to fight this rapidly propagating virus. With virus testing, most countries publish the number of confirmed cases, dead cases, recovered cases, and locations routinely through various channels and forms. This important data source has enabled researchers worldwide to perform different COVID-19 scientific studies, such as modeling this virus’s spreading patterns, developing prevention strategies, and studying the impact of COVID-19 on other aspects of society. However, one major challenge is that there is no standardized, updated, and high-quality data product that covers COVID-19 cases data internationally. This is because different countries may publish their data in unique channels, formats, and time intervals, which hinders researchers from fetching necessary COVID-19 datasets effectively, especially for fine-scale studies. Although existing solutions such as John’s Hopkins COVID-19 Dashboard and 1point3acres COVID-19 tracker are widely used, it is difficult for users to access their original dataset and customize those data to meet specific requirements in categories, data structure, and data source selection. To address this challenge, we developed a toolset using cloud-based web scraping to extract, refine, unify, and store COVID-19 cases data at multiple scales for all available countries around the world automatically. The toolset then publishes the data for public access in an effective manner, which could offer users a real time COVID-19 dynamic dataset with a global view. Two case studies are presented about how to utilize the datasets. This toolset can also be easily extended to fulfill other purposes with its open-source nature.

As common vulnerabilites and exposures are detected, they are also discussed in various social platforms. The problem is that only a few of the posts made about them, are getting enough attention. This leads to an unawareness of potential and critical threats against systems. It is therefore important to look for patterns that make certain vulnerabilites more or less discussed. To do so, a framework was made for collecting discussions around cybersecurity and more specific vulnerabilites/exposures called CVE from Reddit. In addition, some of the desired data was collected from Twitter. Thereafter, the sentiments of the collected posts were calculated to see patterns between popular subreddits and the attitude shown in them. This was done with three methods: Flair, TextBlob and Vader. The results showed for instance that general discussions about information security were considered to be more positive than discussions of common vulnerabilites and exposures. Another result showed that the spread of CVEs that have a partial impact, are higher in Reddit, and is increasing almost exponentially. CVSS scores showed that a CVE with a CVSS score of around 7 is more likely to appear. Many CVEs in Reddit was also discussed before and after they were disclosed. The implication of this work might be that more and more people might use Reddit to discuss specific types of CVEs in a suitable subreddit, as well as being aware of common vulnerabilites and exposures, in order to prevent future threats.

U sklopu ovog rada otkriven je unutarnji način rada nekolicine sustava za komentiranje na webu te je na osnovu tih saznanja implementiran sustav za dohvat članaka i njihovih komentara. Krajnji cilj rada je stvoriti web scraping sustav s elementima web pauka koji ispunjava svoju svrhu, a da je lako nadogradiv, robusan te lak za održavati. Odnosno, potrebno je osigurati da je implementacija provedena u skladu s dobrim praksama programskog inženjerstva. Kako bi se moglo reći da sustav ispunjava svoju svrhu, mora moći samostalno preuzimati članke i komentare u odnosu na definirana ograničenja te održavati rad bez obzira na vanjske promjene sustava komentiranja na webu. U radu je prikazan postupak reverznog inženjerstva sustava komentiranja, arhitektura razvijanog sustava te implementacijski detalji bitni za razumijevanje najvećih problema koje je ovaj rad riješio. This paper serves to explain the inner-workings of comment systems used on the web, and based upon that theoretical background goes through the process of building a system which retrieves articles and associated comments from news sites. The end goal of this paper is to build an easily-upgradeable, robust and easy-to-maintain web scraping and crawling system which fulfills its purpose. In other words, good practices must be enforced during the whole implementation life-cycle. The criteria which tells us if the system has fulfilled it's purpose is when it can automatically download news articles and comments and keep working regardless of outside changes of comment systems it relies on. The paper contains the reverse engineering process required to build a base of knowledge of comment systems, the architecture of the built system, and most importantly the implementation details which are needed to fully understand the problems this paper solved.

El presente proyecto propone el desarrollo de una arquitectura de Big data que permita evaluar la satisfacción de los usuarios del sector turístico del departamento de Boyacá Colombia mediante el análisis de sentimientos multimodales, a través de una serie de parámetros en la extracción de datos a herramientas como Booking, TripAdvisor y twitter, facilitando el análisis de los sentimiento del usuario por medio de variables de polaridad, esto con tal de conocer la opinión de los usuarios frente al sector turístico del departamento de Boyacá. El proyecto atiende a tres puntos fundamentales dentro del desarrollo de la arquitectura: extracción de datos, análisis de datos y visualización. Dichos puntos establecen un punto de partida dentro del desarrollo de la arquitectura; puesto que los subpuntos de los tres parámetros iniciales buscan el buen funcionamiento y desarrollo de la arquitectura. La extracción comprende el uso de la herramienta webs scraper encargada de la extracción de datos de las herramientas ya mencionadas para posterior uso en el segundo parámetro, el análisis; en este punto se pretende que los datos obtenidos de la extracción se sometan al proceso de polaridad que permita determinar las emociones y opiniones de los usuarios frente a sus visitas; esto para finalmente en el proceso de visualización los datos obtenidos como resultados puedan ser plasmados en gráficas que permiten su entendimiento y comprensión. Finalmente, el proyecto presenta una serie de conclusiones frente a los resultados obtenidos con la arquitectura. La arquitectura está basada en el proyecto de investigación Doctoral de la Pontificia Universidad Javeriana “Modelo multimodal para el análisis de sentimientos en un dominio turístico”. This project proposes the development of a Big data architecture that allows evaluating the satisfaction of users of the tourism sector in the department of Boyacá Colombia through the analysis of multimodal sentiments, through a series of parameters in the extraction of data to tools such as Booking, TripAdvisor and twitter, facilitating the analysis of user sentiment through polarity variables, this in order to know the opinion of users regarding the tourism sector of the department of Boyacá. The project addresses three fundamental points within the development of the architecture: data extraction, data analysis and visualization. These points establish a starting point within the development of architecture; since the sub-points of the three initial parameters seek the proper functioning and development of the architecture. The extraction includes the use of the web scraper tool in charge of extracting data from the aforementioned tools for later use in the second parameter, the analysis; At this point, it is intended that the data obtained from the extraction be subjected to the polarity process that allows determining the emotions and opinions of the users regarding their visits; this for finally in the visualization process the data obtained as results can be captured in graphs that allow their understanding and comprehension. Finally, the project presents a series of conclusions regarding the results obtained with architecture based on its objective. Ingeniero Informático Pregrado

New consent management platforms (CMPs) have been introduced to the web to conform with the EU's General Data Protection Regulation, particularly its requirements for consent when companies collect and process users' personal data. This work analyses how the most prevalent CMP designs affect people's consent choices. We scraped the designs of the five most popular CMPs on the top 10,000 websites in the UK (n=680). We found that dark patterns and implied consent are ubiquitous; only 11.8% meet the minimal requirements that we set based on European law. Second, we conducted a field experiment with 40 participants to investigate how the eight most common designs affect consent choices. We found that notification style (banner or barrier) has no effect; removing the opt-out button from the first page increases consent by 22--23 percentage points; and providing more granular controls on the first page decreases consent by 8--20 percentage points. This study provides an empirical basis for the necessary regulatory action to enforce the GDPR, in particular the possibility of focusing on the centralised, third-party CMP services as an effective way to increase compliance., 13 pages, 3 figures. To appear in the Proceedings of CHI '20 CHI Conference on Human Factors in Computing Systems, April 25--30, 2020, Honolulu, HI, USA

Donada la gran quantitat de demanda d'informació que els centres educatius han d'assumir cada cop que comença un nou curs acadèmic, sorgeix la necessitat de facilitar la feina a les persones encarregades d'atendre aquesta demanda. Els chatbots, també coneguts com assistents virtuals, són capaços de simular mantenir una conversa amb un ésser humà donant respostes automatitzades als dubtes plantejats. Aquest treball de fi de grau té com objectiu la realització d'una prova de concepte per generar un prototip d'assistent virtual que pugi atendre la demanda d'informació sobre els estudis oferts per la UAB, i proporcionar de manera automatitzada la informació en funció de les dades indicades pels usuaris. Per tal d'assolir aquest objectiu, en primer lloc s'ha desenvolupat una aplicació per obtenir les dades del web de la UAB fent servir tècniques de scraping i mineria de dades. Després, per tal de donar persistència a les dades tractades, s'ha fet servir una base de dades MongoDB de tipus NoSQL. Finalment s'ha desenvolupat un chatbot amb Microsoft Bot Framework SDK per a Node.js i un algoritme d'aprenentatge profund que permet adaptar els resultats en funció de les dades proporcionades pels usuaris. Dada la gran demanda de información que deben asumir las escuelas cada vez que se inicia un nuevo curso académico, surge la necesidad de facilitar la labor de los responsables de atender esa demanda. Los chatbots, también conocidos como asistentes virtuales, son capaces de simular una conversación con un ser humano dando respuestas automatizadas a las preguntas planteadas. El objetivo de este proyecto de fin de carrera es realizar una prueba de concepto para generar un prototipo de asistente virtual para atender la demanda de información sobre los estudios que ofrece la UAB, y proporcionar la información de forma automatizada según los datos indicados por los usuarios. Para conseguir este objetivo, en primer lugar se ha desarrollado una aplicación para obtener los datos de la web de la UAB mediante técnicas de scraping y minería de datos. Luego, para dar persistencia a los datos tratados, se ha utilizado una base de datos MongoDB de tipo NoSQL. Por último, se ha desarrollado un chatbot con Microsoft Bot Framework SDK para Node.js y un algoritmo de aprendizaje profundo que permite adaptar los resultados en función de los datos proporcionados por los usuarios. Given the large amount of demand for information that schools must assume each time a new academic year begins, the need to facilitate the work of those responsible for meeting this demand arises. Chatbots, also know as virtual assistants, are able to simulate a conversation with a human being by giving automated answers to the questions raised. The aim of this end-of-degree project is to carry out a proof of concept to generate a prototype of a virtual assistant to meet the demand for information on the studies offered by the UAB, and to provide the information in an automated way according to the data indicated by the users. To achieve this objective, firstly an application has been developed to obtain the data from the UAB website using scraping and data mining techniques. Then, in order to give persistence to the treated data, a MongoDB database of type NoSQL has been used. Finally, a chatbot has been developed with Microsoft Bot Framework SDK for Node.js and a deep learning algorithm that allows to adapt the results according to the data provided by the users.

Представлена робота обсягом 87 сторінок містить 26 ілюстрацій, 12 таблиць та 17 джерел за переліком посилань. Актуальність роботи зумовлюється тим, що на даний момент при розробці багатьох інтернет-ресурсів та веб-сайтів веб-розробники та замовники не завжди задумуються про правильне використання своїх ресурсів та захист даних, що знаходяться у відкритому доступі. Перш за все потрібно мінімізувати автоматизований доступ до таких даних та ефективно використовувати наявні ресурси для захисту контенту, особистих даних, що знаходяться у відкритому вигляді, чи інформації, що є об’єктом захисту авторського права. Метою роботи є підвищення рівня захищеності інформації, що розміщена на інтернет-ресурсах різного характеру, шляхом створення ефективної системи, яка буде виявляти та автоматично блокувати виявлене програмне забезпечення для автоматизованого збору інформації з веб-сайтів. Для досягнення даної мети були поставлені наступні завдання: • аналіз роботи веб-скраперів; • аналіз можливих загроз для власників веб-сайтів при відсутності захисту від веб-скраперів; • визначення найефективніших методів захисту; • вдосконалення існуючих методів захисту; • створення власної системи виявлення веб-скраперів на основі вдосконаленого існуючого методу виявлення; • побудова програмної моделі веб-сайту з використанням запропонованої системи виявлення веб-скраперів; • спроба реалізації атаки на побудовану модель веб-сайту з використанням створеної системи; • оцінка захищеної моделі та аналіз результатів. Об’єктами дослідження є веб-сайти та інтернет-ресурси, вразливі до досліджуваної атаки, та можливі методи захисту від неї. Предметом дослідження є виявлення веб-скраперів за допомогою пасток, розміщених на веб-сайті. Методами дослідження було обрано: опрацювання літератури за даною темою, аналіз причин виникнення даної атаки, аналіз методів захисту, проведення експериментів та порівняння деяких методів. Наукова новизна. В ході проведення дослідження було вперше побудовано систему виявлення веб-скраперів, яка є ефективною при виявленні програмного забезпечення під час його запуску лише в певних директоріях веб-сайту. При цьому система не потребує створення додаткових файлів, які будуть завантажувати сервер. Практичне значення полягає в тому, що результати роботи можуть застосовуватись при створенні інтернет-ресурсів для мінімізації можливості крадіжки контенту, особистих даних, що знаходяться у відкритому вигляді, чи інформації, що є об’єктом захисту авторського права. Результати дослідження, які є основою магістерської дисертації, були опубліковані в статті «Удосконалений метод виявлення веб-скраперів з використанням пасток». Стаття опублікована в збірнику, що містить матеріали XVII Всеукраїнської науково-практичної конференції студентiв, аспiрантiв та молодих вчених «Теоретичні і прикладні проблеми фізики, математики та інформатики». This work of 87 pages contains 26 illustrations, 12 tables and 17 literature references. The relevance of the work is due to the fact that at the moment many web developers and customers do not always think about the proper use of their resources and protection of data that is in the open access, when they develop Internet resources and websites. First of all, it is necessary to minimize automated access to such data and to make effective use of available resources to protect content, personal data that is in the public access, or information that is the subject of copyright protection. The aim of the work is to increase the security of information placed on various resources of the Internet by creating an effective system that will detect and automatically block the detected software for automated information gathering from websites. To achieve this goal, the following tasks were set: • analysis of the work of web scrapers; • analysis of possible threats to website owners in the absence of protection from web scrapers; • the most effective protection methods identifying; • existing protection methods improvement; • designing of own web scrapers detection system, based on an improved existing detection method; • creating a program model of a website, using the proposed web scrapers detection system; • an attempt to implement an attack on a built-in website model, using the created system; • evaluation of the protected model and analysis of the results. The objects of the study are websites and Internet resources, vulnerable to the attack being investigated and possible protection methods against it. The subject of the study is the detection of web scrapers, using honeypots, located on the website. The research methods were chosen: studying the literature on this topic, analyzing the causes of this attack, analyzing protection methods, conducting experiments and comparing some methods. Scientific novelty. During the study, a web scrapers detection system, which is effective in detecting software, when it is launched only in certain directories of the website, was built for the first time. The system does not require additional files that will load the server. The practical value is that work results can be applied when creating Internet resources to minimize the possibility of theft of content, personal data that is in the public access, or information that is the subject of copyright protection. The research findings, which are the basis of the master's thesis, were published in the article "Improved web scrapers detecting method using honeypots". The article is published in the collection containing materials of the XVII All-Ukrainian Scientific and Practical Conference of Students, Aspirants and Young Scientists "Theoretical and Applied Problems of Physics, Mathematics and Computer Science".

The modern era is essentially characterized by information technology. Notwithstanding data availability, there is the urge to design systems capable of providing information quickly and effortlessly for the user. This paper presents the development process of creating a content aggregator application that solves data overloading and location issues existing in the Autonomous University of Barcelona web services. This is achieved by implementing a serverless cloud function that scrapes the university web service, parses the results, and stores them to a database to satisfy the application requests. L'època moderna és essencialment caracteritzada per les tecnologies de la informació. Malgrat l'alt nivell de disponibilitat de dades, existeix la necessitat de dissenyar sistemes capaços de proporcionar informació de manera ràpida i amb el mínim nivell d'esforç per part de l'usuari per tal de facilitar el procés de cerca. Aquest article presenta el procés de desenvolupament per crear una aplicació que actua com agregador de contingut que soluciona les dificultats d'ubicació i sobrecàrrega d'informació que existeixen en el web de la Universitat Autònoma de Barcelona. Això s'aconsegueix mitjançant la implementació d'una funció al núvol sense servidor que col·lecciona dades del web de la universitat, analitza i modifica els resultats, i els guarda en una base de dades per tal de satisfer les peticions de l'aplicació. La era moderna está esencialmente caracterizada por las tecnologías de la información. A pesar de la alta disponibilidad de los datos, existe la necesidad de diseñar sistemas capaces de proporcionar la información de forma rápida y con el mínimo nivel de esfuerzo por parte del usuario para facilitar el proceso de búsqueda. Este artículo presenta el proceso de desarrollo para crear una aplicación que actúa como un agregador de información que soluciona las dificultades de ubicación y sobrecarga de información que existen en la web de la Universidad Autonóma de Barcelona. Esto se consigue mediante la implementación de una función en la nube sin servidor que colecciona datos de la web, analiza y modifica los resultados, y los guarda en una base de datos para satisfacer las peticiones de la aplicación.

This thesis describes a web browser plugin called NewsEnhancer, which has the goal to make it more efficient to read the news. The plugin receives data from a server application, which continuously aggregates meta information from articles on various news sites. This meta information is then presented to the user through the plugin. This can help the user to decide whether or not to click a headline link, which then can make the reading process more efficient. The plugin is also community based, and enables users to report headlines, as clickbait or fake news, and submit summaries of articles. The reported headlines will be flagged for other users, as to signal that this has been reported as a clickbait or fake news. The submitted summary will appear alongside the other meta information. Masteroppgave i informatikk INF399

Web forums are a popular way of exchanging information and discussing various topics. These websites usually have a special structure, divided into boards, threads and posts. Although the structure might be consistent across forums, the layout of each forum is different. The way a web forum presents the user posts is also very different from how a news website presents a single piece of information. All of this makes the navigation and extraction of text a hard task for web scrapers. The focus of this thesis is the development of a web scraper specialized in forums. Three different methods for text extraction are implemented and tested before choosing the most appropriate method for the task. The methods are Word Count, Text-Detection Framework and Text-to-Tag Ratio. The handling of link duplicates is also considered and solved by implementing a multi-layer bloom filter. The thesis is conducted applying a qualitative methodology. The results indicate that the Text-to-Tag Ratio has the best overall performance and gives the most desirable result in web forums. Thus, this was the selected methods to keep on the final version of the web scraper. Webforum är ett populärt sätt att utbyta information och diskutera olika ämnen. Dessa webbplatser har vanligtvis en särskild struktur, uppdelad i startsida, trådar och inlägg. Även om strukturen kan vara konsekvent bland olika forum är layouten av varje forum annorlunda. Det sätt på vilket ett webbforum presenterar användarinläggen är också väldigt annorlunda än hur en nyhet webbplats presenterar en enda informationsinlägg. Allt detta gör navigering och extrahering av text en svår uppgift för webbskrapor. Fokuset av detta examensarbete är utvecklingen av en webbskrapa specialiserad på forum. Tre olika metoder för textutvinning implementeras och testas innan man väljer den lämpligaste metoden för uppgiften. Metoderna är Word Count, Text Detection Framework och Text-to-Tag Ratio. Hanteringen av länk dubbleringar noga övervägd och löses genom att implementera ett flerlagers bloom filter. Examensarbetet genomförs med tillämpning av en kvalitativ metodik. Resultaten indikerar att Text-to-Tag Ratio har den bästa övergripande prestandan och ger det mest önskvärda resultatet i webbforum. Således var detta den valda metoden att behålla i den slutliga versionen av webbskrapan.

In Zeiten, in welchen Informationen öffentlich im Internet verfügbar sind und Computer in der Lage sind große Datenmengen zu verarbeiten, ist es sinnvoll News und andere Textquellen zu nutzen um unser Verständnis von Aktienreaktionen zu verbessern. Bei den hier untersuchten Reaktionen handelt es sich um Volatilität, Handelsvolumen und Rendite. Diese Arbeit schlägt ein mögliches Konzept vor, um Informationen mit Bezug zu Aktien, welche aus verschiedenen Quellen wie zum Beispiel News und Investionsartikel, zu extrahieren und zu verarbeiten. Des Weiteren wird die verfolgte Meinung in den Texten anhand eines Finanzlexikons in numerische Werte projeziert um im folgenden mathematische Methoden wie zum Beispiel Panel Regression zu verwenden. Anhand der abgeleiteten Meinungen werden die folgenden Forschungsfragen beantwortet: (i) Hat das verwendete Meinungsmaß einen Einfluss auf die Ergebnisse? (ii) Reagieren die Aktien asymmetrisch, gegeben der Meinungswerte? (iii) Gibt es Belege für die Gültigkeit der Uncertain Information Hypothesis? In an era where information is publicly available on the internet and computers are able to handle large amounts of data, it is only logical to utilize news and other text sources to improve our understanding of stock reactions. Namely, these stock reactions are volatility, trading volume and return. This thesis proposes a guide line on how to extract and process stock related information from different sources such as news and investment articles. Furthermore, sentiment is projected by using a finance specific lexicon that classifies words in either positive or negative. Hence, the sentiment is numerisized and it is possible to utilize it in mathematical models such as panel regression. The derived sentiment is then used to answer the following research questions: (i) Does the nature of the derived sentiment measure play an important role? (ii) Is there an asymmetric response given the sentiment values? (iii) Is there evidence for the validity of the uncertain information hypothesis?

V tem diplomskem delu predstavimo rešitev za strganje besedila iz spletnih mest s poizvedbami AJAX, ki spreminjajo prikazano vsebino spletnega mesta, kljub istemu spletnemu naslovu. Najprej raziščemo in preučimo tehnologije, ki se navezujejo na temo diplomskega dela. Navedemo možne ovire pri preiskovanju spleta in predstavimo nabor tehnologij za razvoj programske rešitve. Na podlagi informacij iz strukture DOM izdelamo aplikacijo za strganje besedila, v kateri implementiramo rešitve za navedene ovire pri preiskovanju spletnih mest. Delovanje implementirane aplikacije je podrobno predstavljeno še s primeri uporabe. In this diploma work we present a solution for web site text scraping using AJAX queries which change content displayed, in spite of the same web adress. First we study technologies related to our work. We list possible obstacles in web crawling procedures and present a set of technologies used to develop a software solution. Based on information gathered from DOM structure, we develop an application for web site text scraping where solutions of the listed obstacles are implemented. The implemented application is also presented in detail by demonstration examples.

L'objectiu d'aquest TFG és implementar un sistema adaptat a Jutge.org que detecti plagi entre les solucions dels estudiants, proporcionant un format de visualització gràfica. Addicionalment, es desenvoluparà un web scraper que recorrerà la xarxa en busca de solucions a problemes de Jutge.org. The aim of this project is to implement a system adapted to Jutge.org that detects plagiarism among the solutions of the students, providing a graphic visualization format. Additionally, a web scraper will be developed that will go through the network in search of solutions to problems of Jutge.org.