Search

Search engines are an indispensable part of a web user's life. A vast majority of these web users experience difficulties caused by the keyword-based search engines such as inaccurate results for queries and irrelevant URLs even though the given keyword is present in them. Also, relevant URLs may be lost as they may have the synonym of the keyword and not the original one. This condition is known as the polysemy problem. To alleviate these problems, we propose an algorithm called automatic discovery and ranking of synonyms for search keywords in the web (ADRS). The proposed method generates a list of candidate synonyms for individual keywords by employing the relevance factor of the URLs associated with the synonyms. Then, ranking of these candidate synonyms is done using co-occurrence frequencies and various page count-based measures. One of the major advantages of our algorithm is that it is highly scalable which makes it applicable to online data on the dynamic, domain-independent and unstructured World Wide Web. The experimental results show that the best results are obtained using the proposed algorithm with WebJaccard.

Objectives: The objective of our proposed study is to eliminate the inference attacks through knowledge discovery process without compromising the accuracy of the systems. Methods/Analysis: In this paper, we have explored an enhanced privacy adoption scheme for twitter users. A supervised learning model that prevents the inference attacks using hit analytics and metadata knowledge derivation systems. By integrating the extracted feature vectors, we have eliminated the inference attacks caused by third party applications and elegantly classified the malignant and benign URLs. Findings: The proposed model is experimented in twitter dataset. The tweet URLs are collected and segmented into different URLs and similar URLs. The performance analysis is in terms of frequency of URLs usage, User’s assurance level via hit analytics and accuracy. It is evident from the results that higher usage of twitter network is employed and as the rate of users increases, the proposed classifier detects the benign and malicious URLs at significant level of 83%. Novelty/Improvement: A recent development made in the digital era has seen a phenomenal growth of social users. The social users make use of social networks to share their opinions about the facts, events etc. Our proposed study has incorporated on twitter users where trained classifier is developed from the behavior analysis of tweets users.

All the information and data on the Internet are connected based on URL. Although many people use URL to share and convey the information, it is difficult to transmit the information when URL is long and special characters are mixed. Short URL service is a service that transforms long URL with information into short form of URL and conveys the information, which makes it possible to access the page with necessary information. Recently, attackers who want to distribute the malicious code abuse the short URL through SMS or SNS to distribute malicious codes. With the short URL information, as it is difficult to predict the original URL, it has the vulnerability to Phishing attacks. In this study, a method is proposed, which writes the destination information when generating a short URL so that a user is able to check whether the destination is a web document or a file. The service provider of short URL monitors the risk of target URL page of the generated short URL and decides whether to provide service. By monitoring the modification of web-document, it measures and evaluates the risk of the webpage and decides whether to block the short URL according to the threshold, which prevents attacks such as "drive by download" through the short URL.

The web graph is the most common mathematical model of a website. Constructing a web graph of a real site requires data about the structure of that site: html-pages and/or documents in the site (in particular, data about URLs of web resources) and hyperlinks linking them. Web servers often use pseudonyms and redirections. They also generate the same pages dynamically via different URL requests. This raises a problem in which there are various URLs but with the same content. Thus, we can get a web graph in which some of its vertices correspond to pages of a site with the same content. The paper describes a crawler called RCCrawler that collects information about websites to build the web graphs of these sites. This crawler largely addresses the above problem as confirmed by a series of experiments conducted.

Genetic Algorithm (GA) is applied for generating the cluster based optimal ranked clicked URLs for effective Personalized Web Search(PWS).The computational overhead associated with the use of GA has no impact on the online performance of the PWS using clustered query sessions.The experimental results were compared on the basis of percent improvement in average precision of PWS both with and without optimal ranked clicked URLs over Classic IR in the domain Academics, Entertainment and Sports.The statistically verified results show more percent improvement in average precision of PWS with optimal ranked clicked URLs than PWS without optimal ranked clicked URLs over Classic IR. In this paper a novel approach is proposed for generating the optimal ranked clicked URLs using genetic algorithm (GA) based on clustered web query sessions for effective personalized web search. Experimental study was conducted on the data set of web query sessions captured in the domains academics, entertainment and sports to test the effectiveness of clusterwise optimal ranked clicked URLs for personalized web search (PWS). The results, which are verified statistically shows an improvement in the average precision of the personalized web search based on optimal ranked clicked URLs over both Classic IR and personalized web search without optimal ranked clicked URLs. Thus the effectiveness of personalized web search using optimal ranked clicked URLs is confirmed for better customizing the web search according to the information need of the user.

In the virus world of Internet, it is a challenging and urgent problem that how can we ensure the safety of search engines. A security subsystem of the search engine based on the research of content-based image search engine system V2.0 is developed. A malicious URL (Uniform Resource Locator) detection method based on BM (Boyer-Moore) pattern matching is proposed. The main research contents and results are as follows: Many malicious URLs could be downloaded by web image search, which may cause unnecessary loses to the users. So the malicious URL detection algorithm based on BM pattern matching is proposed. This method is to let the URL source code match the virus characteristics in the database to confirm whether the URL is safe or not. Web image search detects 203 malicious URLs based on this method. By kaspersky scanning, we confirmed 189 URLs to be malicious URLs, and the error rate is 6.9%, and the accurate rate is 93.1%. The experimental results show that the malicious URL detection algorithm provides secure URLs for web image search engine..

A large number of URLs collected by web crawlers correspond to pages with duplicate or near-duplicate contents. To crawl, store, and use such duplicated data implies a waste of resources, the building of low quality rankings, and poor user experiences. To deal with this problem, several studies have been proposed to detect and remove duplicate documents without fetching their contents. To accomplish this, the proposed methods learn normalization rules to transform all duplicate URLs into the same canonical form. A challenging aspect of this strategy is deriving a set of general and precise rules. In this work, we present DUSTER, a new approach to derive quality rules that take advantage of a multi-sequence alignment strategy. We demonstrate that a full multi-sequence alignment of URLs with duplicated content, before the generation of the rules, can lead to the deployment of very effective rules. By evaluating our method, we observed it achieved larger reductions in the number of duplicate URLs than our best baseline, with gains of 82 and 140.74 percent in two different web collections.

Twitter has emerged as a popular source of sharing and delivering news information. In tweet messages, URLs to web resources and hashtags are often included. This study investigates the potential of the hyperlinks and hashtags as topical clues and indicators to tweet messages. For this study, we crawled and analyzed about 1.5 million tweets for a 3-month period covering any topic or subject. The findings of this study revealed a power law relationship for the ranking and frequency of a the host names of URLs, and b a pair of hashtags and URLs that appeared in the tweet messages. This study also discovered that the most popular URLs used in tweets come from news and media websites, and a majority of the hyperlinked resources are news web pages. One implication of this study is that Twitter users are becoming more active in sharing already published information than producing new information. Finally, our investigation on hashtags for web resource indexing reveals that hashtags have the potential to be used as indexing terms for co-occurring URLs in the same tweet. We also discuss the implications of this study for web resource recommendation.

Currently, the efficiency of the existing focused crawlers is not high because of their unsatisfactory precision. In this article, we analyze the URL analysis methods of the existing focused crawlers, and propose a URL analysis algorithm based on the semantic content and link clustering in cloud environment. In this algorithm, the download URLs are clustered with the philosophy of clustering on the basis of VSM to improve the precision of the focused crawler according to the correlation between download URLs and new URLs. The algorithm is evaluated on Heritrix3.10 compared with Best First Search algorithm and Shark Search algorithm. The experiment results demonstrate that the algorithm proposed can collect web pages related to the given topic accurately and effectively.Moreover, the algorithm has a good ability of learning which proves the possibility of this algorithm.

Twitter is web based application which provide the service of sharing and mentioning the tweets and messages, which is social networking service. Twitter message could be a context information, or a URLs. Spam, scam, malicious distribution can be done easily with these URLs, spams and malicious data are usually a suspicious URLs with share another URLs in it. These URLs are known as URLs redirect chains, it is a repetition URLs. They itself contain too many URLs which is called as spam URLs. Generally there are several schemes to find such spam URLs, and they are evasion using time - based and evasion using crawling. This system is design to find- detect such spam or suspicious URLs- tweets from many other tweets, The URLS redirect chains frequently share the same URLs, and have been reused usually by attackers because they have limited resources. A Run-time Detection System for Malicious URLs in Twitter has method to show and detect the suspiciousness of the correlated URLs redirect chains, which are commonly shared URLs

In a Web application, you can pass without restrictions special network security devices such as IPS and F/W, URL parameter, which is an important element of communication between the client and the server, is forwarded to the Web server. Parameters are modulated by an attacker requests a URL, disclose confidential information or through e-commerce, can take financial gain. Vulnerability parameter manipulation thereof cannot be able to determine whether to operate in only determined logical applica tion, blocked with Web Application Firewall. In this paper, I will pr esent a technique OTACUS(One-Time Access Control URL System) to complement the shortcomings of the measures existing approaches . OTACUS can be effectively blocked the modulation of the POST or GET method parameters passed to the server by preventing the exposure of the URL to the attacker by using clean URL techniq ue simplifies complex URL that contains the parameter. Performance test results of the actual implementation OTACUS proves that i t is possible to show a stable operation of less than 3% increase in the load

Attribute-specific models are quite effective in the detection of web attacks. Nevertheless, none of existing attribute-specific models can be applied directly to modern web applications, which are dominant today and which use RESTful URLs. The barrier is that paths and parameters are mixed in URLs so that detection systems cannot identify which programs and attributes are which. To remove this barrier, this paper is the first one dealing with the identification of paths and parameters in RESTful URLs. The problem is raised, then a simple yet effective method to solve the problem is proposed. We validate our method via experiments.

Many URLs on the Internet point to identical contents, which increase the burden of web crawlers. Techniques that detect such URLs (known as URL de-duping) can greatly save resources such as bandwidth and storage for crawlers. Traditional de-duping methods are usually limited to heavily engineered rule matching strategies.In this work, we propose a novel URL de-duping framework based on sequence-to-sequence (Seq2Seq) neural networks. A single concise translation model can take the place of thousands of explicit rules. Experiments indicate that a vanilla Seq2Seq architecture yields robust and accurate results in detecting duplicate URLs. Furthermore, we demonstrate the efficiency of this framework in the real large-scale web environment.

Nowadays, citizens require high level quality information from public institutions in order to guarantee their transparency. Institutional websites of governmental and public bodies must publish and keep updated a large amount of information stored in thousands of web pages in order to satisfy the demands of their users. Due to the amount of information, the “search form”, which is typically available in most such websites, is proven limited to support the users, since it requires them to explicitly express their information needs through keywords. The sites are also affected by the so-called “long tail” phenomenon, a phenomenon that is typically observed in e-commerce portals. The phenomenon is the one in which not all the pages are considered highly important and as a consequence, users searching for information located in pages that are not condiered important are having a hard time locating these pages.

Publicly posted URLs sometimes contain a wealth of information about the identities and activities of the users who share them. URLs often utilize query strings -- that is, key-value pairs appended to the URL path -- to pass session parameters and form data. Although often benign and necessary to render the Web page, query strings sometimes contain tracking mechanisms, usernames, email addresses, and other information that users might not wish to publicly reveal. In isolation, this isn't particularly problematic, but the growth of Web 2.0 platforms such as social networks and microblogging means URLs, which are often copied and pasted from Web browsers, are increasingly publicly broadcast. To study URL sharing's privacy ramifications, the authors ran a measurement study that looked at 892 million user-submitted URLs, many disseminated in semipublic forums. That corpus contained a trove of personal information, including 1.7 million email addresses. In the most egregious examples, query strings contain plaintext usernames and passwords for administrative and sensitive accounts. The authors identify data leakage via both key-driven and value-driven analysis using manual inspections and automatic detection logic. Additionally, they analyze the click-through rates of sensitive URLs, examine geographical and mobile behavior patterns, and measure the broader statistical properties of key-value pairs. Finally, they propose a CleanURL service that can "scrub"' URLs of privacy-violating content.

Seed URLs selection for focused Web crawler intends to guide related and valuable information that meets a user's personal information requirement and provide more effective information retrieval. In this paper, we propose a seed URLs selection approach based on user-interest ontology. In order to enrich semantic query, we first intend to apply Formal Concept Analysis to construct user-interest concept lattice with user log profile. By using concept lattice merger, we construct the user-interest ontology which can describe the implicit concepts and relationships between them more appropriately for semantic representation and query match. On the other hand, we make full use of the user-interest ontology for extracting the user interest topic area and expanding user queries to receive the most related pages as seed URLs, which is an entrance of the focused crawler. In particular, we focus on how to refine the user topic area using the bipartite directed graph. The experiment proves that the user-interest ontology can be achieved effectively by merging concept lattices and that our proposed approach can select high quality seed URLs collection and improve the average precision of focused Web crawler.

Embedding malicious URLs in e-mails is one of the most common web threats facing the internet community today. Malicious URLs have been widely used to mount various cyber-attacks like spear phishing, pharming, phishing and malware. By falsely claiming to be a trustworthy entity, users are lured into clicking on these compromised links to divulge vital information such as usernames, passwords, or credit card details and unknowingly succumb to identity theft. Hence, the detection of malicious URLs in e-mails is very essential so as to help internet users implement safe practices and as well prevent them from becoming victims of fraud. This paper explores how malicious links in e-mails can be detected from the lexical and host-based features of their URLs to protect users from identity theft attacks. This research uses Naive Bayesian classifier as a probabilistic model to detect if a URL is malicious or legitimate. The Naive Bayesian classifier is used to count up the occurrence of each feature in an e-mail and calculate the cumulative score. If the cumulative score is greater than the given threshold, the URL is considered malicious otherwise the URL is legitimate.

The World Wide Web supports a wide range of criminal activities such as spam-advertised e-commerce, financial fraud and malware dissemination. Although the precise motivations behind these schemes may differ, the common denominator lies in the fact that unsuspecting users visit their sites. These visits can be driven by email, web search results or links from other web pages. In all cases, however, the user is required to take some action, such as clicking on a desired Uniform Resource Locator (URL). In order to identify these malicious sites, the web security community has developed blacklisting services. These blacklists are in turn constructed by an array of techniques including manual reporting, honeypots, and web crawlers combined with site analysis heuristics. Inevitably, many malicious sites are not blacklisted either because they are too recent or were never or incorrectly evaluated. In this paper, we address the detection of malicious URLs as a binary classification problem and study the performance of several well-known classifiers, namely Naive Bayes, Support Vector Machines, Multi-Layer Perceptron, Decision Trees, Random Forest and k-Nearest Neighbors. Furthermore, we adopted a public dataset comprising 2.4 million URLs (instances) and 3.2 million features. The numerical simulations have shown that most classification methods achieve acceptable prediction rates without requiring either advanced feature selection techniques or the involvement of a domain expert. In particular, Random Forest and Multi-Layer Perceptron attain the highest accuracy.

On the World Wide Web, the malicious links are highly problematic in the dissemination channels as a source code to the malware broadcasting. These suspicious malicious links gives full access to the web attackers as an instrument of web pages on internet. It is easily affected by the results of attackers on the system of victim where system is utilized easily for performing the cyber-attacks such as stealing the financial credentials, phishing-spamming, hacking and many more such web attacks. The developed system must be accurate and fast enough to detect these types of such cyber-attacks by observing the ability to find new developed malicious URLs or malicious source code contents. It is the critical task to detect the malicious contents in network of the web pages over the World Wide Web. The various malicious cyber-attacks like spamming, code phishing are done by using the malicious URLs to mount these types of cyber-attacks. Internet unlawful activities are found in Malicious Web sites as cornerstone of the Malicious URLs. The main threat is to identify these attacks so that the suspicious URLs can be easily resolved as malicious URLs along with its source code of the web pages. In this paper, a method has been proposed which is highly useful in the field of World Wide Web networking of domains for detecting the malicious URLs by using BM (Boyer-Moore) [1] string pattern matching algorithm based on word segmentation approach [2]. The nature of attacks is identified as a malicious URL or source code of the web sites questioned on the World Wide Web. The proposed approach is based on the real time system for getting suspicious URL from the DNS server followed by the detection on the basis of word segmentation of source code. The discriminative features of this system are verified by using the proposed method which gives a variety of properties including text and link in the source code as highly powerful and novel approach in the detection of suspicious URLs.

The current URLs are assigned per website so that a user must access the website to utilize specific contents resources. Considering that users are more interested in the resources than the website itself, the current URL is limited in terms of user convenience. This paper proposes a new model for contents location that is easily recognizable as web resources such as digital contents and can be accessed directly. It is expected to contribute to establishing standard infrastructure in contents distribution and better user convenience.

Web servers provide immunity against Man In The Middle (MITM) attacks and eavesdropping by using HTTP Strict Transport Security (HSTS) to force user agents to communicate only over HTTPS connections. However, the initial connection request from a user is made over an insecure HTTP connection. This issue was addressed by user agents; Google Chrome and Firefox, implicitly, by including a static list of URLs to be accessed only over secure HTTPS connections. Since, these user agents maintain their lists independently, the URLs used by one user agent are invisible to another. A user is prone to MITM attacks, especially in public hotspot environments, when accessing a URL present in the list of secure URLs of one browser but not in another, since the initial handshake from that user agent is insecure. Attacks can be initiated by modifying the outgoing HTTP packets and also the HTTPS response packets from the webserver. This motivated us to propose a solution independent of user agents, by merging the static URL lists of different user agents and enforcing HTTPS for all those URLs. In this paper, we propose a solution, SHSHTTPS Enforcer that introduces a local daemon to enforce URL redirection before the request flows out of the client for the URLs in a list compiled from multiple sources. The proposed solution has been demonstrated through a prototype implementation of the Squid Proxy server as our local daemon. The experiment was conducted by providing a URL, which was not present in one browser's list but was present in another browser's list. It was evident that SHS-HTTPS Enforcer enforced HTTPS successfully and MITM attacks were prevented.

Detecting malicious URLs is an essential task in network security intelligence. In this paper, we make two new contributions beyond the state-of-the-art methods on malicious URL detection. First, instead of using any pre-defined features or fixed delimiters for feature selection, we propose to dynamically extract lexical patterns from URLs. Our novel model of URL patterns provides new flexibility and capability on capturing malicious URLs algorithmically generated by malicious programs. Second, we develop a new method to mine our novel URL patterns, which are not assembled using any pre-defined items and thus cannot be mined using any existing frequent pattern mining methods. Our extensive empirical study using the real data sets from Fortinet, a leader in the network security industry, clearly shows the effectiveness and efficiency of our approach.

Shorten URL service is the method of using short URL instead of long URL, it redirect short url to long URL. While the users of microblog increased rapidly, as the creating and usage of shorten URL is convenient, shorten url became common under the limited length of writing on microblog. E-mail, SMS and books use shorten URL well, because of its simplicity. But, there is no relativeness between the most of shorten URLs and their target URLs, user can not expect the target URL. To cover this problem, there is attempts such as changing the shorten URL service name, inserting the information of website into shorten URL, and the usage of shortcode of physical address. However, each ones has the limits, so these are the trouble of automation, relatively long address, and the narrowness of applicable targets. SHRT is complementary to the attempts, as getting the idea from the writing system of Arabic. Though the writing system of Arabic has no vowel alphabet, Arabs have no difficult to understand their writing. This paper proposes SHRT, new method of URL Shortening. SHRT makes user guess the target URL using Relative word of the lowest domain of target URL without vowels. ※ "본 연구는 미래창조과학부 및 정보통신산업진흥원의 대학 IT연구센터 지원사업의 연구결과로 수행되었음"

Twitter is prone to malicious tweets containing URLs for spam, phishing, and malware distribution. Conventional Twitter spam detection schemes utilize account features such as the ratio of tweets containing URLs and the account creation date, or relation features in the Twitter graph. These detection schemes are ineffective against feature fabrications or consume much time and resources. Conventional suspicious URL detection schemes utilize several features including lexical features of URLs, URL redirection, HTML content, and dynamic behavior. However, evading techniques such as time-based evasion and crawler evasion exist. In this paper, we propose WarningBird, a suspicious URL detection system for Twitter. Our system investigates correlations of URL redirect chains extracted from several tweets. Because attackers have limited resources and usually reuse them, their URL redirect chains frequently share the same URLs. We develop methods to discover correlated URL redirect chains using the frequently shared URLs and to determine their suspiciousness. We collect numerous tweets from the Twitter public timeline and build a statistical classifier using them. Evaluation results show that our classifier accurately and efficiently detects suspicious URLs. We also present WarningBird as a near real-time system for classifying suspicious URLs in the Twitter stream.

Web Spoofing lures the user to interact with the fake websites rather than the real ones. The main objective of this attack is to steal the sensitive information from the users. The attacker creates a ‘shadow’ website that looks similar to the legitimate website. This fraudulent act allows the attacker to observe and modify any information from the user. This paper proposes a detection technique of phishing websites based on checking Uniform Resources Locators (URLs) of web pages. The proposed solution is able to distinguish between the legitimate web page and fake web page by checking the Uniform Resources Locators (URLs) of suspected web pages. URLs are inspected based on particular characteristics to check the phishing web pages. The detected attacks are reported for prevention. The performance of the proposed solution is evaluated using Phistank and Yahoo directory datasets. The obtained results show that the detection mechanism is deployable and capable to detect various types of phishing attacks maintaining a low rate of false alarms.

It is becoming increasingly difficult to ignore the importance of using online social networks (OSNs) for various purposes such as marketing, education, entertainment, and business. However, OSNs also open the door for harmful activities and behaviors. Committing financial fraud and propagating malware and spam advertisements are very common criminal actions that people engage in by accessing uniform resource locators (URLs). It has been reported that advanced attackers tend to exploit human flaws rather than system flaws, thus, users are targeted in social media threats by hour. This research aims to understand the state of literature on detecting malicious URLs in OSNs, with a focus on two major aspects: URL and OSN objects. Although the literature presents these two aspects in a different context, this paper will mainly focus on their features in relation to malicious URL detection using classification methods. We firstly review three features of URLs: lexical features, hosts, and domains, then we discuss their limitations. We then introduce social spam analytics and detection models using combined features from both URLs and OSNs, particularly the use of user profiles and posts together with URL features, to enhance the detection of malicious behavior. This combination can help in understanding the interests of the user either explicitly, by stating choices in the profile, or implicitly, by analyzing the post behavior, as the spammers do not maintain a regular interest and tend to exploit events or top trend topics.

Recently, many online attacks have increased due to the growing use of the Internet, Among these the most well-known attack is phishing. Phishing is a demonstration of getting delicate data by persuading the users to uncover their own data by pretending as a trusty source in the web transaction. Most phishing attacks work by sending a forged email that contains a URL which leads you to the fake website by clicking on it. In most of the cases, phisher chooses http URLs for the attack. Along these lines, we are proposing a novel algorithm which will detect whether a given http URL is of phishing site or not. Our algorithm performs Google's updated blacklist check, utilizes Google search engine results, Alexa Ranking and no of URL-based features, for detecting phishing URLs. It displays alert message if the URL found as possible phishing, otherwise it displays safe message. This algorithm enhances the performance when dealing with known/old phishing URLs.

As malware industry grows, so does the means of infecting a computer or device evolve. One of the most common infection vector is to use the Internet as an entry point. Not only that this method is easy to use, but due to the fact that URLs come in different forms and shapes, it is really difficult to distinguish a malicious URL from a benign one. Furthermore, every system that tries to classify or detect URLs must work on a real time stream and needs to provide a fast response for every URL that is submitted for analysis (in our context a fast response means less than 300-400 milliseconds/URL). From a malware creator point of view, it is really easy to change such URLs multiple times in one day. As a general observation, malicious URLs tend to have a short life (they appear, serve malicious content for several hours and then they are shut down usually by the ISP where they reside in). This paper aims to present a system that analyzes URLs in network traffic that is also capable of adjusting its detection models to adapt to new malicious content. Every correctly classified URL is reused as part of a new dataset that acts as the backbone for new detection models. The system also uses different clustering techniques in order to identify the lack of features on malicious URLs, thus creating a way to improve detection for this kind of threats.

With the rapid development of the information technology, internet security has drawn more and more attention. Nowadays, most researchers focus on lexical and host features to classify Phishing URLs. In this paper, we proposed Vulnerable Sites List and a new feature which is named URL Correlation. URL Correlation is based on the similarity of URLs with the List above that we created. In addition, a large improvement of accuracy is observed by comparing methods which use our new feature with the others which use the normal one.

In the era of the mobile Internet, the problem of personal information leakage is becoming more and more serious. It is possible that third-parities (e.g., advertising companies, telecom operators) can easily obtain private user information by analyzing the Internet traffic flows to (or from) users. In this paper, in order to examine the degree of information leakage via the URLs, we propose an efficient Internet metadata analytics framework called URLSight, which attempts to profile user behaviors by simply analyzing the Internet metadata embedded in the URLs. Our analysis is based on a large dataset which contains one-month URL logs generated by around 100 thousand mobile users. After parsing and contextualizing URLs, URLSight can extract all the key-value pairs and perform co-occurrence processing to eliminate information redundancy. We also implement the URLSight framework on the Apache Spark platform to improve its performance. The results show that URLSight can effectively extract user privacy from URLs. Last, we also discuss a few practical approaches to defend against URL information leakage.

Phishing is a considerable problem differs from the other security threats such as intrusions and Malware which are based on the technical security holes of the network systems. The weakness point of any network system is its Users. Phishing attacks are targeting these users depending on the trikes of social engineering. Despite there are several ways to carry out these attacks, unfortunately the current phishing detection techniques cover some attack vectors like email and fake websites. Therefore, building a specific limited scope detection system will not provide complete protection from the wide phishing attack vectors. This paper develops detection system with a wide protection scope using URL features only which is relying on the fact that users directly deal with URLs to surf the internet and provides a good approach to detect malicious URLs as proved by previous studies. Additionally, Anti-phishing solutions can be positioned at different levels of attack flow where most researchers are focusing on client side solutions which turn to add more processing overhead at the client side and lead to losing the trust and satisfaction of the users. Nowadays many organizations make centralized protection of spam filtering. This paper proposes a system which can be integrated into such process in order to increase the detection performance in a real time. The simulation results of the proposed system showed a phishing URLs detection accuracy with 93% and provided online process of a single URL in average time of 0.12 second.

Today, web is used in virtually all the spheres of daily life. Internet is a worldwide storehouse of information. Internet searcher is a data recovery framework that has different site information stored in it. The web index records No. of times the URL accessed; it goes to the web URL to recover its meta data and indicates page mistakes. The web searcher is at first sustained with URLs of various destinations and stores this data in its database. Then the web searcher licenses customers to look for particular data. SEO matches the content with the content given in the URLs reinforced in the database. When client taps on the connections the website page with all its information is displayed. In this paper a comparative analysis of several SEO techniques has been done based on some inducing factors like N gram indices, Key word & link optimization. The analysis shows that crawling & fetching is the best method because of faster and efficient retrieval of data from search engine.

Web security is an important part of information security. This paper proposes a new web anti-attack method based on URL randomization. Adding a random field in the URL leads that the attackers cannot get desired URLs through sniffing and scanning the static URLs. In the section of theoretical analysis, we analyze the probability that the attackers construct the correct URLs. Finally, we implement a prototype of the method we come up with and use it to measure the overhead the method will bring. Experiment results shows that the overhead the method brings is very little and this method has the significance of practical application.

With the expanding application of the two-dimensional barcode, the harmfulness of malicious URLs is much more serious. For preventing malicious URLs of two-dimensional barcode, this paper proposes a detection method -- H-SBH which is utilizing hash function. Firstly, the detection system extracts eigenvalues of malicious and benign URLs by H-SBH, then builds black and white lists library based on the eigenvalues, after that, matches URLs with lists library. Finally, the detection system gives the user safety tips according to match results. The experimental results indicated that this method was efficient and feasible for detecting the malicious URLs in two-dimensional barcodes.

Thanks to the strength of crowdsourcing, there is a lot of useful information on StackOverflow, the most popular Question and Answer (Qaamp;A) platform in software engineering area. This information can be treated as numerous URLs (Uniform Resource Locators), which can be categorized into URLs of Qaamp;As and URLs in Qaamp;As. The domain of former ones is Stack-Overflow itself, while domains of latter ones are miscellaneous, such as some personal blogs and so on. Although each Qaamp;A has been manually assigned tags, relations between URLs and tags are not clear enough. In this paper, we propose SOLinker, a method to build semantic links between various URLs and tags. Firstly, SOLinker identifies proper relations from a predefined relation set between tags and URLs, which is modeled as a text classification problem. Features are extracted from content of Qaamp;A, the URL and the tag list, and classification algorithms are Logistic Regression and Gradient Boosting Decision Tree, depending on the category of URLs. Secondly, there exists a partial tagging problem, which means for a URL in a Qaamp;A, there are only a part of tags of the Qaamp;A relating to the URL. To address this problem, we propose a semantic analysis method to analyze context of this URL and the URL itself from both implicit and explicit aspects. Then SOLinker will infer proper tags by the label propagation technique. Results show that our method is feasible and practical in constructing semantic links between tags and URLs of/in Qaamp;As. In particular, the F-Score of semantic relation identification is around 78%, 5% higher than the other existing method, and F-Score of partial tagging solving is around 88%.

Twitter is one of the most popular Online Social Networks (OSN). It is used by millions of users worldwide everyday. Due to the text limitation on Twitter (140 characters per tweet), URL (Uniform Resource Locator) shortening services are widely used, however they are not free from risks. Shortened URLs are completely different from the original URLs, and hence users have no idea where the short URLs will direct them to. Attackers leverage this knowledge to their advantage to spread malicious URLs. Most of the approaches proposed for classifying malicious URLs utilize information from both social networks and URL shorting service providers. In this paper, we propose a novel approach to detect malicious short URLs using only visible features of tweets and user profiles. We test four machine learning algorithms, i.e., Naive Bayes, random forest, support vector machine, and logistic regression and obtain an accuracy of up to 97% using random forest when classifying malicious short URLs. Our testing results indicate that the approach using visible features from social networks to detect malicious URLs is practicable.

A Uniform Resource Locator (URL) represents the address of a web page in World Wide Web (WWW). A URL grants access to a single web page on the WWW. Here in this paper the main focus is on the URLs addressing the same web page/same content. A web Page can have two or more URLs with which the web page can be accessed. These duplicate URLs can be a serious threat to the entire pipeline of internet searcher administration for indexing and creeping. I am presenting a novel algorithm for detecting canonization rules for normalizing URLs to the original single URL. Here a pattern recognition approach has been used for analyzing textual data. This approach benefits search engines from information about duplicate URLs to optimize the performance of search engine in terms of reduced cost and improved quality.

Cross-site scripting (XSS) is one of the major threats to the security of web applications. Many techniques have been taken to prevent XSS. This paper presents an approach to identify Cross-Site Scripting attacks based on URL analysis. The fundamental assumption of our method is that the URL contains a part that can produce a valid JavaScript syntax tree. First, we extract the parameters of the URL to produce a valid JavaScript syntax tree and weight its parsing depth. If its depth exceeds a user-defined threshold, the URL is considered suspicious. Second, to the exception URLs, a second level of defense is formed by analyzing its structure. The experimental results demonstrate that our approach can effectively distinguish most of the malicious URLs from the benign ones.

Crawler is one of the main components in the search engines which use URLs to fetch web pages to build a repository of web pages starting with entering URL. Each web page is parsed to extract the URLs included in it and store the extracted URLs in the URLs Queue to fetch by the crawlers in sequential. The process of crawling takes long time to collect more web pages, and it has become necessary to utilize the unused computing resources and cost/time savings in organizations. This paper deals with the crawler of search engine using grid computing. This paper presents the grid computing that has been implemented by Alchemi. Alchemi is an open source project developed at the University of Melbourne, provides middleware for creating an enterprise grid computing environment. The crawling processes are passed to Alchemi manager which distribute the processes over a number of computers as executors. The search engine crawler with the grid computing is implemented, tested and the results are analyzed. There is an increase in performance and less time over the single computer.

Crawlers are basic entity that makes search engine to work efficiently in World Wide Web. Semantic Concept is implied into the search engine to provide precise and constricted search results which is required by end users of Internet. Search engine could be enhanced in searching mechanism through semantic Lexical Database such as WordNet, ConceptNet, YAGO, etc; Search results would be retrieved from Lexical and Semantic Knowledge Base (KB) by applying word sense and metadata technique based on the user query. The Uniform Resource Locater (URL) could be added and updated by the user to Semantic knowledge base so that crawlers can easily extract meta data and text which is available in specified web page. The proposed methodology enables web crawler to extract all meta tags and metadata from the web page which are stored in Semantic KB, hence search results are expected to be more significant and effective.

Webometric network analyses have been used to map the connectivity of groups of websites to identify clusters, important sites or overall structure. Such analyses have mainly been based upon hyperlink counts, the number of hyperlinks between a pair of websites, although some have used title mentions or URL citations instead. The ability to automatically gather hyperlink counts from Yahoo! ceased in April 2011 and the ability to manually gather such counts was due to cease by early 2012, creating a need for alternatives. This article assesses URL citations and title mentions as possible replacements for hyperlinks in both binary and weighted direct link and co-inlink network diagrams. It also assesses three different types of data for the network connections: hit count estimates, counts of matching URLs, and filtered counts of matching URLs. Results from analyses of U.S. library and information science departments and U.K. universities give evidence that metrics based upon URLs or titles can be appropriate replacements for metrics based upon hyperlinks for both binary and weighted networks, although filtered counts of matching URLs are necessary to give the best results for co-title mention and co-URL citation network diagrams.

An URL parameter can hold some information that is confidential or vulnerable to illegitimate tampering. We propose Link-E-Param(Link with Encrypted Parameters) to protect the whole URL parameter names as well as their values. Unlike other techniques concealing only some of the URL parameters, it will successfully discourage attacks based on URL analysis to steal secret information on the Web sites. We implement Link-E-Param in the form of a servlet filter to be deployed on any Java Web server by simply copying a jar file and setting a few configuration values. Thus it can be used for any existing Web application without modifying the application. It also supports numerous encryption algorithms to choose from. Experiments show that our implementation induces only 2~3% increase in user response time due to encryption and decryption, which is deemed acceptable.

In this paper we propose a method that identifies and extracts keywords within URLs, focusing on the Greek Web and especially on URLs containing Greek terms. Although there are previous works on how to process Greek online content, none of them focuses on keyword identification within URLs of the Greek web domain. In addition, there are many known techniques for web page categorization based on URLs but, none addresses the case of URLs containing transliterated Greek terms. The proposed method integrates two components; a URL tokenizer that segments URL tokens into meaningful words and a Latin–to–Greek script transliteration engine that relies on a dictionary and a set of orthographic and syntactic rules for converting Latin verbalized word tokens into Greek terms. The experimental evaluation of our method against a sample of 1,000 Greek URLs reveals that it can be fruitfully exploited towards automatic keyword identification within Greek URLs

In service-oriented computing, a recommender system can be wrapped as a web service with machine-readable interface. However, owing to the cross-organizational privacy issue, the internal dataset of an organization is seldom exposed to external services. In this paper, we propose a higher level recommender strategy INSERT that guides the underlying external universal recommender to suggest a set of indexes. INSERT then matches the title of each top-ranked index entry with the domain-specific keywords in the organization's internal dataset, and further directs the universal recommender to verify the popularity of such matching. INSERT finally makes recommendation based on the verification results. INSERT also employs URLs taken from a client as user contexts, which is challenging because URLs contain little content. Our experiment shows that this strategy is feasible and effective.

Malicious Web sites are a cornerstone of Internet criminal activities. The dangers of these sites have created a demand for safeguards that protect end-users from visiting them. This article explores how to detect malicious Web sites from the lexical and host-based features of their URLs. We show that this problem lends itself naturally to modern algorithms for online learning. Online algorithms not only process large numbers of URLs more efficiently than batch algorithms, they also adapt more quickly to new features in the continuously evolving distribution of malicious URLs. We develop a real-time system for gathering URL features and pair it with a real-time feed of labeled URLs from a large Web mail provider. From these features and labels, we are able to train an online classifier that detects malicious Web sites with 99% accuracy over a balanced dataset.

PurposeThe paper aims to cover the institutional decision‐making processes behind the decision to add machine readable cataloging records to the library catalog that link to hundreds of thousands of URLs with freely available web content, the management processes making this possible, as well as other more philosophical concerns.Design/methodology/approachThis is a case study conducted at the University of Denver, Penrose Library.FindingsImplementing a URL tracking system, combined with a philosophical decision to follow a proactive URL maintenance approach rather than a reactive one, has resulted in an improved error rate for URL click‐throughs.Originality/valueThe paper addresses URL maintenance issues, staffing solutions, and error rates not addressed elsewhere in the literature.

The information on the web is growing dramatically. The users have to spend lots of time on the web finding the information they are interested in. Today, he traditional search engines do not give users enough personalized help but provide the user with lots of irrelevant information, In this paper, we present a personalize Web search system, which can helps users to get the relevant web pages based on their selection from the domain list. Thus, users can obtain a set of interested domains and the web pages from the system. The system is based on features extracted from hyperlinks, such as anchor terms or URL tokens, user interest domains and past search results. Our methodology uses an innovative weighted URL Rank algorithm based on user interested domains and user query.

Focused web crawlers collect topic-related web pages from the Internet. Using Q learning and semi-supervised learning theories, this study proposes an online semi-supervised clustering approach for topical web crawlers (SCTWC) to select the most topic-related URL to crawl based on the scores of the URLs in the unvisited list. The scores are calculated based on the fuzzy class memberships and the Q values of the unlabelled URLs. Experimental results show that SCTWC increases the crawling performance.