Search log analysis: What it is, what's been done, how to do it

Website analytics for government user behavior during COVID-19 pandemic

Purpose

The present study aims to clarify the following two research objectives: (1) the user behavior of government websites during the coronavirus disease (COVID-19) period and (2) how the government improved government's website design during the COVID-19 period.

Design/methodology/approach

The authors used website analytics to examine usage patterns and behaviors of the government website via personal computer (PC) and mobile devices during the COVID-19 pandemic. In-depth interviews were conducted to understand the user experience of government website users and to gather users' opinions about how government websites should be redesigned.

Findings

With the rising of the COIVID-19 pandemic, most studies expect that the use of government websites through a mobile device will grow astonishingly. The authors uncovered that the COVID-19 pandemic did not increase the use of government websites. Instead, severe declines in website usage were observed for all device users with the declines being more pronounced in mobile device users than in PC users. This is an admonitory caveat that reveals public health and pandemic prevention information announced on government websites cannot be effectively transmitted to the general public through official online platforms.

Originality/value

The study highlights the gap in information behavior and usage patterns between PC and mobile device users when visiting government websites. Although mobile devices brought many new visitors, mobile devices are ineffective in retaining visitors and continuous long-term use. The results of localize experience is helpful in the improvement of government website evaluation worldwide.

The use of OPAC query logs to support evidence-informed information literacy (IL) training at The University of Dar es Salaam

This paper reports the results of the study which analysed OPAC users’ searching behaviour through the use of search queries. The study analysed how OPAC search queries can be used to assess the effectiveness of the information literacy programme offered by the University of Dar es Salaam. Data were collected unobtrusively and stored in the OPAC web server’s log file. OPAC search logs were qualitatively and quantitatively analysed to determine search patterns and search query formulation. Information literacy course contents were also qualitatively analysed and compared with users’ search activities. The results suggest that search query logs are essential and high quality evidence that can be used to inform information literacy pedagogical methods and course contents improvement. A comparison between OPAC users’ search behaviours and information lteracy course content identified some areas for readjusting the way information literacy instructions are delivered and improving OPAC search functionalities. The study has contributed to the conceptual understanding and the use of evidence to support evidence-informed practices (EIPs) in delivering information literacy courses.

The effects of cognitive load on query reformulation: mental demand, temporal demand and frustration

Purpose

This study explores the effects of cognitive load on the propensity to reformulate queries during information seeking on the web.

Design/methodology/approach

This study employs an experimental design to analyze the effect of manipulations of cognitive load on the propensity for query reformulation between experimental and control groups. In total, three affective components that contribute to cognitive load were manipulated: mental demand, temporal demand and frustration.

Findings

A significant difference in the propensity of query reformulation behavior was found between searchers exposed to cognitive load manipulations and searchers who were not exposed. Those exposed to cognitive load manipulations made half as many search query reformulations as searchers not exposed. Furthermore, the National Aeronautical and Space Administration Task Load Index (NASA-TLX) cognitive load scores of searchers who were exposed to the three cognitive load manipulations were higher than those of searchers who were not exposed indicating that the manipulation was effective. Query reformulation behavior did not differ across task types.

Originality/value

The findings suggest that a dual-task method and NASA-TLX assessment serve as good indicators of cognitive load. Because the findings show that cognitive load hinders a searcher's interaction with information search tools, this study provides empirical support for reducing cognitive load when designing information systems or user interfaces.

Use analysis of the digital library of PhD dissertations defended at the University of Novi Sad

Purpose

The purpose of this paper is to determine the digital library usage patterns as a means of improving the system, as well as the user experience, to give appropriate recognition to the most popular dissertations’ authors and to measure the interest of non-academic users for dissertations defended at the University of Novi Sad (UNS).

Design/methodology/approach

A logging module of the digital library of theses and dissertations of University of Novi Sad (PHD UNS) application has been implemented. The module recorded the messages relating to the search queries and downloads over a three-year period from 2017–2019. These logs are analysed using the Elasticsearch, Logstash and Kibana (ELK) technology stack and the results are shown using graphs and tables.

Findings

The analysis determined the perfect time for weekly maintenance of the system, defined a recommendation for improving the system and revealed the most popular dissertations. A significant number of downloads and queries originated from citizens, i.e. users outside the academic community.

Practical implications

The conducted analysis defined recommendations for the system improvement which can be used by PHD UNS research and development (R&D) team and revealed the most popular dissertations which are used for the promotion of its authors through faculties’ websites.

Originality/value

To the best of the authors’ knowledge, this is the first study of ELK based log analysis of a Serbian language documents’ repository. Besides, the value of results for the PHD UNS R&D team and UNS rector team, the study proves that PhD digital library presents an important Open Science communication channel for presenting scientific results to the citizens.

A transaction log analysis of EBSCO Discovery Service using Google Analytics: the methodology

Purpose

The article presents a methodology that can be used to analyze data from the transaction log of EBSCO Discovery Service searches recorded in Google Analytics. It explains the steps to follow for exporting the data, analyzing the data, and recreating searches. The article provides suggestions to improve the quality of research on the topic. It also includes advice to vendors on improving the quality of transaction log software.

Design/methodology/approach

Case study

Findings

Although Google Analytics can be used to study transaction logs accurately, vendors still need to improve the functionality so librarians can gain the most benefit from it.

Research limitations/implications

The research is applicable to the usage of Google Analytics with EBSCO Discovery Service.

Practical implications

The steps presented in the article can be followed as a step-by-step guide to repeating the study at other institutions.

Social implications

The methodology in this article can be used to assess how library instruction can be improved.

Originality/value

This article provides a detailed description of a transaction log analysis process that other articles have not previously described. This includes a description of a methodology for accurately calculating statistics from Google Analytics data and provides steps for recreating accurate searches from data recorded in Google Analytics.

Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results

To address the challenges of aging societies, various information and communication technology (ICT)-based systems for older people have been developed in recent years. Currently, the evaluation of these so-called active and assisted living (AAL) systems usually focuses on the analyses of usability and acceptance, while some also assess their impact. Little is known about the actual take-up of these assistive technologies. This paper presents a framework for measuring the take-up by analyzing the actual usage of AAL systems. This evaluation framework covers detailed information regarding the entire process including usage data logging, data preparation, and usage data analysis. We applied the framework on the AAL prototype CARIMO for measuring its take-up during an eight-month field trial in Austria and Italy. The framework was designed to guide systematic, comparable, and reproducible usage data evaluation in the AAL field; however, the general applicability of the framework has yet to be validated.

Diverse mobile users: the development of library experts

Purpose

The purpose of this paper is to undertake a formative evaluation of growth over time that would demonstrate diverse library users’ development as they interact with mobile digital library services.

Design/methodology/approach

This paper incorporated a server log analysis to evaluate first, the location of users. To study the nature of diverse user development, users from unique locations were identified and tracked over several years. The type of growth that this paper analyzes is the development of a library user from the beginning stages of use into one who is more experienced. For the purposes of this paper, the authors define library experts as experienced library users. These are users who have come back to the library over multiple sessions of learning and branched out into multiple areas of library functionality and services.

Findings

The findings of modular mobile use over time suggest that, while over half of users only utilized one module, 39 per cent of all users accessed more than one module. This formative approach to assessing student library engagement suggests alternative metrics for assessing outreach and distance learning.

Originality/value

The underlying departure point for this study is that formative models may introduce descriptive data valuable to the learning analytics toolkit. The library research literature on learning analytics, and perhaps library service offerings that support learning, may gain additional value by attending to students’ formative development as they interact with library resources. Describing the way in which mobile app users develop can yield insights about learning over time, both on campus and at a distance.

Use and users of the Minrva mobile app

Purpose

The purpose of this paper is to investigate new undergraduate student library engagement in the Minrva mobile app during the months of May 2015 through December 2015.

Design/methodology/approach

This research investigated what parts of a mobile app new students were using in their first semester after downloading the app. The quantitative study used application programming interface log analysis to better understand what parts of the app new students use in the mobile app.

Findings

By undertaking this study, the author has a better understanding about what students are finding useful within the app and what tools are not being used by this cohort in their first semester.

Originality/value

The value of this research is in helping system designers and first-year experience planners know what mobile support tools students are finding useful in their first semester. Implication for mobile interface design based on module popularity are discussed.

Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data

In electronic health (eHealth) research, limited insight has been obtained on process outcomes or how the use of technology has contributed to the users’ ability to have a healthier life, improved well-being, or activate new attitudes in their daily tasks. As a result, eHealth is often perceived as a black box. To open this black box of eHealth, methodologies must extend beyond the classic effect evaluations. The analyses of log data (anonymous records of real-time actions performed by each user) can provide continuous and objective insights into the actual usage of the technology. However, the possibilities of log data in eHealth research have not been exploited to their fullest extent. The aim of this paper is to describe how log data can be used to improve the evaluation and understand the use of eHealth technology with a broader approach than only descriptive statistics. This paper serves as a starting point for using log data analysis in eHealth research. Here, we describe what log data is and provide an overview of research questions to evaluate the system, the context, the users of a technology, as well as the underpinning theoretical constructs. We also explain the requirements for log data, the starting points for the data preparation, and methods for data collection. Finally, we describe methods for data analysis and draw a conclusion regarding the importance of the results for both scientific and practical applications. The analysis of log data can be of great value for opening the black box of eHealth. A deliberate log data analysis can give new insights into how the usage of the technology contributes to found effects and can thereby help to improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.

Whiplash Syndrome Reloaded: Digital Echoes of Whiplash Syndrome in the European Internet Search Engine Context

Background

In many Western countries, after a motor vehicle collision, those involved seek health care for the assessment of injuries and for insurance documentation purposes. In contrast, in many less wealthy countries, there may be limited access to care and no insurance or compensation system.

Objective

The purpose of this infodemiology study was to investigate the global pattern of evolving Internet usage in countries with and without insurance and the corresponding compensation systems for whiplash injury.

Methods

We used the Internet search engine analytics via Google Trends to study the health information-seeking behavior concerning whiplash injury at national population levels in Europe.

Results

We found that the search for “whiplash” is strikingly and consistently often associated with the search for “compensation” in countries or cultures with a tort system. Frequent or traumatic painful injuries; diseases or disorders such as arthritis, headache, radius, and hip fracture; depressive disorders; and fibromyalgia were not associated similarly with searches on “compensation.”

Conclusions

In this study, we present evidence from the evolving viewpoint of naturalistic Internet search engine analytics that the expectations for receiving compensation may influence Internet search behavior in relation to whiplash injury.

Understanding user-system interactions

This paper presents findings of a study that examines how the Online Public Access Catalogue (OPAC) of the University of Dar es Salaam library is used; the extent to which users succeed in locating information; and the reasons behind search failure. OPAC transaction logs were collected unobtrusively between January and December 2015, which were then subjected to transaction log analysis. The results indicate a relatively low use of OPAC, with default keyword search, author, title and subject terms being the most preferred access points. Only 102,037 (19.2%) of the search queries received zero hits. Reasons for search failure were multiple. They included spelling mistakes, use of wrong syntax, searching in inappropriate search fields, users’ lack of knowledge of Library of Congress Subject Headings (LCSH) and materials unavailability. Thus, the paper suggests redesigning OPAC interface to include features of the ‘next generation’ catalogue and exposing users to evidence-based information literacy.

From query analysis to user information needs: a study of campus map searches

Purpose

– Search engines and web applications have evolved to be more tailored toward individual user’s needs, including the individual’s personal preferences and geographic location. By integrating the free Google Maps Application Program Interface with locally stored metadata, the author created an interactive map search for users to locate, and navigate to, destinations on the University of New Mexico (UNM) campus. The purpose of this paper is to identify the characteristics of UNM map search queries, the options and prioritization of the metadata augmentation, and the usefulness and possible improvement of the interface.

Design/methodology/approach

– Queries, search date/time, and the number of results found were logged and examined. Queries’ search frequency and characteristics were analyzed and categorized.

Findings

– From November 1, 2012 to September 15, 2013, the author had a total 14,097 visits to the SearchUNM Maps page (http://search.unm.edu/maps/). There were total 5,868 searches (41 percent of all the page visits), and out of all the search instances, 2,297 of them (39 percent) did not retrieve any results. By analyzing the failed queries, the author was able to develop a strategy to increase successful searches.

Originality/value

– Many academic institutions have implemented interactive map searches for users to find locations and navigate on campus. However, to date there is no related research on how users conduct their searches in such a scope. Based on the query analysis, this paper identifies user’s search behavior and discusses the strategies of improving searches results of campus interactive maps.

Online information encountering: modeling the process and influencing factors

Purpose

– Information encountering is the serendipitous acquisition of information that requires low or no involvement and expectation of users. The purpose of this paper is to model the explicit process and the implicit factors of online information encountering, i.e. how and why it occurs.

Design/methodology/approach

– The critical incident technique was adopted to collect qualitative data from 16 interview participants. They contributed 27 true incidents of online information encountering which were used to identify the key phases of the encountering process. They also commented on the factors that they thought had an influence on the chance of the occurrence of encountering.

Findings

– The macro-process of information encountering is composed of three phases. First, browsing, searching, or social interaction provides the context for encountering; second, the encountering occurrence consists of three steps – noticing the stimuli, examining the content, and acquiring interesting or useful content; and third, the information encountered will be explored further, saved, used, or shared. The 14 influencing factors of information encountering obtained divide into three clusters. User-related factors include sensitivity, emotions, expertise, attitudes, intentionality, curiosity, activity diversity; information-related factors include type, relevance, quality, visibility, and sources; and environment-related factors include time limits and interface usability.

Originality/value

– This study engenders useful implications for designing information encountering experience. The changeable nature of some influencing factors suggests that encountering can be elicited through the purposive design of encountering support features or even encountering systems, and the macro-process depicts the natural occurring mechanisms of encountering for the design to follow.

Search engine queries used to locate electronic theses and dissertations

Purpose

– The purpose of this paper is to examine two research questions: What search engine queries lead users to the Auburn University electronic theses and dissertations (AUETDs) collection? Do these queries vary for users in different locations and, if so, how?

Design/methodology/approach

– Search engine queries used to locate the AUETDs collection were obtained from Google Analytics and were separated into groups based on user location. These queries were assigned to empirically derived categories based on their content.

Findings

– Most local users’ queries contained person names, variants for thesis or dissertation, and variants for Auburn University. Over a third were queries for the AUETDs collection, while the remainder were seeking theses and dissertations from specific Auburn researchers. Most out-of-state users’ queries contained title and subject keywords and appeared to be seeking specific research studies. Queries from users located within the state but outside of the local area were intermediate between these groups.

Practical implications

– Over two-thirds of visits to the AUETDs collection were made by search engine users which reinforces the importance of having repository content indexed by search engines such as Google. The specificity of their queries indicates that full-text indexing will be more helpful to users than metadata indexing alone.

Originality/value

– This is the first detailed analysis of search engine queries used to locate an ETDs collection. It may also be the last, as query content for the major search engines is no longer available from Google Analytics.

Machine Learning Approach to Search Query Classification

Search query classification is a necessary step for a number of information retrieval tasks. This chapter presents an approach to non-hierarchical classification of search queries that focuses on two specific areas of machine learning: short text classification and limited manual labeling. Typically, search queries are short, display little class specific information per single query and are therefore a weak source for traditional machine learning. To improve the effectiveness of the classification process the chapter introduces background knowledge discovery by using information retrieval techniques. The proposed approach is applied to a task of age classification of a corpus of queries from a commercial search engine. In the process, various classification scenarios are generated and executed, providing insight into choice, significance and range of tuning parameters.

The potential of the Internet for health communication: the use of an interactive on-line tool for diabetes risk prediction

OBJECTIVE

The aim of the study is to assess the use of a web-based tool to calculate the 5-year risk of developing type 2 diabetes mellitus, the German Diabetes Risk Score (DRS), in a real-life setting.

METHODS

Recorded web server log information and user entered data (n=24,495) was analyzed. To distinguish information seeking behavior between high- and low-risk DRS records, a sub-sample was created to compare data entries of the highest and lowest risk quintile (n=5046). To learn more on the information seeking behavior of DRS users we created a dataset on the characteristics of those who calculate more than one DRS (n=11,902).

RESULTS

Most users (61%) entered the DRS without a referring website. Forty percent of users did not know their waist circumference. Visit duration among high-risk DRS records was almost 1.5 min longer compared to low-risk DRS records (p-value<0.001). Those who calculated multiple DRS were associated with a higher first DRS and male gender.

CONCLUSION

Understanding users' website utilization is an important step to evaluate risk calculator websites.

PRACTICE IMPLICATIONS

A range of media should be used conjointly to raise type 2 diabetes mellitus risk awareness. Risk websites should be linked to patient information pages.

An analysis of clinical queries in an electronic health record search utility

PURPOSE

While search engines have become nearly ubiquitous on the Web, electronic health records (EHRs) generally lack search functionality; furthermore, there is no knowledge on how and what healthcare providers search while using an EHR-based search utility. In this study, we sought to understand user needs as captured by their search queries.

METHODS

This post-implementation study analyzed user search log files for 6 months from an EHR-based, free-text search utility at our large academic institution. The search logs were de-identified and then analyzed in two steps. First, two investigators classified all the unique queries as navigational, transactional, or informational searches. Second, three physician reviewers categorized a random sample of 357 informational searches into high-level semantic types derived from the Unified Medical Language System (UMLS). The reviewers were given overlapping data sets, such that two physicians reviewed each query.

RESULTS

We analyzed 2207 queries performed by 436 unique users over a 6-month period. Of the 2207 queries, 980 were unique queries. Users of the search utility included clinicians, researchers and administrative staff. Across the whole user population, approximately 14.5% of the user searches were navigational searches and 85.1% were informational. Within informational searches, we found that users predominantly searched for laboratory results and specific diseases.

CONCLUSIONS

A variety of user types, ranging from clinicians to administrative staff, took advantage of the EHR-based search utility. Though these users' search behavior differed, they predominantly performed informational searches related to laboratory results and specific diseases. Additionally, a number of queries were part of words, implying the need for a free-text module to be included in any future concept-based search algorithm.