Showing 1–50 of 51 results for author: Shaikh, S

BraTS-PEDs: Results of the Multi-Consortium International Pediatric Brain Tumor Segmentation Challenge 2023

Authors: Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Anna Zapaishchykova, Julija Pavaine, Lubdha M. Shah, Blaise V. Jones, Nakul Sheth, Sanjay P. Prabhu, Aaron S. McAllister, Wenxin Tu, Khanak K. Nandolia, Andres F. Rodriguez, Ibraheem Salman Shaikh, Mariana Sanchez Montano, Hollie Anne Lai, Maruf Adewole, Jake Albrecht, Udunna Anazodo, Hannah Anderson, Syed Muhammed Anwar, Alejandro Aristizabal, Sina Bagheri , et al. (54 additional authors not shown)

Abstract: Pediatric central nervous system tumors are the leading cause of cancer-related deaths in children. The five-year survival rate for high-grade glioma in children is less than 20%. The development of new treatments is dependent upon multi-institutional collaborative clinical trials requiring reproducible and accurate centralized response assessment. We present the results of the BraTS-PEDs 2023 cha… ▽ More Pediatric central nervous system tumors are the leading cause of cancer-related deaths in children. The five-year survival rate for high-grade glioma in children is less than 20%. The development of new treatments is dependent upon multi-institutional collaborative clinical trials requiring reproducible and accurate centralized response assessment. We present the results of the BraTS-PEDs 2023 challenge, the first Brain Tumor Segmentation (BraTS) challenge focused on pediatric brain tumors. This challenge utilized data acquired from multiple international consortia dedicated to pediatric neuro-oncology and clinical trials. BraTS-PEDs 2023 aimed to evaluate volumetric segmentation algorithms for pediatric brain gliomas from magnetic resonance imaging using standardized quantitative performance evaluation metrics employed across the BraTS 2023 challenges. The top-performing AI approaches for pediatric tumor analysis included ensembles of nnU-Net and Swin UNETR, Auto3DSeg, or nnU-Net with a self-supervised framework. The BraTSPEDs 2023 challenge fostered collaboration between clinicians (neuro-oncologists, neuroradiologists) and AI/imaging scientists, promoting faster data sharing and the development of automated volumetric analysis techniques. These advancements could significantly benefit clinical trials and improve the care of children with brain tumors. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Adaptive Randomized Smoothing: Certifying Multi-Step Defences against Adversarial Examples

Authors: Saiyue Lyu, Shadab Shaikh, Frederick Shpilevskiy, Evan Shelhamer, Mathias Lécuyer

Abstract: We propose Adaptive Randomized Smoothing (ARS) to certify the predictions of our test-time adaptive models against adversarial examples. ARS extends the analysis of randomized smoothing using f-Differential Privacy to certify the adaptive composition of multiple steps. For the first time, our theory covers the sound adaptive composition of general and high-dimensional functions of noisy input. We… ▽ More We propose Adaptive Randomized Smoothing (ARS) to certify the predictions of our test-time adaptive models against adversarial examples. ARS extends the analysis of randomized smoothing using f-Differential Privacy to certify the adaptive composition of multiple steps. For the first time, our theory covers the sound adaptive composition of general and high-dimensional functions of noisy input. We instantiate ARS on deep image classification to certify predictions against adversarial examples of bounded $L_{\infty}$ norm. In the $L_{\infty}$ threat model, our flexibility enables adaptation through high-dimensional input-dependent masking. We design adaptivity benchmarks, based on CIFAR-10 and CelebA, and show that ARS improves accuracy by $2$ to $5\%$ points. On ImageNet, ARS improves accuracy by $1$ to $3\%$ points over standard RS without adaptivity. △ Less

Submitted 14 June, 2024; originally announced June 2024.

EnterpriseEM: Fine-tuned Embeddings for Enterprise Semantic Search

Authors: Kamalkumar Rathinasamy, Jayarama Nettar, Amit Kumar, Vishal Manchanda, Arun Vijayakumar, Ayush Kataria, Venkateshprasanna Manjunath, Chidambaram GS, Jaskirat Singh Sodhi, Shoeb Shaikh, Wasim Akhtar Khan, Prashant Singh, Tanishq Dattatray Ige, Vipin Tiwari, Rajab Ali Mondal, Harshini K, S Reka, Chetana Amancharla, Faiz ur Rahman, Harikrishnan P A, Indraneel Saha, Bhavya Tiwary, Navin Shankar Patel, Pradeep T S, Balaji A J , et al. (2 additional authors not shown)

Abstract: Enterprises grapple with the significant challenge of managing proprietary unstructured data, hindering efficient information retrieval. This has led to the emergence of AI-driven information retrieval solutions, designed to adeptly extract relevant insights to address employee inquiries. These solutions often leverage pre-trained embedding models and generative models as foundational components.… ▽ More Enterprises grapple with the significant challenge of managing proprietary unstructured data, hindering efficient information retrieval. This has led to the emergence of AI-driven information retrieval solutions, designed to adeptly extract relevant insights to address employee inquiries. These solutions often leverage pre-trained embedding models and generative models as foundational components. While pre-trained embeddings may exhibit proximity or disparity based on their original training objectives, they might not fully align with the unique characteristics of enterprise-specific data, leading to suboptimal alignment with the retrieval goals of enterprise environments. In this paper, we propose a methodology to fine-tune pre-trained embedding models specifically for enterprise environments. By adapting the embeddings to better suit the retrieval tasks prevalent in enterprises, we aim to enhance the performance of information retrieval solutions. We discuss the process of fine-tuning, its effect on retrieval accuracy, and the potential benefits for enterprise information management. Our findings demonstrate the efficacy of fine-tuned embedding models in improving the precision and relevance of search results in enterprise settings. △ Less

Submitted 18 May, 2024; originally announced June 2024.

ACM Class: I.2.7

The Brain Tumor Segmentation in Pediatrics (BraTS-PEDs) Challenge: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)

Authors: Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Deep Gandhi, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, Ujjwal Baid, Timothy Bergquist, Austin J. Borja, Evan Calabrese, Verena Chung, Gian-Marco Conte, Farouk Dako, James Eddy, Ivan Ezhov, Ariana Familiar, Keyvan Farahani, Andrea Franson, Anurag Gottipati, Shuvanjan Haldar, Juan Eugenio Iglesias , et al. (46 additional authors not shown)

Abstract: Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. Here we pr… ▽ More Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. Here we present the CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs challenge, focused on pediatric brain tumors with data acquired across multiple international consortia dedicated to pediatric neuro-oncology and clinical trials. The CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs challenge brings together clinicians and AI/imaging scientists to lead to faster development of automated segmentation techniques that could benefit clinical trials, and ultimately the care of children with brain tumors. △ Less

Submitted 11 July, 2024; v1 submitted 23 April, 2024; originally announced April 2024.

Comments: arXiv admin note: substantial text overlap with arXiv:2305.17033

Predicting Next Useful Location With Context-Awareness: The State-Of-The-Art

Authors: Alireza Nezhadettehad, Arkady Zaslavsky, Rakib Abdur, Siraj Ahmed Shaikh, Seng W. Loke, Guang-Li Huang, Alireza Hassani

Abstract: Predicting the future location of mobile objects reinforces location-aware services with proactive intelligence and helps businesses and decision-makers with better planning and near real-time scheduling in different applications such as traffic congestion control, location-aware advertisements, and monitoring public health and well-being. The recent developments in the smartphone and location sen… ▽ More Predicting the future location of mobile objects reinforces location-aware services with proactive intelligence and helps businesses and decision-makers with better planning and near real-time scheduling in different applications such as traffic congestion control, location-aware advertisements, and monitoring public health and well-being. The recent developments in the smartphone and location sensors technology and the prevalence of using location-based social networks alongside the improvements in artificial intelligence and machine learning techniques provide an excellent opportunity to exploit massive amounts of historical and real-time contextual information to recognise mobility patterns and achieve more accurate and intelligent predictions. This survey provides a comprehensive overview of the next useful location prediction problem with context-awareness. First, we explain the concepts of context and context-awareness and define the next location prediction problem. Then we analyse nearly thirty studies in this field concerning the prediction method, the challenges addressed, the datasets and metrics used for training and evaluating the model, and the types of context incorporated. Finally, we discuss the advantages and disadvantages of different approaches, focusing on the usefulness of the predicted location and identifying the open challenges and future work on this subject by introducing two potential use cases of next location prediction in the automotive industry. △ Less

Submitted 15 January, 2024; originally announced January 2024.

The Rise of Creative Machines: Exploring the Impact of Generative AI

Authors: Saad Shaikh, Rajat bendre, Sakshi Mhaske

Abstract: This study looks at how generative artificial intelligence (AI) can revolutionize marketing, product development, and research. It discusses the latest developments in the field, easy-to-use resources, and moral and social hazards. In addition to addressing mitigating techniques for issues like prejudice and disinformation, the debate emphasizes the significance of responsible development through… ▽ More This study looks at how generative artificial intelligence (AI) can revolutionize marketing, product development, and research. It discusses the latest developments in the field, easy-to-use resources, and moral and social hazards. In addition to addressing mitigating techniques for issues like prejudice and disinformation, the debate emphasizes the significance of responsible development through continual stakeholder communication and ethical principles. △ Less

Submitted 22 November, 2023; originally announced November 2023.

Comments: The impact of generative AI on research, product creation, ethical concerns etc is examined in this six-page article. Figures 1, 2, and 3, which are essential to the analysis, are included in the discussion along with opportunities, hazards, and ethical considerations

ACM Class: I.2.7

Finite Element Analysis and Machine Learning Guided Design of Carbon Fiber Organosheet-based Battery Enclosures for Crashworthiness

Authors: Shadab Anwar Shaikh, M. F. N. Taufique, Kranthi, Balusu, Shank S. Kulkarni, Forrest Hale, Jonathan Oleson, Ram Devanathan, Ayoub Soulami

Abstract: Carbon fiber composite can be a potential candidate for replacing metal-based battery enclosures of current electric vehicles (E.V.s) owing to its better strength-to-weight ratio and corrosion resistance. However, the strength of carbon fiber-based structures depends on several parameters that should be carefully chosen. In this work, we implemented high throughput finite element analysis (FEA) ba… ▽ More Carbon fiber composite can be a potential candidate for replacing metal-based battery enclosures of current electric vehicles (E.V.s) owing to its better strength-to-weight ratio and corrosion resistance. However, the strength of carbon fiber-based structures depends on several parameters that should be carefully chosen. In this work, we implemented high throughput finite element analysis (FEA) based thermoforming simulation to virtually manufacture the battery enclosure using different design and processing parameters. Subsequently, we performed virtual crash simulations to mimic a side pole crash to evaluate the crashworthiness of the battery enclosures. This high throughput crash simulation dataset was utilized to build predictive models to understand the crashworthiness of an unknown set. Our machine learning (ML) models showed excellent performance (R2 > 0.97) in predicting the crashworthiness metrics, i.e., crush load efficiency, absorbed energy, intrusion, and maximum deceleration during a crash. We believe that this FEA-ML work framework will be helpful in down select process parameters for carbon fiber-based component design and can be transferrable to other manufacturing technologies. △ Less

Submitted 22 August, 2023; originally announced September 2023.

Data augmentation and refinement for recommender system: A semi-supervised approach using maximum margin matrix factorization

Authors: Shamal Shaikh, Venkateswara Rao Kagita, Vikas Kumar, Arun K Pujari

Abstract: Collaborative filtering (CF) has become a popular method for developing recommender systems (RSs) where ratings of a user for new items are predicted based on her past preferences and available preference information of other users. Despite the popularity of CF-based methods, their performance is often greatly limited by the sparsity of observed entries. In this study, we explore the data augmenta… ▽ More Collaborative filtering (CF) has become a popular method for developing recommender systems (RSs) where ratings of a user for new items are predicted based on her past preferences and available preference information of other users. Despite the popularity of CF-based methods, their performance is often greatly limited by the sparsity of observed entries. In this study, we explore the data augmentation and refinement aspects of Maximum Margin Matrix Factorization (MMMF), a widely accepted CF technique for rating predictions, which has not been investigated before. We exploit the inherent characteristics of CF algorithms to assess the confidence level of individual ratings and propose a semi-supervised approach for rating augmentation based on self-training. We hypothesize that any CF algorithm's predictions with low confidence are due to some deficiency in the training data and hence, the performance of the algorithm can be improved by adopting a systematic data augmentation strategy. We iteratively use some of the ratings predicted with high confidence to augment the training data and remove low-confidence entries through a refinement process. By repeating this process, the system learns to improve prediction accuracy. Our method is experimentally evaluated on several state-of-the-art CF algorithms and leads to informative rating augmentation, improving the performance of the baseline approaches. △ Less

Submitted 30 September, 2023; v1 submitted 22 June, 2023; originally announced June 2023.

Comments: 21 pages

The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)

Authors: Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, Sina Bagheri, Ujjwal Baid, Timothy Bergquist, Austin J. Borja, Evan Calabrese, Verena Chung, Gian-Marco Conte, Farouk Dako, James Eddy, Ivan Ezhov, Ariana Familiar, Keyvan Farahani, Shuvanjan Haldar, Juan Eugenio Iglesias, Anastasia Janas , et al. (48 additional authors not shown)

Abstract: Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20\%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. The MICCA… ▽ More Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20\%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. The MICCAI Brain Tumor Segmentation (BraTS) Challenge is a landmark community benchmark event with a successful history of 12 years of resource creation for the segmentation and analysis of adult glioma. Here we present the CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs 2023 challenge, which represents the first BraTS challenge focused on pediatric brain tumors with data acquired across multiple international consortia dedicated to pediatric neuro-oncology and clinical trials. The BraTS-PEDs 2023 challenge focuses on benchmarking the development of volumentric segmentation algorithms for pediatric brain glioma through standardized quantitative performance evaluation metrics utilized across the BraTS 2023 cluster of challenges. Models gaining knowledge from the BraTS-PEDs multi-parametric structural MRI (mpMRI) training data will be evaluated on separate validation and unseen test mpMRI dataof high-grade pediatric glioma. The CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs 2023 challenge brings together clinicians and AI/imaging scientists to lead to faster development of automated segmentation techniques that could benefit clinical trials, and ultimately the care of children with brain tumors. △ Less

Submitted 23 May, 2024; v1 submitted 26 May, 2023; originally announced May 2023.

Rescuing the End-user systems from Vulnerable Applications using Virtualization Techniques

Authors: Vinayak Trivedi, Tushar Gurjar, Sumaiya Shaikh, Saketh Maddamsetty, Debadatta Mishra

Abstract: In systems owned by normal end-users, many times security attacks are mounted by sneaking in malicious applications or exploiting existing software vulnerabilities through security non-conforming actions of users. Virtualization approaches can address this problem by providing a quarantine environment for applications, malicious devices, and device drivers, which are mostly used as entry points fo… ▽ More In systems owned by normal end-users, many times security attacks are mounted by sneaking in malicious applications or exploiting existing software vulnerabilities through security non-conforming actions of users. Virtualization approaches can address this problem by providing a quarantine environment for applications, malicious devices, and device drivers, which are mostly used as entry points for security attacks. However, the existing methods to provide quarantine environments using virtualization are not transparent to the user, both in terms of application interface transparency and file system transparency. Further, software configuration level solutions like remote desktops and remote application access mechanisms combined with shared file systems do not meet the user transparency and security requirements. We propose qOS, a VM-based solution combined with certain OS extensions to meet the security requirements of end-point systems owned by normal users, in a transparent and efficient manner. We demonstrate the efficacy of qOS by empirically evaluating the prototype implementation in the Linux+KVM system in terms of efficiency, security, and user transparency. △ Less

Submitted 4 November, 2022; originally announced November 2022.

Comments: 14 pages, 9 figures

Big Tech Companies Impact on Research at the Faculty of Information Technology and Electrical Engineering

Authors: Ahmad Hassanpour, An Thi Nguyen, Anshul Rani, Sarang Shaikh, Ying Xu, Haoyu Zhang

Abstract: Artificial intelligence is gaining momentum, ongoing pandemic is fuel to that with more opportunities in every sector specially in health and education sector. But with the growth in technology, challenges associated with ethics also grow (Katharine Schwab, 2021). Whenever a new AI product is developed, companies publicize that their systems are transparent, fair, and are in accordance with the ex… ▽ More Artificial intelligence is gaining momentum, ongoing pandemic is fuel to that with more opportunities in every sector specially in health and education sector. But with the growth in technology, challenges associated with ethics also grow (Katharine Schwab, 2021). Whenever a new AI product is developed, companies publicize that their systems are transparent, fair, and are in accordance with the existing laws and regulations as the methods and procedures followed by a big tech company for ensuring AI ethics, not only affect the trust and perception of public, but it also challenges the capabilities of the companies towards business strategies in different regions, and the kind of brains it can attract for their projects. AI Big Tech companies have influence over AI ethics as many influencing ethical-AI researchers have roots in Big Tech or its associated labs. △ Less

Submitted 10 April, 2022; originally announced May 2022.

BankNote-Net: Open dataset for assistive universal currency recognition

Authors: Felipe Oviedo, Srinivas Vinnakota, Eugene Seleznev, Hemant Malhotra, Saqib Shaikh, Juan Lavista Ferres

Abstract: Millions of people around the world have low or no vision. Assistive software applications have been developed for a variety of day-to-day tasks, including optical character recognition, scene identification, person recognition, and currency recognition. This last task, the recognition of banknotes from different denominations, has been addressed by the use of computer vision models for image reco… ▽ More Millions of people around the world have low or no vision. Assistive software applications have been developed for a variety of day-to-day tasks, including optical character recognition, scene identification, person recognition, and currency recognition. This last task, the recognition of banknotes from different denominations, has been addressed by the use of computer vision models for image recognition. However, the datasets and models available for this task are limited, both in terms of dataset size and in variety of currencies covered. In this work, we collect a total of 24,826 images of banknotes in variety of assistive settings, spanning 17 currencies and 112 denominations. Using supervised contrastive learning, we develop a machine learning model for universal currency recognition. This model learns compliant embeddings of banknote images in a variety of contexts, which can be shared publicly (as a compressed vector representation), and can be used to train and test specialized downstream models for any currency, including those not covered by our dataset or for which only a few real images per denomination are available (few-shot learning). We deploy a variation of this model for public use in the last version of the Seeing AI app developed by Microsoft. We share our encoder model and the embeddings as an open dataset in our BankNote-Net repository. △ Less

Submitted 7 April, 2022; originally announced April 2022.

Comments: Pre-print

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

Authors: Shikib Mehri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, Chen Zhang

Abstract: This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research. This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research. △ Less

Submitted 18 March, 2022; originally announced March 2022.

Comments: Report from the NSF AED Workshop (http://dialrc.org/AED/)

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

Authors: Erfan Al-Hossami, Samira Shaikh

Abstract: In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also known as Code Intelligence (CI) and Programming Language Processing (PLP). We survey over 287 publications and present a software-engineering centered taxon… ▽ More In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also known as Code Intelligence (CI) and Programming Language Processing (PLP). We survey over 287 publications and present a software-engineering centered taxonomy for CI placing each of the works into one category describing how it best assists the software development cycle. Then, we overview the field of conversational assistants and their applications in software engineering and education. Lastly, we highlight research opportunities at the intersection of AI for code and conversational assistants and provide future directions for researching conversational assistants with CI capabilities. △ Less

Submitted 10 February, 2022; originally announced February 2022.

Comments: 55 pages, 16 Figures, 4 Tables

ACM Class: I.2.2; I.2.7; K.3.1

Can We Generate Shellcodes via Natural Language? An Empirical Study

Authors: Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, Samira Shaikh

Abstract: Writing software exploits is an important practice for offensive security analysts to investigate and prevent attacks. In particular, shellcodes are especially time-consuming and a technical challenge, as they are written in assembly language. In this work, we address the task of automatically generating shellcodes, starting purely from descriptions in natural language, by proposing an approach ba… ▽ More Writing software exploits is an important practice for offensive security analysts to investigate and prevent attacks. In particular, shellcodes are especially time-consuming and a technical challenge, as they are written in assembly language. In this work, we address the task of automatically generating shellcodes, starting purely from descriptions in natural language, by proposing an approach based on Neural Machine Translation (NMT). We then present an empirical study using a novel dataset (Shellcode_IA32), which consists of 3,200 assembly code snippets of real Linux/x86 shellcodes from public databases, annotated using natural language. Moreover, we propose novel metrics to evaluate the accuracy of NMT at generating shellcodes. The empirical analysis shows that NMT can generate assembly code snippets from the natural language with high accuracy and that in many cases can generate entire shellcodes with no errors. △ Less

Submitted 8 February, 2022; originally announced February 2022.

Comments: 33 pages, 5 figures, 9 tables. To be published in Automated Software Engineering journal

The Study of Peer Assessment Impact on Group Learning Activities

Authors: Zhiyuan Chen, Soon Boon Lee, Shazia Paras Shaikh, Mirza Rayana Sanzana

Abstract: Comparing with lecturer marked assessments, peer assessment is a more comprehensive learning process and many of the associated problems have occurred. In this research work, we study the peer-assessment impact on group learning activities in order to provide a complete and systematic review, increase the practice and quality of the peer assessment process. Pilot studies were conducted and took th… ▽ More Comparing with lecturer marked assessments, peer assessment is a more comprehensive learning process and many of the associated problems have occurred. In this research work, we study the peer-assessment impact on group learning activities in order to provide a complete and systematic review, increase the practice and quality of the peer assessment process. Pilot studies were conducted and took the form of surveys, focus group interviews, and questionnaires. Prelimi-nary surveys were conducted with 582 students and 276 responses were received, giving a response rate of 47.4%. The results show 37% student will choose individual work over group work if given the choice. In the case study, 82.1% of the total of 28 students have en-joyed working in a group using Facebook as communication tools. 89.3% of the students can demonstrate their skills through group-working and most importantly, 82.1% of them agree that peer assess-ment is an impartial method of assessment with the help of Facebook as proof of self-contribution. Our suggestions to make group work a pleasant experience are by identifying and taking action against the freeloader, giving credit to the deserving students, educating students on how to give constructive feedback and making the assessment pro-cess transparent to all. △ Less

Submitted 7 January, 2022; originally announced January 2022.

Comments: Regular Research Paper Accepted by FECS'21 (The 17th Int'l Conf on Frontiers in Education: Computer Science and Computer Engineering)

EVIL: Exploiting Software via Natural Language

Authors: Pietro Liguori, Erfan Al-Hossami, Vittorio Orbinato, Roberto Natella, Samira Shaikh, Domenico Cotroneo, Bojan Cukic

Abstract: Writing exploits for security assessment is a challenging task. The writer needs to master programming and obfuscation techniques to develop a successful exploit. To make the task easier, we propose an approach (EVIL) to automatically generate exploits in assembly/Python language from descriptions in natural language. The approach leverages Neural Machine Translation (NMT) techniques and a dataset… ▽ More Writing exploits for security assessment is a challenging task. The writer needs to master programming and obfuscation techniques to develop a successful exploit. To make the task easier, we propose an approach (EVIL) to automatically generate exploits in assembly/Python language from descriptions in natural language. The approach leverages Neural Machine Translation (NMT) techniques and a dataset that we developed for this work. We present an extensive experimental study to evaluate the feasibility of EVIL, using both automatic and manual analysis, and both at generating individual statements and entire exploits. The generated code achieved high accuracy in terms of syntactic and semantic correctness. △ Less

Submitted 1 September, 2021; originally announced September 2021.

Comments: Paper accepted at the 32nd International Symposium on Software Reliability Engineering (ISSRE 2021)

Shellcode_IA32: A Dataset for Automatic Shellcode Generation

Authors: Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, Samira Shaikh

Abstract: We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode_IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with stan… ▽ More We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode_IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with standard methods in neural machine translation (NMT) to establish baseline performance levels on this task. △ Less

Submitted 18 March, 2022; v1 submitted 27 April, 2021; originally announced April 2021.

Comments: Paper accepted to NLP4Prog Workshop 2021 co-located with ACL-IJCNLP 2021. Extended journal version of this work has been published in the Automated Software Engineering journal, Volume 29, Article no. 30, March 2022, DOI: 10.1007/s10515-022-00331-3

Testing Framework for Black-box AI Models

Authors: Aniya Aggarwal, Samiulla Shaikh, Sandeep Hans, Swastik Haldar, Rema Ananthanarayanan, Diptikalyan Saha

Abstract: With widespread adoption of AI models for important decision making, ensuring reliability of such models remains an important challenge. In this paper, we present an end-to-end generic framework for testing AI Models which performs automated test generation for different modalities such as text, tabular, and time-series data and across various properties such as accuracy, fairness, and robustness.… ▽ More With widespread adoption of AI models for important decision making, ensuring reliability of such models remains an important challenge. In this paper, we present an end-to-end generic framework for testing AI Models which performs automated test generation for different modalities such as text, tabular, and time-series data and across various properties such as accuracy, fairness, and robustness. Our tool has been used for testing industrial AI models and was very effective to uncover issues present in those models. Demo video link: https://youtu.be/984UCU17YZI △ Less

Submitted 11 February, 2021; originally announced February 2021.

Comments: 4 pages Demonstrations track paper accepted at ICSE 2021

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Authors: Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak , et al. (31 additional authors not shown)

Abstract: We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it… ▽ More We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate. △ Less

Submitted 1 April, 2021; v1 submitted 2 February, 2021; originally announced February 2021.

Human Computations in Citizen Crowds: A Knowledge Management Solution Framework

Authors: Nadeem Kafi, Zubair Ahmed Shaikh, Muhammad Shahid Shaikh

Abstract: KG (Knowledge Generation) and understanding have traditionally been a Human-centric activity. KE (Knowledge Engineering) and KM (Knowledge Management) have tried to augment human knowledge on two separate planes: the first deals with machine interpretation of knowledge while the later explore interactions in human networks for KG and understanding. However, both remain computer-centric. Crowdsourc… ▽ More KG (Knowledge Generation) and understanding have traditionally been a Human-centric activity. KE (Knowledge Engineering) and KM (Knowledge Management) have tried to augment human knowledge on two separate planes: the first deals with machine interpretation of knowledge while the later explore interactions in human networks for KG and understanding. However, both remain computer-centric. Crowdsourced HC (Human Computations) have recently utilized human cognition and memory to generate diverse knowledge streams on specific tasks, which are mostly easy for humans to solve but remain challenging for machine algorithms. Literature shows little work on KM frameworks for citizen crowds, which gather input from the diverse category of Humans, organize that knowledge concerning tasks and knowledge categories and recreate new knowledge as a computer-centric activity. In this paper, we present an attempt to create a framework by implementing a simple solution, called ExamCheck, to focus on the generation of knowledge, feedback on that knowledge and recording the results of that knowledge in academic settings. Our solution, based on HC, shows that a structured KM framework can address a complex problem in a context that is important for participants themselves. △ Less

Submitted 27 November, 2020; originally announced November 2020.

Journal ref: Mehran University Research Journal of Engineering & Technology, Vol. 37, No. 3, 513-528 July 2018, p-ISSN: 0254-7821, e-ISSN: 2413-7219

Learning to Plan and Realize Separately for Open-Ended Dialogue Systems

Authors: Sashank Santhanam, Zhuo Cheng, Brodie Mather, Bonnie Dorr, Archna Bhatia, Bryanna Hebenstreit, Alan Zemel, Adam Dalton, Tomek Strzalkowski, Samira Shaikh

Abstract: Achieving true human-like ability to conduct a conversation remains an elusive goal for open-ended dialogue systems. We posit this is because extant approaches towards natural language generation (NLG) are typically construed as end-to-end architectures that do not adequately model human generation processes. To investigate, we decouple generation into two separate phases: planning and realization… ▽ More Achieving true human-like ability to conduct a conversation remains an elusive goal for open-ended dialogue systems. We posit this is because extant approaches towards natural language generation (NLG) are typically construed as end-to-end architectures that do not adequately model human generation processes. To investigate, we decouple generation into two separate phases: planning and realization. In the planning phase, we train two planners to generate plans for response utterances. The realization phase uses response plans to produce an appropriate response. Through rigorous evaluations, both automated and human, we demonstrate that decoupling the process into planning and realization performs better than an end-to-end approach. △ Less

Submitted 4 October, 2020; v1 submitted 25 September, 2020; originally announced September 2020.

Comments: Accepted at EMNLP 2020 (Findings)

The Panacea Threat Intelligence and Active Defense Platform

Authors: Adam Dalton, Ehsan Aghaei, Ehab Al-Shaer, Archna Bhatia, Esteban Castillo, Zhuo Cheng, Sreekar Dhaduvai, Qi Duan, Md Mazharul Islam, Younes Karimi, Amir Masoumzadeh, Brodie Mather, Sashank Santhanam, Samira Shaikh, Tomek Strzalkowski, Bonnie J. Dorr

Abstract: We describe Panacea, a system that supports natural language processing (NLP) components for active defenses against social engineering attacks. We deploy a pipeline of human language technology, including Ask and Framing Detection, Named Entity Recognition, Dialogue Engineering, and Stylometry. Panacea processes modern message formats through a plug-in architecture to accommodate innovative appro… ▽ More We describe Panacea, a system that supports natural language processing (NLP) components for active defenses against social engineering attacks. We deploy a pipeline of human language technology, including Ask and Framing Detection, Named Entity Recognition, Dialogue Engineering, and Stylometry. Panacea processes modern message formats through a plug-in architecture to accommodate innovative approaches for message analysis, knowledge representation and dialogue generation. The novelty of the Panacea system is that uses NLP for cyber defense and engages the attacker using bots to elicit evidence to attribute to the attacker and to waste the attacker's time and resources. △ Less

Submitted 20 April, 2020; originally announced April 2020.

Comments: Accepted at STOC

Adaptation of a Lexical Organization for Social Engineering Detection and Response Generation

Authors: Archna Bhatia, Adam Dalton, Brodie Mather, Sashank Santhanam, Samira Shaikh, Alan Zemel, Tomek Strzalkowski, Bonnie J. Dorr

Abstract: We present a paradigm for extensible lexicon development based on Lexical Conceptual Structure to support social engineering detection and response generation. We leverage the central notions of ask (elicitation of behaviors such as providing access to money) and framing (risk/reward implied by the ask). We demonstrate improvements in ask/framing detection through refinements to our lexical organi… ▽ More We present a paradigm for extensible lexicon development based on Lexical Conceptual Structure to support social engineering detection and response generation. We leverage the central notions of ask (elicitation of behaviors such as providing access to money) and framing (risk/reward implied by the ask). We demonstrate improvements in ask/framing detection through refinements to our lexical organization and show that response generation qualitatively improves as ask/framing detection performance improves. The paradigm presents a systematic and efficient approach to resource adaptation for improved task-specific performance. △ Less

Submitted 20 April, 2020; originally announced April 2020.

Comments: Accepted at STOC

GeoFlink: A Distributed and Scalable Framework for the Real-time Processing of Spatial Streams

Authors: Salman Ahmed Shaikh, Komal Mariam, Hiroyuki Kitagawa, Kyoung-Sook Kim

Abstract: Apache Flink is an open-source system for scalable processing of batch and streaming data. Flink does not natively support efficient processing of spatial data streams, which is a requirement of many applications dealing with spatial data. Besides Flink, other scalable spatial data processing platforms including GeoSpark, Spatial Hadoop, etc. do not support streaming workloads and can only handle… ▽ More Apache Flink is an open-source system for scalable processing of batch and streaming data. Flink does not natively support efficient processing of spatial data streams, which is a requirement of many applications dealing with spatial data. Besides Flink, other scalable spatial data processing platforms including GeoSpark, Spatial Hadoop, etc. do not support streaming workloads and can only handle static/batch workloads. To fill this gap, we present GeoFlink, which extends Apache Flink to support spatial data types, indexes and continuous queries over spatial data streams. To enable the efficient processing of spatial continuous queries and for the effective data distribution across Flink cluster nodes, a gird-based index is introduced. GeoFlink currently supports spatial range, spatial $k$NN and spatial join queries on point data type. An extensive experimental study on real spatial data streams shows that GeoFlink achieves significantly higher query throughput than ordinary Flink processing. △ Less

Submitted 2 August, 2020; v1 submitted 7 April, 2020; originally announced April 2020.

Comments: CIKM 2020 Preprint

Detecting Asks in SE attacks: Impact of Linguistic and Structural Knowledge

Authors: Bonnie J. Dorr, Archna Bhatia, Adam Dalton, Brodie Mather, Bryanna Hebenstreit, Sashank Santhanam, Zhuo Cheng, Samira Shaikh, Alan Zemel, Tomek Strzalkowski

Abstract: Social engineers attempt to manipulate users into undertaking actions such as downloading malware by clicking links or providing access to money or sensitive information. Natural language processing, computational sociolinguistics, and media-specific structural clues provide a means for detecting both the ask (e.g., buy gift card) and the risk/reward implied by the ask, which we call framing (e.g.… ▽ More Social engineers attempt to manipulate users into undertaking actions such as downloading malware by clicking links or providing access to money or sensitive information. Natural language processing, computational sociolinguistics, and media-specific structural clues provide a means for detecting both the ask (e.g., buy gift card) and the risk/reward implied by the ask, which we call framing (e.g., lose your job, get a raise). We apply linguistic resources such as Lexical Conceptual Structure to tackle ask detection and also leverage structural clues such as links and their proximity to identified asks to improve confidence in our results. Our experiments indicate that the performance of ask detection, framing detection, and identification of the top ask is improved by linguistically motivated classes coupled with structural clues such as links. Our approach is implemented in a system that informs users about social engineering risk situations. △ Less

Submitted 25 February, 2020; originally announced February 2020.

Comments: Accepted at AAAI 2020

Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents

Authors: Sashank Santhanam, Alireza Karduni, Samira Shaikh

Abstract: Humans quite frequently interact with conversational agents. The rapid advancement in generative language modeling through neural networks has helped advance the creation of intelligent conversational agents. Researchers typically evaluate the output of their models through crowdsourced judgments, but there are no established best practices for conducting such studies. Moreover, it is unclear if c… ▽ More Humans quite frequently interact with conversational agents. The rapid advancement in generative language modeling through neural networks has helped advance the creation of intelligent conversational agents. Researchers typically evaluate the output of their models through crowdsourced judgments, but there are no established best practices for conducting such studies. Moreover, it is unclear if cognitive biases in decision-making are affecting crowdsourced workers' judgments when they undertake these tasks. To investigate, we conducted a between-subjects study with 77 crowdsourced workers to understand the role of cognitive biases, specifically anchoring bias, when humans are asked to evaluate the output of conversational agents. Our results provide insight into how best to evaluate conversational agents. We find increased consistency in ratings across two experimental conditions may be a result of anchoring bias. We also determine that external factors such as time and prior experience in similar tasks have effects on inter-rater consistency. △ Less

Submitted 26 February, 2020; v1 submitted 18 February, 2020; originally announced February 2020.

Comments: Accepted at CHI 2020

'Alexa, Do You Know Anything?' The Impact of an Intelligent Assistant on Team Interactions and Creative Performance Under Time Scarcity

Authors: Sonia Jawaid Shaikh, Ignacio Cruz

Abstract: Human-AI collaboration is on the rise with the deployment of AI-enabled intelligent assistants (e.g. Amazon Echo, Cortana, Siri, etc.) across organizational contexts. It is claimed that intelligent assistants can help people achieve more in less time (Personal Digital Assistant - Cortana, n.d.). However, despite the increasing presence of intelligent assistants in collaborative settings, there is… ▽ More Human-AI collaboration is on the rise with the deployment of AI-enabled intelligent assistants (e.g. Amazon Echo, Cortana, Siri, etc.) across organizational contexts. It is claimed that intelligent assistants can help people achieve more in less time (Personal Digital Assistant - Cortana, n.d.). However, despite the increasing presence of intelligent assistants in collaborative settings, there is a void in the literature on how the deployment of this technology intersects with time scarcity to impact team behaviors and performance. To fill this gap in the literature, we collected behavioral data from 56 teams who participated in a between-subjects 2 (Intelligent Assistant: Available vs. Not Available) x 2 (Time: Scarce vs. Not Scarce/Control) lab experiment. The results show that teams with an intelligent assistant had significantly fewer interactions between its members compared to teams without an intelligent assistant. Teams who faced time scarcity also used the intelligent assistant more often to seek its assistance during task completion compared to those in the control condition. Lastly, teams with an intelligent assistant underperformed on a creative task compared to those without the device. We discuss implications of this technology from theoretical, empirical, and practical perspectives. △ Less

Submitted 30 December, 2019; originally announced December 2019.

Comments: 29 pages, 1 figure

Natural Language Generation Using Reinforcement Learning with External Rewards

Authors: Vidhushini Srinivasan, Sashank Santhanam, Samira Shaikh

Abstract: We propose an approach towards natural language generation using a bidirectional encoder-decoder which incorporates external rewards through reinforcement learning (RL). We use attention mechanism and maximum mutual information as an initial objective function using RL. Using a two-part training scheme, we train an external reward analyzer to predict the external rewards and then use the predicted… ▽ More We propose an approach towards natural language generation using a bidirectional encoder-decoder which incorporates external rewards through reinforcement learning (RL). We use attention mechanism and maximum mutual information as an initial objective function using RL. Using a two-part training scheme, we train an external reward analyzer to predict the external rewards and then use the predicted rewards to maximize the expected rewards (both internal and external). We evaluate the system on two standard dialogue corpora - Cornell Movie Dialog Corpus and Yelp Restaurant Review Corpus. We report standard evaluation metrics including BLEU, ROUGE-L, and perplexity as well as human evaluation to validate our approach. △ Less

Submitted 26 November, 2019; originally announced November 2019.

Comments: Oral Presentation at ICMLA 2019

Emotional Neural Language Generation Grounded in Situational Contexts

Authors: Sashank Santhanam, Samira Shaikh

Abstract: Emotional language generation is one of the keys to human-like artificial intelligence. Humans use different type of emotions depending on the situation of the conversation. Emotions also play an important role in mediating the engagement level with conversational partners. However, current conversational agents do not effectively account for emotional content in the language generation process. T… ▽ More Emotional language generation is one of the keys to human-like artificial intelligence. Humans use different type of emotions depending on the situation of the conversation. Emotions also play an important role in mediating the engagement level with conversational partners. However, current conversational agents do not effectively account for emotional content in the language generation process. To address this problem, we develop a language modeling approach that generates affective content when the dialogue is situated in a given context. We use the recently released Empathetic-Dialogues corpus to build our models. Through detailed experiments, we find that our approach outperforms the state-of-the-art method on the perplexity metric by about 5 points and achieves a higher BLEU metric score. △ Less

Submitted 25 November, 2019; originally announced November 2019.

Comments: Oral Presentation at CCNLG 2019

Towards Best Experiment Design for Evaluating Dialogue System Output

Authors: Sashank Santhanam, Samira Shaikh

Abstract: To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer from the inconsistency of ratings, extant research has also found that the design of the evaluation task affects the consistency and quality of human judgments.… ▽ More To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer from the inconsistency of ratings, extant research has also found that the design of the evaluation task affects the consistency and quality of human judgments. We conduct a between-subjects study to understand the impact of four experiment conditions on human ratings of dialogue system output. In addition to discrete and continuous scale ratings, we also experiment with a novel application of Best-Worst scaling to dialogue evaluation. Through our systematic study with 40 crowdsourced workers in each task, we find that using continuous scales achieves more consistent ratings than Likert scale or ranking-based experiment design. Additionally, we find that factors such as time taken to complete the task and no prior experience of participating in similar studies of rating dialogue system output positively impact consistency and agreement amongst raters △ Less

Submitted 22 September, 2019; originally announced September 2019.

Comments: Accepted at INLG 2019

I Stand With You: Using Emojis to Study Solidarity in Crisis Events

Authors: Sashank Santhanam, Vidhushini Srinivasan, Shaina Glass, Samira Shaikh

Abstract: We study how emojis are used to express solidarity in social media in the context of two major crisis events - a natural disaster, Hurricane Irma in 2017 and terrorist attacks that occurred on November 2015 in Paris. Using annotated corpora, we first train a recurrent neural network model to classify expressions of solidarity in text. Next, we use these expressions of solidarity to characterize hu… ▽ More We study how emojis are used to express solidarity in social media in the context of two major crisis events - a natural disaster, Hurricane Irma in 2017 and terrorist attacks that occurred on November 2015 in Paris. Using annotated corpora, we first train a recurrent neural network model to classify expressions of solidarity in text. Next, we use these expressions of solidarity to characterize human behavior in online social networks, through the temporal and geospatial diffusion of emojis. Our analysis reveals that emojis are a powerful indicator of sociolinguistic behaviors (solidarity) that are exhibited on social media as the crisis events unfold. △ Less

Submitted 18 July, 2019; originally announced July 2019.

A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions

Authors: Sashank Santhanam, Samira Shaikh

Abstract: One of the hardest problems in the area of Natural Language Processing and Artificial Intelligence is automatically generating language that is coherent and understandable to humans. Teaching machines how to converse as humans do falls under the broad umbrella of Natural Language Generation. Recent years have seen unprecedented growth in the number of research articles published on this subject in… ▽ More One of the hardest problems in the area of Natural Language Processing and Artificial Intelligence is automatically generating language that is coherent and understandable to humans. Teaching machines how to converse as humans do falls under the broad umbrella of Natural Language Generation. Recent years have seen unprecedented growth in the number of research articles published on this subject in conferences and journals both by academic and industry researchers. There have also been several workshops organized alongside top-tier NLP conferences dedicated specifically to this problem. All this activity makes it hard to clearly define the state of the field and reason about its future directions. In this work, we provide an overview of this important and thriving area, covering traditional approaches, statistical approaches and also approaches that use deep neural networks. We provide a comprehensive review towards building open domain dialogue systems, an important application of natural language generation. We find that, predominantly, the approaches for building dialogue systems use seq2seq or language models architecture. Notably, we identify three important areas of further research towards building more effective dialogue systems: 1) incorporating larger context, including conversation context and world knowledge; 2) adding personae or personality in the NLG system; and 3) overcoming dull and generic responses that affect the quality of system-produced responses. We provide pointers on how to tackle these open problems through the use of cognitive architectures that mimic human language understanding and generation capabilities. △ Less

Submitted 2 June, 2019; originally announced June 2019.

Real-time Closed Loop Neural Decoding on a Neuromorphic Chip

Authors: Shoeb Shaikh, Rosa So, Tafadzwa Sibindi, Camilo Libedinsky, Arindam Basu

Abstract: This paper presents for the first time a real-time closed loop neuromorphic decoder chip-driven intra-cortical brain machine interface (iBMI) in a non-human primate (NHP) based experimental setup. Decoded results show trial success rates and mean times to target comparable to those obtained by hand-controlled joystick. Neural control trial success rates of approximately 96% of those obtained by ha… ▽ More This paper presents for the first time a real-time closed loop neuromorphic decoder chip-driven intra-cortical brain machine interface (iBMI) in a non-human primate (NHP) based experimental setup. Decoded results show trial success rates and mean times to target comparable to those obtained by hand-controlled joystick. Neural control trial success rates of approximately 96% of those obtained by hand-controlled joystick have been demonstrated. Also, neural control has shown mean target reach speeds of approximately 85% of those obtained by hand-controlled joystick . These results pave the way for fast and accurate, fully implantable neuromorphic neural decoders in iBMIs. △ Less

Submitted 10 December, 2018; originally announced December 2018.

Comments: accepted at Neural Engineering Conference (NER), 2019

Extracting Fairness Policies from Legal Documents

Authors: Rashmi Nagpal, Chetna Wadhwa, Mallika Gupta, Samiulla Shaikh, Sameep Mehta, Vikram Goyal

Abstract: Machine Learning community is recently exploring the implications of bias and fairness with respect to the AI applications. The definition of fairness for such applications varies based on their domain of application. The policies governing the use of such machine learning system in a given context are defined by the constitutional laws of nations and regulatory policies enforced by the organizati… ▽ More Machine Learning community is recently exploring the implications of bias and fairness with respect to the AI applications. The definition of fairness for such applications varies based on their domain of application. The policies governing the use of such machine learning system in a given context are defined by the constitutional laws of nations and regulatory policies enforced by the organizations that are involved in the usage. Fairness related laws and policies are often spread across the large documents like constitution, agreements, and organizational regulations. These legal documents have long complex sentences in order to achieve rigorousness and robustness. Automatic extraction of fairness policies, or in general, any specific kind of policies from large legal corpus can be very useful for the study of bias and fairness in the context of AI applications. We attempted to automatically extract fairness policies from publicly available law documents using two approaches based on semantic relatedness. The experiments reveal how classical Wordnet-based similarity and vector-based similarity differ in addressing this task. We have shown that similarity based on word vectors beats the classical approach with a large margin, whereas other vector representations of senses and sentences fail to even match the classical baseline. Further, we have presented thorough error analysis and reasoning to explain the results with appropriate examples from the dataset for deeper insights. △ Less

Submitted 12 September, 2018; originally announced September 2018.

Vulnerable to Misinformation? Verifi!

Authors: Alireza Karduni, Isaac Cho, Ryan Wesslen, Sashank Santhanam, Svitlana Volkova, Dustin Arendt, Samira Shaikh, Wenwen Dou

Abstract: We present Verifi2, a visual analytic system to support the investigation of misinformation on social media. On the one hand, social media platforms empower individuals and organizations by democratizing the sharing of information. On the other hand, even well-informed and experienced social media users are vulnerable to misinformation. To address the issue, various models and studies have emerged… ▽ More We present Verifi2, a visual analytic system to support the investigation of misinformation on social media. On the one hand, social media platforms empower individuals and organizations by democratizing the sharing of information. On the other hand, even well-informed and experienced social media users are vulnerable to misinformation. To address the issue, various models and studies have emerged from multiple disciplines to detect and understand the effects of misinformation. However, there is still a lack of intuitive and accessible tools that help social media users distinguish misinformation from verified news. In this paper, we present Verifi2, a visual analytic system that uses state-of-the-art computational methods to highlight salient features from text, social network, and images. By exploring news on a source level through multiple coordinated views in Verifi2, users can interact with the complex dimensions that characterize misinformation and contrast how real and suspicious news outlets differ on these dimensions. To evaluate Verifi2, we conduct interviews with experts in digital media, journalism, education, psychology, and computing who study misinformation. Our interviews show promising potential for Verifi2 to serve as an educational tool on misinformation. Furthermore, our interview results highlight the complexity of the problem of combating misinformation and call for more work from the visualization community. △ Less

Submitted 17 March, 2019; v1 submitted 25 July, 2018; originally announced July 2018.

Comments: 11 pages, 7 figures

Anchored in a Data Storm: How Anchoring Bias Can Affect User Strategy, Confidence, and Decisions in Visual Analytics

Authors: Ryan Wesslen, Sashank Santhanam, Alireza Karduni, Isaac Cho, Samira Shaikh, Wenwen Dou

Abstract: Cognitive biases have been shown to lead to faulty decision-making. Recent research has demonstrated that the effect of cognitive biases, anchoring bias in particular, transfers to information visualization and visual analytics. However, it is still unclear how users of visual interfaces can be anchored and the impact of anchoring on user performance and decision-making process. To investigate, we… ▽ More Cognitive biases have been shown to lead to faulty decision-making. Recent research has demonstrated that the effect of cognitive biases, anchoring bias in particular, transfers to information visualization and visual analytics. However, it is still unclear how users of visual interfaces can be anchored and the impact of anchoring on user performance and decision-making process. To investigate, we performed two rounds of between-subjects, in-laboratory experiments with 94 participants to analyze the effect of visual anchors and strategy cues in decision-making with a visual analytic system that employs coordinated multiple view design. The decision-making task is identifying misinformation from Twitter news accounts. Participants were randomly assigned one of three treatment groups (including control) in which participant training processes were modified. Our findings reveal that strategy cues and visual anchors (scenario videos) can significantly affect user activity, speed, confidence, and, under certain circumstances, accuracy. We discuss the implications of our experiment results on training users how to use a newly developed visual interface. We call for more careful consideration into how visualization designers and researchers train users to avoid unintentionally anchoring users and thus affecting the end result. △ Less

Submitted 7 June, 2018; originally announced June 2018.

Using Naive Bayes Algorithm to Students' bachelor Academic Performances Analysis

Authors: Fahad Razaque, Nareena Soomro, Shoaib Ahmed Shaikh, Safeeullah Soomro, Javed Ahmed Samo, Natesh Kumar, Huma Dharejo

Abstract: Academic Data Mining was one of emerging field which comprise procedure of examined students details by different elements such as earlier semester marks, attendance, assignment, discussion, lab work were of used to improved bachelor academic performance of students, and overcome difficulties of low ranks of bachelor students. It was extracted useful knowledge from bachelor academic students data… ▽ More Academic Data Mining was one of emerging field which comprise procedure of examined students details by different elements such as earlier semester marks, attendance, assignment, discussion, lab work were of used to improved bachelor academic performance of students, and overcome difficulties of low ranks of bachelor students. It was extracted useful knowledge from bachelor academic students data collected from department of Computing. Subsequently preprocessing data, which was applied data mining techniques to discover classification and clustering. In this study, classification method was described which was based on naive byes algorithm and used for Academic data mining. It was supportive to students along with to lecturers for evaluation of academic performance. It was cautionary method for students to progress their performance of study. △ Less

Submitted 5 February, 2018; originally announced February 2018.

Comments: 2017

Journal ref: IEEE Proceedings ICETAS 2017

An End-To-End Machine Learning Pipeline That Ensures Fairness Policies

Authors: Samiulla Shaikh, Harit Vishwakarma, Sameep Mehta, Kush R. Varshney, Karthikeyan Natesan Ramamurthy, Dennis Wei

Abstract: In consequential real-world applications, machine learning (ML) based systems are expected to provide fair and non-discriminatory decisions on candidates from groups defined by protected attributes such as gender and race. These expectations are set via policies or regulations governing data usage and decision criteria (sometimes explicitly calling out decisions by automated systems). Often, the d… ▽ More In consequential real-world applications, machine learning (ML) based systems are expected to provide fair and non-discriminatory decisions on candidates from groups defined by protected attributes such as gender and race. These expectations are set via policies or regulations governing data usage and decision criteria (sometimes explicitly calling out decisions by automated systems). Often, the data creator, the feature engineer, the author of the algorithm and the user of the results are different entities, making the task of ensuring fairness in an end-to-end ML pipeline challenging. Manually understanding the policies and ensuring fairness in opaque ML systems is time-consuming and error-prone, thus necessitating an end-to-end system that can: 1) understand policies written in natural language, 2) alert users to policy violations during data usage, and 3) log each activity performed using the data in an immutable storage so that policy compliance or violation can be proven later. We propose such a system to ensure that data owners and users are always in compliance with fairness policies. △ Less

Submitted 18 October, 2017; originally announced October 2017.

Comments: Presented at the Data For Good Exchange 2017

Towards a disaster response system based on cognitive radio ad hoc networks

Authors: Noman Islam, Ghazala Shafi Shaikh

Abstract: This paper presents an approach towards disaster management based on cognitive radio ad hoc network. Despite the growing interests on cognitive radio ad hoc networks, not much work has been reported on using them for disaster management. This paper discusses opportunities for disaster management based on cognitive radio ad hoc networks. In this direction, the paper presents a novel technique for d… ▽ More This paper presents an approach towards disaster management based on cognitive radio ad hoc network. Despite the growing interests on cognitive radio ad hoc networks, not much work has been reported on using them for disaster management. This paper discusses opportunities for disaster management based on cognitive radio ad hoc networks. In this direction, the paper presents a novel technique for disaster detection based on Artificial Neural Network (ANN). The ANN is trained using backward propagation algorithm. An ANN-based spectrum sensing scheme is also presented. Finally, a service discovery scheme is presented for coordination during the time of disaster. The simulation of proposed approach has been performed in NS-2 simulator. The proposed approach shows very low false negative alarm rate using the proposed disaster detection system. The spectrum switching time of spectrum sensing scheme is also analyzed along with an analysis of latency of proposed service discovery scheme △ Less

Submitted 3 October, 2017; originally announced October 2017.

Simultaneous Detection and Quantification of Retinal Fluid with Deep Learning

Authors: Dustin Morley, Hassan Foroosh, Saad Shaikh, Ulas Bagci

Abstract: We propose a new deep learning approach for automatic detection and segmentation of fluid within retinal OCT images. The proposed framework utilizes both ResNet and Encoder-Decoder neural network architectures. When training the network, we apply a novel data augmentation method called myopic warping together with standard rotation-based augmentation to increase the training set size to 45 times t… ▽ More We propose a new deep learning approach for automatic detection and segmentation of fluid within retinal OCT images. The proposed framework utilizes both ResNet and Encoder-Decoder neural network architectures. When training the network, we apply a novel data augmentation method called myopic warping together with standard rotation-based augmentation to increase the training set size to 45 times the original amount. Finally, the network output is post-processed with an energy minimization algorithm (graph cut) along with a few other knowledge guided morphological operations to finalize the segmentation process. Based on OCT imaging data and its ground truth from the RETOUCH challenge, the proposed system achieves dice indices of 0.522, 0.682, and 0.612, and average absolute volume differences of 0.285, 0.115, and 0.156 mm$^3$ for intaretinal fluid, subretinal fluid, and pigment epithelial detachment respectively. △ Less

Submitted 17 August, 2017; originally announced August 2017.

Software Model Checking: A Promising Approach to Verify Mobile App Security

Authors: Irina Mariuca Asavoae, Hoang Nga Nguyen, Markus Roggenbach, Siraj Ahmed Shaikh

Abstract: In this position paper we advocate software model checking as a technique suitable for security analysis of mobile apps. Our recommendation is based on promising results that we achieved on analysing app collusion in the context of the Android operating system. Broadly speaking, app collusion appears when, in performing a threat, several apps are working together, i.e., they exchange information w… ▽ More In this position paper we advocate software model checking as a technique suitable for security analysis of mobile apps. Our recommendation is based on promising results that we achieved on analysing app collusion in the context of the Android operating system. Broadly speaking, app collusion appears when, in performing a threat, several apps are working together, i.e., they exchange information which they could not obtain on their own. In this context, we developed the Kandroid tool, which provides an encoding of the Android/Smali code semantics within the K framework. Kandroid allows for software model checking of Android APK files. Though our experience so far is limited to collusion, we believe the approach to be applicable to further security properties as well as other mobile operating systems. △ Less

Submitted 15 June, 2017; originally announced June 2017.

Comments: 2 pages, 2 figures, FTFJP'17

Towards Automated Android App Collusion Detection

Authors: Irina Mariuca Asavoae, Jorge Blasco, Thomas M. Chen, Harsha Kumara Kalutarage, Igor Muttik, Hoang Nga Nguyen, Markus Roggenbach, Siraj Ahmed Shaikh

Abstract: Android OS supports multiple communication methods between apps. This opens the possibility to carry out threats in a collaborative fashion, c.f. the Soundcomber example from 2011. In this paper we provide a concise definition of collusion and report on a number of automated detection approaches, developed in co-operation with Intel Security. Android OS supports multiple communication methods between apps. This opens the possibility to carry out threats in a collaborative fashion, c.f. the Soundcomber example from 2011. In this paper we provide a concise definition of collusion and report on a number of automated detection approaches, developed in co-operation with Intel Security. △ Less

Submitted 7 March, 2016; originally announced March 2016.

Document clustering using graph based document representation with constraints

Authors: Muhammad Rafi, Farnaz Amin, Mohammad Shahid Shaikh

Abstract: Document clustering is an unsupervised approach in which a large collection of documents (corpus) is subdivided into smaller, meaningful, identifiable, and verifiable sub-groups (clusters). Meaningful representation of documents and implicitly identifying the patterns, on which this separation is performed, is the challenging part of document clustering. We have proposed a document clustering tech… ▽ More Document clustering is an unsupervised approach in which a large collection of documents (corpus) is subdivided into smaller, meaningful, identifiable, and verifiable sub-groups (clusters). Meaningful representation of documents and implicitly identifying the patterns, on which this separation is performed, is the challenging part of document clustering. We have proposed a document clustering technique using graph based document representation with constraints. A graph data structure can easily capture the non-linear relationships of nodes, document contains various feature terms that can be non-linearly connected hence a graph can easily represents this information. Constrains, are explicit conditions for document clustering where background knowledge is use to set the direction for Linking or Not-Linking a set of documents for a target clusters, thus guiding the clustering process. We deemed clustering is an ill-define problem, there can be many clustering results. Background knowledge can be used to drive the clustering algorithm in the right direction. We have proposed three different types of constraints, Instance level, corpus level and cluster level constraints. A new algorithm Constrained HAC is also proposed which will incorporate Instance level constraints as prior knowledge; it will guide the clustering process leading to better results. Extensive set of experiments have been performed on both synthetic and standard document clustering datasets, results are compared on standard clustering measures like: purity, entropy and F-measure. Results clearly establish that our proposed approach leads to improvement in cluster quality. △ Less

Submitted 4 December, 2014; originally announced December 2014.

An improved semantic similarity measure for document clustering based on topic maps

Authors: Muhammad Rafi, Mohammad Shahid Shaikh

Abstract: A major computational burden, while performing document clustering, is the calculation of similarity measure between a pair of documents. Similarity measure is a function that assigns a real number between 0 and 1 to a pair of documents, depending upon the degree of similarity between them. A value of zero means that the documents are completely dissimilar whereas a value of one indicates that the… ▽ More A major computational burden, while performing document clustering, is the calculation of similarity measure between a pair of documents. Similarity measure is a function that assigns a real number between 0 and 1 to a pair of documents, depending upon the degree of similarity between them. A value of zero means that the documents are completely dissimilar whereas a value of one indicates that the documents are practically identical. Traditionally, vector-based models have been used for computing the document similarity. The vector-based models represent several features present in documents. These approaches to similarity measures, in general, cannot account for the semantics of the document. Documents written in human languages contain contexts and the words used to describe these contexts are generally semantically related. Motivated by this fact, many researchers have proposed seman-tic-based similarity measures by utilizing text annotation through external thesauruses like WordNet (a lexical database). In this paper, we define a semantic similarity measure based on documents represented in topic maps. Topic maps are rapidly becoming an industrial standard for knowledge representation with a focus for later search and extraction. The documents are transformed into a topic map based coded knowledge and the similarity between a pair of documents is represented as a correlation between the common patterns (sub-trees). The experimental studies on the text mining datasets reveal that this new similarity measure is more effective as compared to commonly used similarity measures in text clustering. △ Less

Submitted 17 March, 2013; originally announced March 2013.

Comments: 5 pages

A comparison of SVM and RVM for Document Classification

Authors: Muhammad Rafi, Mohammad Shahid Shaikh

Abstract: Document classification is a task of assigning a new unclassified document to one of the predefined set of classes. The content based document classification uses the content of the document with some weighting criteria to assign it to one of the predefined classes. It is a major task in library science, electronic document management systems and information sciences. This paper investigates docum… ▽ More Document classification is a task of assigning a new unclassified document to one of the predefined set of classes. The content based document classification uses the content of the document with some weighting criteria to assign it to one of the predefined classes. It is a major task in library science, electronic document management systems and information sciences. This paper investigates document classification by using two different classification techniques (1) Support Vector Machine (SVM) and (2) Relevance Vector Machine (RVM). SVM is a supervised machine learning technique that can be used for classification task. In its basic form, SVM represents the instances of the data into space and tries to separate the distinct classes by a maximum possible wide gap (hyper plane) that separates the classes. On the other hand RVM uses probabilistic measure to define this separation space. RVM uses Bayesian inference to obtain succinct solution, thus RVM uses significantly fewer basis functions. Experimental studies on three standard text classification datasets reveal that although RVM takes more training time, its classification is much better as compared to SVM. △ Less

Submitted 13 January, 2013; originally announced January 2013.

Comments: ICoCSIM 2012, Medan Indonesia

A Framework for Analysing Driver Interactions with Semi-Autonomous Vehicles

Authors: Siraj Shaikh, Padmanabhan Krishnan

Abstract: Semi-autonomous vehicles are increasingly serving critical functions in various settings from mining to logistics to defence. A key characteristic of such systems is the presence of the human (drivers) in the control loop. To ensure safety, both the driver needs to be aware of the autonomous aspects of the vehicle and the automated features of the vehicle built to enable safer control. In this pap… ▽ More Semi-autonomous vehicles are increasingly serving critical functions in various settings from mining to logistics to defence. A key characteristic of such systems is the presence of the human (drivers) in the control loop. To ensure safety, both the driver needs to be aware of the autonomous aspects of the vehicle and the automated features of the vehicle built to enable safer control. In this paper we propose a framework to combine empirical models describing human behaviour with the environment and system models. We then analyse, via model checking, interaction between the models for desired safety properties. The aim is to analyse the design for safe vehicle-driver interaction. We demonstrate the applicability of our approach using a case study involving semi-autonomous vehicles where the driver fatigue are factors critical to a safe journey. △ Less

Submitted 31 December, 2012; originally announced January 2013.

Comments: In Proceedings FTSCS 2012, arXiv:1212.6574

ACM Class: H.1.2

Journal ref: EPTCS 105, 2012, pp. 85-99

Content-based Text Categorization using Wikitology

Authors: Muhammad Rafi, Sundus Hassan, Mohammad Shahid Shaikh

Abstract: A major computational burden, while performing document clustering, is the calculation of similarity measure between a pair of documents. Similarity measure is a function that assign a real number between 0 and 1 to a pair of documents, depending upon the degree of similarity between them. A value of zero means that the documents are completely dissimilar whereas a value of one indicates that the… ▽ More A major computational burden, while performing document clustering, is the calculation of similarity measure between a pair of documents. Similarity measure is a function that assign a real number between 0 and 1 to a pair of documents, depending upon the degree of similarity between them. A value of zero means that the documents are completely dissimilar whereas a value of one indicates that the documents are practically identical. Traditionally, vector-based models have been used for computing the document similarity. The vector-based models represent several features present in documents. These approaches to similarity measures, in general, cannot account for the semantics of the document. Documents written in human languages contain contexts and the words used to describe these contexts are generally semantically related. Motivated by this fact, many researchers have proposed semantic-based similarity measures by utilizing text annotation through external thesauruses like WordNet (a lexical database). In this paper, we define a semantic similarity measure based on documents represented in topic maps. Topic maps are rapidly becoming an industrial standard for knowledge representation with a focus for later search and extraction. The documents are transformed into a topic map based coded knowledge and the similarity between a pair of documents is represented as a correlation between the common patterns. The experimental studies on the text mining datasets reveal that this new similarity measure is more effective as compared to commonly used similarity measures in text clustering. △ Less

Submitted 17 August, 2012; originally announced August 2012.

Comments: 9 pages; IJCSI August 2012

Association Rule Mining Based On Trade List

Authors: Sanober Shaikh, Madhuri rao

Abstract: In this paper a new mining algorithm is defined based on frequent item set. Apriori Algorithm scans the database every time when it finds the frequent item set so it is very time consuming and at each step it generates candidate item set. So for large databases it takes lots of space to store candidate item set .In undirected item set graph, it is improvement on apriori but it takes time and space… ▽ More In this paper a new mining algorithm is defined based on frequent item set. Apriori Algorithm scans the database every time when it finds the frequent item set so it is very time consuming and at each step it generates candidate item set. So for large databases it takes lots of space to store candidate item set .In undirected item set graph, it is improvement on apriori but it takes time and space for tree generation. The defined algorithm scans the database at the start only once and then from that scanned data base it generates the Trade List. It contains the information of whole database. By considering minimum support it finds the frequent item set and by considering the minimum confidence it generates the association rule. If database and minimum support is changed, the new algorithm finds the new frequent items by scanning Trade List. That is why it's executing efficiency is improved distinctly compared to traditional algorithm. △ Less

Submitted 22 February, 2012; originally announced February 2012.

Comments: 15 pages

Journal ref: http://www.airccj.org/ijdkp/ijdkp2011.html

Comparing SVM and Naive Bayes classifiers for text categorization with Wikitology as knowledge enrichment

Authors: Sundus Hassan, Muhammad Rafi, Muhammad Shahid Shaikh

Abstract: The activity of labeling of documents according to their content is known as text categorization. Many experiments have been carried out to enhance text categorization by adding background knowledge to the document using knowledge repositories like Word Net, Open Project Directory (OPD), Wikipedia and Wikitology. In our previous work, we have carried out intensive experiments by extracting knowled… ▽ More The activity of labeling of documents according to their content is known as text categorization. Many experiments have been carried out to enhance text categorization by adding background knowledge to the document using knowledge repositories like Word Net, Open Project Directory (OPD), Wikipedia and Wikitology. In our previous work, we have carried out intensive experiments by extracting knowledge from Wikitology and evaluating the experiment on Support Vector Machine with 10- fold cross-validations. The results clearly indicate Wikitology is far better than other knowledge bases. In this paper we are comparing Support Vector Machine (SVM) and Naïve Bayes (NB) classifiers under text enrichment through Wikitology. We validated results with 10-fold cross validation and shown that NB gives an improvement of +28.78%, on the other hand SVM gives an improvement of +6.36% when compared with baseline results. Naïve Bayes classifier is better choice when external enriching is used through any external knowledge base. △ Less

Submitted 18 February, 2012; originally announced February 2012.

Comments: 5 pages

Journal ref: Multitopic Conference (INMIC), 2011 IEEE 14th International