Register no:411622149007

SYNTHETICA
Ai efficiency research summarization

Project Created By :
• Haripriya G[411622149007]
• Shruthi Kamal[411622149019]
• Avanthika R[411622243004]
• Lilymalar R[411622243021]
Project Reviewed By:
Ms.DHIVYA T

Project Created Date :05/06/2024

Project Code:PE008

College Code: 4116

Team Name: PE 0772

Register no:411622149007
 Register no:411622149007

Table of Contents:

Project Title: SYNTHETICA: Evaluating AI Efficacy in Research

Summarization

Table of Contents: .................................................................. 1

Executive Summary: ................................................................ 2
Project Objective.................................................................... 2
Scope: .................................................................................... 3
Methodology: ........................................................................... 6
Artifacts used ............................................................................. 9
Technical coverage: ............................................................... 11
Results: .....................................................................................22
Challenges and Resolutions: .................................................. 22
Conclusion ................................................................................ 25
References: ............................................................................ 27
 Register no:411622149007

Executive Summary:

SYNTHETICA is a project aimed at evaluating the effectiveness

of artificial intelligence (AI) systems in summarizing research papers.
With the exponential growth of academic publications, there is a
pressing need for efficient summarization tools to help researchers
keep up-to-date with the latest findings without having to read entire
papers. This project focuses on assessing various AI models designed
for summarization to determine their accuracy, coherence, and utility
in the research community.
This project successfully integrates various AI tools to enhance
the accessibility, understanding, and evaluation of research papers on
neurodegenerative diseases. By comparing AI-generated and human-
generated abstracts and presenting them on a well-designed website,
the project not only showcases the current state of AI capabilities in
academic summarization but also provides a valuable resource for
the research and medical community. The outcomes and implications
underscore the potential of AI in transforming academic research,
offering insights that could guide future developments and
applications of AI in various fields.

Project Objective:

The objective of this project is to enhance the accessibility and

comprehension of research papers on neurodegenerative diseases by
creating an interactive website. This website leverages AI tools to
generate and compare abstracts of research papers, providing a
comprehensive resource for researchers, students, and healthcare
professionals.
 Register no:411622149007

1. Evaluation of AI Models: To systematically evaluate different AI

summarization models, including state-of-the-art systems like GPT-4,
BERT, and other transformer-based models.
2.Accuracy Assessment:To measure the accuracy of AI-generated
summaries compared to human-generated summaries.
3. Coherence and Readability: To assess the coherence,
readability, and overall quality of the summaries produced by AI.
4. Utility in Research: To determine the practical utility of AI
summaries for researchers by conducting user studies and surveys.
5. Benchmark Creation: To create a benchmark dataset consisting
of research papers and their corresponding human and AI-generated
summaries for future research and comparison.

Scope:

1. AI Technologies in Research Summarization

➢ Natural Language Processing (NLP): Explore advancements in NLP

that facilitate research summarization, including models like GPT,
BERT, and other transformer-based architectures.
➢ Machine Learning Techniques: Investigate supervised,
unsupervised, and reinforcement learning methods employed in
summarization tasks.
➢ Hybrid Approaches: Examine combinations of rule-based and
statistical methods used to enhance summarization quality.

2. Summarization Methodologies
 Register no:411622149007

➢ Extractive Summarization: Assess methods that identify and

extract key sentences or phrases from the text.
➢ Abstractive Summarization: Evaluate techniques that generate
new sentences to encapsulate the meaning of the source text.
➢ Hybrid Summarization: Look into systems combining extractive
and abstractive methods to leverage the strengths of both.

3. Evaluation Metrics
Intrinsic Evaluation Metrics:
➢ ROUGE Scores: Analyze recall-oriented measures evaluating
overlap between system-generated and reference summaries.
➢ BLEU Scores: Evaluate precision-based metrics for
summarization, though traditionally used in machine
translation.
➢ METEOR, CIDEr: Other metrics that consider synonyms and
stemming.
Extrinsic Evaluation Metrics:
➢ Task-Based Evaluations: Assess how well the summary aids in
specific tasks like information retrieval or decision-making.
➢ User-Centric Evaluations: Collect feedback from users regarding
the summary’s usefulness, readability, and informativeness.

4. Challenges and Limitations

➢ Quality and Relevance: Evaluate the coherence, relevance, and

factual accuracy of summaries.
➢ Bias and Fairness: Investigate potential biases in AI models and
their impact on summarization quality.
 Register no:411622149007

➢ Domain Specificity: Examine the performance of summarization

models across different domains, such as medicine, law, and
science.
5. Practical Applications and Use Cases

➢ Academic Research: How AI-generated summaries can assist

researchers in quickly grasping the content of papers.
➢ Industry Applications: Use cases in pharmaceuticals, legal firms,
and other industries where research summarization can
enhance productivity.
➢ Education: Application of summarization tools in academic
settings for both students and educators.

6. Human-AI Collaboration

➢ Augmented Intelligence: Explore how AI can augment human

capabilities in summarization tasks.
➢ Interactive Systems: Development of interfaces that allow users
to interact with and refine AI-generated summaries.

7. Ethical and Societal Implications

➢ Privacy Concerns: Evaluate the handling of sensitive information

within research documents.
➢ Intellectual Property: Address concerns related to the use of AI
in summarizing copyrighted research works.
 Register no:411622149007

Methodology:
1. Collection of Research Paper:

• Source: Google Scholar.

• Format: PDF.
• Topic: Neurodegenerative diseases.
• Process: A research paper related to neurodegenerative
diseases is identified and downloaded in PDF format from
Google Scholar. This paper serves as the primary source
document for the subsequent steps.

2. Abstract Generation Using Lightning PDF AI:

• Tool Used: Lightning PDF AI.

• Process:
o The downloaded research paper is uploaded to Lightning
PDF AI.
o The AI processes the document, extracting key points and
summarizing the content to generate an abstract.
o This abstract highlights the main findings, methodology,
and conclusions of the research paper.

3. Human-Generated Abstract:

• Tool Used: GitHub Copilot.

• Process:
o A human expert, using the assistance of GitHub Copilot,
reads the research paper.
o The expert, guided by Copilot's suggestions, creates a
detailed and accurate abstract.
o This abstract is intended to capture the essence of the
research, ensuring clarity and coherence.

4. Comparative Analysis Using Perplexity AI:

• Tool Used: Perplexity AI.

 Register no:411622149007

• Process:
o Both the AI-generated abstract from Lightning PDF AI and
the human-generated abstract are analyzed using
Perplexity AI.
o Metrics for comparison include:
▪ Coherence: The logical flow and understandability of
the abstract.
▪ Relevance: How well the abstract covers the key
points of the research paper.
▪ Readability: The ease with which the abstract can
be read and understood by the target audience.
o The analysis results provide insights into the strengths
and weaknesses of each abstract, highlighting the
effectiveness of AI tools compared to human-generated
content.

5. Website Creation:

• Tool Used: Canva.

• Design:
o The website is designed to be user-friendly and visually
appealing, using Canva.
o It features a clear and intuitive layout with four primary
buttons, each serving a specific purpose:
1. Link to the Original Research Paper: Provides direct
access to the PDF of the research paper, allowing
users to read the full document.
2. Link to AI-Generated Abstract: Directs users to the
abstract generated by Lightning PDF AI.
3. Link to Human-Generated Abstract: Directs users to
the abstract created with the assistance of GitHub
Copilot.
4. Comparative Analysis Results: Directs users to a
page displaying the comparison metrics and insights
generated using Perplexity AI.
 Register no:411622149007

• Content: Each section of the website is designed to provide

comprehensive information and easy navigation, ensuring users
can seamlessly access all resources.

6. Deployment and Access:

• Hosting: The website is hosted on a suitable platform, ensuring

it is accessible to the intended audience.
• User Interface: The interface is designed to be intuitive, with
clear navigation paths and accessible content, making it easy
for users to interact with the website and find the information
they need.

Expected Outcomes:
1. Comprehensive Evaluation Report:A detailed report evaluating
the performance of different AI summarization models, highlighting
which models perform best in specific domains.
2. Recommendations: Practical recommendations for researchers
and developers on using AI summarization tools.
3. Benchmark Dataset: A publicly available benchmark dataset for
the research community to use in future evaluations and model
improvements.
4. Research Publications: Academic papers detailing the findings
and methodologies used in the project.

Timeline:
1. Phase 1 :
• Project planning and dataset collection.
• Initial selection and training of AI models.
 Register no:411622149007

2. Phase 2 :
• Fine-tuning models and initial evaluations.
• Conducting human evaluations and surveys.
3. Phase 3 :
• Data analysis and synthesis of results.
• Preparation of the benchmark dataset.
4. Phase 4 :
• Compilation of the evaluation report and recommendations.
• Submission of research publications.
• Public release of the benchmark dataset.

Team Composition:
Project Lead: Oversees the project, ensures objectives are met,
and coordinates between team members.
AI Researchers: Responsible for selecting, training, and
evaluating AI models.
Data Scientists: Handle data collection, preprocessing, and
analysis.
Human Evaluators: Conduct human evaluations and surveys.
Technical Writers: Compile reports, documentation, and
research publications.

Artifacts used:

We used the following artifacts:

 Register no:411622149007

Al websites used for poster

1. Canva
Use: Canva offers a wide range of templates and design tools,
including AI-driven design suggestions. It is user-friendly and suitable
for creating visually appealing posters.
Website: [Canva.com](https://www.canva.com/)

2. Visme
Use: Visme provides tools for creating infographics, presentations,
and posters. Its AI features can help you design layouts and choose
color schemes.
Website: [Visme.co](https://www.visme.co/)

3. Adobe Spark
Use: Adobe Spark offers easy-to-use design tools for creating
posters, flyers, and social graphics. AI-driven features can help with
layout suggestions and design enhancements.
Website: [Adobe Spark](https://spark.adobe.com/)

4. PosterMyWall
Use: PosterMyWall offers customizable templates for posters. AI
tools assist in optimizing images and text to create professional-
looking designs.
Website: [PosterMyWall.com](https://www.postermywall.com/)

5. DesignCap
 Register no:411622149007

Use: DesignCap provides a wide range of poster templates and

design tools. AI-driven design recommendations help in creating
polished and engaging posters.
Website: [DesignCap.com](https://www.designcap.com/)

AI Tools for Content Creation

1. Scholarcy
Use: Scholarcy can summarize research papers, extracting key
points and creating concise content that can be used in your poster.
Website: [Scholarcy.com](https://www.scholarcy.com/)

2. TLDR This
Use: TLDR This generates brief summaries of long research articles,
which can be useful for creating concise, impactful content for your
poster.
Website: [TLDRthis.com](https://www.tldrthis.com/)

3. Grammarly
Use: Grammarly provides AI-driven writing assistance, ensuring
your poster content is clear, concise, and free of errors.
Website: [Grammarly.com](https://www.grammarly.com/)

4. QuillBot
 Register no:411622149007

Use: QuillBot offers paraphrasing and summarization tools that can

help condense complex research content into digestible points for
your poster.
Website: [QuillBot.com](https://www.quillbot.com/)

Steps to Create the Poster

1. Research Summarization:
• Use tools like Scholarcy or TLDR This to extract key points
from the research papers you want to highlight.

2. Content Creation:
• Refine the summarized content using Grammarly and QuillBot
to ensure clarity and brevity.

3. Designing the Poster:

• Choose a design tool like Canva or Visme to start creating your
poster. Select a template that suits your theme.
• Use AI design suggestions to enhance the visual appeal of
your poster. Incorporate images, charts, and infographics to
illustrate key points.

4. Final Touches:
• Ensure the poster has a logical flow and that all elements are
aligned. Use Adobe Spark or DesignCap for any final design
tweaks.
 Register no:411622149007

• Proofread the content and ensure it effectively communicates

the evaluation of AI efficacy in research summarization.

5. Printing and Sharing:

• Once your poster is complete, download it in high resolution.
You can print it or share it digitally as needed.

AI Tools for Research Summarization

1. Scholarcy
Use: Scholarcy summarizes research papers, highlighting key points
and extracting figures and tables. It can generate concise summaries
that can be converted into video scripts.
Website: [Scholarcy.com](https://www.scholarcy.com/)

2. TLDR This
Use: TLDR This creates short summaries of long articles and papers.
It’s useful for generating brief content suitable for video scripts.
Website: [TLDRthis.com](https://www.tldrthis.com/)

3. Scribendi
Use: Scribendi offers AI tools for summarizing text and refining
content. It can help ensure that your video script is clear and concise.
Website: [Scribendi.com](https://www.scribendi.com/)
 Register no:411622149007

Project output design:

Project link:(click to view the website of the project)
https://amen-h.my.canva.site/researchpaper-anaysis
Register no:411622149007
Register no:411622149007
 Register no:411622149007

Sample images using Bing:

Register no:411622149007
 Register no:411622149007

General Prompts

1. General Efficacy Assessment:

⚫ How effective is SYNTHETICA at summarizing research papers
in the field of [specific field] compared to traditional
summarization methods?
⚫ Evaluate the accuracy and coherence of SYNTHETICA's
research summaries compared to human-generated summaries.

2. Comparative Analysis:
⚫ Compare the summarization quality of SYNTHETICA with other
leading AI summarization tools in terms of relevance, conciseness,
-+and readability.
 Register no:411622149007

⚫ Assess the differences in summary quality when SYNTHETICA

is used for summarizing research articles in various academic
disciplines.
Specific Aspects of Summarization
3. Clarity and Coherence:
⚫ Analyze how well SYNTHETICA maintains clarity and coherence
in research summaries for complex topics in [specific field].
⚫ To what extent does SYNTHETICA preserve the logical flow of
the original research when creating summaries

4. Coverage and Comprehensiveness:

⚫ "Evaluate SYNTHETICA's ability to cover all key points and
findings in research articles about [specific topic]."
⚫ "How comprehensive are SYNTHETICA's summaries in capturing
the main contributions and implications of the research?"

5. Accuracy and Precision:

⚫ Assess the accuracy of SYNTHETICA's summaries in terms of
factual correctness and precision in conveying research outcomes.
⚫ Evaluate the precision of SYNTHETICA's summaries when dealing
with technical terms and specialized jargon in [specific field].

User Experience and Practicality

6. User Experience:
 Register no:411622149007

⚫ "Gather feedback from researchers on the usability and

practicality of SYNTHETICA for daily research summarization
tasks."
⚫ "How does the user experience of SYNTHETICA compare to
manual summarization in terms of time efficiency and ease of
use?"

Customization and Flexibility:

⚫ "Evaluate the flexibility of SYNTHETICA in allowing users to
customize summaries based on length, detail level, and specific
focus areas."
⚫ "How effective is SYNTHETICA in generating summaries tailored
to different user needs, such as executive summaries versus
detailed overviews?"

Field-Specific Prompts
8. Discipline-Specific Evaluation:
⚫ "Examine the performance of SYNTHETICA in summarizing
research papers in [specific discipline, e.g., biomedical sciences,
social sciences, engineering]."
⚫ "How well does SYNTHETICA handle the unique summarization
challenges present in interdisciplinary research papers?"

9. Case Studies and Examples:

⚫ Provide case studies of SYNTHETICA summarizing landmark
research papers in [specific field] and evaluate the summaries'
quality.
 Register no:411622149007

⚫ Compare SYNTHETICA's summaries of high-impact research

articles with those from traditional review papers in the same
field."
Longitudinal and Scalability Aspects
10. Longitudinal Studies:
⚫ Assess the consistency and reliability of SYNTHETICA's
summarizations over time when applied to a continuous stream
of research publications in [specific field]
⚫ Evaluate how SYNTHETICA adapts to new research trends and
evolving terminologies in [specific field] over a period of time.

11. Volume HandScalability aling:

⚫ Examine SYNTHETICA's performance and accuracy when
summarizing large volumes of research papers in [specific field].
⚫ How scalable is SYNTHETICA in handling the increasing volume
of research outputs in rapidly growing fields like artificial
intelligence and biotechnology?

Results:
Synthetica: Evaluating AI Efficacy in Research Summarizatio"
reveal a nuanced landscape in which AI systems demonstrate notable
capabilities alongside identifiable limitations. Through rigorous
evaluation against human-generated summaries, AI models exhibit
promising proficiency in generating concise and coherent research
summaries. However, the study highlights areas where AI
performance falls short, particularly in accurately capturing nuanced
contexts and ensuring the fidelity of information. These findings
underscore the ongoing need for refinement and innovation in AI
 Register no:411622149007

algorithms to enhance their effectiveness in research summarization

tasks. Additionally, the study emphasizes the importance of critical
evaluation and collaboration in leveraging AI to augment scholarly
discourse and knowledge dissemination.

Challenges and Resolutions:

Challenge 1: Quality and Accuracy of AI-Generated Abstracts
Description:
One of the primary challenges encountered in this project was
ensuring the quality and accuracy of the AI-generated abstracts. AI
tools can sometimes produce summaries that lack coherence, omit
critical details, or include irrelevant information.

Resolution:
To address this challenge, the project incorporated a comparative
analysis using Perplexity AI to evaluate the coherence, relevance, and
readability of the AI-generated abstracts. This analysis provided
empirical data that helped identify the strengths and weaknesses of
the AI-generated content. Additionally, human oversight was included
through the use of GitHub Copilot to ensure that the human-
generated abstracts served as a high-quality benchmark. This dual
approach ensured that any deficiencies in the AI-generated abstracts
could be systematically identified and addressed.

Challenge 2: Integration of Multiple AI Tools

Description:
 Register no:411622149007

Integrating various AI tools—such as Lightning PDF AI for abstract

generation, GitHub Copilot for human-assisted abstract creation, and
Perplexity AI for comparative analysis—posed technical and logistical
challenges. Each tool had different interfaces, output formats, and
processing requirements.

Resolution:
The project team established a streamlined workflow for
integrating these tools. Clear protocols were defined for how each
tool would be used, including standardized input and output formats.
This involved preprocessing the research paper to ensure
compatibility with Lightning PDF AI and post-processing the outputs
to facilitate easy comparison using Perplexity AI. Regular testing and
iteration helped in ironing out any integration issues, ensuring
smooth and efficient workflow across different tools.

Challenge 3: User-Friendly Website Design

Description:
Creating a website that is both visually appealing and easy to
navigate was a significant challenge. The site needed to present
complex information in an accessible manner without overwhelming
the users.
Resolution:
Using Canva for website design allowed the team to create an
intuitive and aesthetically pleasing layout. User experience (UX)
principles were applied to design clear navigation paths and visually
distinct sections for each key resource (original paper, AI-generated
abstract, human-generated abstract, and comparative analysis). User
feedback was solicited during the design phase to ensure that the
 Register no:411622149007

website met the needs and expectations of its intended audience.

This iterative design process ensured a balance between functionality
and aesthetics.

Challenge 4: Ensuring Comprehensive Comparative Analysis

Description:
A thorough comparative analysis required careful consideration of
various metrics to ensure that the comparison between AI-generated
and human-generated abstracts was meaningful and comprehensive.

Resolution:
The use of Perplexity AI enabled a detailed evaluation based on
multiple metrics: coherence, relevance, and readability. By defining
clear criteria for each metric and using a robust analytical tool, the
project could generate detailed and actionable insights. Additionally,
involving domain experts in the evaluation process provided a
qualitative layer of analysis, ensuring that the findings were not only
quantitatively robust but also contextually relevant.

Challenge 5: Maintaining Consistency and Standardization

Description:
Ensuring consistency and standardization across all abstracts and
analyses was essential to provide reliable and comparable results.

Resolution:
Standardized templates and guidelines were created for abstract
generation and analysis. This included clear instructions for using AI
 Register no:411622149007

tools and human-generated content processes. Regular reviews and

quality checks were conducted to maintain high standards of
consistency and accuracy. Documentation of the methodology and
protocols ensured that all team members followed the same
procedures, thereby minimizing variability and ensuring reliable
outcomes.

Conclusion:

Addressing these challenges requires a multifaceted approach,

combining technical improvements in AI models with robust
evaluation frameworks and ethical considerations. By engaging
domain experts, employing diverse datasets, and developing better
metrics, the efficacy of AI in research summarization can be
systematically enhanced.
Contributions:
This project significantly enhances the accessibility of
research papers on neurodegenerative diseases by curating and
hosting them on a user-friendly website. The integration of Lightning
PDF AI for abstract generation demonstrates how AI can efficiently
summarize complex documents, saving valuable time and effort for
researchers. By comparing AI-generated abstracts with those created
by humans, with the assistance of GitHub Copilot, the project
provides critical insights into the strengths and weaknesses of AI
tools in replicating human-level understanding and summarization.
The use of Perplexity AI for comparative analysis adds a quantitative
dimension to the evaluation process, offering a structured approach
to assessing the efficacy of AI tools. The comprehensive and
interactive website, designed with Canva, enriches the user
experience by providing access to the original research paper,
 Register no:411622149007

abstracts, and comparative analysis results in one centralized

location.

Outcomes:
The project has successfully created a centralized resource that
hosts the original research paper, AI-generated abstract, human-
generated abstract, and comparative analysis results. This not only
facilitates quick access to important research findings but also
provides empirical data on the performance of AI in abstract
generation. By offering these resources, the project improves the
understanding of neurodegenerative diseases and highlights the
current capabilities and limitations of AI tools. The comparative
analysis using Perplexity AI provides valuable insights into how well
AI tools can replicate human-like summaries, informing future
developments in AI for academic purposes.
Implications:
For researchers and academics, this project showcases the
practical application of AI in academic research, encouraging the
integration of AI tools into their workflow for more efficient literature
reviews and summaries. For AI and tech developers, the findings
offer feedback on the performance of tools like Lightning PDF AI and
GitHub Copilot, highlighting areas for improvement and further
development. In the context of education and training, the project
serves as an educational tool, promoting digital literacy and
awareness of modern technological tools. For the medical
community, improved access to summarized research aids healthcare
professionals in staying updated with the latest findings in
neurodegenerative diseases, potentially informing clinical practices
and patient care. Additionally, the methodology and findings of this
project can serve as a foundation for further studies on the
 Register no:411622149007

application of AI in various research domains, fostering innovation

and interdisciplinary collaborations.

References:
Ai poster:
https://www.canva.com/
https://www.visme.co/
https://spark.adobe.com/
https://www.postermywall.com/
https://www.designcap.com/
Content Creation:
https://www.scholarcy.com/
https://www.tldrthis.com/
https://www.scribendi.com/
https://www.quillbot.com/
Final project:(click to view the website of the project)

https://amen-h.my.canva.site/researchpaper-anaysis