Upgrading to

DSpace 7.4

Tim Donohue, LYRASIS

Corrado Lombardi, 4Science
DSpace 7.4 Workshop series: November 16, 2022
 Workshop Schedule
• 15:00-15:05 UTC : Intro (Natalie Baur, LYRASIS)

• 15:05-15:35 UTC : Overview of Upgrade Process

(Tim Donohue, LYRASIS)

• 15:35-16:00 UTC : Overview of New Configurations

(Corrado Lombardi, 4Science)

• 16:00-16:30 UTC : Questions & Answers

 Your Support Works to Sustain
DSpace!
• Membership
– https://tinyurl.com/dspace-membership
• Registered Service Providers
– https://dspace.lyrasis.org/rsp/
• DSpace Development Fund and SCOSS
Pledge
– https://dspace.lyrasis.org/fundraising/
Presenter Introductions
 Questions during Workshop

DSpace 7 Q&A Public Doc

https://tinyurl.com/dspace7-4-questions
Why upgrade to DSpace 7?
 Why DSpace 7?
● Redesigned, modernized, themeable

01 Single User Interface (built on

Angular.io)
●
●
Enhanced User Privacy (GDPR alignment)
WCAG 2.1 AA alignment

02 Fully-featured, integration-ready
REST API 03 Support for OpenAIRE v4

04 Configurable Entities (Person,

Project, Publication, etc) 05 Can upgrade from any prior release of
DSpace (after installing the new UI)

06 Aligning with
COAR Next Generation Re
positories
● Coming soon: Signposting, ResourceSync, Notify Project
implementation

07
Backwards compatible with existing ● Still includes OAI-PMH, SWORD (v1/v2), old REST
integrations API (v6), etc

●
08
Every code change has automated tests (TDD)
Secure & stable out-of-the-box
● Every code change is scanned for security vulnerabilities
Exciting 7.x features not available in 6.x
❖ Full-featured REST API (supports all features of DSpace)
❖ Configurable Entities (Person, Publication, Journal, Project, etc)
❖ OpenAIRE v4 support
❖ ORCID authentication & synchronization *
❖ Researcher Profiles *
❖ IIIF Support and/or Basic Image/Video viewer
❖ OpenID Connect (OIDC) authentication
❖ Many scripts can now be run from Admin User Interface
❖ Import metadata directly from many external sources (arXiv, CrossRef,
Scopus, Web of Science, PubMed, etc.)
* Requires enabling Configurable Entities
7.x Releases (so far)
★ 7.0 (Aug 2021)
○ New User Interface, New REST API, OpenAIRE v4 support, Configurable Entities
(Person, Project, Publication, etc.)
★ 7.1 (Nov 2021)
○ IIIF Support, Request a Copy, Item Versioning, OAI Harvesting
★ 7.2 (Feb 2022)
○ Runtime configs for UI, Item Embargos, Feedback form, OpenID Connect Auth
★ 7.3 (June 2022)
○ ORCID, Importing data from 9 new sources, Versioning of Entities, Control Panel,
Sherpa/Romeo, type-based submission, RSS, etc.
★ 7.4 (Oct 2022)
○ Batch export/import (Zip), Recent Submissions, Thumbnails, Google Captcha,
Google Analytics v4, Markdown/HTML/MathJax in metadata, etc.
Likely coming in 7.5 (Feb 2023)
❏ Search support for hierarchical controlled vocabularies
❏ Supervision orders (like 6.x JSPUI)
❏ Disabling self-registration of users
❏ Full support for custom workflow steps
❏ Managing email subscriptions (to collections)
❏ Context-specific help (like 6.x JSPUI)
❏ Starting Curation Tasks during submission process
❏ Item counts for Communities/Collections

And likely many other improvements, fixes and minor features.

What features are coming in a later 7.x release?
Understanding the DSpace 7
architecture
DSpace 6.x Architecture Front & Back End

JSPUI

XMLUI

OAI-PMH

Database

V4-6
Assetstore

7 separate webapps:
2 UIs, 4 web services &
Solr
DSpace 7.x Architecture Backend
Frontend Server
Webapp
v7

User Interface
OAI-PMH

Database

SEO via Angular

Assetstore
server-side rendering

(optional)
1 backend webapp v4-6
(Solr is now installed separately)
A Few Technical Design Goals

Frontend (UI) is themeable by a

Bootstrap designer (CSS/HTML)
○ WCAG 2.1 AA alignment

Backend is now one webapp

○ Easier to install/configure
 Preparing for a
DSpace 7 Upgrade
 Can I upgrade from
1.x/3.x/4.x/5.x/6.x to 7.x?

YES!

➔ Your backend data (metadata + files)

will be updated automatically
➔ Versions <6.x may wish to start with
fresh configs (in local.cfg)
➔ Review Release Notes for each version
 Planning your Upgrade
Upgrade “in place”?
1. Upgrade backend prerequisites (Java, Postgres, etc)
2. Upgrade to DSpace 7 backend
3. Install DSpace 7 frontend

Start fresh?
4. Install DSpace 7 frontend & backend
5. Migrate existing data into it
Installing DSpace 7
(starting fresh)

https://wiki.lyrasis.org/display/DSDOC7x/Installing+DSpace
 Installing Backend (Java)
Similar to a 5.x/6.x install
● Prerequisites:
○ Java, Maven, Ant, Tomcat
○ Database (PostgreSQL)
○ Solr
 Installing Backend (Java)
1. Download Backend (codebase)
2. Configure local.cfg
3. Build (`mvn package`)
4. Install (`ant fresh_install`)
5. Deploy webapp to Tomcat
6. Copy DSpace Solr configs to Solr
7. Startup Tomcat & Solr
8. http://localhost:8080/server/
Installing Frontend (Angular)

1. Install Node & Yarn

2. Download UI (codebase)
3. Install deps (`yarn install`)
4. Configure config.prod.yml
5.Build (`yarn build:prod`)
6. Startup (`yarn serve:ssr`) } `yarn start`
(quick start)
or use
7. http://localhost:4000/
 Production / HTTPS support
Backend
Proxy / HTTPS
HTTP
port 8080

OR
http://localhost:8080/server

(HTTPS, port 443) Frontend

HTTP
1. Install Apache (with mod_proxy) or Nginx
port 4000

http://localhost:4000/
 Production / HTTPS support
Backend
https://my-dspace.edu
HTTP
)
server/* port 8080
REST (/

OR
https://my-dspace.edu/server

UI (
(HTTPS, port 443) /*) Frontend

HTTP
1 1. Install Apache (with mod_proxy) or Nginx
port 4000
2. Proxy /server URLs to localhost:8080
3. Proxy / (root) URLs to localhost:4000
4. Setup HTTPS in Apache/Nginx like any other
website https://my-dspace.edu/
 Security is Priority
Backend security protections
● HTTPS required in production
● CORS & CSRF protection
● Only trusts ‘dspace.ui.url’ by default
Hints for common settings:
Backend’s local.cfg Frontend’s config.*.yml Example Valid URL Setting

dspace.server.url “rest” section https://my-dspace.edu/server

dspace.ui.url - https://my-dspace.edu

- “ui” section http://localhost:4000/

 Finally, Migrate your data!

1. Migrate your database (using pg_dump)

2. Migrate your assetstore (files) directory
3. Migrate/recreate your configs (local.cfg)
4. Run `./dspace database migrate ignored`
5. Reindex everything
6. Swap your domain to new server
Migrating DSpace to a new server
 Upgrading
to DSpace 7
(in place)

https://wiki.lyrasis.org/display/DSDOC7x/Upgrading+DSpace
 Upgrading Backend
Similar to past (v5.x/6.x) upgrades
● Upgrade or Reinstall Prerequisites:
○ Java, Maven, Ant, Tomcat
○ Database (PostgreSQL)
○ Solr
 Upgrading Backend
1. Download Backend (codebase)
2. Update/create local.cfg
3. Build (`mvn package`)
4. `ant update`
5. `dspace database migrate ignored`
6. `dspace submission-forms-migrate`
(input-forms.xml -> submission-forms.xml)
7. Deploy webapp to Tomcat
8. Copy DSpace Solr configs to Solr
 Upgrade Frontend?

There is no upgrade path from XMLUI

or JSPUI to the DSpace 7 UI.

Instead, install the new UI, and

customize or brand it how you see fit
(via HTML, CSS, Bootstrap)
 Upgrade Tips

https://wiki.lyrasis.org/display/DSDOC7x/Upgrading+DSpace
Starting Fresh vs Upgrade in Place?

Because of scale of upgrade, this may be

a good opportunity to start fresh.
1. Get comfortable with new setup
2. Review/Update your configs
3. Migrate data when you are ready!
 PM2? What is it? Do I need it?

= process manager for Node.js apps

● PM2’s cluster mode (-i max) allows you
to scale DSpace UI across multiple CPUs
○ Improves Server-Side Rendering (SSR) performance
● Not required, but highly recommended

If you use something different, make sure it

supports similar clustering / scaling
Could I run DSpace 7 via Docker?

YES!

• Frontend & backend have (development)

Docker scripts in their codebases
• You are welcome to repurpose these
scripts!
– Scripts likely require updates for production
(e.g. closing dev ports)
 Can I continue to use Oracle DB?

● Not recommended. Oracle is deprecated.

Support ends in June 2023.
● Options to migrate to PostgreSQL:
○ Use AIP Backup & Restore*
○ Use ora2pg to migrate database
○ Hire a Service Provider

* AIPs cannot migrate in-progress submissions

 Do I need to use
Configurable Entities?

https://wiki.lyrasis.org/display/DSDOC7x/Configurable+Entities
 What are Configurable Entities?
• In DSpace 7, an Entity is…
An Item
With a specific defined “type”
With relationship(s) to other Entities
• Entities are optional. Items are default.
• Entities do not replace Items.
• Collections accept either Entities or
Items
Default Entity Types in DSpace 7

Entity Types related to Journal hierarchy

○ Journal
○ Volume
○ Issue
○ Publication* (Article)
Entity Types related to Research
○ Publication*
○ Person
○ Project
○ OrgUnit
* Publication is the “link” between these two models
Should I use Configurable Entities?

Entities are an advanced feature

Known limitations:
● AIPs don’t support Entities (yet)
● No bulk migration from Item -> Entity
● Must create at least one Collection for
each Entity Type you wish to use.
You may wish to enable Entities if…

● You need one of the new features

which require Entities
○ Researcher Profiles
○ Sync profile with ORCID account

● The default Entities align well with the

content you wish to add to DSpace
 You may wish to keep Entities
disabled if…
● You depend on AIP Backup & Restore

● You aren’t interested in being an “early

adopter”

● This upgrade already looks complex.

(Keep it simple for now.)
 Common Install/Upgrade Issues
➢ The UI spins / doesn’t load!
➢ 500 (Service Unavailable) error!
➢ Blocked by CORS policy error!
➢ 403 Forbidden / Invalid CSRF token / cannot login!

Don’t worry! Help is available:

● Common Installation Issues
● Troubleshoot an error
● Ask for Help / Support
New Configurations in
DSpace 7
User Interface
Configurations

config/config.*.yml
 Language Selector

languages:
- code: en
label: English
active: true

…
 Home Page Items

homePage:
recentSubmissions:
pageSize: 5
sortField: 'dc.date.accessioned'
topLevelCommunityList:
pageSize: 5
 Browse By Options

browseBy:
oneYearLimit: 10
fiveYearLimit: 30
defaultLowerLimit: 1900
showThumbnails: true
pageSize: 20
 Community List

communityList:
pageSize: 20
 Basic Embedded Media Player
(videos or images)

mediaViewer:
image: true
video: true
 Backend
Configurations

[dspace]/config/* [dspace]/config/modules/*
 Important Backend Configs
(local.cfg)
dspace.dir (Backend install folder)
dspace.server.url (Backend URL)
dspace.ui.url (UI’s URL / default trusted client)
db.* (Database connection settings)
solr.server (Solr webapp/URL)
rest.cors.allowed-origins
(Add additional trusted clients)

Hint: Untrusted clients CANNOT login via REST API

 “Nice to Have” Backend Configs
(local.cfg)
ads.key
epo.consumerKey Keys to be set for
integrations with external
epo.consumerSecretKey services requiring them.
cinii.appid
scopus.apiKey
scopus.instToken
wos.apiKey
https://github.com/DSpace/DSpace/blob/main/dspace/config/modules
/external-providers.cfg
 LiveImport keys

ads.key epo.consumerKey

epo.consumerSecretKey
cinii.appid
scopus.apiKey
scopus.instToken
wos.apiKey
 Configuring
Submission Forms

‘item-submission.xml’ and ‘submission-forms.xml

 Configuring Submission
item-submission.xml
• Similar to 6.x
• Still allows to map submissions to collections
• Still defines steps of submission
<step> now represents sections of form (accordion)
<step> may be mandatory or optional
<step> has a type
• submission-form, upload, license, etc
 item-submission.xml
<step-definitions>
....
<step-definition id="traditionalpageone" mandatory="true">
<heading>submit.progressbar.describe.stepone</heading>
<processing-class>org.dspace.app.rest.submit.step.DescribeStep</processing-class>
<type>submission-form</type>
</step-definition>
....
</step-definitions>
<submission-definitions>
<submission-process name="traditional">
<step id="collection"/>
<step id="traditionalpageone"/>
<step id="traditionalpagetwo"/>
<step id="upload"/>
<step id="license"/>
</submission-process>
</submission-definitions>
 item-submission.xml
Some optional steps
• cclicense
• itemAccessConditions
 submission-forms.xml

● Replacement for input-forms.xml (v6.x)

● Define metadata fields that appear in
each submission-form <step>
● Define rows of fields (optionally)
● Define fields to describe bitstreams
 submission-forms.xml
★ Customize submission-form named “traditionalpageone”
<form-definitions>
<form name="traditionalpageone">
<row>
....
</row>
....
 submission-forms.xml
★ Customize submission-form named “traditionalpageone”
<form-definitions>
<form name="traditionalpageone">
<row>
<field>
<dc-schema>dc</dc-schema>
<dc-element>contributor</dc-element>
<dc-qualifier>author</dc-qualifier>
<repeatable>true</repeatable>
<label>Authors</label>
<input-type>onebox</input-type>
<hint>Enter the author’s name (Family name, Given names).</hint>
<required></required>
</field>
</row>
....
 submission-forms.xml
★ Multiple fields per row
<row>
<field>
<dc-schema>dc</dc-schema>
<dc-element>date</dc-element>
<dc-qualifier>issued</dc-qualifier>
...
<input-type>date</input-type>
</field>
<field>
<dc-schema>dc</dc-schema>
<dc-element>publisher</dc-element>
...
<input-type>onebox</input-type>
</field>
</row>
 submission-forms.xml
★ Customize metadata fields requested for Bitstreams!
<form name="bitstream-metadata">
<row>
....
</row>
....
</form>
 submission-forms.xml
★ Customize metadata fields requested for Bitstreams!
<form name="bitstream-metadata">
<row>
<field>
<dc-schema>dc</dc-schema>
<dc-element>title</dc-element>
<dc-qualifier></dc-qualifier>
<repeatable>false</repeatable>
<label>Title</label>
<input-type>onebox</input-type>
<hint>Enter the name of the file.</hint>
<required>You must enter the name of the file.</required>
</field>
</row>
....
</form>
 Submission Forms Migration
● script provided
submission-forms-migrate
● Input-forms.xml -> submission-
forms.xml
● Item-submission.xml -> item-
submission.xml (7)
~ <dspace install dir>/bin/dspace submission-forms-migrate
-f <path to source input-forms.xml>
-s <path to source item-submission.xml>
Configuring Workflow
Approval System
 Configuring Workflow

● DSpace 7 uses Configurable Workflow

○ Available since v1.8.x (in XMLUI)
● Default is same 3-step process as
“traditional” workflow
○ Review -> Edit/Review -> Edit only
● Workflow steps/roles configurable as
Spring Beans in a workflow.xml
● Workflow configurable per collection
 workflow.xml
<beans>
....
<bean name="defaultWorkflow" class="org.dspace.xmlworkflow.state.Workflow">
<property name="firstStep" ref=”reviewStep”/>
<property name="steps">
<util:list> Default config
<ref bean="reviewstep"/>
<ref bean="editstep"/>
<ref bean="finaleditstep"/>
</util:list>
</property>
</bean>
<bean name="reviewstep" class="org.dspace.xmlworkflow.state.Step">
...
</bean>
... https://wiki.lyrasis.org/display/DSDOC7x/
</beans> Configurable+Workflow
 workflow.xml
<beans>
....
<bean name="reviewstep" class="org.dspace.xmlworkflow.state.Step">
<property name="userSelectionMethod" ref="claimaction"/>
<property name="role" ref="reviewer"/>
<property name="outcomes"> Default config
<util:map>
<entry key="#{ T(org.dspace.xmlworkflow.state.actions.ActionResult).OUTCOME_COMPLETE}"
value-ref="editstep"/>
</util:map>
</property>
<property name="actions">
<util:list>
<ref bean="reviewaction"/>
</util:list>
</property>
</bean>
...
https://wiki.lyrasis.org/display/DSDOC7x/
</beans>
Configurable+Workflow
 workflow.xml
<beans>
...
<bean id="reviewer" class="org.dspace.xmlworkflow.Role">
<property name="scope"
value="#{ T(org.dspace.xmlworkflow.Role.Scope).COLLECTION}"/>
<property name="name" value="Reviewer"/>
<property name="description"
value="The people responsible for this step are able to edit the metadata of
incoming submissions, and then accept or reject them."/>
</bean> Default config
...
</beans>

https://wiki.lyrasis.org/display/DSDOC7x/
Configurable+Workflow
Wrap-Up
What’s next for 7.x?

https://wiki.lyrasis.org/display/DSPACE/DSpace+Release+7.0+Status
 7.x : Goal is to rapidly modernize
➢ New features in every 7.x release
○ Add a few missing 6.x features in each release
○ If necessary, code refactors may occur
➢ 7.5 due Feb 2023, 7.6 due June 2023
○ 3 releases per year (Feb, June, Oct)
➢ Once 7.x is “feature compatible” with 6.x
(around 7.6/7.7), planning for 8.x will begin

https://wiki.lyrasis.org/display/DSPACE/DSpace+Release+7.0+Status
 How can I contribute?
Join DSpace 7 Working Group (Dev Team)
➢ Meets every Thursday at 15:00UTC
➢ Help review / test Pull Requests!

Join DSpace Community Advisory Team (DCAT)

➢ Repository manager interest group
➢ Meets monthly: Tues, Dec 13 at 17:00UTC
➢ Provide feedback to developers & governance
Time for your questions!
 Questions?
Workshop Slides
https://tinyurl.com/dspace7-4-upgrade

DSpace 7 Q&A Public Doc

https://tinyurl.com/dspace7-4-questions
 More info on…
Configurable Entities

https://wiki.lyrasis.org/display/DSDOC7x/Configurable+Entities
Default Entity Types in DSpace 7

Entity Types related to Journal hierarchy

○ Journal
○ Volume
○ Issue
○ Publication* (Article)
Entity Types related to Research
○ Publication*
○ Person
○ Project
○ OrgUnit
Entities representing Journal Hierarchy
Person Entity
 Entities are Configurable!
★ Configure the model in relationship-types.xml
<relationships>
<type>
<leftType>JournalIssue</leftType>
<rightType>Publication</rightType>
<leftwardType>isPublicationOfJournalIssue</leftwardType>
<rightwardType>isJournalIssueOfPublication</rightwardType>
<leftCardinality>
<min>0</min> isJournalIssueOf..
</leftCardinality>
<rightCardinality>
<min>0</min>
<max>1</max> isPublicationOf..
</rightCardinality>
</type>
</relationships>
Issue Article
 Entities are Configurable!
★ Define 2 entity type names (to create or link entities)
Hint: Generic “Publication” type allows this Entity to be
<relationships>
reused for more than just “Articles”
<type>
<leftType>JournalIssue</leftType>
<rightType>Publication</rightType>
<leftwardType>isPublicationOfJournalIssue</leftwardType>
<rightwardType>isJournalIssueOfPublication</rightwardType>
<leftCardinality>
<min>0</min>
isJournalIssueOf..
</leftCardinality>
<rightCardinality>
<min>0</min>
<max>1</max> isPublicationOf..
</rightCardinality>
</type>
</relationships>
Issue Article
 Entities are Configurable!
★ Define unambiguous name for relation (in both directions)
<relationships>
<type>
<leftType>JournalIssue</leftType>
<rightType>Publication</rightType>
<leftwardType>isPublicationOfJournalIssue</leftwardType>
<rightwardType>isJournalIssueOfPublication</rightwardType>
<leftCardinality>
<min>0</min>
isJournalIssueOf..
</leftCardinality>
<rightCardinality>
<min>0</min>
<max>1</max> isPublicationOf..
</rightCardinality>
</type>
</relationships>
Issue Article
 Entities are Configurable!
★ Optionally: define the cardinality for the relationship
<relationships>
<type>
<leftType>JournalIssue</leftType>
<rightType>Publication</rightType>
<leftwardType>isPublicationOfJournalIssue</leftwardType>
<rightwardType>isJournalIssueOfPublication</rightwardType>
<leftCardinality>
<min>0</min>
</leftCardinality>
Publication can belong to
<rightCardinality> only one Journal issue
<min>0</min>
<max>1</max>
</rightCardinality>
</type>
</relationships>
 Entities are Configurable!
Configure the Entity model(s) in relationship-types.xml

Import relationship-types.xml into DSpace (using CLI

tool) to initialize the Entities and Relationships in
database.

○ If an Entity (of same type/name) exists, it will be

updated with any new relationships
○ If an Entity (of same type/name) doesn’t exist, it will
be created along with its relationships
 Remember, Entities are Items!

Just like Items, Entities...

● may use custom metadata
○ e.g. ‘person’ schema (out of the box)
● may use custom submission forms
○ e.g. ‘Person’ submission-process in item-
submission.xml (out of the box)
● may define a custom UI display
○ UI supports separate templates per entity type,
e.g. /src/app/entity-groups/*
 Entities support Virtual Metadata

★ No duplicated storage of metadata

★ “Populate” an entity with metadata from
related entities
★ Works wherever metadata is used (OAI, REST,
etc)
★ Example:
○ isAuthorOfPublication relationship can be displayed on
a Publication as dc.contributor.author
○ Value populated via related Person’s metadata
 virtual-metadata.xml
<!-- Define the mapper for each relationship name -->
<bean class="org.dspace.content.virtual.VirtualMetadataPopulator">
<property name="map">
<map>
<entry key="isAuthorOfPublication" value-ref="isAuthorOfPublicationMap"/>
</map>
</property>
</bean>

<!-- Define which metadata field the value is mapped into -->
<util:map id="isAuthorOfPublicationMap">
<entry key="dc.contributor.author" value-ref="publicationAuthor_author"/>
</util:map>
 virtual-metadata.xml
<!-- Value is concatenation of three Person metadata fields. Comma-separate multiple values -->
<bean class="org.dspace.content.virtual.Concatenate" id="publicationAuthor_author">
<property name="fields">
<util:list>
<value>person.familyName</value>
<value>person.givenName</value>
<value>organization.legalName</value>
</util:list>
</property>
<property name="separator">
<value>, </value>
</property>
</bean>
} Regular metadata

virtual metadata
(from related
{
Persons)