List of Abbreviations

1. Introduction

The European Maritime Single Window environment (EMSWe) ship database (ESD), common location database (ECLD) and common hazmat database (ECHD) are referred to in articles 14, 15 and 16 of the EMSWe Regulation (EU) 2019/1239.

The ESD allows the collection, storing, updating and provision of the following:

• Ship data comprising ship identification information and particulars, composed of elements of the EMSWe data set related to a ship that are static or subject to limited changes over time and which are submitted by declarants to the MNSWs.

• Information of ship reporting exemptions issued by Member States and reported in the Union Maritime Information and Exchange System (SafeSeaNet) referred to in Directive 2002/59/EC.

The ECLD allows the collection, storing, updating and provision of the following types of location and port facility codes:

• United Nations Code for Trade and Transport Locations (UN/LOCODE),

• IMO GISIS port facilities,

• SafeSeaNet specific codes.

The ECHD allows the collection, storing, updating and provision of information on dangerous and polluting goods stemming from the following conventions and agreements, including voluntary versions (where applicable):

• International Maritime Dangerous Goods (IMDG),

• International Maritime Solid Bulk Cargoes (IMSBC),

• International Code of the Construction and Equipment of Ships Carrying Liquefied Gases in Bulk (IGC),

• International Code for the Construction and Equipment of Ships carrying Dangerous Chemicals in Bulk (IBC),

• International Convention for the Prevention of Pollution from Ships (MARPOL Annex I),

• Tripartite Agreements (MEPC.2/Circular).

The technical specifications, standards, and procedures for setting up the ESD, ECLD and ECHD are provided in Annexes V, VI and VII of Implementing Regulation (EU) 2023/204.

2. Purpose

This document forms the System Interfaces Guide (SIG) of the EMSWe databases: ESD, ECLD and ECHD. The purpose of this document is to define the system-to-system protocols and respective messages, commissioning measures and procedures for establishing connection of the Member States’ Maritime National Single Windows (MNSW) with the EMSWe databases.

The intended audience of this document is the National EMSWe Coordinators of the Member States (referred to in Article 18 of the EMSWe Regulation) and the persons responsible for the development of the MNSWs.

3. Services

The following RESTful services are provided to the MNSWs:

• Ship data notification service: the service is used by the MNSW to provide the ESD with ship data as reported by the declarant to the MNSW.

• Data retrieval service: the service is used by the MNSW to retrieve an extract of the data from the ESD, ECLD and ECHD.

The following Kafka-based event message service is exposed to the MNSWs:

• Announcement service: the ESD, ECLD and ECHD expose this service to distribute to the MNSWs the data updates from the three databases.

The REST services are described in section 3.1 below. The event-based message service is described in section 3.2 below.

All elements which are strings in the JSON and AVRO messages use a UTF-8 character encoding.

3.1 REST services

Ship data notification service:

The ESD offers a REST endpoint designed to receive ship data notifications from the MNSWs. When ship data is received from a Declarant, the MNSW sends a REST PUT HTTP request with JSON body to the ESD through this endpoint. The content of the message is validated by the ESD against schemas and business rules, and feedback provided in the HTTP response to the MNSW if the validation was successful (with descriptive failure message in case it was not ok). Following its validation, the incoming update is compared with the existing data and is finally stored in the ESD (if differences are found).

The ship data notification should be sent by the MNSW as soon as it receives a formality containing ship data in scope of the ESD . The ship data notification contains a unique identifier of the notification defined by the MNSW, the authentication date of the formality submitted in the MNSW, the identification number of the port call (Visit ID), the EORI number and name of the declarant, and the ship identification information and particulars as reported in the formality.

Data retrieval service:

The data retrieval service should be used by the MNSW to initiate their local copies of the common databases.

For retrieval of the complete dataset, the EMSWe databases offer three REST endpoints which allow downloading the latest full dataset of ship data from the ESD, location data from the ECLD and hazmat data from the ECHD. These endpoints support only GET HTTP requests. The data is fetched from the ESD, ECLD and ECHD storages, serialized into binary AVRO format and is immediately available for downloading by the client that invoked the respective REST endpoint. In case of successful processing of a request, the content type of the response is AVRO/binary, as defined by the AVRO specification. In any other case, the system responds with a message in JSON format, which includes the error code and a descriptive message indicating the reason of failure in the communication.

The ship data contains for each ship a unique number assigned by the ESD (ESD Id), the latest ship identification information and particulars, and information regarding the ship’s reporting exemptions.

The location data contains the location data and related port facility data, the date-time when the location was created, the date-time of the last update of the location and port facility data.

The hazmat data contains the dangerous and polluting good information and a unique identifier of the good item defined by the ECHD.

The specifications of the REST APIs are provided in the sections below and rely on the OpenAPI specification standard The specifications include the schemas as well as examples of the messages to be exchanged, while the YAML files provided are the machine-readable version of the specifications that can be used by developers (in conjunction with open-source tools) to create mock-ups of the services for supporting and testing the implementation (see Appendix A for more details).

3.1.1 API specification

• ESD specification

• ECLD specification

• ECHD specification

3.1.2 YAML

• ESD yaml file

• ECLD yaml file

• ECHD yaml file

3.2 Announcement service

The distribution of the data updates from the ESD, ECLD and ECHD relies on ”publisher-consumer events topics”. This allows decoupling between publishers (EMSWe databases) and consumers (MNSWs) since the event messages are stored and made available for consuming for a configurable amount of time (current envisioned period of retention of messages in topics is 7 days).

The EMSWe databases provide three different events topics which MNSWs may use to consume update events related to the updates of data in the ESD, ECLD and ECHD databases:

• ESD: Each time that a ship record is updated (update of at least one ship attribute or at least one ship exemption) or created, the ESD will publish an announcement message to the events topic containing all the latest attributes values of the ship and the ship exemptions information. In case that a ship record is deactivated, the corresponding announcement message will contain only the Ship ID, with the attribute “deactivated” equal to “true”. In all the other cases, the attribute “deactivated” will be equal to “false”.

• ECLD: Each time that a location record is updated or created, the ECLD will publish an announcement message to the events topic containing all the latest location’s attributes values. In the case that a location record is deactivated, the corresponding announcement message will only contain the Location ID, with the attribute “deactivated” equal to “true”. In all the other cases, the attribute “deactivated” will be equal to “false”.

• ECHD: Each time that a hazmat record is created or updated, the ECHD will publish an announcement message to the events topic containing all the latest attributes values of the hazmat record. In the case that a hazmat record is deleted, the announcement message will only contain the Hazmat ID and an attribute “Deleted” equal to “true”. In all the other cases, the attribute “Deleted” will be equal to “false”.

All AVRO schemas related to the above messages will be stored in the Azure Schema Registry. It is a schema repository service hosted by Azure Event Hubs, providing schema storage, versioning, and management. With this approach, both the EMSWe databases (as producer) and the MNSWs (as consumers) will use the same schema for producing and validating the messages, without having to manage and share the schemas manually.

The specifications of the API are provided in the sections below and rely on the AsyncAPI specification standard. The specifications include the schemas as well as examples of the messages to be exchanged, while the YAML files provided are the machine-readable version of the specifications that can be used by developers (in conjunction with open-source tools) to create mock-ups of the services for supporting and testing the implementation (see Appendix A for more details).

3.2.1 API specification

• ESD specification

• ECLD specification

• ECHD specification

3.2.2 YAML

• ESD yaml file

• ECLD yaml file

• ECHD yaml file

4. Access Rights Policies and Registration Process

This section presents the process for the registration of external entities to the EMSWe databases, as well as the applicable access rights policies and the authorisation process.

4.1 Access rights policies and authorisation

MS shall establish and maintain a system-to-system connection between their MNSW systems and the EMSWe databases for exchanging data. This connection is secured with Microsoft Azure high security integrated access control management for resources and applications, based on Microsoft Entra ID. This is realized by utilizing an Oauth2.0 client credentials flow (on top of SSL encrypted HTTP transport layer) using access token which is the recommended approach for the system-to-system access.

In this flow, the client application of a MS, which is registered in Microsoft Entra ID, should request an access token from Microsoft Entra ID by providing its client id and the client secret. The authorization server will respond providing the corresponding access token.

Once having received the access token, the client application should perform a call to a specific EMSWe databases’ REST API including the token in the Authorization header of the call. Following the successful validation of the token and access rights, the client application is provided with the appropriate response from the API. The same approach must also be followed for the announcement service, where the same token must be included in the connection process.

4.2 Registration process

In order to establish a connection between the EMSWe databases and the MNSW, the National EMSWe Coordinators must appoint a reference person for the EMSWe databases who will be the contact person for EMSA to establish the connection of the MNSW with the EMSWe databases and to coordinate the execution of the commissioning tests. The National EMSWe Coordinator must send a registration request by email to EMSA’s Maritime Support Services (maritimesupportservices@emsa.europa.eu), indicating the name and contact details (email address and telephone number) of the appointed reference person. The email address of the reference person must be registered in EU Login in order to proceed with the commissioning tests as indicated in section 6

Once the commissioning tests are completed, EMSA will register the MNSW in Microsoft Entra ID and will communicate to the reference person the client id and client secret to interface with the EMSWe databases in the production environment.

After the above steps, the MNSW will be able to obtain an access token from Microsoft Entra, using the client id and client secret, and establish the connection with the EMSWe database’s API.

EMSA will also grant access to the EMSWe databases within the EMSA Portal to the national EMSWe coordinator and the reference person.

5. Connection process

All system connections with the EMSWe databases are protected by enforcing the OAuth2.0 client credentials flow as described in section 4 above. All connections are encrypted using TLS 1.2.

MNSWs can communicate with the EMSWe databases as follows:

• The MNSWs will communicate with the ship data notification service of ESD by executing the appropriate HTTP PUT calls to https://apim.emswe-db.emsa.europa.eu/esd/v1.0/update.

• For the data retrieval service, ship data, location data and hazmat data will be accessed by the MNSWs by executing simple HTTP GET requests to the following endpoints:

  • Ship Data from https://apim.emswe-db.emsa.europa.eu/esd/v1.0/export;

  • Location Data from https://apim.emswe-db.emsa.europa.eu/ecld/v1.0/export;

  • Hazmat Data from https://apim.emswe-db.emsa.europa.eu/echd/v1.0/export.

• Announcement service: Updates of data in the EMSWe databases produce update announcement messages in an Azure Event Hub. The MNSWs can use the external Kafka client libraries (e.g. Confluent’s Python client) for establishing Kafka connections to the Event Hub in order to receive the announcement messages. These libraries should be configured to connect to emswe-prod-mnsw-broker.servicebus.windows.net:9093 endpoint and with the respective topics names as described in API specifications of section 3.2.1. Also in order to connect to the Event Hub's schema registry the https://emswe-prod-mnsw-broker.servicebus.windows.net endpoint should be set and the schema group value set to "esd"

• The MNSWs will communicate with EMSA’s Entra ID service to obtain the required access tokens used by the systems to connect to EMSWe databases services, by connecting to the https://login.microsoftonline.com/acfcf9f2-6d3b-4411-915b-327cdf780e79/oauth2/v2.0/token and setting the scope to:

  • api://3c951a43-1b1b-4e11-9a16-a4055d96a524/.default - for the REST API endpoints
  • https://emswe-prod-mnsw-broker.servicebus.windows.net/.default - for the Kafka endpoint

6.Commissioning measures

The Interoperability Test Bed (ITB) will be used (accessible at https://www.itb.ec.europa.eu/itb) for the commissioning test. The ITB is a service offered by the Commission DG DIGIT to facilitate the conformance testing of IT systems. Online documentation on how to use the ITB is available at https://joinup.ec.europa.eu/collection/interoperability-test-bed-repository/solution/interoperability-test-bed/documentation. Please refer to the Organisation User Guide and the Organisation Admin Guide.

6.1 Initial configuration

The reference person must inform the EMSA’s Maritime Support Services (maritimesupportservices@emsa.europa.eu) when ready to perform the commissioning tests.

EMSA will then create user account in ITB for the reference person using the email provided in the registration process (refer to Section 4.2). The reference person will be able to add additional users, if needed, for the execution of the tests.

IMPORTANT: The email addresses of each user in ITB must be registered as well in EU Login.

EMSA will configure the test suite in ITB. The test suite will include the following test cases:

1. MNSW sends an ESD notification to create/update a ship with an IMO number with all ship attributes.
2. MNSW sends an ESD notification to create/update a ship with Call Sign and MMSI number with all ship attributes.
3. MNSW receives an announcement message after ESD updates.
4. MNSW requests to retrieve a full copy of the latest ESD ship data.
5. MNSW receives an announcement message after ECLD updates.
6. MNSW requests to retrieve a full copy of the latest ECLD data.
7. MNSW receives an announcement message after ECHD updates.
8. MNSW requests to retrieve a full copy of the latest ECHD data.

EMSA will communicate to the reference person the client id and client secret to be used by the MNSW to access the EMSWe databases mock APIs used for commissioning.

NOTE: To facilitate the development of the interface with the EMSWe databases, it is possible to run the tests locally in the development environment without connecting to the ITB (refer to Appendix C). Please note that executing the tests locally does not replace the commissioning tests in ITB.

6.2 First login in ITB

1. At the first login in ITB click on the “Confirm your new role” link.

   

2. This will redirect to the EU Login page.
3. Once logged in, a page "Link a new role to your account" will appear."
4. Select the option “Confirm a role assigned to me by an administrator”.

   

Once the first login to ITB has been completed, please inform EMSA MSS.

NOTE: This role confirmation is done only once and will not be repeated at the following logins. If a specific EU Login account has roles in more than one community in ITB, the user will be prompted to select the community for which he or she wants to log in each time that the login is done.

6.3 Tests execution

Before executing the commissioning tests, the MNSW must be configured to connect to the endpoint(s) of the EMSWe databases mock APIs. The landing page in the ITB portal provides all the necessary information for configuring the MNSW to run the commissioning tests.

For executing the commissioning tests, the following steps must be performed:

1. Login to the ITB with the respective EU Login account.
2. Click on the left-hand navigation to access the “My Conformance Statements” and then click on the desired conformance statement.

   

3. This opens a screen with all the available test suites and the included test cases. The test cases can run individually or all together. Each test case will contain documentation that will describe the required steps that the tester needs to perform for the test to pass. Next to each test, there is an indication of the test case status, as well as of the overall test suite status.

   

4. When a test case fails, there will be an error message with information about the reason. The error message should provide enough details to be able to debug the problem. In case that the tester needs assistance for debugging the problems, the MNSW reference person can contact EMSA to request additional support.

   

   

5. When all tests have been successfully executed, the tester can download a conformance certificate in pdf format. The reference person shall inform EMSA that the commissioning tests have all passed.

   

All communications with EMSA must be done at the following email address:

maritimesupportservices@emsa.europa.eu

Appendix A - Usage of YAML specifications

The YAML files provided for the OpenAPI and AsyncAPI specifications, in Sections 3.1 and 3.2, serve a dual purpose. They are used to generate HTML documents for human readers (also available in Sections 3.1 and 3.2), providing a readable interface for understanding the API's capabilities. Simultaneously, they are also used by developers with open-source tools to generate clients and mock-ups for development.

OpenAPI and AsyncAPI specifications in YAML format provide a standard, language-agnostic interface to RESTful and asynchronous APIs respectively. These specifications can be processed by a variety of open-source tools to generate code, documentation, and even test cases.

For instance, Swagger UI and ReDoc are popular tools that generate interactive API documentation from OpenAPI specifications. They allow developers and non-developers to interact with the API’s endpoints without any implementation logic in place. For AsyncAPI specifications, the AsyncAPI Generator is a powerful tool. It can generate documentation and even code for different languages and protocols.

To generate mock servers, Prism is a good tool for OpenAPI specs, and Microcks is a good tool for AsyncAPI specs. These tools help you create realistic mock-ups of your API, aiding in efficient and effective development.

For a full list of supporting tools refer to OpenAPI Tooling (openapis.org) and Tools | AsyncAPI Initiative for event-driven APIs.

In conclusion, YAML files for OpenAPI and AsyncAPI are not just for generating human-readable documents, but they are also a powerful resource for developers, enabling them to leverage open-source tools to accelerate and enhance their development process.

Appendix B - Translation of Codes list used in APIs

Appendix C - Local execution of Test Cases

For the local execution of the test cases, developers and testers can download this zip file that contains docker composed yamls and other necessary files, to run locally the mock APIs, the ITB instance and Apache Kafka.

The steps required for local execution are as follows:

1. Ensure that the local machine has Docker installed.
2. Extract the contents of the zip file.
3. Open a terminal to the location of the ITB docker-composed .yaml file (in the "itb" subfolder of the extracted zip archive) and run the following command:
   docker compose up -d
4. Open a terminal to the location of the Apache Kafka docker-composed .yaml file (in the "kafka" subfolder of the extracted zip archive) and run the following command:
   docker compose up -d
5. Set up the base URL for calls to the mock APIs from the system under test as:
   http://localhost:8097/{countryCode}/
   where {countryCode} is the member state's 3-letter country code in uppercase letters (e.g., GRC). For example, a call to export ESD data for Greece would be performed to the endpoint:
   http://localhost:8097/GRC/esd/export
6. For listening to Kafka messages in the system under test, set the client to connect to the running Kafka instance at localhost:9092.
7. Use the following credentials to log in to the test bed by visiting: localhost:9000
   Username: localtester@itb.com
   Password: X^Dk%$9zwMUPPid
8. Navigate to "My conformance statements" on the left-hand side, select the "EMSWe Specification" and run the tests by following the documentation on each test case.

Since the ITB testing environment for local executions should be isolated, there are two differences compared with the ITB test executions in the commissioning environment:

• There is no authentication for accessing the specific EMSWe DB mock endpoints.
• Since the protocol for consuming the EMSWe DB announcements is Kafka, the Azure Event Hub, responsible for streaming the announcements, is emulated by running the Apache Kafka instance locally. Therefore, the schema registry is not used in this case. Nevertheless, the approach for consuming the announcements will remain the same as described in the Azure documentation and examples (e.g., https://github.com/Azure/azure-event-hubs-for-kafka).